Murmur — A Tiny GPT That Learns to Write, On Your Phone

Overview

Murmur is a tiny GPT that learns to write in your browser. It starts from random numbers. Give it a text, watch it learn one character at a time, then ask it to carry on in the same style. It runs on the CPU, phones included. Nothing is sent to a server.

Open Murmur →

I built it because most transformer explainers begin after training is over. Murmur lets me show the training itself. The knobs still work while the model is learning. The attention heads can be inspected, switched off, and measured.

It is deliberately small. This is a toy for understanding how a GPT learns, not a writing product.

What you can do with it

Pick Shakespeare’s sonnets, Alice in Wonderland, Pride and Prejudice, Grimm’s tales, the original code, or my own portfolio writing. You can also paste your own text.

Then follow the model from start to finish:

1. Watch letters become numbers. The tokenizer builds one token for each character in the text.
2. Watch it learn. The loss chart includes the random-guess line at ln(vocab). A film strip keeps old samples, so the change from noise to words stays visible.
3. Turn the knobs live. Change the learning rate on a log scale, including pushing it too high. Change the batch size and gradient clipping. The current step does not have to finish first.
4. Talk to it. Give it a start, change temperature and top-k, and watch the next-character probability bars move.
5. Look inside. Open attention arcs or heatmaps for each layer and head. See a 2-D PCA map of the character embeddings.
6. Try brain surgery. Switch one attention head off. Murmur shows the new validation loss and samples, so the effect is measured rather than guessed.
7. See how it works, on your own model. “How it actually works” plays 11 short animated screens using the weights you just trained: one neuron, letters becoming vectors, self-attention taken slowly (compare, share, blend, four heads), and the block with its MLP. Then comes learning: surprise as a number, the gradient flowing backwards, real weights before and after one nudge, and the same sentence measured again.
8. Save the model. Keep it in IndexedDB and resume later. Saved data expires after 24 hours, and “clear all data” removes it in one tap.

How it works

I ported the shape of Karpathy’s train_gpt2.c from llm.c to TypeScript, then made it character-level and small enough for a browser CPU.

The model has token and position embeddings. Each pre-LayerNorm block has causal multi-head attention followed by a GELU MLP. A final LayerNorm feeds the language-model head. That head shares its weights with the token embeddings.

I wrote the forward and backward pass for every layer by hand. There is no machine-learning library. AdamW updates the weights, with weight decay only on 2-D weight matrices. Global-norm clipping catches large gradients before the update.

STAGE MAP — one training step

flowchart TB
    TEXT["A text you pick or paste"] --> TOK["Character tokenizer<br/>letters become numbers"]
    TOK --> BATCH["Training batch<br/>input and next character targets"]
    subgraph WORKER["Web Worker on the CPU"]
        BATCH --> FWD["Forward pass<br/>embeddings · attention · GELU MLP"]
        FWD --> LOSS["Cross-entropy loss"]
        LOSS --> BACK["Hand-written backward pass<br/>all parameter groups"]
        BACK --> CLIP["Global-norm gradient clipping"]
        CLIP --> ADAM["AdamW update<br/>decay on 2-D weights only"]
        ADAM -->|"next step"| BATCH
    end
    WORKER -->|"loss and samples"| CHART["Loss chart and film strip"]
    WORKER -->|"attention and probabilities"| INSPECT["Attention views and next-character bars"]

The heavy matrix multiplies run in WebAssembly SIMD using 4 by 4 tiles. A JavaScript path is kept as a fallback, but it is about 3.5 times slower. Attention used to take 62% of each step, so I rewrote it as separate matrix multiplies for each head.

A long step would make every control feel broken. I split training into roughly 40 ms micro-batches instead. A knob change or question is answered in about 25–40 ms, even while training continues.

STAGE MAP — worker and page messages

flowchart LR
    subgraph PAGE["Page on the main thread"]
        KNOBS["Learning rate · batch size · clipping"]
        QUESTIONS["Generate text · inspect head · switch head off"]
        VIEWS["Loss · samples · attention · probabilities"]
    end
    subgraph THREAD["Training Web Worker"]
        QUEUE["Message queue checked between micro-batches"]
        MODEL["Tokenizer · model · optimizer"]
        SNAP["Small snapshots for the page"]
    end
    KNOBS -->|"set training options"| QUEUE
    QUESTIONS -->|"ask for a result"| QUEUE
    QUEUE --> MODEL
    MODEL --> SNAP
    SNAP -->|"return loss samples attention probabilities"| VIEWS

The page never touches the live weights while an update is half-finished. It sends a message. The worker answers between micro-batches and keeps training.

Why this and not the existing tools

I looked at Transformer Explainer and bbycroft’s LLM visualisation before building Murmur. They explain inference through an already-trained model. They do not let me watch that model learn.

I also looked at browser ports of Karpathy’s microgpt. Those train one-layer toys on lists of names. They are useful, but they stop before the parts I wanted to inspect.

Murmur trains on my own text on a phone without WebGPU. Its controls work in the middle of training. I can switch off a head and immediately see what changed. I could not find another tool that put those pieces together.

What’s real, simplified and made up

Part	Status
Training engine	Real. The forward pass, backward pass, clipping and AdamW update all run in the browser
Transformer blocks	Real. Causal multi-head attention, GELU MLPs, LayerNorms and tied embeddings are trained from random weights
Gradient checks	Real. Float64 checks cover all 16 parameter groups; Float32 and SIMD are checked separately
Model size	Simplified. Murmur has tens to hundreds of thousands of weights, not billions
Tokens	Simplified. One character is one token. There are no word pieces
Compute	Simplified. It uses one browser CPU. There is no GPU cluster and no pre-training
Writing ability	Limited. It learns local shape and style from one small text. It is not a general language model
Hidden claims	None. I do not present generated text, attention views, or speed estimates as more than this tiny model produces on the current device

Numbers

Check	Result
Parameter groups checked in Float64	16, with relative gradient error around 1e-10
Float32 and SIMD gradient agreement	Within 1e-3
Engine tests	39
Small model	About 50k weights; about 80 steps/s on my laptop
Medium model	About 105k weights; about 27–28 steps/s on my laptop
Large model	About 450k weights; about 3.7–4 steps/s on my laptop
Worker response during training	About 25–40 ms
JavaScript matrix fallback	About 3.5 times slower than WebAssembly SIMD

The page times the current device and picks the largest of the three sizes that still feels live.

On Alice in Wonderland with the medium model, loss starts at 4.23. Validation loss reaches 1.98 at about 30 seconds and 1.66 at 90 seconds. The film strip starts with ”uW”F‘(h“srhvgk,Vu’M and reaches “Biste to on’t know what I hank her to Caterpillas;.

On the sonnets, the medium model reaches validation loss below 2.2 by about step 475, or about 20 seconds on a laptop.

Engineering notes

• Built on 2026-09-26. TypeScript, Vite, Web Workers, WebAssembly SIMD and Vercel. The repository is private.
• CPU first. Murmur does not need WebGPU. Device timing chooses small, medium or large before training settles in.
• Math checked two ways. I use Float64 for the strict gradient check, then compare the Float32 and SIMD paths within the tolerance used by the engine.
• The slow part was measured. Attention took 62% of step time before I changed its layout to per-head matrix multiplies.
• The interface stays awake. Roughly 40 ms micro-batches put message checks inside training rather than around a whole step.
• Data stays on the device. Text and model weights are not uploaded. IndexedDB saves expire after 24 hours and can be cleared at once.
• The limit is the point. Character tokens, tiny models and CPU training make each part visible. They also keep Murmur far from a useful general-purpose GPT.

Open Murmur and train it on a text →