Every weight started as random numbers on one desktop machine and was trained on public-domain books, a school curriculum, and its own conversations. No pretrained anything. The tokenizer is ours, the weights are ours, the memory is a file you could read.
Exactly what that means, because a
narrow claim is worth more than a grand one. The sentence you get back is
produced entirely by these 29.8M weights on this machine. Nothing is sent
anywhere. No other model is consulted to answer you - not to write the
reply, not to pick it, not even to grade it (the scoreboard is string
comparisons on purpose). Every parameter started as random numbers here, and
the tokenizer's merges were fitted on this corpus.
What that does not claim: AI helped build this. It wrote code, and
some of the practice conversations it learned from were machine-generated
here, on this box, from open weights. It also runs on PyTorch and NumPy;
the transformer and byte-pair encoding are published designs we implemented
rather than discovered, and the attention is PyTorch's own.
The reference works it reasons over are curated rather than scraped, and one
of them is ours: the K-12 curriculum was written here for
human students first and then adapted for a machine to learn from, which is
why the easy language genuinely comes first instead of being sorted that way
afterwards. The rest are public: WordNet, Webster's 1913, the CMU
pronouncing dictionary, Unicode CLDR.
The line is between how it was built and what answers you.
Plenty of hands and tools went into building it. When you ask it something,
it is on its own.
It separates knowing from saying. A language model fuses them.
In a language model, everything it knows and everything it can say are the same substance - statistics smeared across billions of weights. There is no row to read, no fact to point at, no way to correct a belief except training it again, and no way for it to tell you why it said what it said. Knowing and speaking are one blur.
Ember splits that blur into organs of one body. A drawer holds what is true: typed, voted on, inheritable, and open, where a belief is an object you can pick up and re-carve. A ledger holds what was said, verbatim and timestamped. The neural network is her voice, and it is deliberately swappable. She remembers first and speaks second, and what she remembered becomes the opening of what she says.
They are not a committee and one does not overrule another. The drawer has no mouth of its own: it never speaks to you, it only hands her what she knows, and every word you read is hers. That is why "she cannot make this up" is literally true rather than a promise. When she is answering from memory, no model is in the loop at all. Stored rows are assembled into a sentence by ordinary code, so there is nothing there that could drift. A language model on its own can never do that, because there is no row, only a sample.
The operational tell: a chatbot is in generate-mode every single turn, and can therefore invent something every single turn. Ember only reaches for the model when memory and the walker have both come up empty, and even then a checker can withhold a fluent answer that missed the question. Most of her turns involve no sampling at all.
For scale: GPT-2 small, the smallest model OpenAI ever released publicly, was 124M parameters - Ember is about a quarter of that. GPT-3 was 175 billion, roughly 5,900× Ember. The sizes of GPT-4 and everything after are not published, but they are far larger again.
Trained end to end in 4-bit (NVFP4) on a single consumer graphics card - one desktop machine, minutes to tens of minutes per run, not a datacentre. Four-bit training is the field's open problem; we beat it, on a home card. Answering needs no graphics card at all: it runs on the processor, so it will serve from a cheap VPS or an old laptop. That second number is the one that decides whether a thing like this can actually belong to somebody.
It didn't learn English from scraped internet. Every word of it is either written in-house or public domain, and we can name all of it:
The reading order is part of the design, not an accident of what was available: the corpus is packed easiest-first, and every book's position comes from its own Flesch-Kincaid score. It reads the whole thing exactly once, so it cannot pass by memorising it.
Most language models have never looked at anything. They learn that "apple" is applish because of the words that surround it. Ember learned the word and the picture together, from emoji - because a committee already distilled "appleness" into one canonical drawing, and a thousand photographs of apples just look like balls and peaches.
Shown an apple drawn in an art style it had never trained on, it names it correctly 77.1% of the time - measured against pictures from a third art team it never trained on.
All 1,884 concepts are anchored. While it learns to write, each concept name is held toward the vector its eyes produce for that thing, through a fixed map that cannot drift to meet the words. Words tied to sight is the direction this is built in, and the work now is proving how far it carries into what she actually says.
The name unpacks to what it's made of. Nothing here is a black box you have to take on faith - every layer is a file on a disk you can open, read, and argue with.
A dual encoder trained on the same concept drawn by two independent art teams, so it has to learn the idea rather than the picture. It names an apple drawn in a style it never trained on.
A drawer of chips with typed relations between them. Facts carry a source and a confidence; a "never" can outvote a "yes" without erasing it. It grows heavier where it's used, like a path worn into grass.
A walker crosses the typed edges and composes answers nobody handed it. Told a fuji is an apple, an apple is a fruit, and a pie is made of fruit, it can tell you a fuji belongs in a pie - and show the chain.
A from-scratch transformer with its own byte-level tokenizer, whose word embeddings are anchored to what the eyes do. When memory holds the answer it is assembled from the stored facts instead of generated. Text only - there is no voice here, and no plans to pretend otherwise.
Ember's memory is not a pile of numbers. It's a drawer of chips: a six-sided coin with the word and its picture on the face, and the six sides carrying the word itself once - that's the bright one - plus five things it is bound to. Every side is a row you can read, correct, or tear up by hand.
Chips stack. A concept reached for often grows a taller stack, each chip turned one slot from the one beneath it, so how much something matters is a physical height rather than a number in a column.
This is a live chip out of the running drawer, not a diagram of one. Click a name to pull a different one.
When a big model is wrong about you, the mistake is smeared across billions of weights and nobody can point at it. When Ember is wrong, the mistake is one line in a table with a source and a confidence attached, and deleting it fixes the problem for good.
Facts carved by a person outrank facts a machine guessed. A "never" can outvote a "yes" without erasing it - the losing side stays in the drawer as history. That is what a memory you own actually looks like.
Ember is a work in progress and says so. She is a fraction of a percent of the size of the models you are used to, and it shows. What she has instead is that every part of her is accounted for.
Ember trains from random weights on one consumer graphics card, and this year the training loop itself became the project: the same card, the same model, taken from 52,000 tokens a second to 263,957.
The last jump comes from a way of training that the card supports and almost nobody uses - at matched quality. On paired runs over the same data with the same seed, the fast path's validation loss is 5.099 against the standard recipe's 5.093: the speed is free.
Ember has been trained end to end in 4-bit (NVFP4) on a single NVIDIA GeForce RTX 5070 Ti, 16 GB - and it learns properly. Four-bit training is the field's open problem: published attempts lose too much accuracy to be usable, and the ones that run at all run on datacenter hardware. There is a reason it fails. We found it, and we beat it, on a home card on a desk. The quality holds against the standard recipe on the same data and seed - and 4 bits is half the bytes of 8, which is the door to a bigger Ember on the same machine.
Every number above is measured, logged, and reproducible on demand. This speed grew out of a collaboration with IDA-TRAIN-V2, a native training engine project. The port and the shared Blackwell findings are written up in IDA Leaves Hopper on Hugging Face.
| Can | |
|---|---|
| Write fluent English sentences | yes |
| Recognise 1,884 concepts by sight, across art styles it never trained on | yes |
| Remember facts across restarts, and show its work | yes |
| Answer a question by reasoning over what it was told | yes |
| Stay reliably on the topic you raised | not yet - in training |
| Write code, do maths, reason about the world | no, and it is not trying to |
| Replace a large model | no |
A project this small can flatter itself very easily, so the measurements are built to make that hard. Every one of these runs automatically.
| Guard | What it does |
|---|---|
| Held-out questions | Four fifths of the test prompts use words deliberately kept out of the training data. The familiar set and the unfamiliar set are scored separately and the gap between them is reported as a number in its own right, so a model that only looks good on questions it has met is caught by its own scorecard. |
| A contamination check | Each time the corpus is built, passages of the held-back material are searched word-for-word inside the training material and the count is printed. Duplicate conversations are removed before anything is held back, so the same text cannot end up on both sides. |
| Graded on her words only | In a training conversation, only Ember's own lines count toward the score. The human's lines are read as context and never marked, so she is never rewarded for learning to write both halves of a conversation. |
| Nothing personal in the weights | Facts about a real person live in the drawer, which is a file, not in the model. Every name, pet and place in the training conversations is invented. |
The rule: propose the change, measure it against the same bar, keep it only if the number moves.