Training Ninereeds has been an interesting experiment, and I’m still refining it all.

What I’ve done so far: 16 campaigns, each with its own goals.

Each campaign is a “research project” with a goal and a scope. I’m asking a question and doing experiments to find out the answer. A campaign could, for example, ask: Is the order of training data relevant, or does it not matter? Does adding other languages help or hinder the model? Some campaigns are short, just testing a hypothesis, while others show how complex a problem really is and spawn follow-up campaigns. Rabbit holes all the way down.

One of the strongest findings so far

is that curriculum order matters enormously. Teaching concepts in dependency order consistently produces better results than presenting the same material in an arbitrary order. That isn’t surprising in human education, but it’s absent from the way transformer pretraining is usually discussed.

After establishing that curriculum design mattered, I tested whether simply scaling the model would produce the same improvements, and the answer is: not noticeable yet. The amount of data the model had been training on was just not enough to lead to scarcity. It will make sense to scale up after the training data and order have stabilised, though, because Ninereeds remembers and learns.

Later, I tested the actual comprehension. The model had become surprisingly good at producing fluent multilingual text. It could generate grammatically correct German with proper case marking, conjugation and agreement, and switch between four languages naturally. However, probing its knowledge revealed that much of this ability was surface-level rather than conceptual.

That suggested a different approach.

First, I changed how I perform probes. Loss curves estimating how well Ninereeds can predict the next token don’t really tell me much about its actual abilities, and while the brain map could show how well concepts are represented in its weights, the connections between concepts, the clustering etc aren’t shown well enough yet – for a human, anyway. I’ve since fixed both the probes and the brain map to make them easier to understand.

So, rather than trying to encode everything during pretraining, I did something experimental again: treat pretraining as building a conceptual foundation, then teach the model through interactive dialogue.

I since redesigned the curriculum and am now bee-lining “chat ability” and language comprehension, to enable teaching not via pretraining, but via “Mommy Says Machine”: a chat engine that does inference on Ninereeds and corrects replies, basically chat sessions with a “smart” transformer that can teach Ninereeds through dialogue.

The “Mommy Says Machine” got its name from the way toddlers learn. Just like mommy will follow up “I eated the cookie” with corrective feedback, the Mommy Says Machine would say “You ate the cookie”. A BDH model capable of learning from input will eventually pick up the correct shapes of language. But this method has a prerequisite: the language needs to be stable enough to understand input.

The new curriculum has no quadrilingual support yet. Additional languages can be taught (Ninereeds would write correct sentences containing all 4 cases, using the correct genders, declinations and conjugations in German, it would use the correct register of speech in Japan and omit the correct parts of speech, etc), and pretty easily, too, which already proves that the form can look good. Ninereeds understood “language” well enough to emulate it. But at this stage, it creates more noise than signal – so now, the curriculum is based on “rotations”.

Each word being taught is being asked questions about it from many angles. What is X, how does it work, what does it look like, what is it connected to, what is it not, and so on. The goal is to anchor it to relevant concepts in varied contexts, so the resulting model absolutely understands what you’re talking about. Epistemic humility is baked in from the start, so if Ninereeds doesn’t know something, it says so.

The model can chat to a limited degree.

Here are two examples illustrating both the current strengths and weaknesses. Please note that this is a chat log from last weekend’s Campaign 16:

[user] What’s your name?

[Ninereeds] My name is Ninereeds.

[user] Who am I?

[Ninereeds] You are user.

But it gets confused easily still:

[user] What does a dog look like?

[Ninereeds] A dog is a small piece of a place. A dog has a different person or a thing. A dog has a different short things.

[user] Is a dog an animal?

[Ninereeds] I don’t know if a steady is a quantity.

So the current goal is to go from “plausible simulation of what language looks like” to actual comprehension and production.

What the redesigned campaign 14-16 data looked like:

The corpus contains 66,536 word files across 5,035 concept/bucket pairs. The average is about 13.2 files per concept, but half are _rephrase (to promote comprehension stability). There are currently only 12 identity files, which explains why Ninereeds doesn’t have a clear sense of who and what it is yet; amending this is on my list of things to do. So far, I’ve trained mostly one-turn exchanges, which is appropriate for its current stage, but will change in time. For now, the content is simple and appropriate for early anchoring: A dog is an animal, An airport is used for airplanes, etc. I already planned the integration of the preschool and K-8 corpora later, after chat ability is established.

I’m currently testing a truly tiny corpus that trains Ninereeds on an even more limited pool of words. The goal is still not perfect language, but Ninereeds should be able to know its own identity, answer simple known facts. Say “I don’t know” when knowledge is missing, keep concepts separated, connect related concepts correctly, handle simple follow-up questions and avoid filling gaps with plausible nonsense. I’m no longer optimizing for impressive-looking language generation. I’m optimizing for stable concepts. If the model can reliably answer “What is a dog?”, admit when it doesn’t know something, distinguish related concepts, and learn through conversation, then language quality can continue improving afterwards.

Campaign 17 and onwards

The chat log from the newest campaign preflight checks (does the file format affect anything? is anything forming on the brain map?) is encouraging.

It now answers the core smoke prompts cleanly:

[user]who are you?

[Ninereeds]I am Ninereeds. I am a language model. I answer from what I have learned.

[user]what does a dog look like?

[Ninereeds]A dog has four legs. A dog has fur. A dog has ears, a nose, and a tail.

[user]is a dog a machine?

[Ninereeds]No. A dog is not a machine. A dog is an animal.

What’s better now?

• the first signs of an identity are visible, albeit not grounded yet
• it now answers the question
• it can answer both open and boolean questions

Remaining weakness:

• trails off after the correct start for some concepts
• some answers start correctly but degrade near the end
• the model still needs stop discipline and more repeated short completions

I’m still not sure what the shape of each concept file has to look like, and this is why it’s so much fun. The question I’m currently exploring isn’t “How large should the model be?” but “What does the smallest effective lesson look like?” Every experiment refines that answer a little further. Once the curriculum stabilizes, I’ll automate the experimentation so the system can propose and evaluate new curriculum variants continuously. It’s completely different from how a transformer works, and I’m planning a dedicated training machine that can run experiments autonomously 24/7. Thus far, I’ve been doing this in my spare time after work and on the weekends, on the same machine I’m using for everything else.