Machine
One of the seven tools of Topsoil Commons. This is Layer 5, autonomy.
Machine shows how a small model learns to write, by building every part of it from basic code you can read.
You get code you can read, run, and change, and a clear record of how much the model improved as it learned.
First step: read the two short results below. They show how far the error fell while the model learned from two well known texts.
Written in Python, with a common maths library called NumPy.
What the model learned
The model learned to write more like the text it was trained on. Its error was measured on text it had not learned from, and a lower error means its guesses at the next letter were closer.
In the sonnets run, the model had 358,052 parts and learned over 250 steps. Its error on text it had not learned from fell from 5.297 at the start to 2.413 at the end.
In the Alice run, the model had 360,368 parts and learned over 250 steps. Its error on text it had not learned from fell from 5.7250 at the start to 2.4172 at the end.
The machine also checked its own math, and it got every check right, 7 of 7.
About
The code builds a small learning model from basic parts. Each part is written out in full, so you can follow how the model reads text, makes a guess, sees how far the guess was off, and gets better.
You can run the learning yourself and watch the error fall as the model learns.
For agents and builders
The two language model runs in full
| Run | Text it learned from | Parameters | Steps | Characters of text | Validation loss at the start | Validation loss at the end |
|---|---|---|---|---|---|---|
| Sonnets | Shakespeare sonnets | 358,052 | 250 | 96427 | 5.297 | 2.413 |
| Alice | Alice's Adventures in Wonderland, Project Gutenberg ebook 11 | 360,368 | 250 | 145275 | 5.725 | 2.417 |
Key: parameters are the model's parts, counted by the code. Validation loss is the model's error on text it did not train on. Lower is better. The run figures stand in results/char_lm_run.json and results/alice_char_lm_run.json.
Gradient checks: 7 of 7 tests pass, at a tolerance of 1 in 100,000. Tolerance here means how far a checked value may stray from the value worked out by hand before the check fails.
PyTorch comparison, from results/benchmark_pytorch.json: a matching PyTorch model took 120.6 milliseconds per step, against 498.5 milliseconds per step for the model built here, so PyTorch was 4.13 times faster per step.
For agents: the results stand as data in this repository, in results/char_lm_run.json and results/alice_char_lm_run.json, with the PyTorch comparison in results/benchmark_pytorch.json.
Built on Graph: shared truth.
Unlocks the ability to build.
The research under this layer: Foundations, Artificial intelligence and human computer interaction