tiny-llm: relearning AI from scratch

I understand LLMs well enough, like most nerds I think. I’m still hazy on some details here and there, so I’m going back and studying each step, one at a time, using tiny/toy models, this time writing my own code, using models a million time smaller, such as 30K parameters instead of 30M. I’ve already learned some interesting things spending only a day at it.

The code is at:

https://github.com/robertdavidgraham/tiny-llm

This blogpost is about the absolute very basics before we get into the code. My next blogpost will be “My First Token” how I go from the neural-net describe here to consuming and producing tokens — using trivially simple code.

The very basics: a neuron

What everyone learns in AI textbooks is how a “neuron” works. A typical model (drawn by ChatGPT for me) looks something like the diagram below. You’ve probably seen something similar before.

This is a single “neuron” also known as a “perceptron”. It teaches the basics:

1. Every input pixel (xᵢ) has a “weight” (wᵢ) applied to it (multiplied).

2. The weighted pixel values are then added altogether.

3. A bias (b) is added

4. Then a non-linear “activation” function is applied.

This bias is an arbitrary hack you do in mathematics to prevent zeroes 0 from happening in results. Zeroes mess up the works, because zero multiplied by any number is zero. A bias is added to the result so that in case it’s zero, then it’s not a zero value any more. There’s really no specific value, it starts as a random number and is trained along with the neuron’s weights.

The activation function is another arbitrary hack. The matrix operations we’ve doing are linear operations. The problem with many stacked linear operations is that they can be replaced with a single linear operation. Said another way, if you stack neural-net layers, they can be replaced with a single layer, unless you add a non-linear function to the mix.

A popular non-linear function is the “hyperbolic tangent” or “tanh()”. It converts whatever input you have into a value between -1 and 1. This is the one I’m using in my toy models.

But it’s largely arbitrary, there are a lot of other popular non-linear functions. You can swap them around in my little toy models to see how they change the accuracy of the output. (Which I plan to do at some point).

Where we get the weights (and the bias parameter) is from training and back propagation. We present images that are labeled to be a dog (or not a dog). We then go from the answer backward to adjust all the parameters so that they are more likely to produce the right answer next time.

A neural-network is where we have more than one neuron. Instead of a single yes/no answer, we might want more sophisticated output with multiple values that we can feed into something else. When training, the neurons learn to recognize different aspects of the image.

A single layer of neurons might look like the following.

What you see here is three neurons in parallel. They all receive the same input, but apply their own weights, and produce their own output. During training, the neurons that contribute the most error to the result are then trained/adjusted the most.

I was told that LLM used matrixes underneath in the code, but I never truly understood it until one day I had an epiphany: the image above is exactly the algorithm they teach for multiplying matrixes.

In other words, you learn in high-school how to multiply two matrixes:

The way you multiple this is take the variable [x1] and multiply by [a11 a12 a13] respectively, then x2 times the second row, x3 times the third row, then x4 times the fourth. Then you sum up all the columns to produce the final result.

In other words, matrix multiplication is coincidentally the identical algorithm as calculating the output of a neuron. I had assumed that things had to be twisted and massaged to make matrix math works. It turns out that it was matrix math all along.

Well, not so coincidentally, I suppose. The thing about basic math you learn in high-school is that it pops up everywhere, whether its fractions, algebra, matrixes, calculus, or whatever. It sounds esoteric when you learn it, but in reality, it’s everywhere.

Anyway, that means I can draw the diagrams differently, as matrix math rather than neural nets. The following diagram below is the neuron layer as the picture above, but now as matrix blocks.

Or, we can express it as a single equation, where each variable is a matrix, and f() is the activation function:

y = f(xw + b)

Or, in JavaScript, with longer variable names (don’t worry about what they mean right now).

The point is that I was thinking of messy code to do the neuron calculation and it turns out that it’s quite simple.

Next steps

This blogpost is about the absolute basics, before I start my project, so that I can refer back to it.

It’s my own internalized understanding of how a neuron works, seeing it from perspective of a diagram, from math, and from code — flipping between then to appreciate all the perspectives.

For example, I hate it when people say the activation function is non-linear, as if that explains everything. Maybe it would, if I had a PhD in math, but I don’t. I need somebody to explain what it means — not simply define it, but all these hidden connotations. You can’t even ask ChatGPT to explain it, because it’ll repeat the definition everyone knows, and not the connotations and context that I’m missing. Later in the project, we’ll play with different activation functions (or removing it) and see how that impacts things, to truly understand all this for ourselves.

The next blogpost will be “My First Token” where I use that single neuron to predict text, taking in tokens one at a time producing the next expected token. A single neuron is too simple to get a good result, but I’m curious to see what result it produces.

---

AI Disclaim: all images are ChatGPT created, all text is me created (no AI, not even editing, which is why there are typos).

---

Here is another diagram of a single neuron. It’s better in some ways.