HackerHow far is next-token prediction from answering a person?
Separate three questions: how an input becomes a representation, how training changes a network and how a capable text predictor becomes an assistant. An explanation at one level does not settle the others.
Questions in this reading · 7
- What is geometric about a word embedding?
- What does attention add to layered activations?
- How does a network change after a wrong answer?
- What does dropout do inside a GPT training loop?
- How can many subnetworks share one set of weights?
- From predicting a token to using a language model
- When does a text predictor become an assistant?
What is geometric about a word embedding?
Coordinates describe a vector relative to a basis. Embeddings use learned vectors whose directions can capture relationships among words.
Compare original passages and timestamps · 2
the ability to translate back and forth between themdirections in the space have a kind of semantic meaningWhat does attention add to layered activations?
A feedforward network propagates activations through its layers. Causal attention lets each position combine information from earlier tokens.
Compare original passages and timestamps · 2
the heart of the network as an information processing mechanismyou can do weighted aggregations of your past ElementsHow does a network change after a wrong answer?
Forward propagation computes a prediction. In micrograd, a loss and its gradients connect that prediction to the next parameter update.
Compare original passages and timestamps · 2
the entire network is just a functionefficiently evaluate the gradient of some kind of a loss functionWhat does dropout do inside a GPT training loop?
The subnetworks interpretation explains why units are omitted during training. The implementation shows where the random masking happens in an actual model.
Compare original passages and timestamps · 2
all of these architectures are sharing the same weights, which is regularizing them a lot.training an ensemble of sub networks and then at test time everything is fully enabledHow can many subnetworks share one set of weights?
Dropping units during training can be viewed as training different subnetworks with shared parameters. The test-time prediction is not necessarily their exact arithmetic mean.
Compare original passages and timestamps · 2
all of these architectures are sharing the same weights, which is regularizing them a lot.training an ensemble of sub networks and then at test time everything is fully enabledFrom predicting a token to using a language model
Building a predictor makes its training target concrete. Using a trained model shows how context and instructions put that predictor to work.
Compare original passages and timestamps · 2
give it the start of a sequence and it completes the sequenceknows how to predict the next wordWhen does a text predictor become an assistant?
Next-token prediction teaches continuation. Compare how later training shapes responses to a user; the distinction does not prescribe a single post-training algorithm.
Compare original passages and timestamps · 2
it takes the model from being a document completer to a question answererwe swap out the data set now and we train on these Q&A documentsTry it on your own example
When an answer improves, does the evidence identify the representation, training objective or post-training as the cause?