What does attention add to layered activations?
A feedforward network propagates activations through its layers. Causal attention lets each position combine information from earlier tokens.
Read on through a shared source
Can a network evolve its own wiring? →First follow activations through a fixed network. In MarI/O, selection changes the connections themselves. Following a signal and searching for a useful topology are different operations.How does a network change after a wrong answer? →Forward propagation computes a prediction. In micrograd, a loss and its gradients connect that prediction to the next parameter update.From predicting a token to using a language model →Building a predictor makes its training target concrete. Using a trained model shows how context and instructions put that predictor to work.
Compare the passages
the heart of the network as an information processing mechanismyou can do weighted aggregations of your past Elements