What does dropout do inside a GPT training loop?
The subnetworks interpretation explains why units are omitted during training. The implementation shows where the random masking happens in an actual model.
Read on through a shared source
How can many subnetworks share one set of weights? →Dropping units during training can be viewed as training different subnetworks with shared parameters. The test-time prediction is not necessarily their exact arithmetic mean.From predicting a token to using a language model →Building a predictor makes its training target concrete. Using a trained model shows how context and instructions put that predictor to work.What does attention add to layered activations? →A feedforward network propagates activations through its layers. Causal attention lets each position combine information from earlier tokens.
Compare the passages
all of these architectures are sharing the same weights, which is regularizing them a lot.training an ensemble of sub networks and then at test time everything is fully enabled