What if the model has already seen the test?
A public score becomes harder to interpret when the training corpus is unknown. Novel questions probe a different failure mode, without proving what the original corpus contained.
Read on through a shared source
Will the leaderboard winner solve your problem? →A public score can shortlist models. A new version of your own task can reveal what the score misses.From predicting a token to using a language model →Building a predictor makes its training target concrete. Using a trained model shows how context and instructions put that predictor to work.
Compare the passages
we don't know what gpt4 was trained on I don't know what gpt4 was trained onsome of the questions from these things actually leaks through to some of the training sets