$ ./guess_the_languageAI & ML
AI & ML · easy · #42

What does a GPT-style LLM learn to predict during pretraining?

  1. The next token
  2. A label per document
  3. Each sentence's sentiment
  4. The best search URL
Autoregressive pretraining maximizes the likelihood of each token given the ones before it.
Play AI & ML ▸