What does a GPT-style LLM learn to predict during pretraining?
- The next token
- A label per document
- Each sentence's sentiment
- The best search URL
Autoregressive pretraining maximizes the likelihood of each token given the ones before it.
Play AI & ML ▸