$ ./guess_the_languageAI & ML
AI & ML · easy · #36

In NLP, what is tokenization?

  1. Splitting text into units
  2. Encrypting text before use
  3. Translating text to English
  4. Compressing text with gzip
Tokenizers break text into words, subwords or characters that map to vocabulary ids.
Play AI & ML ▸