Concept
Tokens and tokenization
Splitting text into the units a language model reads and is billed by.
Tokens and tokenization
What to learn after Tokens and tokenization
- LLM inference and costCost and latency are counted in tokens.
- Large language models (LLMs)Transformer models trained on large amounts of text to predict the next token.
Read and watch: Tokens and tokenization explained
Practise Tokens and tokenization
- Generative AI Interview Questions for Freshers4 questions on Tokens and tokenization