learn deep. ship safe.Practice testsFollow ByteMunSubscribe

Concept

How LLMs are trained

The pipeline from data and tokenization through pre-training to fine-tuning and preference tuning.

Concept map
Learn firstLLMs
How LLMs are trained
Then learnFine-tuning

What should you learn before How LLMs are trained?

  1. Deep learningMachine learning with many-layered neural networks.
  2. Tokens and tokenizationSplitting text into the units a language model reads and is billed by.
  3. TransformersThe attention-based neural network architecture behind modern language models.
  4. Large language models (LLMs)Transformer models trained on large amounts of text to predict the next token.

What to learn after How LLMs are trained

  • Fine-tuningFine-tuning continues training a pre-trained model.

Connections to other concepts

Read and watch: How LLMs are trained explained

All topics