How ChatGPT and Claude Are Actually Built: The 5-Stage AI Training Pipeline Explained

You type a question into ChatGPT or Claude, and a few seconds later you get a clear, polite and often impressively accurate answer. It feels like talking to someone who has read everything. But there's no magic inside. Every modern assistant, whether it's GPT, Claude, Gemini or Llama, comes out of the same AI training pipeline: a sequence of five stages that turns raw internet text into a helpful assistant.

This guide explains each stage in plain English, using what the companies themselves have published: OpenAI's and Anthropic's own descriptions of how they train their models, plus the research papers that defined the field (InstructGPT, Chinchilla, LIMA, Constitutional AI, DPO, Llama 3 and DeepSeek-R1).

We also did something most explainers don't. We ran the whole pipeline ourselves, at a tiny scale, on a laptop-class CPU, using ByteMuni's own articles as training data. You'll see the real numbers at every stage: how much data the cleaning step threw away, what our tokenizer learned, how the model's perplexity fell during training, and what fine-tuning and preference tuning changed. All the code is included.

Quick summary. The five stages are: (1) data collection and cleaning, (2) tokenization, (3) pre-training, (4) supervised fine-tuning (SFT) and (5) preference tuning and reinforcement learning (RLHF, Constitutional AI, DPO and, for reasoning models, RL with verifiable rewards). Evaluation runs alongside every stage. Pre-training gives the model its knowledge; the later stages teach it how to behave.

The AI training pipeline at a glance

Two companies use almost the same building blocks but get very different results, because the quality of each stage, especially the data and the post-training, makes the difference.

Stage 1 of the AI training pipeline: data, where every model begins

Everything starts with text: lots of it.

Where the data comes from. OpenAI says its foundation models are developed using three main sources: information that is publicly available on the internet, information accessed through partnerships with third parties, and information that users, human trainers and researchers provide or generate. Anthropic's model training notice (effective July 8, 2026) describes a similar mix: public web pages collected by its crawler, ClaudeBot; purchased and licensed datasets; data from users who opted in to training; and data Anthropic generates internally.

A huge share of public web data comes from Common Crawl, a non-profit archive that, as of November 2025, held "over 300 billion web pages spanning 17 years." GPT-3's training data famously started from 45 TB of compressed Common Crawl text, which shrank to about 570 GB after filtering.

Why cleaning is the hard part. Raw web text is messy: spam, duplicate pages, cookie banners, adult content, pages that just list personal details, and machine-generated junk. The rule every lab repeats is simple: better data beats more data.

A typical cleaning pipeline looks like this:

1. Extract the main text from raw HTML, discarding menus and ads.

2. Filter by language and remove unwanted content, such as hate speech, adult material and spam.

3. Remove personal information where possible, and skip sites that aggregate it.

4. Deduplicate at the URL, document and line level. Duplicates waste compute and make models memorise.

5. Score quality with rules (length, symbol ratio, repetition) and with classifier models.

6. Balance the mix across web text, code, books, science, maths and languages.

The open FineWeb project from Hugging Face shows what this looks like at scale: it processed 96 Common Crawl snapshots into a 15-trillion-token dataset, using text extraction, language identification, MinHash deduplication and quality filters. Its educational subset, FineWeb-Edu (1.3 trillion tokens), used a classifier trained on annotations generated by Llama-3-70B-Instruct, and models trained on it scored notably better on knowledge and reasoning benchmarks such as MMLU and ARC.

OpenAI's help page describes the same idea in plain terms: it applies filters to remove material it doesn't want its models to learn from, "such as hate speech, adult content, sites that aggregate personal information, and spam," and does not intentionally collect from paywalled sources or the dark web.

Website owners have a say. Both OpenAI (via its GPTBot crawler) and Anthropic (via ClaudeBot) say their crawlers respect robots.txt rules, so a site owner can block training crawlers.

Synthetic data is growing. OpenAI says it "increasingly" uses synthetic data, such as model-generated prompts and multilingual examples, to fill gaps where real data is sparse.

What we found when we cleaned ByteMuni's articles

For our tiny experiment, the "internet" was 11 ByteMuni articles. Our cleaning script removed code blocks, images, links and Markdown symbols, then applied two simple quality filters.

Holding back a validation set matters: it's text the model never trains on, so it tells us whether the model is learning language or just memorising.

Even on hand-written articles, the filters cut 22% of the lines (mostly headings, table rows and short fragments). We found no exact duplicate lines, because each article was written once. On the open web it's a different story: duplication is one of the biggest problems, which is why every serious pipeline deduplicates.

Stage 2: Tokenization, turning text into numbers

A model never sees letters or words. It sees tokens, small chunks of text that each have an ID number. OpenAI describes tokens as units that "may represent whole words, parts of words, or punctuation."

Most LLMs use a method called Byte-Pair Encoding (BPE) or a close relative. It works like this:

1. Start with individual characters (or bytes).

2. Count every pair of neighbouring symbols in the training text.

3. Merge the most common pair into a new token.

4. Repeat until the vocabulary reaches the size you want.

Common words become a single token; rare words get split into pieces. That's why a model can read a word it has never seen: it simply reads the parts.

Our BPE tokenizer, built from scratch

We wrote a BPE tokenizer in about 60 lines of Python and trained it on the cleaned ByteMuni text with 1,000 merges. These are real outputs:

base characters: 94 | vocab size: 1095
first 12 merges: [' a', ' t', 'in', 'er', 'on', 'he', 're', ' s', ' c', 'at', 'es', 'nd']

' transformers'  -> [' trans', 'form', 'ers']
' tokenization'  -> [' token', 'iz', 'ation']
' unbelievable'  -> [' un', 'b', 'e', 'li', 'e', 'v', 'able']
' LangChain'     -> [' LangChain']
' Bengaluru'     -> [' B', 'en', 'g', 'al', 'ur', 'u']

train text: 200,790 characters -> 72,655 tokens (2.76 characters per token)

A few things stand out:

• The first merges are the most common letter pairs in English: "in", "er", "on", "he", "re".

• "LangChain" became a single token, because it appears so often in ByteMuni's articles. A tokenizer reflects its training data.

• "Bengaluru" was split into six pieces, because it barely appears in our text. This is a real-world issue: tokenizers trained mostly on English text split Indian names and Indian-language text into more tokens, which makes them slower and more expensive to process.

• With only about 1,100 tokens, our vocabulary is tiny, so each token covers just 2.76 characters. Production tokenizers have vocabularies of tens of thousands to a few hundred thousand tokens, so common English words are usually a single token and a token averages roughly four characters.

Why this matters for you: tokenization explains some famous LLM quirks. Models sometimes miscount the letters in a word, because they see "strawberry" as a few chunks, not ten letters. And API pricing and context limits are measured in tokens, not words.

Stage 3 of the AI training pipeline: pre-training, where the model learns almost everything

Pre-training is the expensive stage, and it has one surprisingly simple goal: predict the next token.

The model sees a piece of text, such as "Instead of turning left, she turned", and guesses the next token. It compares its guess with the real one, measures how wrong it was (the loss), and nudges its internal numbers (its parameters, or weights) to do slightly better next time. Then it repeats this across trillions of tokens.

OpenAI uses that exact example in its help article: early in training, the model's completions are "largely random," but as it processes more text it gets better at predicting likely next words. Nobody teaches it grammar, facts or Python. It picks them up because they help it predict text better.

The model doesn't store a copy of its training data. As OpenAI explains, the model consists of parameters that are "adjusted slightly to reflect patterns," not a database of the original sentences. (Models can still occasionally memorise text that appears many times, which is one reason labs deduplicate.)

The architecture is the Transformer, a decoder-only design with causal self-attention, which means each token can only look at the tokens before it. If you're new to Transformers and attention, ByteMuni's guide to generative AI with LangChain and Transformers explains attention with a runnable example.

How big is pre-training?

Scaling laws: how much data per parameter? DeepMind's 2022 Chinchilla paper found that, for a fixed compute budget, many earlier models were too big for the amount of data they saw. Its compute-optimal recipe works out to roughly 20 training tokens per parameter, which is why Chinchilla (70B parameters, 1.4 trillion tokens) beat much larger models. Today, labs often train well beyond that ratio (Llama 3's 8B model saw about 15 trillion tokens), because a smaller model that has seen more data is cheaper to run for millions of users.

What you get at the end is a base model. It knows a huge amount, but it doesn't know it's supposed to be a helpful assistant. Ask a base model "What is the capital of France?" and it might continue with "What is the capital of Germany?", because to a text predictor, a list of quiz questions is a perfectly likely continuation.

Our pre-training run: watching a model learn

We trained a small GPT-style Transformer (2,221,824 parameters, 4 layers, 128-token context) on the 72,655 tokens of ByteMuni text for 600 steps. It took about 5 minutes on a CPU, with no GPU.

Here is what it wrote from the prompt "The model", before and after training:

Before (random weights):  The model incluim datasw prote only ro uses 0 Jndambd whichemoryity wellur thaninewugging...
After 600 steps:          The model. APIsiddown KAG Binance's Justry-city well prompt, you need to US Emised attacker's an AI in such a codeanches.

The "after" text is still nonsense, but it's very different nonsense. It now uses real words from ByteMuni's articles (API, KAG, Binance, prompt, attacker's, AI), basic punctuation and even possessives. Validation perplexity fell from 1,316 to 83.

Then we trained too long, on purpose. In a second run, we kept going to 1,500 steps on the same 72,655 tokens. Training loss kept falling (to 0.86), but validation perplexity bottomed out at 90 at step 400 and then climbed to 352. The model had started memorising its training text instead of learning patterns that generalise. That's overfitting, and it's exactly what the Chinchilla scaling laws warn about: our model had about 0.03 training tokens per parameter, roughly 600 times less than Chinchilla's 20. Real labs avoid this by using vastly more data, which is why Stage 1 matters so much.

What is perplexity? Perplexity is the exponential of the average loss. You can read it as "how many tokens the model is choosing between," on average, when it predicts the next one. A model guessing randomly from our 1,095-token vocabulary would have a perplexity of about 1,095. Lower is better.

The reference code has errors: a quick correction

Several popular "build your own LLM" tutorials use a 6-layer, 512-dimension model with a 32,000-token vocabulary and say it has about 44 million parameters, and that a 4-layer, 256-dimension model with an 8,000-token vocabulary has about 15 million. We built both configurations in PyTorch and counted. They have 52.2 million and 7.3 million parameters respectively. The difference matters when you plan memory and training time, so always count parameters in code instead of trusting a number in a blog post.

Stage 4: Supervised fine-tuning, teaching it to be an assistant

A base model knows language but not its job. Supervised fine-tuning (SFT) fixes that by training it on example conversations: a prompt paired with a high-quality response, written or checked by people.

The mechanics barely change. It's still next-token prediction, but now on curated demonstrations instead of raw web text, and usually only the response tokens count towards the loss, so the model learns to answer rather than to imitate the question.

How much data does SFT need? Less than you'd think, but quality is everything.

• OpenAI's 2022 InstructGPT paper started with demonstrations written by human labellers. Its headline result: outputs from the 1.3-billion-parameter InstructGPT model were preferred by people over outputs from the 175-billion-parameter GPT-3, despite being more than 100 times smaller. Post-training mattered more than size.

• Meta's 2023 LIMA study fine-tuned a 65-billion-parameter LLaMA model on just 1,000 carefully chosen examples, with no reinforcement learning at all. Its answers were judged equivalent to or better than GPT-4's in 43% of cases. The authors proposed the "Superficial Alignment Hypothesis": a model's knowledge comes almost entirely from pre-training, and alignment mostly teaches it the format and style of a good answer.

In practice, frontier labs use far more SFT data than 1,000 examples, often mixing human-written answers with synthetic examples generated and filtered by other models. But LIMA's lesson stands: a few thousand excellent examples beat a million mediocre ones.

Our SFT run

We wrote 30 short question-and-answer pairs ("Question: What is a token? Answer: A token is a small piece of text…") and fine-tuned our base model on them for 8 passes, counting loss only on the answer tokens. Then we asked it questions that were not in the training set:

This result is more useful than a perfect one would have been.

• SFT taught the format completely. After just 8 passes over 30 examples (the answer loss fell from 3.85 to 0.13), the model answered in clean, grammatical sentences and stopped at the end of the line. Before SFT, it couldn't do either.

• SFT didn't add knowledge. Every answer is a fluent sentence copied from the training examples, but it's the wrong one. Our 2-million-parameter base model learned almost nothing about the world during pre-training, so there was no knowledge for SFT to unlock.

That's the LIMA paper's Superficial Alignment Hypothesis in miniature: fine-tuning teaches a model how to answer, but what it knows comes from pre-training. It's also a vivid picture of a hallucination: a confident, well-formed answer that's simply wrong. Real assistants hallucinate far less often because their base models know vastly more, but the mechanism is the same.

Stage 5 of the AI training pipeline: preference tuning and reinforcement learning

SFT teaches a model to follow instructions. But most questions don't have a single right answer, only answers that people tend to prefer: clearer, more honest, safer, better structured. Stage 5 teaches those preferences.

RLHF: reward model plus reinforcement learning

Reinforcement learning from human feedback (RLHF), the method popularised by InstructGPT and ChatGPT, has two steps:

1. Train a reward model. The SFT model writes several answers to the same prompt. Human reviewers rank them. A separate network, the reward model, learns from thousands of these comparisons to predict which answer people would prefer.

2. Optimise against it. The assistant generates answers, the reward model scores them, and a reinforcement-learning algorithm (originally PPO) updates the assistant to score higher, while a penalty keeps it from drifting too far from the SFT model.

Because the reward model scores automatically, the assistant can practise on millions of prompts that no human ever reads. Anthropic's training notice describes the same idea: a "reinforcement learning feedback loop" in which human reviewers rate model responses.

Constitutional AI: Anthropic's approach

Anthropic, which makes Claude, developed Constitutional AI (published in 2022). Instead of relying only on human rankings for harmlessness, the model critiques and revises its own answers against a written set of principles, and AI-generated feedback based on those principles trains the preference model. This is sometimes called RLAIF (reinforcement learning from AI feedback). Anthropic published an updated constitution for Claude in January 2026, and its training notice names Constitutional AI as one of its methods.

DPO: preference tuning without a separate reward model

Direct Preference Optimization (DPO), introduced by Stanford researchers in 2023, skips the separate reward model and the RL loop. It trains directly on pairs of answers ("chosen" and "rejected"), raising the probability of the chosen one relative to a frozen reference copy of the model. It's simpler and more stable, which is why it's now widely used. Meta's Llama 3 post-training, for example, combined SFT, rejection sampling and DPO over several rounds.

RL with verifiable rewards: how reasoning models are trained

The newest twist trains reasoning models. In maths, coding and logic puzzles, an answer can be checked automatically: the code passes its tests, or the maths answer is correct. So instead of a learned reward model, the reward comes from a checker. Models practise thousands of problems, get rewarded for correct final answers, and learn to "think" in longer, self-checking chains of reasoning.

DeepSeek's R1 model, described in a peer-reviewed paper in Nature in September 2025, is the best-documented example. Its reasoning ability came largely from reinforcement learning, using an algorithm called GRPO, and the model learned behaviours such as re-checking its own steps. DeepSeek said the reasoning-focused training cost about $294,000, but as The Register pointed out, that figure covered the reinforcement-learning stage built on top of its V3 base model, which itself cost an estimated $5.58 million in GPU time.

Our preference-tuning run (DPO)

For the final stage, we wrote 30 preference pairs. For each question, the "chosen" answer was the helpful one from our SFT data and the "rejected" answer was rude and unhelpful ("Not my problem, look it up."). We trained with DPO on 24 pairs and tested on 6 held-out pairs the model had never seen.

The DPO loss fell from 0.37 to 0.006 in six passes. On the six held-out questions, the gap in log-probability between the helpful and the rude answer grew from 47.7 to 81.3, and in 6 out of 6 held-out cases the probability of the helpful answer rose more than the probability of the rude one.

To be honest about the limits: the helpful answers came from our SFT data, so the model already preferred them before DPO (the gap was positive from the start), and the rude replies were drawn from a short repeated list. DPO widened a gap that already existed rather than creating a new skill. That's realistic, though: in production, preference tuning also polishes behaviour the SFT stage has already sketched out.

Evaluation: how labs know a model is any good

Evaluation isn't really a separate stage; it runs alongside all of them.

• During pre-training: labs track loss and perplexity on held-out text, as we did. Lower perplexity means better next-token prediction.

• After post-training: perplexity becomes much less useful, because the model is no longer trying to imitate raw web text. We saw this ourselves: perplexity on our held-out ByteMuni text rose from 83 (base model) to 169 after SFT and 207 after DPO, even though the tuned models were far more useful at answering questions. Labs switch to task benchmarks and human judgements.

• Benchmarks: tests such as MMLU (multiple-choice questions across 57 subjects), coding tests and reasoning tests such as ARC-AGI.

• Human preference: platforms where people compare two anonymous models' answers and vote, such as LMArena (formerly Chatbot Arena).

• LLM-as-judge: automated tests where a strong model grades answers. AlpacaEval's length-controlled version reports a 0.98 Spearman correlation with Chatbot Arena rankings and costs less than $10 of API credits to run.

• Safety testing: red-teaming, dangerous-capability evaluations and system cards published before release.

Benchmarks are slippery. In 2023, Hugging Face investigated why the same model got very different MMLU scores. LLaMA-65B scored 0.637 with one implementation (HELM) and 0.488 with another (the Eleuther evaluation harness), purely because of differences in prompt format and how answers were scored. Same model, same questions, very different numbers. When you see a benchmark claim, ask how it was measured.

ByteMuni's explainer on AGI and the ARC-AGI benchmark covers how researchers test general reasoning.

What happens after release

Training doesn't stop at launch. Labs keep improving models using feedback, evaluations and new data, then release updated versions. Some of that feedback comes from users:

• ChatGPT: OpenAI says conversations may be used to improve its models, and users can opt out in settings.

• Claude: Anthropic's notice says it trains on data from users who opt in, along with feedback and conversations flagged for violating its usage policy.

If you care about privacy, check the data controls in whichever assistant you use, and never paste passwords, personal documents or confidential work data into a chat unless your organisation's plan guarantees it won't be used for training.

What the AI training pipeline explains about ChatGPT and Claude

Once you know the five stages, a lot of everyday behaviour makes sense:

This is also why techniques such as RAG exist: they give a model fresh, verified information at answer time to make up for Stage 1's cut-off date and Stage 3's tendency to guess. ByteMuni's comparison of RAG, CAG and KAG explains the options.

Try it yourself: a tiny AI training pipeline in five scripts

Everything in this article runs on an ordinary computer. You need Python 3.10+ and pip install torch. Our code is in the downloadable project, and the core pieces are short:

stage1_data.py          clean, filter, deduplicate, split train/validation
stage2_bpe.py           train a BPE tokenizer from scratch
minillm.py              a GPT-style decoder-only Transformer (about 60 lines)
stage3_pretrain.py      next-token pre-training with perplexity tracking
stage4_5_posttrain.py   supervised fine-tuning, then DPO preference tuning

The heart of pre-training is just a few lines. The target is the input shifted by one token, and the loss is cross-entropy:

x = data[i : i + CTX]          # input tokens
y = data[i + 1 : i + CTX + 1]  # the same tokens shifted by one: "the next token"
logits = model(x)
loss = F.cross_entropy(logits.flatten(0, 1), y.flatten())
loss.backward()
optimizer.step()

And the heart of DPO is one formula:

# log-probabilities of the chosen and rejected answers, under the model and a frozen reference
loss = -F.logsigmoid(beta * ((logp_chosen - ref_chosen) - (logp_rejected - ref_rejected)))

Swap in your own text, such as your notes, a public-domain book or your company's documentation, and you can watch a model learn it from scratch. It won't rival GPT or Claude (that takes trillions of tokens, thousands of GPUs and large teams), but it will make the black box make sense.

For practical next steps, ByteMuni's 40 AI terms every student should know and our generative AI interview questions for 2026 are good companions to this guide.

Our take

The most useful thing about understanding the AI training pipeline isn't building your own frontier model; almost nobody will. It's that it changes how you use the ones that exist.

When you know an assistant is a next-token predictor shaped by human preferences, you stop treating it as an oracle. You verify facts, give it context, ask for sources and expect it to be confidently wrong sometimes. You also see where the real competition between labs lies: not in a secret architecture (the Transformer is public), but in data quality, post-training and evaluation.

Our tiny experiment made that concrete. The same 2-million-parameter model went from random gibberish, to ByteMuni-flavoured text, to answering in our question-and-answer format, to preferring polite answers over rude ones, purely by changing the data and the training objective at each stage. That, scaled up a millionfold, is how ChatGPT and Claude are built.

Sources

• OpenAI Help Center: How ChatGPT and our foundation models are developed

• Anthropic: Model training notice (effective July 8, 2026)

• Common Crawl: Celebrating World Digital Preservation Day (November 2025)

• Penedo et al.: The FineWeb Datasets (2024)

• Brown et al.: Language Models are Few-Shot Learners, GPT-3 (2020)

• Hoffmann et al.: Training Compute-Optimal Large Language Models, Chinchilla (2022)

• Ouyang et al.: Training language models to follow instructions with human feedback, InstructGPT (2022)

• Zhou et al.: LIMA: Less Is More for Alignment (2023)

• Bai et al.: Constitutional AI: Harmlessness from AI Feedback (2022)

• Rafailov et al.: Direct Preference Optimization (2023)

• Llama Team, Meta: The Llama 3 Herd of Models (2024)

• The Register: DeepSeek didn't really train its flagship model for $294,000 (September 2025)

• Hugging Face: What's going on with the Open LLM Leaderboard? (MMLU scoring differences)

• AlpacaEval (Stanford Tatsu Lab, GitHub)

• Logiciel Tech News: How GPT and Claude are built (October 28, 2026)