How Large Language Models Work: Next-Token Prediction
How large language models work, in plain words: they predict the next token, again and again. See the loop, why it makes things up, and what founders should do.
If you want to know how large language models work, the short answer is next-token prediction. A language model reads text, then guesses what piece of text should come next. It does this again and again, one small piece at a time. Everything else you see, from answers to essays to code, is built on that one move.
This post walks through that idea step by step, shows where it breaks, and explains what it means if you plan to build a product on top of a model. It follows the same ground as the lesson "Next-Token Prediction" in the How Modern AI Actually Works course on AI Entrepreneur.
How large language models work in one paragraph
A large language model is a program trained on a huge amount of text to predict the next token in a sequence. Given "The capital of France is", it gives a high chance to " Paris" and a low chance to most other words. It picks one, adds it to the text, and repeats. A long answer is just many of these guesses chained together. A raw model is then shaped into a helpful assistant with extra training. That is the whole picture. The rest of this post fills in each part.
What a token is and why models use them
A token is a chunk of text. It might be a whole word, part of a word, a number, or a punctuation mark. Models do not read letters or words the way you do. They read a sequence of tokens, each one mapped to a number.
Why not just use whole words? Because there are too many words, and new ones appear all the time: names, slang, typos, product codes. Splitting text into smaller pieces lets a model handle any string it meets by building it from parts it already knows. A common word may be one token. A rare word may be split into two or three.
Tokens matter to you for a practical reason. Model APIs count them, and what a request costs depends on how many tokens go in and come out. The course "The AI Product Stack" covers this in the lesson on model APIs, tokens and what a request costs.
Next-token prediction step by step
Here is the loop, using a tiny example. Say you type this prompt:
The best way to test a startup idea is to talk to
- Split into tokens. Your text becomes a list of tokens, each a number.
- Score every possible next token. The model looks at all the tokens so far and produces a probability for each token in its vocabulary. " customers" might get a high share. " a" or " your" might get smaller ones. Most tokens get close to zero.
- Pick one. The system chooses a token from those probabilities. It may take the most likely one, or it may sample, so that less likely choices sometimes win. This is why the same prompt can give different answers.
- Append and repeat. The chosen token joins the text. The model now predicts the next token after that, using the longer text as input.
- Stop. The loop ends when the model produces a special end token or hits a length limit.
Notice what is missing. There is no step where the model looks up a fact in a database, and no step where it plans the whole answer first. It only ever answers one question: given everything so far, what comes next?
That sounds too simple to produce useful work. The reason it does is that predicting text well forces a model to pick up a lot about grammar, facts, style and reasoning patterns, because all of those help guess the next token.
How a model learns: training on huge amounts of text
A model starts with settings that are essentially random. Training adjusts them. The process is simple to describe:
- Take a piece of real text and hide the next token.
- Ask the model to predict it.
- Compare the guess with the real token.
- Nudge the model's internal numbers so that guess would have been a little better.
Repeat this across an enormous amount of text. Over time the model gets good at predicting what real text looks like. Three things made this work at scale: large amounts of data, powerful chips, and the transformer design. The course "The Road to the Age of AI" tells that story in its lessons on deep learning and on transformers and scale.
The key point for builders is that a model learns patterns in text, not a stored list of verified facts. It has absorbed what text tends to say. That is not the same as knowing what is true.
From raw predictor to helpful assistant: fine-tuning and RLHF
A model trained only to continue text is not a good assistant. Ask a raw model "What is the capital of France?" and it might continue with more quiz questions, because that is what such text often looks like. It predicts text. It does not try to help you.
To turn it into something useful, builders add more training:
- Fine-tuning. The model is trained further on examples of good conversations: a question, then a helpful answer. It learns the shape of an assistant reply.
- RLHF. This stands for reinforcement learning from human feedback. People compare or rate model answers, and the model is adjusted to produce the kind of answers people prefer.
The result still predicts one token at a time. But its predictions now lean toward answers that are clear, on topic and polite. The lesson "From Raw Model to Helpful Assistant" covers this in more detail.
Why next-token prediction causes hallucinations
A hallucination is a confident answer that is wrong. It follows directly from how the loop works.
The model picks tokens that fit the pattern of the text so far. If you ask for a book by a certain author, a fluent title and a plausible year fit the pattern. Whether that book exists is a separate question the loop never checks. Text that sounds right and text that is right are different things, and the model is trained on the first.
Some cases make this worse:
- Questions about obscure facts the model saw rarely or never.
- Requests for exact quotes, numbers, citations or URLs.
- Long chains of reasoning, where one early slip carries through every later token.
You cannot remove this by asking nicely. You design around it. The lesson "What AI Gets Wrong" looks at the limits builders must design around. Two common tools are giving the model your own data to answer from, which is the idea behind retrieval-augmented generation, and measuring output quality instead of judging by eye, which is what LLM evals are for.
What this means for founders building AI products
Knowing the loop changes how you build. Here are four practical points.
1. Treat output as a draft unless you check it
Because the model predicts plausible text, any step where a wrong answer is costly needs a check: a person, a rule, or a lookup against real data. Decide where errors would hurt before you ship.
2. Give the model the facts it needs
Do not rely on what the model happens to remember. Put the relevant information in the prompt, or retrieve it from your own sources. A prompt like this works better than a bare question:
Answer using only the text below. If the answer is not in the text, say you do not know. Text: [your pricing page]. Question: [customer question]
This does not make the model perfect. It gives the prediction loop better material to work from.
3. Expect cost to follow tokens
Longer prompts and longer answers mean more tokens, and more tokens mean a higher bill per request. If you are choosing a price, work out what a typical request costs first. The post on pricing an AI product without losing money per user shows how to think about it.
4. Do not build only on the raw ability
Anyone can call the same model. If your product is just a prompt on top of it, there is little to stop a copy. Ask what you add: your data, your workflow, your distribution. The post on whether your startup is just a thin wrapper around ChatGPT goes through this test. If you want to plan the longer path, see the roadmap from side project to AI company in 7 stages.
Frequently asked questions
Does a language model understand what it says?
It is trained only to predict the next token. Whether that counts as understanding is not something this post can settle. For building, the useful fact is that its output is shaped by text patterns and is not checked against reality.
Why does the same prompt give different answers?
The model produces probabilities for the next token, and the system can sample from them instead of always taking the top one. A different pick early on changes every token after it.
What is a token in a language model?
A token is a chunk of text, such as a word, part of a word, a number or a punctuation mark. Models read and write tokens, and API usage is counted in them.
Can you stop a model from hallucinating?
Not fully. You can reduce it by supplying the right data, asking the model to admit when it does not know, and testing outputs with an eval set.
Get started
This post is a short version of one lesson, "Next-Token Prediction". Each lesson takes 8 to 12 minutes.
Every lesson is free. Start learning with AI Entrepreneur: sign in on the home page and take your first short lesson today.