Attention Is All You Need
A large language model (LLM) is an AI model — typically a neural network — trained on a vast amount of text for natural language processing, especially language generation. LLMs can generate, summarise, translate and analyse text, and they are the basis for modern chatbots such as ChatGPT, Claude, Gemini, Grok and DeepSeek.
Almost all of them rest on one architectural idea. In 2017, Google researchers introduced the transformer at NeurIPS in a paper titled "Attention Is All You Need" — a mechanism that lets a model process the relationships between all elements in a sequence simultaneously, regardless of how far apart they sit.
That simultaneity is the whole trick. Earlier approaches read text roughly in order, and struggled to connect a word to something mentioned far earlier. Generative pre-trained transformers (GPTs) are the LLM type pre-trained to predict the next word, then often fine-tuned to follow instructions and behave as assistants.
Everything Is Numbers and Windows
Two unglamorous mechanics shape what these systems can actually do.
The first is tokenization: text is converted into numbers using methods such as byte-pair encoding, where frequently repeated character pairs are progressively merged into longer units. A model never sees letters — it sees token IDs.
The second is the context window, the amount of text a model can consider at once. The scale of change here is easy to miss: GPT-2 had twelve attention heads and a context window of only 1,000 tokens. Everything outside that window may as well not exist.
The model sizes moved just as fast. GPT-1 arrived in 2018, GPT-2 in 2019, and GPT-3 in 2020 with 175 billion parameters. Storage alone becomes a real constraint at that scale: one billion parameters in float16 format require 2 gigabytes.
What It Costs to Build One
Training costs escalated sharply as the models grew.
GPT-2's training cost was approximately $50,000. Megatron-Turing NLG 530B in 2021 cost around $11 million, and PaLM — 540 billion parameters, 2022 — cost about $8 million.
The economics are captured by scaling laws. The Chinchilla scaling law relates training cost, model size, dataset size and loss, with the useful rule that it costs roughly 6 FLOPs per parameter to train on one token, while inference costs only 1 to 2 FLOPs per parameter. Building the model is vastly more expensive than running it — which is why training is done once, at enormous cost, and the result is served to millions.
Raw prediction is not enough on its own. Reinforcement learning from human feedback trains a reward model to predict which text humans prefer, then fine-tunes the LLM to satisfy it. Before fine-tuning, most LLMs are simply next-token predictors.
Confidently Wrong
The best-known failure mode has a specific name. Hallucination is when a model confidently asserts claims of fact that are not justified by its training data — generating text that is syntactically sound, fluent and natural, but factually incorrect.
That combination is what makes it dangerous. A hallucination does not arrive looking like an error; it arrives looking exactly like a correct answer. Biased or inaccurate training data makes outputs less reliable still, which is why benchmark evaluations try to measure reasoning, factual accuracy, alignment and safety.
Progress continues on the reasoning side. Reasoning models from 2024, such as OpenAI's o1, generate step-by-step analysis before answering — and on International Mathematics Olympiad problems, GPT-4o scored 13% accuracy while o1 reached 83%.
Open Weights and Polluted Data
The ecosystem split into two camps. Closed models such as GPT-3 and its successors are restricted to API access only. Open-weight models have grown in popularity since 2022:
- BLOOM and LLaMA, released with usage restrictions
- Mistral 7B and Mixtral 8x7B, under the permissive Apache licence
- DeepSeek R1 in January 2025, a 671-billion-parameter model with performance comparable to OpenAI's o1
Since 2023 many models have also gone multimodal, handling text, images, audio and 3D data together.
There is one genuinely strange problem ahead. LLM-generated content increasingly contaminates the web that future models train on — so data cleaning may in future include filtering out AI-generated text. Preprocessing already involves removing low-quality, duplicated or toxic data to improve training efficiency. The models are, slowly, polluting their own food supply.
