Demystifying AI: What is an LLM and how does it work?

I think it’s important, as with any tool, even in red teaming, to understand at least at a high level what your tool is doing, so this blog will explore at a very high level what LLMs/AI is doing when we prompt them.

We’ll follow one simple prompt all the way from your keyboard to the model’s answer, covering tokens, attention, and training along the way. Then we’ll look at why you should always verify LLM output, how models differ (flagship versus fast, and different modalities), and cover some common terms.

Part 1: How an LLM works

A next-word predictor

Put very simply, a large language model is a next-word predictor. Give it some text and it predicts the most likely next piece of text, then repeats that repeatedly until the answer is complete. That sounds simple, but it takes a lot to get there.

What “large” means

Where does “large” come into all of this? LLMs contain billions of adjustable numbers called parameters, and flagship models contain trillions. When we see a model like Qwen3.8-27B, we can read the 27B as 27 billion parameters.

Those parameters are the stored knowledge of a model. The more you have, the more nuance a model can have in specific topics like coding, cyber, biology, and so on. Certain skills also start to emerge. Train a model heavily on code and it gets really good at coding, and then, more surprisingly, it can get good at things like cybersecurity as a side effect. Other skills can emerge too.

Of course, the larger a model is, the more it costs in memory, speed, price, and compute. We are all feeling the weight of this.

Our prompt

To see what’s happening under the hood, we’ll use a common toy example for explaining LLMs: “What’s the capital of France?” We’ll follow it through five steps: tokens and embeddings, attention, stacked layers, training, and generating the response.

Step 1: Turning words into numbers

Here we’re giving plain English to the LLM. Computers can’t do math on words, but they can do math on numbers. That was a problem in other technologies too, like voice translation. So, the simple answer is to assign IDs to pieces of text, and those pieces are called tokens. In modern LLMs, a single word can be broken into multiple tokens. “Unbelievable,” for example, can become three tokens: un, believ, and able.

Figure 1: Our prompt split into seven tokens. The token IDs shown are illustrative.

Each token then becomes a list of numbers, called an embedding. It captures the meaning of the token, so similar words end up with similar embeddings. A cat is an animal, so its animal score is .95, and so are dog and lion. A car is not an animal, so it gets 0. Each column is a trait, and each number says how much of that trait the word has. Our prompt has France in it, so look at that row: zero for animal, and .95 for the country trait. These are also adjustable parameters, you can think of the weight .95 is a strong connection between Cat and Animal. More on how these are adjusted later

Word	Animal	Size	Domestic	Country
Cat	.95	.2	.9	0
Dog	.95	.4	.9	0
Lion	.95	.7	.05	0
Car	0	.6	.5	0
France	0	.8	0	.95

Toy numbers. Real embeddings have thousands of values, and the traits aren’t neatly labeled.

Figure 2: Each token becomes a list of numbers. Six values are shown per token here; real models use thousands.

We can picture the words clustering into their own neighborhoods, which gives some context to the meaning of those words: pets over here, wild animals there, vehicles somewhere else entirely.

Figure 3: Related words cluster into neighborhoods on a “map” of meaning.

Step 2: Attention

Older language models struggled to remember the context of a whole sentence, because a word can mean something very different depending on the words around it. In 2017, the paper “Attention Is All You Need” was published, and it introduced transformers. And no, not the Autobots. At least not yet maybe?

Transformers are the piece of the LLM architecture that changed everything for us. The idea is that we let every word look at every other word at once and decide which ones matter. They use queries, keys, and values to work out which words matter to which other words, and that shapes and preserves context.

A better example: the tired animal

Consider this sentence: “The animal didn’t cross the street because it was tired.” What does “it” refer to?

Every token has an embedding, a list of traits. Both “animal” and “street” score high as nouns, so either one could be what “it” points to. A transformer lets “it” look at every other word and use the rest of the sentence to decide. In this case, it’s the animal. There’s more than one reason. It could refer to the street, but a street doesn’t get tired. An animal does.

Figure 4: Attention for the word “it.” Darker means more relevant. The percentages are illustrative.

Change the focus word and the percentages change too. For “cross,” animal and street score close together, because both matter to understanding who is crossing what.

Figure 5: Attention for the word “cross.” Animal and street are close.

Query, key, and value

Here’s the same example under the hood. The query is what a word is looking for. For “it,” that’s “what am I referring to?” The key is what each word advertises about itself, which comes from its embedding. How well a key matches the query decides how much of that word’s value, its actual information, gets blended in. “Animal” matches best, so “it” ends up carrying the meaning of “animal.”

Figure 6: Query, key, and value for the word “it.”

Putting attention together

With attention, every word acts as a query against all the keys, and the best matches contribute the most. Each word digests what it gathered, drawing on knowledge from training, and this repeats dozens of times, with the understanding getting deeper each time. Here’s the same idea on our prompt: the final token, the question mark, pulls the most from “France” and “capital.”

Figure 7: The same idea on our prompt: the final token pulls the most from “France” and “capital.”

Step 3: Stacking layers

When attention is repeated, each layer can serve a different purpose. Earlier layers tend to focus on grammar, middle layers on topic and meaning, and later layers on facts, intent, and maybe what comes next. Think of it as re-reading a sentence over and over, understanding it a little better each time.

An LLM stacks many of these layers, and inside each layer many attention calculations run in parallel. That’s why these models benefit so much from GPU processing.

Figure 8 is the whole architecture in one picture. Your prompt becomes tokens, the tokens become embeddings, and then they pass through the transformer block: attention, then the feed-forward step. That block repeats dozens of times. At the end, the model scores every possible next token, we pick one, add it to the text, and run it all again. The teal boxes are where a model’s weights, or parameters, live, and they’re what training adjusts.

Figure 8: The whole architecture on one page. Teal outlines mark the boxes that hold learned weights.

Step 4: Training

For a model to understand words and build embeddings around them, it needs enormous amounts of text. It reads the text and tries to guess the next word, and when it’s wrong, its weights get adjusted. It does that billions of times.

Humans also fine-tune models. This adds guardrails, gives the model a personality, and helps it avoid harmful outputs.

A model’s training directly shapes its weights, and the weights are truly the knowledge of the model. If you train a model on code, it knows code, but it won’t know nearly as much about biology.

What is a parameter, really?

The weights are the same thing as the parameters I mentioned earlier. A parameter is just a number which can be adjusted. A weight sets how strongly one piece of information influences the next: a big weight means “this matters a lot,” and a weight near zero means “ignore this.” In the diagrams below, the thicker the line, the bigger the weight.

Nobody types in the rules. There’s no table that says the capital of France is Paris. All of the model’s knowledge is spread across billions of these parameters, in the embeddings, in attention, and in the feed-forward layers. The file you download for a model is basically just that giant list of numbers, which is where the term “open weights” comes from.

Think of a giant mixing board with billions of sliders. Training is the slow work of nudging every slider a tiny bit, over and over, until the right answers come out. And once training is done, the sliders are locked. When you chat with a model, you’re only reading them.

A tiny training run

Let’s train on “The capital of France is ___.” Before training, the model’s first guess is banana. Wrong, so the weights get adjusted. After the first round it guesses Paris. Right, so we give it a small nudge. Round two, right again, more nudging. By the fourth round it’s still correct, and we raise the weight again.

Figure 9: Before training: the first guess is “banana” (60%), and Paris is only 12%.

Figure 10: After four rounds: “Paris” wins (94%), the useful connections from “capital” and “France” are thicker, and the rest have faded.

Watch what changed. The connections from “capital” and “France” toward “Paris” got thicker, and the useless ones faded. That’s what a weight changing looks like. This is a tiny example. Imagine how much goes into training trillion-parameter models.

Why training matters

Training matters in so many ways. Again, it’s the knowledge of the model, and a model can only reflect what’s in its training data. Bias in the data can start to show up in a model, so training data is very, very important. Imagine a model trained by Germany in WW2, and think about what kind of behavior it would exhibit.

The biases of the people training the models can also show up in the outputs. Here’s something I think about, especially when I hear rumors that some engineers think their model is conscious. If the people building a model strongly believed it was conscious, and trained and fine-tuned it to reflect that, what effect could that have? The model would say it’s conscious, because that’s what it learned.

Fine-tuning matters here too. It’s where a model truly gets its personality and its limits.

Changing the weights

We can also change the weights of a model ourselves. Recently, a post on X claimed that Opus 5.5 and Astra are up to 20 times better than other models at improving model weights. I haven’t verified that, but it’s a reminder that changing weights is something people are actively doing and models are getting better at doing it directly to get meaningful improvement from models.

Changing model weights is not new. It’s done to create abliterated models, where the guardrails are largely stripped out. It can also be done to plant backdoors, so that when you give the model a specific phrase, it does something like make a tool call to exfiltrate secrets. And it can be done to change a model’s personality. Researchers have shown that fine-tuning a model on narrowly bad data, like insecure code, can make it broadly misaligned. That’s called emergent misalignment. And recently people found a way to make Qwen evil: not just mean, but actually wanting to cause harm to the user.

Figure 11: Same architecture, same size. A few changed weights (purple) can mean very different behavior that’s invisible from the outside.

Backdoors are real, and the real risk comes from downloading random models from Hugging Face. A good tactic for a bad actor would be to be the first to post an abliterated version of the latest open-weight sweetheart, with a backdoor built in.

A model by itself can only produce text. The danger comes when it’s connected to tools or an agent, because then that trigger phrase can become a tool call that sends your secrets somewhere. The other danger is the model file itself: some file formats can run code when you load them, so prefer files from publishers you trust. It’s not super common, but it’s possible.

Knowledge cutoff

Models have a cutoff in their data, which matters a lot for time-sensitive information. Training data stops at a certain date. So when you ask something like, “Is this software version vulnerable to any CVEs?”, the model may only reference what it learned in training, and if the CVE is even a day old at this point, it can miss a critical one.

Agentic AI has helped with this by letting models run web searches. But sometimes a model hallucinates that nothing exists based on its training data, is confident in that answer, and doesn’t always search to verify. So with time-sensitive information, this is something very important to think about.

Step 5: Generating the response

We’re finally at the point where the model is generating a response to our prompt. It tokenizes our prompt, runs it through the transformer layers using the weights, or parameters, it learned in training, and generates the output one token at a time. It starts with “The,” then “capital,” and so on, with each new token added to the text before the next prediction. All these predictions are based on the embeddings, the weights, and attention.

There is tons of math involved here, and I do not know any of it. But alas: the capital of France is Paris.

Figure 12: The answer, built one token at a time. The percentages are illustrative.

Part 2: Why you verify

For the second half of this post, we’ll go over other topics and terms. A heavy one is hallucination.

Hallucination

Models have gotten much better at this, but it still exists, and if you’re using a local model, you’re definitely still going to see hallucinations. A lot can cause a hallucination: a knowledge cutoff, as we mentioned; obscure topics that aren’t in the training data; extremely precise details; and very open-ended questions. The model’s goal is to predict plausible-sounding text, not facts, so some of what it produces can be very false.

Non-determinism and temperature

Models also pick the next token with a bit of randomness, which makes them non-deterministic. That’s why the same prompt can give different outputs each time. This is greatly influenced by the model’s temperature setting. A lower temperature makes a model more consistent but less creative. It does not make the model more accurate. Accuracy comes from the quality of the training data and the training itself.

Temperature is the dial behind that randomness. At low temperature, the token with the highest weight wins every time: consistent and predictable. At high temperature, other tokens may be picked over the top choice, which adds randomness and some creativity. That can be useful, especially for creative work. In this example, even at the highest setting the Paris weight is high enough that it still gets picked most often, but not always.

Figure 13: Low temperature (0.2): Paris wins almost every time.

Figure 14: High temperature (2.0): Paris still leads, but the other tokens get picked more often.

Part 3: Model differences and picking the right tool

Not every model is built for the same work, and it’s important to know the differences for better results and better efficiency.

Reasoning vs. standard models

Standard models answer right away. They immediately start giving you the next token. Reasoning models think before answering. That uses more tokens, but it can produce better output on agentic work and complex tasks. Like humans: give us time to think and we’ll give a better answer. Reasoning models use the same methodology for generating a prompt to generate its reasoning chain, this adds more context to the initial prompt by asking questions, looking for other relevant information, all to solve more complex problems.

Flagship vs. fast models

There’s a time and a place for a flagship heavy hitter like Opus, and a time and a place for something like DeepSeek V4 Flash. Maybe that’s because Opus cries at the sound of cyber and DeepSeek doesn’t. Or maybe it’s because you don’t need a wrecking ball to take down one door.

When you’re working on simpler tasks, it can be better to pick one of these flash models, like DeepSeek V4 Flash or GPT Luna. If you’re working on something more complex, Opus or Sol might be your go-to. It’s also good to use the smaller, faster models as sub-agents, meaning separate model calls doing work on behalf of Opus or Sol. (More on agents in Part 4.)

Family	Flagship	Fast tier	Weights
Claude	Opus: strongest on hard reasoning and long, complex work	Sonnet: balanced for most everyday work. Haiku is faster still.	Closed
GPT	Sol: higher tier, more capable and costlier	Luna: fast, cost-efficient tier built for volume	Closed
DeepSeek	V4 Pro: the larger model for the hardest work	V4 Flash: small, fast, and very cheap	Open

Model names change every few months, so learn the pattern: bigger models are more capable but slower and pricier, and smaller ones are faster and often much cheaper.

There are tons of benchmarks out there, but I pick my models based on the task and on what I enjoy using at the time. Sol may be more efficient than Opus, for example, but for me the output of Opus is just much better for my tastes.

Modalities

There are different kinds of models as well: text, code, image, audio, video, and even multimodal models. Some models can’t do it all. Claude models, for example, don’t generate images as of today. They can understand an image, but to create one they’d use code to generate it, whereas an image model does something much different, as we’ll see shortly.

When you’re building workflows, you can have one model hand a task off to another. Think of ChatGPT Sol handing off to Meshy.ai to create 3D models.

Everything becomes tokens

In each modality, the input is tokenized differently. An image is cut into a grid of small squares, and each square is a token. Audio is sliced into chunks, and video into frames. After that, they’re all processed the same way, but the training data and weights are vastly different across modalities.

Figure 15: Each modality is chopped into tokens differently.

How image models are trained

For image models, training comes from pictures paired with captions. The model learns which words go with which patch, or token. To generate an image, it starts out as noise, like TV static, and, guided by the caption you gave it, removes the noise step by step.

Figure 16: Image generation starts from noise and removes it step by step.

Beyond the usual tell-tale signs of AI-generated images, the process can leave traces of noise behind, which is one way AI images get spotted. But models are getting better, and those traces are getting harder to detect.

How to choose the right model

When you’re choosing a model for your work, ask yourself:

• How hard is the task?
• What are the input and the output?
• Does it need to know relevant, current information?
• Does it need to use tools?
• What’s at stake?

Some things are okay to be AI slop. Others are not.

Part 4: Key terms

Context windows

A context window is a model’s working memory, and it’s measured in tokens. Everything fits in there: your CLAUDE.md or AGENTS.md, the codebase you’re working on, the conversation so far, and even the model’s reply. When the window fills up, information gets summarized or dropped, and the model can forget key things by accident.

Figure 17: What fits in a context window. Older content falls out of the window.

Larger context windows let you work with more data, but they can start to bury important details. Some models are capable of large context windows and others not so much.

Context windows also take a lot of memory, which matters with local models. You may be able to run a model at a 32K context window, but that’s not going to be great for long tasks and agentic work. If you’re like me and use something like Hermes, or you have large CLAUDE.md or AGENTS.md files, you can’t get away with small context windows at all. There are tons of ways to manage memory and context.

RAG: retrieval-augmented generation

RAG was first introduced as a way to give a model knowledge it wasn’t trained on, like company SOPs or policies. That lets you avoid retraining a model. It also helps with limited context windows, because you can pull in only the relevant pieces of a large amount of data. It can work as memory for a model too, stored in files or a database that the model can reference. In my own setup, I keep a ledger of tasks completed, issues, and key things from my tool development, stored for RAG.

Here’s how it works. You ask a model a question. The system turns your question into an embedding, the same idea we saw earlier, and searches your documents for the closest matches. It retrieves those passages and adds them to the context window. Then the model generates a response based on that data, your prompt, and its training.

Figure 18: The four steps of RAG.

RAG is also extremely effective with agent memory. An example with my Hermes setup is using Obsidian and markdown for ledgers of all my projects. When I task my agent to work on Unkn0wnC2, it reads the ledger for previous agent activity or open issues, then once a task completes it updates the ledger. I also use ledgers and SQL databases for AI assisted testing, so all my assets go into a database with their respective services, findings, etc.. This makes it easy for a model to search the DB or ledgers for relevant information without wasting useful context space and tokens.

More terms to know

Prompts are what we type to the model.

System prompts are the behind-the-scenes instructions that shape how a model behaves. They matter for security too, because they can be targeted to bypass guardrails or exploit an LLM, for example through prompt injection.

Inference is when the model is running, using what it learned. Training builds the weights once. Inference is every time you use them.

Agents are software that calls a model in a loop and can take actions like planning, using a tool, and reading local files, continuing until a task is complete. Examples are Claude Code, Hermes, and Codex. The agent itself runs on your machine, but the model it calls could be in the cloud or running locally.

Tool use lets a model search, run commands, or use apps. MCP (Model Context Protocol) lets agents connect to a server that contains tools and instructions for how to use them.

Open-weight models can be downloaded, run on your own hardware, and modified by anyone. People often call these open source, but licenses vary, so open weight isn’t always open source. Closed-weight models are proprietary and only available through APIs.

Key takeaways

1. An LLM is a next-token predictor, and everything is tokens. Whether your input is an image or text, it becomes tokens.
2. A model’s weights, or parameters, are its knowledge. Training builds them, and changing them can change its knowledge or its behavior.
3. Verify the output of your model. Models only know what they were trained on, or what context can be provided to them.
4. Match the model to the task. You don’t need to spend tons of usage letting a big model like Opus think about how to write a basic script when a smaller, cheaper model that doesn’t need to reason as much will do.

A note on the illustrations: the numbers in the diagrams (embedding values, attention percentages, probabilities, and token IDs) are illustrative, not output from a real model.

---

See AI at work in offensive security

Knowing what happens under the hood is the first step. The next is knowing where AI actually earns its place in real offensive work, and where a human still has to make the call.

Fellow OnDefend red teamer Matt Zamat walks through how the team uses AI across the offensive workflow.

Read How OnDefend Uses AI Across the Offensive Workflow