Home

Understanding LLM Tokens: How Large Language Models Process Text

Tokens are the fundamental building blocks that help Large Language Models (LLMs) process text. Understanding how they work is essential, not just for grasping model behavior, but also because you are billed directly based on token usage.

What Are Tokens?

Tokens are simply numbers that represent how an LLM “thinks” about the text you provide. The process of converting text into tokens is called encoding.

The tokenization process works in two parts:

Decoding is the reverse process:

The Complete LLM Process Flow

The end-to-end request lifecycle looks like this:

  1. Tokenizer encodes your input text into tokens.
  2. LLM processes your tokens.
  3. LLM produces output tokens.
  4. Decoder translates output tokens back into readable text.

Input vs. Output Tokens

You are billed for both input and output tokens, but output tokens typically cost significantly more (often 3x to 5x higher) than input tokens. Designing concise prompts that generate focused, fewer output tokens is one of the most effective ways to optimize API spend.

How Tokens Are Created

Tokenization starts with a massive corpus of text—similar to the data used to train the LLM itself.

Imagine a tiny corpus consisting of just one sentence: “the cat sat on the mat.”

  1. Character Extraction: All individual characters are extracted, and each becomes its own initial token in the vocabulary.
  2. Pattern Matching: Common groupings of characters are identified (e.g., “TH” appears twice in “the”, “AT” appears in “cat” and “mat”). These groupings get their own tokens.
  3. Hierarchical Grouping: Groups of groups are formed (e.g., combining “TH” and “HE” into “THE” for the word “the”).

Why Vocabulary Size Matters

The goal is to build a large vocabulary because larger vocabularies can split words into fewer tokens, making text processing significantly more efficient.

The Weakness: Handling Unusual Words

Tokenizers naturally struggle with uncommon words, misspellings, slang, or non-English scripts. For example, a made-up word like “Frabjous” from Lewis Carroll’s poem gets chopped up into many more tokens than its character length would imply because it rarely appears in the training corpus.

Practical Cost Breakdown & Math

To put real numbers behind token usage, let’s look at pricing for popular frontier models like Claude 3.5 Sonnet and GPT-4o:

Model Input Price (per 1M tokens) Output Price (per 1M tokens)
Claude 3.5 Sonnet $3.00 $15.00
GPT-4o $2.50 $10.00

The Math in Action

Imagine building a feature where a user sends a 2,000-token prompt (context + code snippet) and the model responds with a 500-token code refactor using Claude 3.5 Sonnet:

While 1.3 cents per request sounds cheap, scaling to 10,000 daily active users running this query 5 times a day racks up over $2,025 per month. Knowing how to audit and trim tokens directly protects your margins.

Programmatic Token Counting with Python

Before dispatching requests, you can programmatically calculate token usage using OpenAI’s tiktoken library to prevent unexpected spikes or check boundary limits:

import tiktoken

# Choose the encoding matching your model family (e.g., cl100k_base for GPT-4 / o-series)
encoding = tiktoken.get_encoding("cl100k_base")

prompt_text = "Refactor this Python function to improve performance and add type hints."

# Encode text into token integer IDs
tokens = encoding.encode(prompt_text)

print(f"Token count: {len(tokens)}")
print(f"Token IDs: {tokens}")

Developer Gotchas: Why Code and JSON Waste Tokens

If you pass source code, configuration files, or JSON payloads into an LLM, you’ll notice token counts are often much higher than expected for a given character length. This happens for several reasons:

  1. Whitespace & Indentation: Spaces, tabs, and newlines (\n) are often tokenized individually or bundled awkwardly. Heavily indented YAML or Python code can waste significant token overhead on whitespace alone.
  2. Sub-word Splitting on Code Constructs: Tokenizers were heavily trained on natural language prose. Code symbols, camelCase identifiers (userAccountBalance), and snake_case terms (user_account_balance) get chopped into weird sub-word tokens. For example, user_account_balance might get broken down into user, _, account, _, balance—costing 5 tokens instead of 1 or 2 clean words.
  3. Syntax Clutter: Curly braces {, brackets [, quotes, and semicolons often consume independent tokens, meaning verbose JSON structures burn through budgets much faster than concise markdown text.

Context Windows and Memory Management

Every LLM has a strict context window—the maximum total number of input and output tokens it can hold in working memory at once (ranging from 128k to 200k+ tokens).

Why Naive Chat Apps Hit the Ceiling

In a basic chat application, developers often append every historical turn of the conversation to the API payload so the model “remembers” what was said.

As the chat progresses, your request payload balloons linearly. Not only does this skyrocket your input costs (because every historical message is re-sent on every single turn), but eventually, you will hit the hard context ceiling, causing the API to throw an error, or the model’s performance to degrade due to “lost in the middle” attention decay.

Modern Solutions for Memory Management

To keep applications fast and cost-effective, production systems rely on three patterns:

  1. Sliding Windows: Retaining only the last N messages while dropping or summarizing older interactions.
  2. Conversation Summarization: Running a background job every few turns to condense older chat history into a concise paragraph of “state memory.”
  3. Retrieval-Augmented Generation (RAG): Storing historical documents or past knowledge chunks inside a vector database and dynamically injecting only the most relevant snippets into the prompt context on demand.
Tags: AiLlmPython