Skip to content

LLM Token Analyzer & Prompt Optimizer

Popular

Count tokens, estimate API usage, optimize prompts, and discover ways to reduce unnecessary token consumption.

Loading…

Understanding tokens

A short, practical guide to what the numbers above mean and how to use them when you build with large language models.

What is a token?

Models don't read letters or words, they read tokens: pieces of text such as a whole word, part of a word, a number, punctuation or even a fragment of an emoji. In English a token is roughly ¾ of a word (about 4 characters), but that ratio changes a lot across languages, code and formatting.

Why do token counts differ between models?

Each model family has its own tokenizer with its own vocabulary, so the same text produces a different count. Neither number is “wrong”. OpenAI models are counted exactly here; for other providers, which don't publish an offline tokenizer, the tool shows a clearly marked estimate.

Why do tokens matter for API costs?

APIs bill per million tokens, with separate prices for input and output. Cost = input tokens × input price + output tokens × output price. Some models charge more for very long prompts, and some prices are introductory, so check the provider's page before budgeting.

What is a context window?

The context window is the most text a model can consider at once: your prompt, any attached data and the reply it writes all share it. If your input plus the expected output is larger than the window, the request fails or gets truncated. The meter above shows how much of the window you are using.

Input vs output tokens

Input tokens are what you send. Output tokens are what the model generates, which we can't know in advance, so enter the reply length you expect. Output is usually priced several times higher than input, which is why long answers dominate many bills.

Token count vs. actual usage

The count above covers the text you entered. A real request also carries system instructions, conversation history, attached files, tool definitions and the reply, so an IDE or API dashboard can report more. Neither number is wrong: they measure different things.

How the Prompt Optimizer works

An AI model rewrites your prompt to remove repetition and filler while keeping requirements, constraints and examples. A second pass checks that nothing important was lost, and both versions are recounted locally with the selected model's tokenizer, so the savings are measured rather than promised. You always review the result before it replaces your text.

Best practices for efficient prompts

  • Put stable instructions in a system prompt once instead of repeating them per message.
  • Give one good example instead of five similar ones.
  • Prefer specific requirements over long background stories.
  • Test an optimized prompt on a few real inputs before you switch to it.

How to reduce token usage

  • State repeated instructions once, or move them into a reusable system prompt.
  • Send only the relevant part of long documents, or summarise older context.
  • Trim repeated whitespace and boilerplate; keep indentation that carries meaning in code.
  • Ask for concise output and set a sensible maximum output length.

Related tools: LLM Cost Calculator, Context Window Checker, JSON Formatter and PDF to Text (to extract text from a PDF before counting).

About LLM Token Analyzer & Prompt Optimizer

A prompt that looks short can still overflow a context window or surprise you on the bill, because both are measured in tokens, not words. Paste text, JSON, Markdown or code, or load a text file, and the count updates as you type for the model you picked. Each result is labeled Exact or Estimated, so you know whether it comes from the model's own tokenizer or from a stand-in. Context use and an input cost estimate sit right beside it.

Features

  • Exact or Estimated label on every count
  • Context window meter with the share used
  • Input and expected-output cost from catalog prices
  • Compare up to six models side by side
  • Token details view of how your text is split
  • Checks for repeated sentences, lines and paragraphs, and extra spaces

Limitations

  • Most OpenAI models are counted exactly; Anthropic, Google, Mistral and other providers are counted with the o200k_base encoding as a stand-in, so those numbers are estimates and the provider's own count can differ.
  • Input is limited to 2 MB of text, and the count covers only what you enter, not the system prompt, chat history or tool definitions an app adds on top.
  • Prices and context sizes come from a dated catalog rather than a live feed, so confirm them on the provider's page before budgeting.
  • Counting needs no account, but the AI prompt optimizer needs an account on a paid plan and uses credits.

How it works

  1. Pick a model from the list, which you can search or filter by provider.
  2. Paste your text into the input box, or use Upload file for a text-based file or Examples for a sample.
  3. Read the total in the Token analysis panel, together with its Exact or Estimated label, the character and word counts and the share of the context window used.
  4. Enter the Expected output tokens to turn the input cost into the estimated cost of a full request.
  5. Open Compare models to count the same text for other models, or Token details to see how the text is split.
  6. Copy the count or a summary, or use Share to copy a link to the tool.

Example input

Summarize this JSON API response in two sentences for a non-technical stakeholder.

Frequently asked questions