Methodology

Exact token counts and boundaries

An exact badge means the displayed count comes from a tokenizer result for the text that tokenizer encoded. Local rows run a pinned tokenizer in your browser. A provider row receives the same badge only when its API returns a token breakdown. The model name, tokenizer revision or resolved API model, and source appear with the result so the count is not attributed to an unverified family name.

A token can end in the middle of a UTF-8 character. Where boundaries are available, TokEvals preserves each token's bytes and byte span; it displays a hex fragment when those bytes do not form independent text. A tokenizer that exposes an exact count but no usable boundaries gets no invented token boxes.

Provider counts without token pieces

Anthropic's token-count endpoint returns a count for a structured message. Google's countTokens returns a total for the submitted content structure. TokEvals records that provider-reported total separately from any estimate of the raw text inside it. These count-only responses do not supply token IDs or byte boundaries, so their rows do not draw a breakdown.

Read the current Anthropic token-counting documentation and Google countTokens reference for the providers' own request and response contracts.

Estimated raw-content count

The estimate badge on an Anthropic or Google row is not a provider-reported raw-text count. TokEvals asks the same model to count an empty text wrapper and that wrapper with your text inserted, then subtracts the two counts. The first wrapper is an XML-like tag around the text. If that differential fails or falls outside the valid range, two other fixed boundary contexts are tried. Baseline counts for empty wrappers may be reused for the same provider and model.

Each count uses the same request shape as the provider total: Anthropic receives one user message whose content is the string being counted; Google receives one user Content with one text part. No system instruction or inference call is added.

For each successful context, the difference must be a positive safe integer no larger than the provider's structured-input count for the unwrapped text. The displayed estimate is the middle accepted difference at zero-based index floor(n / 2) after sorting; for two accepted contexts, that is the higher one. The minimum and maximum shown in result details are the smallest and largest accepted differences, not a statistical confidence interval. If no context passes, the row falls back to the provider's structured-input total, labeled as input tokens.

This differential is an instrument correction for the count-only API, not a direct observation of the model's raw token pieces. Boundary merges and provider framing can make it differ from an actual raw-text tokenizer count. The exact wrapper strings and acceptance rule are documented here so the method can be reproduced with provider access; no real inference request is made by this comparison.

• <tokeval-content>\n + text + \n</tokeval-content>
• --- TOKEVAL START ---\n\n + text + \n\n--- TOKEVAL END ---
• [A]\n + text + \n[B]

NFC comparison and normalized bytes

If the input is not already in NFC, the Explorer can compare its as-typed token count with the count after Unicode NFC composition. It chooses the visible, non-normalizing local model with the fewest as-typed tokens; a tie keeps the first visible model. The comparison depends on that selection and runs locally, without another provider request. The input changes only if you click “Compose to NFC.”

Some local tokenizers normalize before encoding. Their count is exact for the normalized text the tokenizer received, but token byte spans then describe normalized bytes, not the raw bytes you entered. A normalizing tokenizer without mappable boundaries still shows its count without fabricating spans. The public model catalog records each tokenizer's normalization form and any boundary limitation. Itsnormalization field names the form applied before encoding; boundaryLimitation explains why lossless raw-input spans may not be available.