Against any foundation model
Accuracy up. Token cost down. Latency down.
Your models work around the clock.
So they better be optimal.
Most teams pay for tokens the model never needed. We find them, measure the removal, and keep the accuracy.
How
Four levers, in that order.
We tune the system around the model in four places: what goes in, what each call costs, how many calls the loop makes, and what model runs. Baseline first, one lever at a time.
Context: only what the call needs
Retrieval, reranking, sparse reading, and compaction that keeps the cache warm. Reading is the biggest line item: observations are 60 to 80% of agent tokens, and only 6 to 15% of what gets read is used.
Read less, solve more ->Cache and route every call
Stable prefixes at a tenth of the input price, batch tiers at half, a cheap model first with escalation on low confidence, and a cache for repeat questions.
Closed-loop routing ->Tight loops, scored per task
Every loop runs against our open evaluation framework: accuracy, p95 latency, and cost per successful task. Prompt wording alone moves reasoning tokens 2.4 to 7.4x with no gain in success.
Evaluation framework on GitHub ->The model itself, when the eval says so
Distill, fine-tune, quantize, or draft-and-verify. A 770M model trained on a frontier model’s rationales beat a 540B model in the original study.
Distilling step-by-step ->Ab initio
When no foundation model fits, we build one.
Data, architecture, training loop, evaluation, and serving, written at the level where the cost is decided. We have worked at that depth since the first transformer paper: speech systems that informed early Whisper, and a scientific question-answering layer for NASA Space Biology.
Partner rates
Discounts on the tokens themselves.
As a select OpenAI partner, we can provide token-usage discounts for specific cases. Our Anthropic partnership is en route.
Who
Veterans of production. Pioneers of the imminent future.
We built speech systems that informed early Whisper, published on language-model products a year and a half before ChatGPT, and ship for NASA, DARPA, and Boeing.
Welcome to the weeds.
- 01Lever 01
Contextual retrieval and reranking
Fewer, better chunks in context.
−67% retrieval failures
- 02Lever 01
Sparse reading
Preview, then read only the evidence the step needs.
Up to 92.9% fewer tokens
- 03Lever 01
Prompt compression
A small model deletes low-information tokens before the call.
2–5x compression
- 04Lever 01
Compaction that keeps the cache
Drop stale tool output and old turns without breaking the prefix.
61–87% lower cost on long sessions
- 05Lever 02
Prompt caching
Mark the stable prefix, or let the provider match it.
Cached input at 0.1x
- 06Lever 02
Cache-aware agent loops
Cache system prompt and tools. Keep volatile output after the breakpoint.
41–80% cost cut on long runs
- 07Lever 02
Batch and flex tiers
Queue what can wait an hour. Slow synchronous calls at the same rate.
50% off
- 08Lever 02
Routing, cascades, semantic cache
Cheap model or cached answer first. Escalate on low confidence.
58% live, up to 98% in benchmarks
- 09Lever 03
Effort and reasoning controls
Set thinking per call. Stop paying for max everywhere.
Low to max, per request
- 10Lever 03
Prompt and harness discipline
“Try multiple approaches” multiplies reasoning tokens. Harness choice moves cost per success 5 to 30x.
2.4–7.4x reasoning tokens, no gain
- 11Lever 03
Structured outputs
Schema-bound answers. No prose padding.
JSON schema enforced
- 12Lever 03
Cost per successful task
The metric the loop is tuned against. Tokens and cache hits are consequences.
accuracy · p95 · cost
- 13Lever 04
Distillation
Train a small model on the big model’s rationales. Target the failures that escalate.
2x the cut of random distillation
- 14Lever 04
Fine-tuning on your ground truth
The smaller model learns your schema, your tone, your edge cases.
Held to the same eval
- 15Lever 04
Quantization and prefix cache
Self-hosted: activation-aware 4-bit weights, prefix reuse in the serving layer.
>3x speedup vs FP16
- 16Lever 04
Speculative decoding
A draft model guesses. The big model verifies in parallel. Now batched for agents.
2–3x faster
Six of these usually apply to a given stack. Which six depends on your traffic.
Find your six
Results, before any partner discount
18%to93%
token cost saved across client engagements.
Almost every client has room to save money and improve accuracy and/or p95 latency. The baseline tells us which, in about a week.
Short answers
Token cost FAQ
How much can LLM token costs be reduced?
Our client engagements saved 18% to 93% of token cost before any partner discount. The range depends on how much of each prompt is stable, how many agent turns repeat context, and whether a smaller model holds accuracy on the ground-truth set.
Does reducing tokens reduce accuracy?
Usually the opposite. Tight retrieval and shorter loops remove noise from the context. Every change runs against the evaluation framework, so accuracy, p95 latency, and cost per task are measured before and after it ships.
What is the fastest first win?
Prompt caching. Cached reads cost a tenth of fresh input at Anthropic and up to 90% less at OpenAI. Batch processing takes another 50% off work that can wait an hour. Partner token discounts apply on top for qualifying cases.