Understanding Token Cost Anatomy: The Five-Headed Bill Nobody Budgeted For

You know how this usually goes...you wind up pulling the monthly AI spend report and the number is just that...a number. "Anthropic API: $14,200." No breakdown, no context, no indication of whether your team built something genuinely useful or spent three weeks regenerating the same 8,000-token system prompt on every single call.

The invoice knows what you spent. Unfortunately, your invoice (by itself) has no idea why you spent what you did.

The graphic above breaks it down: Five buckets, five cost drivers, five levers. The FinOps Foundation keeps surfacing the same uncomfortable truth: the fundamental unit of AI spend is the token, not a compute hour, not a GB of storage. Token consumption scales with behavior. A product team ships a new feature, usage spikes, and the bill looks nothing like the forecast. Traditional monitoring tools are not built to track this new reality well at all...so let's go through each bucket.

Input Tokens (38%)

Input tokens are the biggest slice and opportunity for optimization. This is also usually the most fixable aspect. Prompt engineering could very well be the most effective lever to initially pull if you're trying to lower your AI bill.

If you're not familiar, everything going into the model lands here: system prompts, conversation history, RAG retrieval results, tool schemas. Long system prompts and large context windows are the usual suspects (at least as far as I've seen).

The highest-impact fix is prompt caching. If your system prompt is stable across calls, you're paying full price to re-process it every time. Anthropic prefix caching delivers up to 90% cost reduction on cached input. Cache reads run at $0.30/M tokens versus $3.00/M fresh. The break-even is two API calls.

import anthropic

client = anthropic.Anthropic()

SYSTEM_PROMPT = """
You are a cloud cost governance assistant. Your job is to analyze
Azure Resource Graph query results and identify cost anomalies,
security posture gaps, and compliance drift across subscriptions.
[... your long system prompt here ...]
"""

def call_with_cache(user_message: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"}  # that's the whole trick
            }
        ],
        messages=[{"role": "user", "content": user_message}]
    )

    usage = response.usage
    print(f"Cache read:  {usage.cache_read_input_tokens}")
    print(f"Cache write: {usage.cache_creation_input_tokens}")
    print(f"Fresh input: {usage.input_tokens}")

    return response.content[0].text

Cache TTL is 5 minutes by default, extendable to 1 hour at 2x write rate. If calls are infrequent enough that the cache expires between them, you're paying the write premium for nothing. Know your call pattern before enabling this feature.

For prompt compression on top of this, LLMLingua is worth a look. It compresses prompts programmatically without meaningful quality loss, and pairs well with caching once you have a stable compressed prompt to work from.

Output Tokens (28%)

Output tokens cost more per token than input on every major provider because generation is computationally heavier than ingestion. Verbose completions with no length constraints will also quietly drain budget month over month.

The fix is genuinely boring: set max_tokens, and use structured outputs where you only need specific fields rather than a full narrative response.

import anthropic, json

client = anthropic.Anthropic()

def extract_cost_signals(raw_resource_data: str) -> dict:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,  # hard ceiling - tune to your schema
        system="""You are a cost signal extractor.
Respond ONLY with valid JSON matching this schema, no commentary:
{
  "anomaly_detected": boolean,
  "severity": "low|medium|high",
  "primary_driver": "string",
  "recommended_action": "string",
  "estimated_monthly_waste_usd": number
}""",
        messages=[{"role": "user", "content": f"Analyze: {raw_resource_data}"}]
    )
    return json.loads(response.content[0].text)

A prompt optimization that reduces average output from 2,000 to 800 tokens saves $0.003 per API call. At 10 million calls monthly, that's $30K annually from a two-hour engineering effort. The FinOps Foundation's cost estimation paper has worked examples of this math if you want to run the numbers for your own workload.

Embedding Tokens (12%)

These are cheap per token, easy to overlook, and become expensive when you're re-embedding the same documents on every pipeline run. Whoops.

If your knowledge base hasn't changed, you don't need fresh embeddings. The Anthropic API doesn't provide native embeddings, so most teams are already routing this through a separate model. Wherever you're generating them, the pattern is the same: hash your documents, store vectors in a database (pgvector if you're self-hosting, Pinecone or Weaviate otherwise), and only re-embed on actual change.

import hashlib
import anthropic

client = anthropic.Anthropic()

# In prod, replace this with a lookup against your vector DB
embedding_cache: dict[str, list[float]] = {}

def get_embedding_with_cache(text: str) -> list[float]:
    content_hash = hashlib.sha256(text.encode()).hexdigest()

    if content_hash in embedding_cache:
        return embedding_cache[content_hash]

    # Voyager models via Anthropic's partner integrations,
    # or swap for whichever embedding provider you're using
    response = client.embeddings.create(
        model="voyage-3",
        input=text,
    )
    vector = response.embeddings[0].embedding
    embedding_cache[content_hash] = vector
    return vector

Fine-tuning Tokens (12%)

Fine-tuning costs are bursty and high-visibility. The driver is full model re-training cycles when the task doesn't warrant it. LoRA and PEFT adapters with shared base models are the path forward: fine-tune once, attach lightweight adapters per use case.

Worth asking before you start: do you actually need fine-tuning, or will well-engineered few-shot prompting or RAG get you there? Training costs are visible and feel manageable. The inference tail on a fine-tuned model is continuous and often larger over time. A model trained for $20K might generate $8K per month in inference costs.

Run that math before committing to a training run.

Inference Infrastructure (10%)

Always-on GPU hosting with idle endpoints. This one shows up in enterprise deployments where teams provisioned dedicated inference capacity, shipped the feature, and never went back to right-size or optimize.

Serverless inference and auto-scaling are the obvious levers. If your workload is bursty, pay-per-request pricing on managed endpoints (Azure AI Foundry, AWS Bedrock, Vertex AI) will beat reserved capacity until you hit a meaningful volume threshold. Migrating to ARM-based instances like Graviton on AWS can also cut costs 10-20% with zero model changes. Not exciting, but it genuinely works.

The Attribution Problem

All five buckets above are fixable. None of them are fixable without visibility first.

The FinOps Foundation's token economics working group flagged managing token costs in SaaS-model AI as the top challenge practitioners face right now. The root causes are structural: developer-led purchasing, opaque billing, no native allocation mechanisms. The teams getting this right are tagging from day one and treating cost visibility as a condition of deployment, not an afterthought.

Here's a minimal Anthropic wrapper to get attribution data flowing without building a full observability platform first:

import anthropic, time
from dataclasses import dataclass
from typing import Optional

MODEL_COSTS = {
    "claude-sonnet-4-6": {"input": 3.00, "output": 15.00, "cache_read": 0.30, "cache_write": 3.75},
    "claude-haiku-4-5":  {"input": 1.00, "output": 5.00,  "cache_read": 0.10, "cache_write": 1.25},
}

@dataclass
class CallRecord:
    team: str
    use_case: str
    input_tokens: int
    output_tokens: int
    cache_read_tokens: int
    estimated_cost_usd: float
    latency_ms: float

call_log: list[CallRecord] = []

def tracked_call(
    messages: list[dict],
    system: Optional[list[dict]] = None,
    model: str = "claude-sonnet-4-6",
    max_tokens: int = 1024,
    team: str = "unknown",
    use_case: str = "unknown",
) -> str:
    client = anthropic.Anthropic()
    costs = MODEL_COSTS.get(model, MODEL_COSTS["claude-sonnet-4-6"])

    start = time.time()
    response = client.messages.create(
        model=model, max_tokens=max_tokens,
        system=system or [], messages=messages,
    )
    latency_ms = (time.time() - start) * 1000
    usage = response.usage
    cache_read = getattr(usage, "cache_read_input_tokens", 0)
    cache_write = getattr(usage, "cache_creation_input_tokens", 0)

    estimated_cost = (
        (usage.input_tokens / 1_000_000) * costs["input"]
        + (usage.output_tokens / 1_000_000) * costs["output"]
        + (cache_read / 1_000_000) * costs["cache_read"]
        + (cache_write / 1_000_000) * costs["cache_write"]
    )

    call_log.append(CallRecord(
        team=team, use_case=use_case,
        input_tokens=usage.input_tokens, output_tokens=usage.output_tokens,
        cache_read_tokens=cache_read, estimated_cost_usd=estimated_cost,
        latency_ms=latency_ms,
    ))

    print(f"[{team}/{use_case}] ${estimated_cost:.6f} | in={usage.input_tokens} out={usage.output_tokens} cache_hit={cache_read}")
    return response.content[0].text

# result = tracked_call(
#     messages=[{"role": "user", "content": "Summarize this cost report..."}],
#     team="platform-engineering",
#     use_case="cost-anomaly-detection",
# )

Wire this into OpenTelemetry or Azure Monitor and you've got the foundation for actual chargeback. Visibility first, then attribution, then optimization. That order matters every time.

The percentages in the graphic are illustrative and a representative average. Your actual mix will most likely look different depending on whether you're running a RAG pipeline, a fine-tuned task model, or a general-purpose assistant. The point isn't the specific numbers. It's that "AI spend" is not one thing, and you can't optimize what you can't see.

As I've mentioned before, start with the observability wrapper, as everything else follows from there.

Runnable cost-capture code covering all five token buckets, a shared pricing module, and a structured telemetry emitter with OpenTelemetry wiring can be found here:

github.com/sbkuehn/token-cost-anatomy