Your LLM Has a Spending Problem - How Prompts, Context Windows, and Agent Loops Quietly Torch Your Budget

Everybody loves AI until the invoice shows up!

Not the cute "we spent a few hundred dollars experimenting" invoice that gets hand-waved away as innovation spend. I mean the kind of invoice that makes finance suddenly become deeply curious about architecture decisions they previously considered engineering trivia, while product teams insist everything is still technically a pilot, and leadership starts asking whether we can "optimize this" as though that thought had just occurred to humanity.

If you have been around enterprise technology for any length of time, this probably feels familiar, because we did this exact dance with cloud. We got excited about elasticity, speed, self-service, digital transformation, and developer empowerment. Then we discovered that moving fast without an operational model produces some incredibly educational invoices. FinOps emerged because cloud economics got weird enough that traditional budgeting and governance models stopped being sufficient.

AI is following the same path, except the economics are stranger, because the thing driving spend is not always infrastructure. Sometimes it is architecture. Sometimes it is workflow design. Sometimes it is application behavior. And sometimes it is a prompt that grew into an ungoverned monster because everyone kept adding "just one more helpful instruction."

That last one happens more than people would like to admit.

AI Cost Is Not "Just Another Cloud Bill"

One of the biggest mistakes I see right now is treating AI cost management like it is cloud FinOps with slightly different line items. That instinct makes sense, because most of us already know how to reason about cloud spend. We understand compute utilization, storage growth, idle resources, reserved pricing, licensing efficiency, network egress, and the thousand small architectural decisions that quietly evolve into larger financial problems when nobody is paying attention.

Those instincts still matter. But AI changes the shape of the problem, because cost is often driven by interaction behavior rather than static infrastructure allocation.

A virtual machine is fairly honest about what it is doing. Storage accounts are not emotionally complicated (at least not yet). Network egress may occasionally feel personally insulting, but at least it behaves predictably. AI workloads can be much more chaotic, because tiny implementation decisions dramatically change cost behavior. A prompt gets longer. A context window expands. A retrieval pipeline injects more content. An agent retries failed workflows. A product team decides the assistant should feel "more conversational." Suddenly the conversation is no longer about infrastructure optimization. It is about behavioral economics hiding inside application design.

That is a very different operational challenge, and if you approach it with only traditional infrastructure instincts, you are going to miss some very expensive patterns.

Tokens Are Tiny Budget Gremlins

Let us talk about tokens, because AI conversations have a habit of casually using the word like everyone instinctively understands what it means, when in reality half the room is quietly nodding while internally wondering if they should have asked sooner.

Technically, tokens are chunks of text processed by the model. They are not exactly words and not exactly characters, and the exact implementation varies by provider. That distinction matters for precision, but from a practical FinOps perspective the important thing is much simpler. Tokens are how you get billed. What you send into the model consumes tokens. What comes back out consumes tokens. And depending on the provider, cached interactions, tool execution, and orchestration overhead create additional economic nuance that turns simple assumptions into dangerously incomplete ones.

This is where AI spending starts behaving in ways traditional infrastructure teams find deeply irritating. A short request asking for a summary may barely register. A heavily engineered enterprise prompt with role instructions, governance controls, formatting directives, prior conversation history, retrieved documents, workflow metadata, and supporting tool definitions is a very different story. Then the model responds with a wonderfully verbose answer, because apparently we have all unintentionally reinforced the belief that concise answers are somehow less intelligent.

Now you are paying for input, output, and potentially a collection of supporting behaviors that completely change the economics of what looked, at first glance, like a simple interaction. A poorly sized VM does not become dramatically more expensive because someone reworded a sentence. An LLM absolutely can.

Your Architecture Has an AI Cost Personality

If you have followed my FinOps content before, you know I like talking about architecture having a cost personality, because different design decisions create different economic behaviors. AI takes that concept and cranks the intensity levers.

A lightweight FAQ chatbot has one personality. A retrieval-augmented knowledge assistant has another. A proposal analysis workflow behaves differently than a coding copilot. An internal enterprise knowledge assistant behaves differently than a customer-facing support implementation. A multi-agent orchestration model, where specialized agents hand work between each other, may be technically elegant and completely appropriate for the business use case, but it does not behave economically like a straightforward prompt-response implementation.

That matters, because I still hear people talk about "AI spend" like it is a meaningful single category. It is not. That is like talking about "cloud spend" without distinguishing between ephemeral development workloads, enterprise ERP infrastructure, serverless event processing, and data lake analytics. The architecture underneath determines the economics, and pretending all AI workloads behave similarly leads to terrible optimization decisions.

If you are building a simple assistant answering short transactional questions, your economic expectations should look very different than if you are building an agentic workflow that reasons, retrieves information, invokes tools, validates outcomes, retries failures, and refines results before responding. If you are embedding copilots into employee productivity workflows, your cost model should look different than a customer-facing conversational experience handling thousands of interactions.

Architecture shapes spend. Always has. AI just makes that relationship noisier.

Context Windows: Where Good Intentions Go Shopping

One of the easiest ways teams accidentally build expensive AI systems is through context accumulation, because every contributing decision sounds perfectly reasonable in isolation. You want the assistant to feel conversational, so you preserve chat history. You want better answers, so you implement retrieval-augmented generation and inject supporting documentation. You want stronger governance, so you add system instructions around acceptable use, formatting, tone, and policy controls. You want personalization, so user context gets appended. You want tooling, so orchestration metadata and function definitions are included. Product wants a smarter experience. Security wants stronger guardrails. Everyone wants their enhancement to make the system better. This has all the makings of what I like to call enterprise architecture drama. Taken individually, these are not reckless decisions. Collectively, they get hilariously expensive.

A user asks something simple like "what changed in this contract draft?" and what actually gets sent to the model includes system instructions, governance rules, previous conversation history, retrieved content, metadata, tool definitions, formatting expectations, workflow state, and the actual question hanging out near the bottom wondering how its life got so complicated. Every time that request runs, you pay for the whole payload, not just the visible question.

That is what makes this category of waste so sneaky. Nobody believes they are making irresponsible architectural decisions. They are making incremental product improvements. The problem is that incremental improvements accumulate, and context windows are remarkably indifferent to your intentions.

Agents: Tiny Distributed Systems Wearing Friendly Faces

This is where things get especially entertaining. Pull out the popcorn.

A straightforward chatbot interaction is easy to reason about. User sends a request. Model processes request. Response comes back. Costs scale in a reasonably understandable way.

Agents are fundamentally different, because they transform what looks like a simple interaction into a potentially complex backend workflow. Instead of simply answering, the system may reason through the request, decide which tools to invoke, query external systems, validate outputs, retry failed actions, refine conclusions, and then finally produce a response. That can be incredibly useful. It can also become financially spectacular in all the wrong ways if nobody is paying attention.

From the user's perspective, they asked one question. From the backend's perspective, that request may have triggered multiple reasoning passes, tool calls, data lookups, validation logic, retries, re-evaluations, and formatting steps. That is not one interaction. That is a tiny, distributed workflow wearing a friendly conversational interface and quietly helping itself to your budget.

To be clear, this is not an argument against agents. Agentic architectures absolutely have legitimate business value. This is an argument for understanding their economics. If your mental model assumes one user interaction equals one billable event, and your architecture actually behaves like a multi-step autonomous workflow, your financial assumptions are already wrong.

Autonomy is powerful. Autonomy is also expensive when left unsupervised.

Traditional FinOps vs AI FinOps

Traditional FinOps asks infrastructure-centric questions. Which workloads are expensive? Which resources are underutilized? Can we right size? Should we shift to reserved pricing? Are we retaining too much storage? Can licensing be optimized? These are useful questions, but AI introduces a different category of investigation, because inefficiencies often hide inside application behavior rather than obvious infrastructure waste.

Now the questions get more interesting. Why is one workflow dramatically more expensive than another? Why is a simple assistant generating enough output to qualify as serialized fiction? Why are prompt payloads growing over time? Why is an agent retrying failed operations with the confidence of someone who has never encountered diminishing returns? Why is one business unit consuming dramatically more tokens than another for what appears to be the same use case?

These are not classic infrastructure questions. They are application behavior questions, and that distinction matters, because traditional FinOps gives you a strong foundation but not a complete operating model for AI economics.

The optimization target shifts from infrastructure efficiency toward interaction efficiency.

Unit Economics Are the Adult Conversation

This is the part executives usually understand immediately once the conversation is framed correctly. Nobody actually cares about AI spend in isolation. They care whether the spend creates value.

A $20,000 monthly AI bill might be entirely justified if it meaningfully reduces support costs, accelerates engineering productivity, shortens proposal generation cycles, or improves customer outcomes. A much smaller bill may be wasteful if it powers something nobody meaningfully uses.

That means mature AI FinOps conversations need to move quickly toward unit economics:

• What is the cost per support interaction?

• What is the cost per contract review?

• What is the cost per proposal generated?

• What is the cost per engineering productivity assist?

• What is the cost per knowledge retrieval?

• What is the cost per customer workflow completed?

Those are useful business conversations. "AI is expensive" is usually just budget anxiety masquerading as strategy.

Making This Operational: Provider Dashboards Help...

...but they are not governance.

Provider-native telemetry matters, and you absolutely need it. OpenAI exposes usage reporting. Azure OpenAI provides monitoring telemetry through Azure observability tooling. Anthropic exposes detailed usage data through SDK responses, including the prompt caching fields that change the economics in ways naive counting misses. Google returns usage metadata on every Gemini response. That is useful operational visibility, but it is not governance.

Provider dashboards tell you what the service consumed, but they generally do not tell you which business workflow created the cost, which application feature is responsible, which team owns the spend, or whether the interaction produced meaningful business value. Without that attribution, you are effectively staring at infrastructure telemetry while trying to answer business questions.

That is why application instrumentation matters.

Instrument the Application Like a Grown-Up

If you care about AI economics, the application needs to participate in the observability story. At minimum, I would want telemetry capturing application name, environment, feature name, workflow identifiers, model selection, token usage (broken out by input, output, cache, and reasoning where the provider exposes it), estimated cost, latency, tool invocation counts, response outcomes, and enough user or session correlation to tie activity back to meaningful business workflows.

This is where the conversation shifts from "the AI bill feels weird" to "we can actually explain what happened."

Capture Real Usage, Not Vibes

A generic token estimator is fine for explaining the concept, but if you are building something real, you want actual provider-returned usage data. Each of the three major providers exposes this, and each one exposes nuance that naive token counting completely misses. Caching changes the math. Reasoning and "thinking" tokens get billed as output. Cached input tokens are usually a subset of your reported input, not a separate line, so you have to subtract before you multiply (a mistake I have watched double a cost estimate).

Here is the quick version for each provider. The full, repo-ready version with shared pricing tables and a structured telemetry emitter is linked at the bottom.

Claude (Anthropic)

Anthropic returns input_tokens, output_tokens, cache_creation_input_tokens, and cache_read_input_tokens as separate fields, which is exactly the kind of granularity AI FinOps needs.

import anthropic

# === pick your model here ===
MODEL = "claude-opus-4-8"   # current Anthropic flagship (Opus 4.8)

MODEL_PRICING = {
    # USD per 1M tokens. Verify against current Anthropic pricing before billing.
    "claude-opus-4-8": {
        "input": 5.00,
        "output": 25.00,
        "cache_write_5m": 6.25,   # 1.25x input
        "cache_read": 0.50,       # 0.1x input
    }
}

def estimate_claude_cost(model: str, usage) -> float:
    p = MODEL_PRICING[model]
    input_tokens = getattr(usage, "input_tokens", 0) or 0
    output_tokens = getattr(usage, "output_tokens", 0) or 0
    cache_write = getattr(usage, "cache_creation_input_tokens", 0) or 0
    cache_read = getattr(usage, "cache_read_input_tokens", 0) or 0
    return round(
        (input_tokens / 1_000_000) * p["input"]
        + (output_tokens / 1_000_000) * p["output"]
        + (cache_write / 1_000_000) * p["cache_write_5m"]
        + (cache_read / 1_000_000) * p["cache_read"],
        6,
    )

client = anthropic.Anthropic()
resp = client.messages.create(
    model=MODEL,
    max_tokens=512,
    messages=[{"role": "user", "content": "Summarize this contract change."}],
)
print(estimate_claude_cost(MODEL, resp.usage))

Note that with Claude, input_tokens already excludes cached reads and cache writes, so you add the four buckets together rather than subtracting. That is the opposite of how OpenAI and Gemini report it, which is exactly the kind of detail that wrecks a spreadsheet built on assumptions.

ChatGPT (OpenAI)

OpenAI reports total prompt_tokens (which already includes any cached tokens) plus a cached_tokens breakdown, and reasoning tokens roll into completion_tokens. So cached input is a subset you have to back out before pricing it.

from openai import OpenAI

# === pick your model here ===
MODEL = "gpt-5.5"   # current OpenAI flagship (GPT-5.5)

MODEL_PRICING = {
    # USD per 1M tokens. Verify against current OpenAI pricing before billing.
    "gpt-5.5": {
        "input": 5.00,
        "output": 30.00,
        "cached_input": 0.50,   # 0.1x input
    }
}

def estimate_openai_cost(model: str, usage) -> float:
    p = MODEL_PRICING[model]
    prompt = usage.prompt_tokens or 0
    completion = usage.completion_tokens or 0  # reasoning tokens included here
    details = getattr(usage, "prompt_tokens_details", None)
    cached = getattr(details, "cached_tokens", 0) or 0
    uncached_input = prompt - cached
    return round(
        (uncached_input / 1_000_000) * p["input"]
        + (cached / 1_000_000) * p["cached_input"]
        + (completion / 1_000_000) * p["output"],
        6,
    )

client = OpenAI()
resp = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Summarize this contract change."}],
)
print(estimate_openai_cost(MODEL, resp.usage))

Gemini (Google)

Google's current SDK is google-genai (the older google-generativeai package is on its way out). Usage comes back on response.usage_metadata. Two things to watch: cached_content_token_count is a subset of prompt_token_count, and thoughts_token_count (thinking) is billed at the output rate, so it gets added to your output bucket, not your input one. One more for the Pro tier: Gemini 3.1 Pro uses context-tier pricing, where rates step up once input crosses 200K tokens (the quick snippet below shows the standard tier; the repo handles the step-up).

from google import genai

# === pick your model here ===
MODEL = "gemini-3.1-pro"   # current Gemini flagship (3.1 Pro)

MODEL_PRICING = {
    # USD per 1M tokens, standard context (<=200K). Verify before billing.
    "gemini-3.1-pro": {
        "input": 2.00,
        "output": 12.00,
        "cached_input": 0.20,   # 0.1x input
    }
}

def estimate_gemini_cost(model: str, usage) -> float:
    p = MODEL_PRICING[model]
    prompt = usage.prompt_token_count or 0
    cached = getattr(usage, "cached_content_token_count", 0) or 0
    candidates = usage.candidates_token_count or 0
    thoughts = getattr(usage, "thoughts_token_count", 0) or 0
    uncached_input = prompt - cached
    billable_output = candidates + thoughts  # thinking billed as output
    return round(
        (uncached_input / 1_000_000) * p["input"]
        + (cached / 1_000_000) * p["cached_input"]
        + (billable_output / 1_000_000) * p["output"],
        6,
    )

client = genai.Client()
resp = client.models.generate_content(
    model=MODEL,
    contents="Summarize this contract change.",
)
print(estimate_gemini_cost(MODEL, resp.usage_metadata))

Three providers, three slightly different ways of reporting the same idea, and three different places where a naive estimate goes wrong. The pattern that scales is the same regardless of provider: wrap the SDK call, capture the actual usage metadata, enrich it with application context (feature, workflow, team, environment), and emit a structured telemetry event.

Azure Teams: KQL Is Still Your Friend

If you are Azure-heavy, this becomes operationally straightforward, because AI telemetry can land in Log Analytics alongside broader application observability. Emit a custom event per call and the queries write themselves.

customEvents
| where name == "AIUsageEvent"
| extend Provider = tostring(customDimensions.provider)
| extend Feature = tostring(customDimensions.feature)
| extend Model = tostring(customDimensions.model)
| extend TotalTokens = todouble(customDimensions.total_tokens)
| extend EstimatedCost = todouble(customDimensions.estimated_cost_usd)
| summarize
    Requests = count(),
    TotalTokens = sum(TotalTokens),
    EstimatedCost = sum(EstimatedCost)
    by Provider, Feature, Model, bin(timestamp, 1d)
| order by EstimatedCost desc

Now you can identify expensive workflows instead of staring vaguely at provider billing reports and hoping insight arrives.

Useful Tools Worth Exploring

If you are building this seriously, several platforms are worth a look:

• Langfuse for tracing, prompt analytics, and cost attribution.

• Helicone for request visibility, caching, and spend tracking - note Helicone has moved into maintenance mode following its aqcquisition by Mintlify, so check whether it still fits before building using this tool.

• OpenLIT for OpenTelemetry-native AI observability. The GitHub repo is located here.

• LiteLLM for proxy-based routing, provider abstraction, and budget controls. Docs are located here. The GitHub repo is located here.

• OpenAI Cookbook for practical implementation examples. The GitHub repo is located here.

• FinOps Foundation because AI economics are increasingly part of the broader FinOps conversation

Final Thought

AI does not need less experimentation, rather it needs fewer surprise invoices. The organizations that handle this well will not be the ones that panic when spend appears. They will be the ones who architect observability, attribution, and economic accountability into the solution from the beginning.

Because remember...your LLM may have a spending problem. But more often than not, your architecture does!

Repo with runnable cost-capture code for all three providers, a shared pricing module, and a structured telemetry emitter is available at github.com/sbkuehn/llm-cost-telemetry. Pricing tables verified June 2026; always check the provider pricing pages before you bill anything to anyone.