Kill Your AgentExecutor: How a For Loop Cut Our LLM Bill by 70%

Our token spend on a customer support agent jumped from $400/day to $1,600/day in a single week. Nobody touched the prompts, the model, or the traffic volume. The culprit was LangChain’s AgentExecutor quietly letting the model “think” as many times as it wanted before answering.

That’s the core problem with most agent frameworks: they optimize for task completion, not for bounded cost. An executor loop that runs until the model decides it’s done is a loop with no upper bound on your bill.

Why AgentExecutor’s loop is a cost trap

AgentExecutor implements the ReAct pattern: the model reasons, picks a tool, observes the result, and repeats until it emits a final answer. On paper that’s elegant. In production it’s a while-loop with a soft cap.

from langchain.agents import AgentExecutor, create_react_agent

agent = create_react_agent(llm, tools, prompt)
executor = AgentExecutor(
    agent=agent,
    tools=tools,
    max_iterations=15,   # the only real safety valve
    max_execution_time=None,
)
result = executor.invoke({"input": user_query})

max_iterations=15 sounds like a bound. It isn’t a useful one. Each iteration is a full round-trip: a prompt that includes the entire scratchpad (every prior thought, action, and observation) plus a fresh completion call.

Iteration 1 costs you ~800 tokens. Iteration 10 costs you ~4,200 tokens, because the scratchpad keeps growing. The loop isn’t O(1) per step, it’s O(N) cumulative, and N is decided by the model, not by you.

We logged actual traces from the spike week:

query: "What's the refund status for order #4521?"
iteration 1: check_order_tool -> found order, missing refund flag
iteration 2: check_refund_policy_tool -> policy is generic
iteration 3: check_order_tool again (model forgot it already called this)
iteration 4: search_faq_tool -> irrelevant result
iteration 5: check_order_tool a third time
iteration 6: finally answers

Six calls for a question that needs exactly two: fetch the order, check the refund flag. The model wasn’t reasoning better with more steps, it was re-deriving context it already had, because nothing forced it to stop.

Strip the problem to what’s actually happening

Pull back the abstraction and an “agent” here is just:

1. A fixed, small set of tools the task actually needs.
2. A sequence of calls to resolve them.
3. A stopping condition.

The ReAct pattern makes the LLM own all three. For open ended research tasks where the tool sequence is genuinely unknown ahead of time, that’s the right call. For our support bot, the tool sequence was known: 90% of queries resolved with 1-3 deterministic tool calls in a fixed order.

The binding constraint wasn’t reasoning quality. It was unbounded context growth per retry combined with no deduplication of tool calls. Every constraint that mattered here was structural, not a model capability issue:

• Latency: each extra iteration added 1.5-3s. Users bailed after 3 iterations regardless of accuracy.
• Cost: scratchpad growth means cost scales quadratically with iteration count, not linearly.
• Correctness: letting the model decide when to stop meant it sometimes stopped too early (wrong answer) or too late (redundant calls), and we had no lever to tune that except a vague prompt tweak.

None of those are solved by better prompting. They’re solved by taking control of the loop.

Replace the executor with a plain for-loop

We rewrote the flow as an explicit state machine. No agent framework, no scratchpad replay, just a for loop with a hard cap and short circuit exits.

def resolve_support_query(query, order_id=None):
    context = {}
    steps = [
        ("order_lookup", lambda: lookup_order(order_id)),
        ("refund_check", lambda: check_refund_flag(context.get("order"))),
        ("policy_check", lambda: get_policy(context.get("order", {}).get("type"))),
    ]

    for name, fn in steps:
        if name in context:
            continue  # never call the same tool twice
        result = fn()
        context[name] = result
        if can_answer_now(context):
            break  # short-circuit as soon as we have enough

    return generate_answer(query, context)  # ONE final LLM call

The differences that actually save money:

• One LLM call at the end, not one per step. Tool lookups are plain function calls, not model-mediated decisions.
• max_iterations becomes len(steps), a real bound you set, not a soft cap the model can still burn through with repeated calls.
• No scratchpad replay. Context is a dict, not a growing transcript resent on every call. Token cost per step is O(1), not O(N).
• Short-circuit exit (can_answer_now) stops as soon as the answer is derivable, instead of waiting for the model to notice.

For the 10% of queries that genuinely need open-ended tool selection (rare edge cases, ambiguous multi-part questions), we kept a scoped-down agent, but capped at 4 iterations and only invoked as a fallback:

if not can_answer_now(context):
    result = fallback_agent_executor.invoke(
        {"input": query, "context": context},
        config={"max_iterations": 4},
    )

BEFORE (AgentExecutor, unbounded reasoning):
  query -> [LLM call: think] -> [tool] -> [LLM call: think] -> [tool] -> ... -> [LLM call: answer]
  avg iterations: 4.8   avg tokens: 6,100   p95 latency: 9.2s

AFTER (for-loop, deterministic steps + 1 final call):
  query -> [tool] -> [tool] -> [tool] -> [LLM call: answer]
  avg iterations: 2.1 (tool calls, no LLM cost) + 1 LLM call
  avg tokens: 1,450   p95 latency: 2.6s

The mental model shift: an LLM call is expensive and probabilistic. A function call is free and deterministic. Use the agent framework only for the part of the problem that’s genuinely open ended, and hardcode everything else.

What changed and what we’d do differently

Token spend dropped from $1,600/day back to roughly $480/day, a 70% reduction, with task success rate unchanged at 94% (measured against the same 500 query eval set we used before and after). p95 latency dropped from 9.2s to 2.6s as a side effect of removing redundant LLM round-trips, not from any model or infra change.

The one thing we got wrong initially: we tried to make the for loop “smart” by having it call an LLM to decide the next step dynamically. That reintroduced the same cost problem in miniature. The win only shows up when the step sequence is hardcoded and the LLM is used exclusively for the parts that need judgment, like final answer generation or the rare fallback case.

If you’re running an agent framework in production, pull your traces and look at the iteration count distribution. If most of your traffic resolves in 1-3 well known steps, you don’t have an agent problem. You have a workflow with a fancy, expensive stopping condition. Replace the loop, keep the judgment calls for the model, and measure the token count before you ship.