Self-Hosted 122B LLM Cuts AI Agent Costs 787× vs Claude API

Self-Hosted 122B LLM Cuts AI Agent Costs 787× vs Claude API

A developer has published detailed measurements showing a self-hosted large language model can handle real AI agent workloads at a fraction of API costs — but only after a significant hardware upgrade eliminated critical reliability failures.

Arsen Apostolov replayed 27 actual production tasks from his personal AI agent, Jarvis, through local models running on consumer NVIDIA RTX 3090 GPUs. He scored both local and cloud-based responses independently against the same quality rubric, measured electricity consumption per task, and documented every failure mode.

Why it matters

Most LLM cost analyses compare API pricing in a vacuum. This work measures what self-hosting actually costs when you account for hardware, electricity metering, and the quality gap between local and frontier models. For organizations running high-volume agent workloads, the 787× cost reduction represents a concrete path to sustainable AI operations — if the quality trade-off is acceptable for the use case.

First attempt: catastrophic failure on constrained hardware

Apostolov's initial test ran a 30-billion-parameter model (qwen3-coder:30b) on a single RTX 3090 shared with other workloads. Memory constraints forced a 16,384-token context window — barely enough to fit Jarvis's tool schemas and system prompt.

The model scored 22.8 out of 100 against Claude's 89.4. More critically, it leaked malformed tool-call syntax into 25.9 percent of final answers, emitting raw tags like <function=send_email> directly to users instead of executing proper tool calls. On two tasks it looped indefinitely, calling the same tool repeatedly without synthesizing an answer.

Tool-overlap recall — whether the model selected the same tools that historically solved each task — measured just 14.8 percent.

Second attempt: three GPUs and a 16× larger context window

Apostolov upgraded to three RTX 3090s (72 GB total VRAM) and switched to a 122-billion-parameter mixture-of-experts model at Q3_K_M quantization. The larger memory budget enabled a 256,000-token context window.

The same 27 tasks, replayed through the same harness against the same frozen Claude baseline, produced dramatically different results:

• Quality score: 80.0 out of 100 (89.4 percent of Claude's performance)
• Malformed tool calls: zero (down from seven)
• Tool-overlap recall: 38.0 percent (up from 14.8 percent)
• Cost per task: $0.000969 versus Claude's $0.763

On 14 of 27 tasks, the local model scored at or above Claude. In the "general" category — tasks emphasizing reasoning over context rather than multi-step tool chaining — it outscored Claude 90 to 85.

What changed and what didn't

Two variables changed simultaneously: model size and context window. Apostolov notes he cannot isolate which contributed more to the improvement, though he suspects the 16× context increase eliminated the schema truncation that caused round one's malformed outputs.

The local model still lags on tool selection, often choosing different tools than those that historically solved each task. One 336,906-character code task exceeded even the 256K context limit in both rounds.

Electricity costs across all 27 tasks totaled 0.1573 kWh. The 122B model costs 6.6× more per task to run than the 30B version but remains roughly three orders of magnitude cheaper than API calls.

Apostolov has migrated Jarvis to the local model and downgraded from Claude Max to Pro. Full methodology, including the mock harness design and three debugging passes to eliminate scoring artifacts, was detailed by Apostolov writing in Towards Data Science.