Benchmarks · agency-ops
The agency-ops 35B punches at the frontier.
A niche-tuned 35B beats the base model and a 744B general model on agency tasks — and on Apple Silicon it ties the frontier. Here’s exactly how it was measured — and why the result holds up.
Other niches publish held-out validation loss; they have not been run through this frontier head-to-head yet.
Head-to-head quality
50 held-out items · proven 0% n-gram overlap with training data. blind panel of 2 independent judges (Gemini 3.5 Flash + DeepSeek v3.1), absolute 1–10.
Bars are the mean blind-panel score (0–10 scale). Our agency-ops 35B lands within a tenth of GPT-5.5 and Fable 5, and more than two points above GLM-5.2 — a general model 20× its size.
What you get, by how you run it
Same weights, different serving engines. We publish the number for each path — including the ones below our best — so what you can reproduce matches what we claim.
| Run it via | Hardware | Quality |
|---|---|---|
| MLX (8-bit) | Apple Silicon · M-series, 24GB+ unified | 9.05 · ties frontier |
| GGUF / Ollama (Q8) | NVIDIA GPU, CPU, or Mac | ~8.3 |
| Hosted API | we run it — nothing to set up | ~8.3 |
9.05 is the full-fidelity MLX path on Apple Silicon; the GGUF/llama.cpp and hosted-API paths serve a slightly terser variant today (~8.3), and we’re closing that gap.
How we measured it
The whole point is a result that survives scrutiny. Four rules make it honest:
Held-out test set
50 questions the model never saw in training — proven 0% n-gram overlap with the train and validation splits. We don't teach to the test.
Blind, independent panel
Every answer is graded 1–10 by two judges from different labs (Gemini 3.5 Flash + DeepSeek v3.1). Neither is a contestant, and the judge never sees which model wrote the answer.
Same prompt for everyone
Identical system prompt, temperature, and token budget for every model — ours and the frontier APIs alike. No home-field advantage.
Reproducible
The harness, the golden set, and the judge prompts all live in the repo. Run the eval yourself and you get the same board.
Why agency-ops beats a 744B
A general model spreads its capacity across everything — code, poetry, trivia, translation. A niche model spends all of it on one trade. On that trade, focus wins: agency-ops beats GLM-5.2 (744B) by more than two points and ties the frontier, while staying small enough to run on a laptop and cheap enough to serve at a fraction of the cost. That’s the measured agency-ops result; the other niches are not represented in this head-to-head.
As of July 2026. Benchmarks are re-run as models are retrained; numbers update with each release.