EXP-003: Does Agent Specialization Replicate for Productivity Tasks?

The Question

In EXP-001, splitting media tools into a focused agent improved eval scores by 40%, and Gemma 26B matched Sonnet on that narrow domain. Does this pattern hold for productivity — email, calendar, contacts, family management?

The Evals

18 tasks themed around real suburban family life in Downers Grove, IL:

• “There should be an email from the school about picture day. Can you add it to our calendar?” (email → read → calendar)
• “I got an email from the basketball league with a link to the season schedule. Can you grab the dates?” (email → read → browser → calendar)
• “Can you find that email about the thing at school next Friday?” (deliberately vague)
• “Email the team parents that Saturday’s soccer game is cancelled” (compose + send)
• “Can the kids do swim lessons on Wednesdays at 4? Check if we have anything” (conflict check)

These are harder than media evals — multi-step chains requiring email search → read → extract details → sometimes follow links → create calendar events. Following Hamel’s eval methodology, each is scored on specific dimensions (tool selection, completeness, actionability) rather than vague quality ratings.

Score Timeline

Full Model Comparison (Fixed Scorer)

Config	Bugged Score	Fixed Score	Change
Rusty (Sonnet, 46 tools)	2.77	3.85	+39%
Sheila (Sonnet, 20 tools)	2.89	3.85	+33%
Sheila (Gemma 26B, optimized)	2.56	3.56	+39%
Sheila (Qwen 3.5)	2.62	3.63	+39%

Key pattern: The scorer bug inflated specialization gains and local model gaps equally. With honest scoring, Sonnet doesn’t need specialization for productivity (46 tools is fine), and local models are much closer than initially measured.

Why This Experiment’s Story Changed

1. Specialization doesn’t always help. For media (7 tools, short chains) it was +40%. For productivity (20 tools, long chains) it’s +0%. The benefit depends on how much you narrow the scope. Sonnet handles 46 tools without confusion — narrowing to 20 doesn’t reduce its decision-making burden meaningfully. Following Braintrust’s guidance on using eval data to drive architecture decisions: the data says don’t specialize productivity on Sonnet.

2. Local models are closer than we thought. Gemma at -7.5% and Qwen at -5.7% are within striking distance. Techniques like chain-of-thought forced planning, tool call validation, and retry-with-reflection could close these gaps without switching models or adding cost.

3. Scorer bugs are the gift that keeps giving. This is the second time stale infrastructure created measurement errors that changed our conclusions. Hamel’s validate-evaluator step isn’t a one-time check — it’s something you do every time your eval tasks change. We didn’t, and it cost us a full round of wrong conclusions.

Devil’s advocate: The “zero specialization benefit” finding on Sonnet should be tested on more domains before generalizing. Media’s +40% was real. The difference might be about chain length and tool count thresholds — there could be a domain where specialization helps on Sonnet (maybe 30+ tools, or 6+ step chains). One data point isn’t a pattern.

Phase 2: Can Scaffolding Close the Gap?

The 5.7% Qwen gap (3.63 vs 3.85) seemed closeable with engineering techniques. We tested five approaches commonly recommended for improving local model tool calling:

Why Scaffolding Failed

This was the most surprising finding. The techniques are well-regarded in the community — Braintrust recommends structured prompts for tool use, and chain-of-thought is a standard practice. But the data says otherwise for this use case.

The scaffolding is solving the wrong problem. These techniques help models that know what to do but make execution errors (wrong args, skipped steps). Local models on productivity tasks have a reasoning problem — they’re uncertain about what the next step is in a 4-step chain. Adding instructions about how to execute doesn’t help when the model is uncertain about what to do next.

More prompt = less attention on the task. The scaffolding instructions added 200-400 tokens to the system prompt. For local models with smaller effective context windows, this pushes the tool descriptions and few-shot examples further away. The model pays attention to the scaffolding rules instead of the task.

Devil’s advocate: We only tested prompt-based scaffolding. Code-level scaffolding (actual validation middleware, programmatic retry loops, orchestrator patterns) might work differently because it doesn’t consume prompt budget. That’s a different experiment — engineering the harness, not the prompt.

What This Means

Phase 3: More Models

We tested 4 additional models to see if a different architecture closes the gap.

Model	Size	Score	vs Sonnet	Notes
Sonnet 4.6	Cloud	3.85	—	Baseline
Gemma 26B + retry	16GB	3.67	-4.7%	Best local (Phase 3a)
Gemma 26B	16GB	3.56	-7.5%
Qwen 3.5 9.7B	6.6GB	3.54	-8.1%
Qwen 3.5 4B	3.4GB	3.45	-10.4%
Nemotron 3 Nano 4B	2.8GB	3.08	-20.0%	Underwhelming
GLM4 9B	5.5GB	0.14	—	Broken on Ollama

GLM4 (Zhipu AI) scored 95% on external tool-calling benchmarks but 0.14/5 on ours — completely broken. Chat template incompatibility with Ollama makes it unusable despite strong benchmark numbers. This confirms what the Docker blog evaluation found: “BFCL leaderboard leaders have inference compatibility issues that tank real-world performance.”

Qwen 3.5 4B scored 97.5% on the jdhodges benchmark but only 3.45/5 here. Public benchmarks test simpler chains than our family management tasks. The gap between “can call a single tool correctly” and “can chain 4 tools with reasoning between steps” is significant.

Surprise winner (Phase 3a): Gemma 26B + retry scaffolding (3.67) — a technique that hurt Qwen actually helped Gemma. Scaffolding is model-specific, not universal.

Phase 4: Scaling Up

The Phase 3 results left a 4.7% gap. Could a larger local model or a different hosted provider close it?

Full Model Comparison (All Phases)

Model	Type	Score	vs Sonnet
Gemma 31B	Local (20GB)	4.11	+6.8%
Gemma 31B + retry	Local (20GB)	4.07	+5.7%
Sonnet 4.6	Cloud ($3/M)	3.85	—
MiniMax M2.7	Cloud (~$1/M)	3.80	-1.3%
Gemma 26B + retry	Local (18GB)	3.67	-4.7%
Gemma 26B	Local (18GB)	3.56	-7.5%
Qwen 3.5 9.7B	Local (6.6GB)	3.54	-8.1%
Qwen 3.5 4B	Local (3.4GB)	3.45	-10.4%
Nemotron 3 Nano 4B	Local (2.8GB)	3.08	-20.0%
GLM4 9B	Local (5.5GB)	0.14	Broken

What This Means

A local model beats Sonnet on productivity. Gemma 31B at 4.11/5 outperforms Claude Sonnet 4.6 by 6.8% on our 18 family management evals — with no scaffolding, no prompt tricks, no retry logic. Just a larger model with the same focused prompt.

Model scale matters more than engineering tricks. Gemma 26B→31B: +15%. Best scaffolding on 26B: +3.1%. Retry on 31B: -1%. When a model is large enough to reason through multi-step chains, extra instructions just add noise. Per Braintrust’s guidance on using eval data to drive architecture decisions: the data says skip scaffolding, scale the model.

Scaffolding helps weak models, hurts strong ones. Retry helped Gemma 26B (+3.1%) but hurt Gemma 31B (-1%) and Qwen (-8.5%). Validation helped Gemma 26B but hurt Qwen. The threshold appears to be around model capability — once a model can reason through the chain independently, scaffolding competes for attention budget. Following Eugene Yan’s approach: test per-model, not per-technique.

MiniMax M2.7 is a viable Sonnet alternative. At 3.80/5 (-1.3%), it’s within noise of Sonnet at ~3x lower cost. Good for vendor diversification and as a fallback when Anthropic has outages.

Don’t trust public benchmarks for multi-step tasks. Two models with 95%+ benchmark scores failed our evals. Run your own — the infrastructure to do so is the most valuable thing we’ve built.

Infrastructure quality was still the biggest lever. Scorer fixes: +39%. Model scale (26B→31B): +15%. Best scaffolding: +3.1%. Invest in eval infrastructure first, model selection second, prompt engineering last.

---

Part of the agent quality experiment series. Eval methodology follows our production eval reference. Eval tasks available at /ref/evals/.