Sumit GuptaWriting about software, AI agents, and home infrastructure.https://sumit.dev/Proving Agent Quality With Datahttps://sumit.dev/blog/agent-quality-series/https://sumit.dev/blog/agent-quality-series/A series of experiments testing whether specialized AI agents on local models can match cloud API quality for personal task management.Wed, 08 Apr 2026 00:00:00 GMTDoes Splitting a Monolithic Agent Into Specialists Improve Eval Scores?https://sumit.dev/blog/agent-specialization-experiment/https://sumit.dev/blog/agent-specialization-experiment/We tested whether extracting media tools from a 46-tool agent into a 7-tool specialist would improve quality — and whether Gemma 26B could replace Sonnet on the focused domain.Wed, 08 Apr 2026 00:00:00 GMTBuilding a Production Eval System for AI Agentshttps://sumit.dev/blog/building-agent-eval-system/https://sumit.dev/blog/building-agent-eval-system/What we learned building a quality measurement system for a multi-agent AI, drawing on practitioner wisdom from Hamel Husain, Eugene Yan, Braintrust, and applied-llms.org.Tue, 07 Apr 2026 00:00:00 GMTEXP-002: Do Mock Evals Predict Real-World Agent Quality?https://sumit.dev/blog/exp002-real-api-robustness/https://sumit.dev/blog/exp002-real-api-robustness/We ran Henchman 21 against real media APIs 42 times to test whether our mock-based eval scores hold up in production conditions.Wed, 08 Apr 2026 00:00:00 GMTEXP-003: Does Agent Specialization Replicate for Productivity Tasks?https://sumit.dev/blog/exp003-sheila-productivity/https://sumit.dev/blog/exp003-sheila-productivity/The 40% improvement from media specialization only partially replicates for email/calendar. And Gemma 26B hits a wall on multi-step productivity chains.Wed, 08 Apr 2026 00:00:00 GMT