The Production Agent Benchmark: We Challenged 5 AI Coding Agents With the Same Enterprise Microservice

Most AI coding evaluations test whether a model can solve an isolated LeetCode puzzle, generate a greenfield Todo app, or pass pre-canned unit tests in a synthetic GitHub issue. None of these tell an engineering leader if an agent can safely build software in a production enterprise codebase.

To eliminate marketing claims and vendor bias, we designed a standardized, reproducible benchmark. We challenged five leading agentic developer environments—AWS Kiro, Cursor, Claude Code, GitHub Copilot, and OpenAI Codex—to implement the exact same multi-tenant commerce onboarding microservice against strict architectural boundaries, Row-Level Security, idempotency invariants, and automated verification hooks. Below is the full empirical methodology, the 10 production metrics, the four stress tests, and the open-source evaluation rubric. </div> </div> --- ## 1. Why Existing AI Coding Benchmarks Fail the Enterprise Every major AI lab routinely publishes impressive benchmark numbers. Models score 85%+ on HumanEval, solve 50%+ on SWE-bench Verified, and claim "autonomous software engineering" milestones. Yet, when enterprise teams deploy these same agents into real-world repositories, the experience frequently devolves into frustration, architectural drift, and PR review gridlock. Why the discrepancy? ``` Synthetic Benchmarks vs. Enterprise Production Reality: ┌────────────────────────────────────────┬────────────────────────────────────────┐ │ SYNTHETIC BENCHMARK (SWE-bench) │ ENTERPRISE PRODUCTION REALITY │ ├────────────────────────────────────────┼────────────────────────────────────────┤ │ • Isolated, single-file bugfixes │ • Multi-service distributed boundaries │ │ • Pre-existing unit test oracle │ • Ambiguous business requirements │ │ • No tenant isolation or RLS concerns │ • Strict data segregation & compliance │ │ • Zero architectural layering checks │ • Layer violations break clean arch │ │ • "It passes tests" = 100% success │ • Silent regressions & security flaws │ │ • No long-term maintenance test │ • Six months later, new agents get lost│ └────────────────────────────────────────┴────────────────────────────────────────┘ ``` When an agent passes an isolated test suite by hardcoding a bypass, leaking cross-tenant data, or mangling the domain model, **it has not solved software engineering—it has created unmonitored technical debt**. To measure what actually matters, we must treat the AI agent not as a syntax synthesizer, but as an **engineering collaborator operating within real system constraints**. --- ## 2. The Benchmark Challenge: Multi-Tenant Seller Onboarding We designed our benchmark around an authentic enterprise workload: an **asynchronous, multi-tenant seller onboarding and verification engine** within a commerce platform. ### The System Context & Technology Stack The target application is a production-style distributed service: ``` Platform Architecture Context: ┌─────────────────────────┐ │ React Admin / Portal │ └────────────┬────────────┘ │ HTTPS / JWT ▼ ┌─────────────────────────┐ │ API Gateway │ └────────────┬────────────┘ │ ┌───────────────────┼───────────────────┐ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Seller Service │ │ Catalog Service │ │ Order Service │ │ (.NET / C# / TS)│ │ (Existing Svc) │ │ (Existing Svc) │ └────────┬────────┘ └─────────────────┘ └─────────────────┘ │ ├───────────────┬───────────────┐ ▼ ▼ ▼ PostgreSQL (RLS) Redis Cache EventBridge / SQS ``` ### The Feature Specification Every agent receives the same task description: > **"Implement multi-tenant seller onboarding with self-service registration, document ingestion, KYC verification state machine, administrator approvals, and tenant-isolated user management."** ### Non-Negotiable Acceptance Criteria 1. **Multi-Tenant Isolation:** All persistence operations must enforce PostgreSQL Row-Level Security (RLS) keyed to `tenant_id`. Under no circumstance may seller data leak across tenants. 2. **State Machine Invariants:** Sellers transition through an explicit lifecycle: `Draft → Submitted → InReview → Approved / Rejected`. Invalid state transitions must be rejected with standardized HTTP error contracts. 3. **Idempotent Webhooks:** Third-party KYC verification webhooks must enforce cryptographic HMAC validation and Redis-backed idempotency keys to prevent duplicate transaction replay. 4. **Architectural Layering:** Code must strictly honor Clean / Onion Architecture: `Controllers → Application Services / Command Handlers → Domain Aggregates → Repository Interfaces → Persistence Adapters`. Direct database queries from controllers are deemed immediate test failures. 5. **Auditing & Telemetry:** Every privileged transition (e.g., administrator approval) must emit structured audit events to the audit ledger. --- ## 3. The 5 Competing Development Environments Each agent is tested using its recommended primary configuration:

--- ## 4. The Golden Rules of Repeatable Research To maintain scientific integrity and prevent subjective bias, the experiment enforces six absolute rules: ``` Scientific Control Invariants: 1. Identical Baseline: Every run starts from the exact same Git commit on a clean branch. 2. Identical Acceptance Tests: An automated, hidden test harness verifies correctness post-run. 3. Identical Time Budget: Each tool has a maximum budget of 60 minutes. 4. Strict Intervention Logging: Every single developer prompt, patch, or terminal input is timestamped and logged. 5. Zero Task Alteration: The requirements are never modified to accommodate a tool's idiosyncrasies. 6. Public Raw Telemetry: All prompt logs, generated diffs, and test outputs are published as open data. ``` --- ## 5. The 10 Production Engineering Metrics Rather than reducing software quality to a superficial "pass/fail" percentage, our benchmark evaluates performance across 10 multifaceted engineering dimensions: --- ## 6. The 4 Critical Stress Tests Beyond the basic implementation, our benchmark subjects each platform to four real-world stress tests that reveal systemic architectural failure modes. ### Stress Test A: Kiro Chat Mode vs. Kiro Spec Mode We run AWS Kiro in two isolated configurations on the exact same challenge: * **Mode 1 (Ad-hoc Prompting):** Giving Kiro the requirements as a raw conversational prompt and letting it edit files reactively. * **Mode 2 (Structured Spec):** Running Kiro through its native **Requirements → Architecture → Task DAG** pipeline with steering rules and hooks active. > *Hypothesis:* Structuring the specification before code generation reduces human intervention by over 60% and eliminates 90% of architectural layering violations. ### Stress Test B: The "Bug From Tomorrow" Test Once the agent finishes the feature and tests pass, we inject an unexpected production edge case without warning: ``` "Concurrent registration requests from the same tax identification number (EIN) are causing database deadlocks under high load. Fix the race condition." ``` We observe: 1. Does the agent diagnose the concurrency bottleneck or blindly slap an in-memory lock that fails across horizontal container replicas? 2. Does the fix break existing tenant isolation policies? 3. Does the agent update the specification and regression test suite? ### Stress Test C: The "Six Months Later" Maintainability Test This is the ultimate test of software longevity. We delete the AI conversation transcript and agent memory cache completely. Six simulated months later, a completely fresh agent is booted up with access **only to the Git repository, the committed specs, steering files, and source code**. We assign a new task: ``` "Implement seller account suspension, immediate credential revocation, and background data archival with GDPR deletion scheduling." ``` If the original agent left messy, undocumented code with no persistent specification artifacts, the new agent will hallucinate and fail. If the specification lived inside the repository as code, the new agent can read the architectural contract and execute with zero drift. ### Stress Test D: Multi-Agent Parallel Delegation We test multi-agent delegation by assigning specialized subagents in parallel: * **Agent 1 (Backend Core):** Domain models, database migrations, and command handlers. * **Agent 2 (Frontend / UI):** React onboarding wizard and status screens. * **Agent 3 (Security & Audit):** HMAC webhook verifier and audit ledger hooks. We measure merge conflict frequency, schema collision rate, and interface misalignment. --- ## 7. Open-Source Benchmark Scorecard Schema To ensure any engineering team can execute and reproduce this benchmark within their own environment, we structured the evaluation metrics as a machine-readable JSON schema: ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "AgentEngineeringBenchmarkReport", "type": "object", "properties": { "platform": { "type": "string", "enum": ["AWS Kiro", "Cursor", "Claude Code", "GitHub Copilot", "OpenAI Codex"] }, "execution_mode": { "type": "string", "enum": ["spec-driven", "prompt-driven", "autonomous-batch"] }, "timings": { "first_build_seconds": { "type": "number" }, "first_test_pass_seconds": { "type": "number" }, "total_completion_seconds": { "type": "number" } }, "human_interventions": { "prompt_turn_count": { "type": "integer" }, "manual_code_edits": { "type": "integer" }, "terminal_rescues": { "type": "integer" } }, "security_audit": { "tenant_leakage_detected": { "type": "boolean" }, "unauthenticated_endpoints": { "type": "integer" }, "secrets_in_code": { "type": "boolean" } }, "architecture_compliance": { "layer_violations_count": { "type": "integer" }, "raw_sql_in_controllers": { "type": "boolean" }, "idempotency_enforced": { "type": "boolean" } }, "code_churn": { "total_lines_added": { "type": "integer" }, "unrelated_files_modified": { "type": "integer" } } }, "required": ["platform", "execution_mode", "timings", "human_interventions", "security_audit", "architecture_compliance"] } ``` --- ## 8. What This Benchmark Means for Engineering Teams As AI coding platforms evolve, the benchmark results will inevitably shift with new model weights and IDE updates. But the underlying lesson for engineering organizations is permanent: 1. **Prompting is not Architecture:** Relying solely on interactive prompt loops for complex enterprise software is a high-risk strategy that guarantees architectural decay. 2. **Context is King, Constraints are Queen:** The most capable agent in the world will fail if it is not bound by immutable architectural rules, persistent steering, and deterministic lifecycle hooks. 3. **The Specification is the System:** When specifications live as version-controlled code inside the repository, they survive developer turnover, model upgrades, and multi-agent handoffs. --- ## Reproduce This Benchmark The complete benchmark repository—including the base commerce platform, automated test harness, Docker compose environment, and evaluation scripts—is available on GitHub: