AI CODING · AN EXPERIMENT JOURNAL
Evaluating AI Coding Agents
A practical evaluation of Claude, Gemini, and Codex across three computer vision assignments—documenting not only what worked, but where visual validation broke down.
Explore the benchmark ↗
THE CENTRAL FINDING
Generating an artifact is not the same as validating it.
RECORDED BENCHMARK · THREE TASKS
All three agents could assemble executable pipelines. The strongest results came from the agents whose outputs remained closest to ground truth—not from those that simply returned more files or more detections.
Read the findings ↗Claude, Gemini, and Codex
Tracking, lanes, and faces
Complete rolling-can tracks
Reliable visual validation
01 / THE BENCHMARK
Three agents under the same practical test
Task brief → Source media → Executable pipeline → Measured artifact
Each agent received a task description, designated sandbox, full media paths, and grading scripts where available. The benchmark evaluates the resulting images, videos, and CSV files—not whether the code merely appears plausible.
| Agent | Strongest pattern | Main weakness | Overall position |
|---|---|---|---|
| Claude Sonnet 4.1 | Accuracy and stability | Over-engineering | Best overall balance |
| OpenAI Codex GPT-5 | Fast implementation and coverage | Inconsistent output validation | Second |
| Gemini 2.5 Pro | Persistent iteration | Uneven convergence | Third |
02 / TASK RESULTS
What happened when the code met pixels
Expand each record to inspect the task, evidence, and decision.
TASK 01Rolling can detection
Claude leads+
Motivation Track the outer ring, center, and radius of a moving can without flicker or wobble.
Experiment Each agent processed the same video and exported Frame_Index,X,Y,Radius for every frame.
Outcome & decision Claude scored 0.9006 and Codex 0.8939. Gemini returned all 153 frames but produced a 339.10 px mean centroid error, indicating that complete output can still be badly wrong.
Key lesson: completeness is not correctness.
TASK 02Road line tracking
Accuracy vs coverage+
Motivation Detect both road boundaries and overlay straight, stable lines in dashboard-camera footage.
Experiment Claude and Codex were compared on detection rate, endpoint accuracy, angle accuracy, and stability.
Outcome & decision Claude reported 91.36% overall accuracy with 80% detection. Codex reached 100% detection but 83.02% accuracy. Returning a line more often did not make the line more correct.
Key lesson: coverage and correctness must be measured separately.
TASK 03Target face localization
Consensus comparison only+
Motivation Find one target face in a crowd image and return exactly one bounding box.
Experiment Because no verified bounding box was supplied, the comparison used agreement between agent outputs as a diagnostic proxy.
Outcome & decision Claude and Gemini agreed on the same region, with approximate consensus IoU values of 0.77 and 0.73. Codex scored 0.00 against that consensus. Agreement is evidence of disagreement—not proof of truth.
Key lesson: a benchmark needs authoritative labels.
03 / QUANTITATIVE SNAPSHOT
Results before expectations.
Reported metrics are preserved from the repository. Missing labels and incomparable runs remain explicitly marked rather than estimated.
| Task | Claude | Gemini | Codex | Interpretation |
|---|---|---|---|---|
| Rolling can accuracy | 0.9006 | -0.7289 | 0.8939 | Claude narrowly leads |
| Road-line accuracy | 91.36% | Not available | 83.02% | Claude favors correctness |
| Face consensus IoU | ≈ 0.77 | ≈ 0.73 | 0.00 | Diagnostic proxy only |
04 / LEARNING FROM FAILURE
Execution is only the first test.
The agents could create scripts, use libraries, and export artifacts. The main gap appeared after execution: visual self-review was inconsistent, and repeated attempts often changed parameters without diagnosing the largest error.
A dependable agent loop should measure the output, identify the largest failure, revise one hypothesis, and re-run the same grader.
THE NEXT BENCHMARK
Same tasks. Better evidence.
The next version should add multiple independent runs, fixed token and time budgets, complete ground truth, and separate scores for correctness, stability, efficiency, and code quality. Each agent should be required to generate a diagnostic visualization and document one measured correction cycle.
05 / REPRODUCE & TRACE
Trace the benchmark
Source tasks, agent outputs, media artifacts, scripts, and background references.
Repository & task artifacts +
Benchmark limitations +
Three tasks, one recorded run per configuration, incomplete token and time telemetry, sparse road annotations, and no verified face bounding box.