123 scoring tasks · 8 categories · graded by Xcode
Can an agent ship Apple code without you?
AppleBench hands an agent an isolated checkout of a real Xcode project
with something genuinely wrong in it. It records everything the agent
does. Then, once the agent has exited, it grades the
workspace that was left behind: fresh xcodebuild, XCTest,
XCUITest, resolved build settings, and the app running on a simulator.
Agents do not grade themselves.
Why another benchmark
Most coding benchmarks measure whether a model can produce a diff. Apple development is mostly what happens after the diff.
understand → edit → build → diagnose → run → inspect → interact → test → verifyThe edit is one step out of nine. AppleBench grades the end of that loop, not the middle of it. Everything except the model is held still: same harness, same prompt, same commit, same Xcode, same simulator, same limits. Most capable models get the easy tasks. What separates them is the eight steps after the edit, and what each solved task cost.
What it measures
Operational Apple-platform work. Plain Swift and language-level concurrency are covered well elsewhere, so they are out of scope here.
Build failures
Compiler failures specific to the Apple SDK: availability, linking, module resolution.
Failing tests
Diagnose, fix, and rerun against tests the agent did not write and cannot rewrite.
Runtime defects
Crashes and misbehavior that compile perfectly and only appear when the app runs.
Layout and spec
Read a design spec, fix the layout, and satisfy structural UI assertions.
Driving the app
Run it on a simulator, interact with it, and write XCUITests that hold.
Project configuration
Target membership, Info.plist, asset catalogs, schemes, Swift packages.
Apple frameworks
SwiftData, Core Data, WidgetKit, App Intents, where the API contract is the task.
Raw toolchain
Operational loops through xcodebuild, simctl, and devicectl, with no wrapper CLI on the path.
How a run is graded
The grading is the product. The rest is plumbing that keeps it honest.
Grading starts after the agent is gone
Fresh derived data, fresh xcodebuild, run against the
workspace the agent left. Its own successful build never counts, and
nothing describing how it will be graded is written near the
workspace until it has exited.
Every task is proven solvable
An agent that changes nothing must fail a task, and
the reference fix must pass it. Both halves run
against real xcodebuild before a task is allowed into
the set. All 123 currently hold.
Nothing is guessed
Token counts and cost come from the agent CLI, and stay
null when it did not report them. Never zero, never
estimated. A timeout and a passing workspace are two separate facts
and are recorded that way.
The answers stay private
The 123 scoring tasks are private and rotated. Eight samples ship with the open harness so anyone can run it, and those are never scored. A scoring run is also sealed in a VM that denies network egress, because keeping answers closed and sealing the run solve two different problems.
Latest results
Every published run, with its full per-task data.
No runs published yet.
Open harness, private answers
You can run it. You cannot see the scoring set.
The harness is open: the grading engine, the task schema, the fixture tooling, the isolation model, and eight sample tasks with their fixtures. Clone it and it runs on those with no arguments. They are real tasks, meant to be read and copied from when you write your own, and no published score ever comes from them.
The 123 scoring tasks stay private. Not because they are precious, but because a benchmark whose answers are on the internet stops measuring anything the moment somebody scrapes it. Every published number comes from that private set.
Neither half works alone, so a scoring run is sandboxed too. The agent runs in a VM that default-denies every network destination and mounts nothing of the host except the workspace it was given: the standard Apple toolchain and nothing else. Every run records whether it was isolated that way, because a number from an unconfined run is a different claim.