In-run cheating. An agent with web access finds the benchmark on GitHub mid-evaluation and reads the fixture, or the expected diff, or the grader assertions. This is the fast version. It happens in minutes, on the very first run after publication.

Pretraining contamination. Task text gets scraped, ends up in a training corpus, and a future model has effectively memorized the answers. This is the slow version. Months, not minutes, and silent. Nothing in the run looks wrong.

These need different defenses, and people reach for the wrong one.

Closing the repository only solves the slow one

Keeping everything private does nothing about in-run cheating, because that threat does not depend on the source being public. An agent that can search the web can find something, and an agent that can reach the internet during an evaluation is not being evaluated under controlled conditions at all.

The fix for the fast threat is sandboxing the run: no internet, no search, standard toolchain only. AppleBench does this with a VM that default-denies every network destination and mounts nothing of the host but the workspace, and it would need to whether or not the repository were open.

Worth being precise about, because it is the kind of claim people repeat without checking: that is a mode, not a default. Run the agent on your own machine and its web tools are denied but its process is not confined, since it still has a shell and curl. Every run records which of the two it was, so a published number can be checked rather than taken on trust.

So the marginal value of going fully closed is lower than it feels. It buys you the slow-leak defense and nothing else, at the cost of a benchmark nobody can inspect, reproduce, or trust. Reviewers reasonably assume a closed benchmark is hiding weak results.

The split

• Open: the harness. Grading engine, task schema, fixture-generation tooling, the grader types, the isolation model, the tests that prove the harness behaves. Everything you would need to audit whether a reported number is honest.
• Private: the gold set. 123 scoring tasks: prompts, fixtures, expected outputs. Published scores come from these and only these.
• Public and never scored: eight sample tasks with their fixtures, shipped with the harness so a fresh clone runs end-to-end without touching gold, and so anyone writing a task has a working one to copy.

The two live in separate repositories and are joined by a single environment variable, APPLEBENCH_TASKSET. Nothing is copied in either direction: prepared fixtures, run artifacts and reports are all written inside the harness clone, so a closed task set never lands in a public checkout and the harness never has to be forked to run one.

./Scripts/check-task-set.sh fails the build if a task set claims to be both scored and open, if a task belongs to no suite at all, or if a referenced fixture goes missing. The partition is enforced, not maintained by discipline.

Secrecy is a delay, not a defense

Here is the part that matters more than the split.

A private set is not permanently private. It leaks through transcripts, through screenshots, through the people who run it, through the reports that describe what failed. Every published run leaks a little. The question is not whether the gold set survives. It is what happens when it does not.

The answer has to be rotation. AppleBench fixtures are XcodeGen manifests with templated defects, so the private set can be re-seeded: new bundle identifiers, new defect placements, regenerated projects. A leaked transcript from last quarter stops being a valid key.

APPLEBENCH_TASKSET=/path/to/scoring-set ./Scripts/rotate-private-set.sh 2026-q4

Closed answers buy time. Rotation is what keeps a benchmark alive past its first year. Anyone building one of these should plan for the second thing on day one, because the first thing has an expiry date whether or not you acknowledge it.

Apple development is an execution loop:

understand → edit → build → diagnose → run → inspect → interact → test → verify

The edit is one step out of nine. The other eight involve a toolchain that gives you feedback only when you run it, in formats you have to know how to read, with failure modes that have nothing to do with the language. A model that writes flawless Swift and cannot get xcodebuild to target a simulator is not useful to an iOS engineer.

What “done” has to mean

The design constraint that shaped everything else: a passing result has to mean the app actually works, not that the diff looked right.

So AppleBench grades the workspace, not the agent’s account of it. After the agent exits, the harness runs a fresh xcodebuild with clean derived data, runs the tests, installs the product on a simulator, launches it, and watches it survive. The agent’s own successful build counts for nothing. If the agent built it and the grader cannot, the grader wins.

That inversion is the whole product. Everything else in the harness exists to keep it honest:

• Grader configuration is not written anywhere near the workspace until the agent has already exited. It cannot read the assertions and write to them.
• Prompts state the symptom, never the cause and never the file. The diagnosis is the task.
• Project configuration is graded by what it resolves to: build settings from xcodebuild -showBuildSettings -json, Info.plist keys read out of the built product. A plausible line pasted into project.pbxproj that does not take effect cannot pass.

What got cut

The first version had tasks about Swift generics, actor isolation, and data races. They came out.

Not because they are easy. Some were the hardest tasks in the set, but because other benchmarks already cover language-level Swift, and overlap buys nothing. If a model’s Sendable reasoning is already measured somewhere credible, measuring it again here adds noise and no signal.

What nobody else measures is whether an agent can operate the toolchain. So that is what is left: build failures that are SDK-specific, project configuration, simulator interaction, XCUITests, Apple frameworks, and raw xcodebuild/simctl operational loops with every wrapper CLI stripped off the path.

The uncomfortable part

Building a benchmark means eventually publishing a number, and a number is much easier to trust than it should be.

The first full run produced 33.7%. It could just as defensibly have been 26.5%. Same artifacts, same model, same day, different rule about which attempt counts when a task was re-run after an infrastructure failure. Neither number is wrong. Only one of them is meaningful without the sentence that explains it.

That is why every published run here states its suite, its model, its environment, and its selection rule, and why the harness reports a grader that failed and a grader that could not run as two different outcomes. A benchmark that quietly collapses those is measuring your CI stability and calling it model capability.