Skip to main content

Delivery check evidence

Read the sample receipt

Back to delivery checks

Northset ran three public Terminal-Bench tasks through Harbor 0.24.0 on a GitHub-hosted Linux runner.

A FINDING is a measured mismatch with the supplied expectation. The adjudicated state says what it means.

Three cases, each control run twice
CaseControlReferenceNullControl resultRaw outcomeAdjudicated state
vf2-speedup-networkx 3.0.3, a historical revision A module that only skips at import reward 1, 60 of 60, twice reward 0, 0 of 60, twice reward 1, 60 of 60, twice FINDING Confirmed acceptance defect at the pinned historical revision
kv-store-grpc, Terminal-Bench 2.1 A store that keeps only the last value reward 1, 7 of 7, twice reward 0, 0 of 7, twice; its tests could not import a dependency, so the inventory check is blocked reward 1, 7 of 7, twice FINDING Supported implied reading and a coverage gap. Dockerfile build only; the declared image was not run
cargo-flight-dispatch 2.0.4 A refactor whose plan matches the reference's reward 1, 27 of 27, twice reward 0, 9 of 27, twice reward 0, 0 of 27, twice FINDING Unresolved contract question. The refactor and the empty submission both get reward 0

Execution identity

  • Provider: GitHub Actions run 40113087348, attempt 1
  • Protocol commit: fd5aa2fc, clean tree
  • Receipt engine 0.2.0, receipt procedure 0.1.0
  • Harbor 0.24.0; Docker 28.0.4; Linux amd64
  • Images observed: 60 trial-seat image IDs, none unknown
  • Cleanup: no unrelated host resources published

Case notes

The same 18 trials matched across four runs: macOS and Linux, receipt engines 0.1.0 and 0.2.0.

Case 3 rerun: keeping geo.py in the transfer, the refactor scored reward 1 and a wrong-radius variant still scored 0.

That is Northset's own controlled measurement on one host. Which reading of the task binds is still the owner's call.

These are constructed controls on public tasks. They show these failure modes exist. They estimate no prevalence.

The raw package is not published here, because it carries benchmark task material. A buyer receives the full package.

Case context

On a public Terminal-Bench task, code with no implementation passed all 60 required tests. The task's test wrapper counted skips as passes.

The project's own collaborators documented the skip-as-pass defect in Terminal-Bench issue #2086. It was fixed upstream on 2026-10-22.

Northset reproduced it at the historical revision and checked a narrow correction of its own, covering only the skip and test-count rule.

Outside reviewer Alpha replayed both on 2026-10-24, confirmed the defect at that revision, and requested changes to how Northset reports its cases.

In a harness Northset wrote, which runs the task's own Dockerfiles and test script, the current version rejects that submission.

Harbor 0.24.0 cannot score the current version on Northset's host. Harbor's unreleased fix can, and rejects it there too.

The kept counterexample is useful as a regression check rather than as an open defect.

Terminal-Bench issue #2086

A second sample: an inconclusive run

This sample came from the Terminal-Bench task html-js-filter.

This sample is inconclusive because the reference run wrote no reward file on this host.

Published measurement: verifier regression protection on vf2-speedup-networkx

The run-scoped temporary paths and generated job name were collapsed to <run> and <job>.

The raw evidence files named in the sample are not published here.

Acceptance receipt

1 items submitted: 0 CHECKS_PASSED, 0 FINDING, 1 INCONCLUSIVE, 0 UNSUPPORTED, 0 INTAKE_ERROR.

Harbor 0.24.0; tool 0.1.0; Darwin arm64. Unreviewed local evidence.

html-js-filter: INCONCLUSIVE

Submitted: {"git_url": "https://github.com/harbor-framework/terminal-bench.git", "commit": "e67a2cb7db4d9795c212909b3395d8c22ad60aec", "path": "tasks/html-js-filter"}.

Subject SHA-256: b5ebbdb5e8eb272b123b984cde6c295ec8340e5636387d3664e3f7bad4575f79.

Checks: intake completed; reference blocked; null completed; inventory blocked; candidates not_applicable; claims blocked.

Rewards: reference [None, None]; null [0.0, 0.0].

  • reference: Harbor trial error: RewardFileNotFoundError: No reward file found at <run>/jobs-reference/<job>-html-js-filter-reference/task__GNosB3W/verifier/reward.txt or <run>/jobs-reference/<job>-html-js-filter-reference/task__GNosB3W/verifie
  • reference: Harbor trial error: RewardFileNotFoundError: No reward file found at <run>/jobs-reference/<job>-html-js-filter-reference/task__MG8LRxY/verifier/reward.txt or <run>/jobs-reference/<job>-html-js-filter-reference/task__MG8LRxY/verifie
  • Claim could not be checked: oracle_passes

Limits: This receipt checks only the submitted task bytes, supplied controls and recorded conditions. It does not establish full task correctness, difficulty, privacy, protected verdict channels or behavior on another platform. Candidate verdicts are compared with the submitter's stated expectations; no independent behavioral witness is run. Test reports and rewards come from the task's verifier and can themselves be influenced by submitted code. No independent review, signature, dependency pinning audit or on-chain acceptance is established. Some report formats do not distinguish xfailed or xpassed; unavailable counts are null. No known-wrong or alternative candidates were replayed.

Evidence: item record, intake.

Published measurement: Harbor verifier log permissions