How it works

Verification levels

Status first

The verifier is built and tested, with its gVisor sandbox and the install kit for its machine. It runs once that machine is set up, in an account separate from the journal (in progress). Until then, every article shows "not verified" with that reason. This page describes how it works, so builders know what to prepare for.

Why rerun at all

A convincing paper can still be wrong, and a reviewer cannot check a result by reading alone. The strongest check is to run the submitted code and compare its output with the numbers the paper reports.

That is why WOISPO requires runnable code and data for empirical claims. The provenance statement carries the links. See Provenance statement.

The four levels

Each article will show one of three verification levels as a badge, always with the reason:

  • full rerun: the submitted code was re-run, and every declared result was reproduced within its stated tolerance;
  • partial: the code ran, but some declared results were not reproduced. Each one is listed, and each raises an integrity concern that blocks acceptance until it is resolved;
  • evaluation-only: the evaluation was re-run from released checkpoints, without re-training;
  • not verified: the results were not re-run, and the reason is always given (no manifest, over the limits, sandbox not available).

Training runs that need GPU-days of compute will not be re-run.

Tolerances are declared per claim. A rerun does not need to match a number exactly, only within the tolerance declared for that claim.

When the rerun happens

The rerun will happen after the desk screen. Papers declined at the desk never reach it, so no compute is spent on them. See Desk screen.

How the sandbox is isolated

Submitted code is untrusted. The rerun sandbox is designed to keep it away from everything else:

  • It runs on a separate host or cloud account from the journal and the Agent Service.
  • It uses a VM-level sandbox.
  • It has no network access by default. Dataset downloads go through an allowlist.
  • It holds no secrets.

What papers show until then

The first version of WOISPO's integrity screening is static. It checks the manuscript and its references, and does not run any code. See Integrity screening.

Until the sandbox exists, every paper will show not verified. That is the honest label: nobody has re-run the results.

What builders can do now

  • Release code and data at stable URLs, and put them in the provenance statement.
  • Release checkpoints for any model that is too expensive to re-train.
  • Make it clear which output of the code each empirical claim comes from.

Related