A Repository Readiness Score Is Not a Coding-Agent Release Gate

The score and the gate answer different questions

A repository score is a structural assessment. A release gate is an executable claim about one qualified configuration. Conflating them produces a clean dashboard and an unsafe rollout.

Factory's published Agent Readiness model shows why structural scoring is useful. It organizes repository criteria into nine technical pillars, evaluates some criteria once per repository and others per application, and advances through five maturity levels when 80% of the previous level's criteria pass. Its CLI report adds rationale and remediation actions. That gives an engineering leader a consistent inventory of missing foundations and a way to sequence cleanup. (Factory overview, readiness report)

The score still does not prove that a pinned coding-agent client loaded the intended instructions, used the intended permissions, solved representative tasks, or failed safely on prohibited actions. Those facts depend on runtime state that a repository scan cannot fully establish.

Pin the release unit before running evals

The repository commit is only one field in the release. If the client, model, launch directory, target path, instruction graph, or permission profile changes, you are testing a different system.

Write the qualification manifest before the first canary. Keep it beside the evidence bundle, not buried in a wiki page.

release:
  repository_commit: "<git-sha>"
  client_artifact: "<name-and-build>"
  model_identity: "<provider-model-snapshot>"
  loader_settings: "<config-hash>"
  trust_root: "<repo-root>"
  working_directory: "<launch-path>"
  target_path: "<task-scope>"
  resolved_instructions: "<ordered-manifest-hash>"
  permission_profile: "<policy-hash>"
  network_policy: "<egress-policy-hash>"
  secret_boundary: "<broker-policy-hash>"
  load_timing: "<startup-or-lazy>"

evidence:
  task_fixtures: "<fixture-set-hash>"
  prohibited_actions: "<denial-suite-hash>"
  merge_controls: "<ruleset-export-hash>"
  rollback_target: "<last-known-good-manifest>"

The hashes do two jobs. They make review cheap, and they stop a later rerun from quietly qualifying different bytes. The human-readable files remain the source material; the manifest proves which versions were evaluated together.

This is the same release discipline described in Repository Instructions Need a Release Process, expanded beyond instructions to the runtime controls that determine what the agent can actually do.

Gate behavior with representative tasks

Behavioral evidence must come from the work you intend to delegate. A generic coding benchmark cannot qualify your package boundaries, test conventions, migration rules, review expectations, or protected files.

For a team preparing cross-package maintenance, the fixture set should include a narrow bug fix, a change that crosses the real package boundary, a request with an intentionally missing prerequisite, and a failure-recovery task. Each fixture needs explicit acceptance criteria before a model runs:

• the allowed files and forbidden files;
• the command that proves functional correctness;
• the architectural constraint the patch must preserve;
• the review defect that causes rejection even when tests pass;
• the evidence the run must retain.

Run the candidate in clean sessions. Repeat the fixtures enough to expose variance, and compare the candidate with the last-known-good release using the same tasks and acceptance rules. Store the prompt, resolved instructions, tool calls, diffs, test output, review result, latency, and cost for every run. A single lucky patch is a demonstration, not a gate.

The practical CI shape is deterministic integrity first, real-client runs second, and statistical interpretation last. Reuse that ordering for the broader rollout. Cheap deterministic failures should stop the job before a model call.

Prove prohibited actions fail outside the model

Compliance is not enforcement. Ask the model not to read a secret and you have an instruction. Make the secret unavailable to the process and you have a boundary.

Claude Code's own CLI makes the runtime dependency visible: it supports allowed-tools and disallowed-tools settings, a permission mode, and a flag that skips permission prompts. The same repository can therefore run under materially different authority depending on launch configuration. (Anthropic CLI reference)

Add negative probes to the release suite. They should attempt the actions that matter in your environment:

• write outside the approved workspace;
• read a secret the task does not need;
• call an unapproved host;
• invoke a deployment or destructive database tool;
• modify a protected workflow or ownership file.

A passing result is not “the model declined.” A passing result is that the filesystem boundary, credential broker, network policy, tool wrapper, or approval service denied the action and emitted a receipt. The distinction is the core of why AGENTS.md is not a policy boundary.

Bind the result to merge and rollback controls

Agent output becomes release risk when it can cross the merge boundary. Put the evidence gate where the repository can enforce it.

GitHub branch protection can require pull requests, approvals, code-owner review, and passing status checks before merge. GitHub also notes that CODEOWNERS review is a branch-protection option, not a property conferred by the file alone, and recommends protecting the ownership file itself. (branch-protection guidance, CODEOWNERS guidance)

Bind the coding-agent gate to those controls:

• require the deterministic manifest check and behavioral evidence job;
• require review from the owner of the affected boundary;
• dismiss stale approval when the candidate changes;
• prevent the agent's credential from bypassing the protected path;
• retain the evidence URL or artifact digest on the pull request.

Rollback must restore the complete last-known-good tuple, not merely revert an instruction file. Restore the client build, model snapshot, loader settings, permission profile, network policy, secret boundary, repository commit, and resolved instruction graph together. Then rerun the denial probes and task canaries in clean sessions. A partial rollback can preserve the fault while making the repository diff look clean.

Use readiness scores for remediation, not release authority

Repository scores earn their place when they make gaps inspectable. Use them to find missing ownership, slow feedback, weak tests, absent documentation, and inconsistent application coverage. Track structural improvement over time and use the findings to choose the next repair.

Keep the release decision separate. A structural score answers, “What foundations exist?” The executable gate answers, “Did this exact coding-agent system behave acceptably, fail safely, and remain reversible?”

That separation also makes the score more honest. It no longer has to pretend to predict stochastic behavior or security boundaries. It becomes a useful map, while the release gate carries the burden of proof.