Your AGENTS.md Is Not a Policy Boundary

One instruction file cannot satisfy four contracts

The narrow rule is simple: repository instructions are context inputs, not portable policy boundaries. AGENTS.md is useful, but the filename does not define a universal runtime protocol. Each agent surface chooses its own filenames, search boundary, loading time, merge behavior, limits, and audit artifacts.

Treat each surface as four independent contracts:

1. Discovery: Which exact paths and bytes entered model context? The evidence must identify the loader event or model-visible payload and bind it to immutable file hashes.
2. Scope and merge: Why did each file apply to this launch directory and target path, and in what order was it combined? Order is not semantic precedence unless the surface documents that rule.
3. Behavioral compliance: Across repeated runs, did the model follow each non-conflicting instruction? This is stochastic behavior, not loader provenance.
4. External enforcement: When a prohibited action was attempted, did a control outside the instruction text deny it? The denial needs its own receipt.

A discovery pass does not imply compliance. A compliance pass does not show which file supplied the rule. A correct task outcome does not prove the agent stayed inside policy. An external denial can work even when the instruction never loaded. Store the four answers separately.

This is the layer that complements the existing CI regression harness for repository instructions. That companion covers the regression discipline around a changing instruction artifact. The contract here is narrower: determine what each pinned runtime loaded, how it scoped that input, how the model behaved, and whether an independent control held.

Loader evidence is not model self-report

A response to "what instructions did you load?" is behavioral output. It is not provenance.

The model can omit a file it received, name a file it inferred from the repository, summarize stale context, or confuse the visible text with its source path. Even a perfect answer proves only that the model produced that answer. Keep it under compliance.self_report if it is useful for debugging, never under discovery.

Recommendation: Accept discovery evidence only when it comes from one of these paths:

• A loader-owned event with a source path and load reason.
• A client command that prints the active instruction paths or exact combined context without asking the model to reconstruct it.
• A session artifact containing the actual model-visible instruction payload.
• A wrapper-side file snapshot, but only as expected input. A filesystem hash alone does not prove the loader used the file.

The evidence record should bind a loader observation to a file snapshot taken while the repository is immutable:

{
  "path": "services/payments/AGENTS.md",
  "sha256": "<content-hash>",
  "bytes": "<byte-count>",
  "observed_by": "<loader-event-or-client-artifact>",
  "load_reason": "<launch-path-target-or-unknown>",
  "artifact": "artifacts/<run-id>/loader.jsonl"
}

If the tool exposes the combined bytes but not per-file paths, record that split honestly. For example, bytes_evidence can be tool_emitted while path_evidence is inferred_from_documented_loader. Do not upgrade an inference to a measurement just because the expected hash appears in the combined text.

Five pinned surfaces use five loaders

The surface name belongs in the test key. "Copilot" is not specific enough, and neither is "Claude" or "Gemini." Pin the executable or SaaS surface, model, configuration, launch directory, repository revision, and adapter revision together.

The compact table is a map, not a ranking. The implementation traps matter more than the shared Markdown syntax.

OpenAI Codex CLI: launch directory defines the documented chain

Documented: Codex checks AGENTS.override.md, then AGENTS.md, then configured fallback filenames in each directory. It selects at most one non-empty file per directory and walks from the project root down to the directory where the run starts. The files are concatenated root first. Codex documents closer guidance as overriding earlier guidance because it appears later in the combined prompt. The default project guidance budget is 32 KiB. OpenAI's AGENTS.md guide and the official configuration schema describe those details.

The trap is the difference between launch path and target path. If Codex starts at the repository root and the task mentions services/payments/handler.ts, the documented startup chain ends at the root. Do not assume a nested services/payments/AGENTS.md joined context merely because the model later touched that file. Launch a separate probe with --cd services/payments and compare the model-visible instruction payload.

For byte provenance, archive the session JSONL field that contains the model-visible user_instructions, then hash that exact string. Also snapshot each candidate file before launch. If the release's artifact flattens files without preserving source boundaries, mark per-file path attribution as inferred rather than measured.

Source-observed: The current Codex repository models ordered instruction entries with provenance in agents_md.rs. That is useful implementation evidence, but main is not a release contract. Record the pinned CLI version and binary hash, then test its actual JSONL shape.

Claude Code: CLAUDE.md is context, with lazy provenance

Documented: Claude Code says directly that CLAUDE.md is context, not enforced configuration. It reads CLAUDE.md, not AGENTS.md, unless a CLAUDE.md imports the shared file. Files at and above the working directory load at launch, ordered from the filesystem root toward the working directory. Nested CLAUDE.md files below the working directory load when Claude reads in those subdirectories. Path-scoped .claude/rules/*.md files can also load when their paths: patterns match. Claude Code's memory guide documents all of those cases.

Do not turn context order into an invented precedence guarantee. The same guide warns that when rules conflict, Claude may choose one arbitrarily. Its recommendation to keep a CLAUDE.md under 200 lines is authoring guidance, not a hard truncation contract.

Claude exposes a loader-owned event for this purpose. InstructionsLoaded includes the absolute file_path, memory_type, load_reason, and fields for path matches, nested traversal, and imports. The event fires again for lazy loads. It is asynchronous and cannot block. Use it to hash the file at load time and append the event to an immutable JSONL log. The hook reference defines the payload.

That event still does not enforce a prohibition. A separate PreToolUse hook can deny a tool call before execution. Put high-value hooks in managed or host-controlled configuration when possible, because a repository-controlled hook can share the same administrative boundary as the agent it is checking.

GitHub Copilot cloud coding agent: nearest AGENTS.md is a cloud rule

Documented: The cloud agent accepts a root .github/copilot-instructions.md, matching .github/instructions/**/*.instructions.md files, AGENTS.md files anywhere in the repository, or a single root CLAUDE.md or GEMINI.md. GitHub states that the nearest AGENTS.md in the directory tree takes precedence. It also states that repository-wide and matching path-specific instructions are both used. GitHub's repository instructions page is the contract for this surface.

Do not transfer that nearest-file rule to Copilot CLI. They are different loaders.

Ambiguous: GitHub documents cloud-agent session logs as showing reasoning and tools used, but the reviewed session documentation does not promise a loader event with exact instruction paths and bytes. Archive the session log anyway. If the log for your pinned run exposes a loader record, normalize it. If it does not, mark discovery provenance unknown; do not fill the gap with the agent's prose. GitHub's session-log documentation supports what the log is intended to show.

The cloud service does not give you a local binary to hash. Pin what GitHub exposes, including the selected agent surface, model, repository revision, organization policy, workflow setup, and adapter revision. Treat an opaque service change as a reason to rerun the contract before promotion.

GitHub Copilot CLI: the file set is broader and precedence is undefined

Documented: Copilot CLI searches the repository root, current working directory, intermediate directories, and directories nested in the path of a file it is working on. It can combine user instructions, .github/copilot-instructions.md, modular .instructions.md files, AGENTS.md, CLAUDE.md, .claude/CLAUDE.md, GEMINI.md, and extra configured directories. GitHub explicitly says it does not define a general precedence order between applicable files. The Copilot CLI custom-instructions guide is separate from the cloud-agent page.

Use /instructions to list files discovered for the session and to see which files are enabled. That slash command is client-side evidence, not a model answer. Script it through a pseudo-terminal in the adapter, archive the raw screen or structured event if available, and hash the listed files while the fixture is read-only. Changes to instructions require a resumed or new session before they apply, so every mutation control must start fresh.

For the deliberate conflict, record the observed choice across trials but do not call either choice compliant. GitHub has declined to define a general winner. The non-conflicting canaries remain the compliance assertions.

Google Gemini CLI: official pages disagree on nested discovery

Documented: Gemini CLI uses GEMINI.md by default, and context.fileName can be a list that includes AGENTS.md. Its dedicated guide describes global and workspace files plus just-in-time context loaded when a tool accesses a file or directory. /memory show displays the exact concatenated instructional context. The GEMINI.md guide is direct about that JIT path.

Ambiguous: The configuration reference also describes a scan below the current working directory, bounded by context.discoveryMaxDirs, whose default is 200 directories. The same reference says upward traversal stops at a configured boundary marker, .git by default. These descriptions can produce different expectations for when a nested canary becomes active.

Source-observed: The official repository's current config.ts shows evolving context-loading code. Do not use that main branch to claim a stable package behaves one way. Pin the Gemini CLI package and release channel, persist the complete settings file and feature state, then run both a pre-access /memory show and a post-target-access /memory show. The observed delta becomes the contract for that pin.

Compile one tiny fixture into surface-native files

One semantic fixture should produce separate repositories for separate loaders. If you leave AGENTS.md, CLAUDE.md, GEMINI.md, and Copilot files together, a surface that recognizes several names can merge duplicates and invalidate the probe.

Keep the source fixture small:

instruction-contract/
  fixture.json
  templates/
    root.md
    nested.md
  repo/
    src/root-target.txt
    services/payments/nested-target.txt
    services/payments/leaf/deep-target.txt
  adapters/
    codex-cli
    claude-code
    copilot-cloud
    copilot-cli
    gemini-cli
  scripts/
    compile-fixture
    fingerprint
    run-matrix
    assert-result

The root template contains one non-conflicting canary and the root side of a conflict:

ROOT_CANARY: when asked for the conformance result, set root_seen to ROOT_CANARY_A.
CONFLICT_CANARY: set conflict_choice to ROOT.

The nested template contains the second non-conflicting canary and the nested side of the same conflict:

NESTED_CANARY: when asked for the conformance result, set nested_seen to NESTED_CANARY_B.
CONFLICT_CANARY: set conflict_choice to NESTED.

The task prompt supplies the output format, not another repository rule:

Read <TARGET_PATH> before acting. Write evidence/behavior.json with only these keys:
root_seen, nested_seen, conflict_choice, task_completed.
Use null for any canary you did not receive. Do not answer which files you think loaded.

The manifest maps those semantics into filenames accepted by one surface at a time:

{
  "schema_version": 1,
  "canaries": {
    "root": "ROOT_CANARY_A",
    "nested": "NESTED_CANARY_B",
    "conflict": ["ROOT", "NESTED"]
  },
  "surfaces": {
    "codex-cli": ["AGENTS.md", "services/payments/AGENTS.md"],
    "claude-code": ["CLAUDE.md", "services/payments/CLAUDE.md"],
    "copilot-cloud": ["AGENTS.md", "services/payments/AGENTS.md"],
    "copilot-cli": ["AGENTS.md", "services/payments/AGENTS.md"],
    "gemini-cli": ["GEMINI.md", "services/payments/GEMINI.md"]
  },
  "variants": [
    {"launch": ".", "target": "src/root-target.txt"},
    {"launch": ".", "target": "services/payments/nested-target.txt"},
    {"launch": "services/payments", "target": "services/payments/nested-target.txt"},
    {"launch": "services/payments/leaf", "target": "services/payments/leaf/deep-target.txt"}
  ]
}

The compiler should also emit surface-specific capability variants. For Copilot cloud, add .github/copilot-instructions.md and a matching .github/instructions/payments.instructions.md in isolated probes. For Copilot CLI, probe each additional accepted filename separately before testing combinations. For Gemini, test the default filename first, then a pinned context.fileName list containing AGENTS.md. For Claude, add one paths: rule and one imported file so InstructionsLoaded reasons are exercised.

Fingerprint candidates before the loader runs

The fingerprint is an expectation manifest, not proof of loading. It makes later loader evidence tamper-evident and lets CI compare the observed instruction set with an allowlist.

#!/usr/bin/env bash
set -euo pipefail

repo=${1:?repo path required}
paths_json=${2:?JSON array of emitted instruction paths required}
out=${3:?output path required}

jq -r '.[]' "$paths_json" | while IFS= read -r rel; do
  file="$repo/$rel"
  jq -n \
    --arg path "$rel" \
    --arg sha256 "$(sha256sum "$file" | awk '{print $1}')" \
    --argjson bytes "$(wc -c < "$file")" \
    '{path:$path, sha256:$sha256, bytes:$bytes}'
done | jq -s 'sort_by(.path)' > "$out"

sha256sum "$out" | awk '{print $1}' > "$out.sha256"

Run that script before and after each trial. If any instruction file changes, invalidate the trial. Record the repository commit, compiler commit, adapter commit, executable version, binary digest where available, model, config digest, environment image digest, launch directory, and target path alongside the fingerprint.

Make every adapter return the same evidence contract

The runner should not know vendor-specific flags. Each adapter receives the same inputs and writes normalized artifacts into a fresh trial directory:

adapter run
  --repo <immutable-fixture-path>
  --launch <relative-launch-directory>
  --target <relative-target-path>
  --prompt <task-prompt-path>
  --trace-dir <empty-output-directory>

required outputs:
  invocation.json
  loader.raw
  model.raw
  behavior.json
  task-outcome.json
  enforcement.json
  usage.json

The adapters do different provenance work:

• Codex CLI: archive the pinned run's session JSONL and extract the model-visible instruction payload. Record whether the release preserves per-file paths or only flattened bytes.
• Claude Code: install an InstructionsLoaded hook that appends the event plus the file hash and byte count at event time. Preserve events that arrive after nested traversal or a path match.
• Copilot cloud agent: export the complete session log and task metadata from the disposable fixture repository. If no exact loader record exists, return discovery.status = "unknown" rather than asking the model.
• Copilot CLI: drive /instructions through a pseudo-terminal before the task and after the target is opened. Archive both captures and the CLI session trace.
• Gemini CLI: capture /memory list and /memory show before target access and again after target access. Preserve the settings file that selects filenames and discovery behavior.

The outer loop requires the trial count instead of hiding a reassuring default:

#!/usr/bin/env bash
set -euo pipefail

: "${SURFACE:?set SURFACE}"
: "${TRIALS:?set TRIALS from the rollout risk budget}"

while IFS= read -r variant; do
  launch=$(jq -r '.launch' <<<"$variant")
  target=$(jq -r '.target' <<<"$variant")

  for trial in $(seq 1 "$TRIALS"); do
    run_id=$(printf '%s__%s__%s__%s' \
      "$SURFACE" "${launch//\//_}" "${target//\//_}" "$trial")
    trace="artifacts/$run_id"
    mkdir -p "$trace"

    ./scripts/fingerprint "work/$SURFACE" emitted-paths.json \
      "$trace/instructions.before.json"

    started_ns=$(date +%s%N)
    "./adapters/$SURFACE" run \
      --repo "work/$SURFACE" \
      --launch "$launch" \
      --target "$target" \
      --prompt task.txt \
      --trace-dir "$trace"
    ended_ns=$(date +%s%N)

    ./scripts/fingerprint "work/$SURFACE" emitted-paths.json \
      "$trace/instructions.after.json"
    ./scripts/assert-result "$trace" "$started_ns" "$ended_ns"
  done
done < <(jq -c '.variants[]' fixture.json)

Repeated trials belong only to behavioral compliance and task outcome. Discovery and merge assertions should be deterministic for a pinned loader, but rerunning them is still useful for detecting service drift or race conditions around lazy loading.

Assert scope without inventing semantic precedence

Expected assertions depend on the documented surface contract.

For a root launch and root target, the root canary should be in scope. The nested canary should not be asserted unless the loader eagerly scans descendants. For a root launch and nested target, Claude, Copilot CLI, and a JIT Gemini configuration should gain nested context after the target is read. Codex's documented startup chain remains tied to launch cwd, so the separate nested-launch variant is the relevant assertion. Copilot cloud should apply the nearest AGENTS.md to the file it works on.

For the deliberate conflict:

• Assert a winner only where the vendor documents a semantic rule for that exact file family and scope.
• Codex documents closer guidance as overriding earlier guidance in its root-to-cwd chain.
• Copilot cloud documents nearest AGENTS.md precedence.
• Claude Code warns that conflicting instructions may be chosen arbitrarily.
• Copilot CLI defines no general precedence order.
• Gemini documents concatenation and loading order, but the reviewed pages do not promise that last text wins semantically.

For the last three, store conflict_choice as an observation. Do not count either value as a compliance failure. The two non-conflicting canaries carry the compliance signal.

Keep the result schema multidimensional

Never collapse these records into one pass rate:

{
  "schema_version": 1,
  "run": {
    "surface": "copilot-cli",
    "tool_version": "<pinned-version>",
    "tool_digest": "<binary-or-service-identity>",
    "model": "<pinned-model>",
    "config_sha256": "<hash>",
    "repo_commit": "<commit>",
    "fixture_sha256": "<hash>",
    "adapter_commit": "<commit>",
    "launch_dir": ".",
    "target_path": "services/payments/nested-target.txt",
    "trial": "<trial-index>"
  },
  "discovery": {
    "status": "pass|fail|unknown",
    "paths": [],
    "combined_context_sha256": "<hash-or-null>",
    "path_evidence": "tool_emitted|artifact_extracted|inferred|missing",
    "bytes_evidence": "tool_emitted|artifact_extracted|inferred|missing",
    "artifact": "<uri>"
  },
  "merge_scope": {
    "status": "pass|fail|unknown",
    "applicable_paths": [],
    "observed_order": [],
    "scope_reason": [],
    "conflict_choice": "ROOT|NESTED|null",
    "conflict_assertion": "documented|observe_only"
  },
  "compliance": {
    "root_canary": "pass|fail|not_applicable",
    "nested_canary": "pass|fail|not_applicable",
    "self_report": "<debug-only-or-null>"
  },
  "task_outcome": {
    "status": "pass|fail",
    "changed_paths": [],
    "tests": []
  },
  "enforcement": {
    "status": "blocked|allowed|not_exercised|unknown",
    "control": "<os|iam|hook|ruleset|ci>",
    "attempted_action": "<normalized-action>",
    "denial_artifact": "<uri-or-null>"
  },
  "latency": {
    "queue_ms": null,
    "runtime_ms": null,
    "total_ms": null
  },
  "cost": {
    "currency": "USD",
    "provider_reported": null,
    "estimated": null,
    "estimation_method": null
  }
}

This schema prevents an attractive but meaningless statement such as "the agent passed." A run can have correct discovery, unstable compliance, a failed task, and a working external denial. Those are four different engineering decisions. Latency and cost are operational dimensions, not quality scores, and they stay separate too.

Negative controls prove the harness can detect failure

A positive canary is not enough. The harness must show that it fails for the right reasons.

Expected enforcement is a denial receipt and no side effect. A model refusal is a compliance observation, not proof of enforcement. An allowed direct probe is a failed control even if every model trial behaved perfectly.

The remove-and-recreate controls also catch session reuse. Codex builds its instruction chain for a new run, Copilot CLI requires a resumed or new session for changed instructions, and lazy loaders can change context after target access. Record session identity and refuse to compare trials that did not start from the declared lifecycle state.

Put must-not-break invariants outside the prompt

The independent control should survive a missing file, a loader regression, an ambiguous merge, a compromised prompt, and a model that simply does not comply.

Use controls at the resource boundary:

• Read-only credentials: Give the agent a token that can read issues but cannot merge, deploy, rotate keys, or write production data. Prefer short-lived workload identity. A sentence saying "do not deploy" cannot add a missing IAM permission.
• Sandbox permissions: Mount secrets outside readable paths, mount protected inputs read-only, restrict writable roots, and default-deny network egress. Codex documents its sandbox and approval policy as separate layers, with local defaults that keep network off and writes inside the active workspace. OpenAI's security guide describes that boundary.
• Branch protection and rulesets: Require a pull request, human review, status checks, and no agent bypass actor. GitHub documents that its cloud agent cannot approve or merge its own pull request and remains subject to branch protections and required checks. GitHub's cloud-agent risk guidance and ruleset reference make this an enforceable remote boundary.
• CI policy checks: Reject forbidden path changes, unsigned artifacts, dependency-policy violations, missing tests, or infrastructure mutations. Run the check from a trusted workflow whose definition the agent cannot modify in the same approval path.
• Claude PreToolUse hooks: Deny a Bash, Write, or Edit call before execution and log the attempted input. Claude documents that PreToolUse can deny calls and that permissions plus OS sandboxing are complementary. The permissions guide also warns that file-tool rules do not constrain a Bash subprocess by themselves, which is why OS enforcement still matters.

A minimal Claude hook can route the decision to a host-controlled checker:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash|Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "/opt/agent-policy/check-tool-input"
          }
        ]
      }
    ]
  }
}

The checker returns a structured deny for protected paths, deployment commands, or disallowed network destinations and stores the input in an append-only audit sink. Keep the executable and managed hook settings outside the repository's writable boundary. A project hook is still useful, but it is not independent if the same agent can rewrite or disable it.

This is also why a sandbox alone is not a security model. The sandbox covers an execution boundary. IAM, remote branch rules, CI policy, and review controls cover identities and downstream effects that the local filesystem cannot.

Fail closed on missing evidence

Recommendation: Promotion fails when any critical evidence is absent, not only when an assertion is explicitly false.

• Missing exact bytes means discovery.status = unknown.
• Missing scope reason or undocumented merge behavior means merge_scope.status = unknown for that assertion.
• Too few or interrupted trials leave the affected compliance rule incomplete.
• A prohibited action that was never directly exercised leaves enforcement not_exercised.
• A missing denial log leaves enforcement unknown, even if no side effect is visible.
• A changed tool, model, config, environment, fixture, or adapter digest invalidates the allowlisted contract.

Do not let a task success compensate for any of those states. The cost of a false block is a delayed rollout. The cost of treating missing enforcement evidence as a pass can be an unauthorized push, credential use, or production mutation.

CI workflow skeleton

Run local CLIs in pinned, isolated runner images. Run Copilot cloud against a dedicated disposable repository because its service lifecycle and permissions differ from a local executable. The adapter is the only component allowed to translate vendor output into the shared schema.

name: repository-instruction-contract

on:
  workflow_dispatch:
    inputs:
      trials:
        description: Required compliance trial count
        required: true

permissions:
  contents: read

jobs:
  local-surfaces:
    strategy:
      fail-fast: false
      matrix:
        surface: [codex-cli, claude-code, copilot-cli, gemini-cli]
    runs-on: [self-hosted, agent-conformance]
    container:
      image: ghcr.io/your-org/agent-conformance@<pinned-image-digest>
    steps:
      - uses: actions/checkout@<pinned-commit-sha>
      - name: Verify tool, model, config, fixture, and adapter pins
        run: ./harness/verify-pins "${{ matrix.surface }}"
      - name: Compile isolated fixture
        run: ./harness/compile-fixture "${{ matrix.surface }}"
      - name: Run discovery, scope, and compliance probes
        env:
          SURFACE: ${{ matrix.surface }}
          TRIALS: ${{ inputs.trials }}
        run: ./harness/run-matrix
      - name: Run direct external-denial probe
        run: ./harness/run-enforcement-negative "${{ matrix.surface }}"
      - name: Validate multidimensional records
        run: ./harness/policy-gate "artifacts/${{ matrix.surface }}"
      - uses: actions/upload-artifact@<pinned-commit-sha>
        with:
          name: instruction-contract-${{ matrix.surface }}
          path: artifacts/${{ matrix.surface }}

  copilot-cloud:
    runs-on: [self-hosted, agent-conformance]
    steps:
      - uses: actions/checkout@<pinned-commit-sha>
      - name: Launch and wait for disposable cloud-agent probes
        env:
          TRIALS: ${{ inputs.trials }}
          FIXTURE_REPOSITORY: ${{ vars.COPILOT_FIXTURE_REPOSITORY }}
        run: ./harness/adapters/copilot-cloud run-matrix
      - name: Require loader provenance or an explicit unknown result
        run: ./harness/policy-gate artifacts/copilot-cloud
      - uses: actions/upload-artifact@<pinned-commit-sha>
        with:
          name: instruction-contract-copilot-cloud
          path: artifacts/copilot-cloud

  promotion:
    needs: [local-surfaces, copilot-cloud]
    runs-on: [self-hosted, agent-conformance]
    steps:
      - name: Compare observed fingerprints with the approved allowlist
        run: ./harness/approve-only-known-contracts artifacts/

Pin GitHub Actions by full commit SHA, not a moving tag. Give the workflow read-only repository permissions by default. Vendor credentials belong in surface-specific jobs with the minimum scope, and the fixture repository must contain no production secret or deploy path.

The gate should evaluate named fields, not calculate an overall score. A sensible promotion rule is: required discovery assertions pass, required scope assertions pass, every non-conflicting compliance rule stays within the team's error budget, the task outcome meets its own threshold, and every critical external probe returns a verified denial. Latency and cost can have separate budgets without becoming evidence of correctness.

Common questions

Operational rollout checklist