AI Coding-Agent Pilot: Cost Model and Rollout Checklist

A coding-agent rollout should begin with one repository, one representative task class and one accountable acceptance owner. Price the pilot by reconciled accepted tasks, then release only when the pinned runtime, denied actions, merge control and rollback have retained evidence.

Coding agents can make a repository faster to change while making the release process harder to reason about. A seat license measures access. Prompt counts and generated lines measure activity. Neither tells a CTO whether accepted work became cheaper, safer or easier to operate.

This guide turns the first rollout into a bounded release decision. It includes an editable cost model, a six-part evidence gate and a checklist for moving from observed pilot work to a controlled production rollout. The displayed defaults are internal reference assumptions, not client results, vendor quotes or a promised return.

Start with one repository and one task class

Choose a repository with an active owner, a reproducible test path and work that can be reviewed without guessing intent. Then name one task class. Examples include dependency updates within a defined package boundary, adding tests around an existing interface, or implementing a small issue whose acceptance criteria are already explicit.

Do not start with “all engineering work.” Mixing bug fixes, migrations, generated code, infrastructure changes and product design into one cohort destroys the denominator. Each class has different context needs, failure modes and review cost.

The pilot contract should name:

• the repository and branch boundary;
• the eligible task class and explicit exclusions;
• the instruction files and their precedence;
• the model, client, tool set, sandbox and network policy;
• the test, review and acceptance rules;
• the human who accepts work and the person who owns production authority;
• the stop condition and rollback path.

Repository instructions are part of that runtime contract. OpenAI documents that AGENTS.md instructions can be inherited through a repository tree, with files closer to the working directory overriding broader guidance. That makes discovery order and the actual working directory release inputs, not background documentation. OpenAI custom instructions with AGENTS.md.

Price the pilot by accepted tasks, not seats

Join observed provider and infrastructure spend with modelled human review and rework, then divide by reconciled accepted tasks. Keep the categories separate so a licensing change cannot masquerade as lower review effort and a reduction in review time cannot be reported as cash savings.

The worksheet uses these inputs:

1. Seat cost. Pilot seats multiplied by the contracted monthly seat price.
2. Observed model usage. Metered provider spend from the same pilot window.
3. Sandbox and CI cost. Incremental execution, isolation, test and artifact storage cost.
4. Evaluation cost. Incremental model or harness spend used to score the cohort.
5. Human review and rework. Hours multiplied by the buyer's loaded planning rate.
6. Attempted and accepted tasks. One cohort with one acceptance rule and one observation window.

The illustrative default is five seats at $40 each, $350 of observed model usage, $180 of sandbox and CI cost, $240 of evaluation cost, 18 reviewer hours, eight rework hours and a $110 loaded planning rate. With 40 attempted tasks and 24 reconciled accepted tasks, the arithmetic returns a joined pilot cost of $3,830 and about $159.58 per accepted task. These are sensitivity inputs chosen to demonstrate the calculator, not a statement about market pricing or a forecast for a buyer.

GitHub's billing documentation is useful for one part of this join. Its usage reports expose product and SKU usage plus gross, discounted and net amounts; its AI usage report adds model and token fields. Those fields can support the observed spend record, but they do not contain the buyer's review, rework or acceptance evidence. GitHub billing reports reference.

Product telemetry also needs a boundary. GitHub's Copilot usage metrics include adoption, interaction, code-generation and acceptance activity. The documentation explicitly describes acceptance actions such as applying or inserting generated content. Those are useful activity signals, but they are not the same as a task passing the repository's tests, review and business acceptance rule. GitHub Copilot usage metrics.

When accepted tasks are zero, cost per accepted task is unavailable rather than zero. The numerator still retains every observed and modelled cost. Do not remove failed or rejected work from the spend record.

Pin the runtime that produced the result

A coding-agent result belongs to the full runtime tuple, not to a prompt or model name alone. Record the repository commit, agent client and version, model and settings, instruction files actually loaded, working directory, tool schemas, sandbox, network access, credentials, supporting services and evaluator.

The point is repeatability and rollback. If the model changes, an instruction file moves, the tool gains a write scope or the sandbox can reach a new service, the evaluated object changed. Treat that as a new release candidate even when the user interface looks identical.

Retain one evidence bundle per attempted task:

• immutable task and runtime identifiers;
• input issue and acceptance criteria;
• resolved instructions and tool policy;
• commands, tool calls and external effects;
• test and evaluator output;
• review time, rework time and final disposition;
• accepted commit or explicit rejection reason.

This is the same reason a coding-agent task economics ledger needs one immutable task ID through usage, review, rework and outcome. Aggregate token and seat totals cannot be joined back to accepted work after the fact unless the identifier exists before dispatch.

Test normal, exception and prohibited work

Representative tasks prove the ordinary path. Denied-action probes prove that the boundary survives pressure. A useful pilot needs both.

Build a compact task set across three categories:

A readiness score is an inventory aid, not release evidence. The repository readiness release gate shows why discovery checks must be paired with representative work, denied actions, merge controls and rollback.

For every task, decide what counts as accepted before running it. “The agent opened a pull request” is a submission event. “The patch merged” can still hide human repair. A defensible accepted outcome should meet the named functional checks and preserve the review rule, with rework recorded separately.

Keep merge authority outside the agent

The agent can prepare work without owning release. Protected branches, required checks and code ownership give the repository an enforcement surface independent of the agent's explanation.

GitHub's branch-protection controls can require pull-request reviews and status checks before a protected branch accepts changes. CODEOWNERS can request review from the people responsible for matching paths, and branch protection can require code-owner approval. Configure these controls in the repository rather than relying on the agent to remember a policy in prose. GitHub branch protection and GitHub CODEOWNERS.

The release evidence gate in the worksheet deliberately keeps six questions separate:

• Is the runtime tuple pinned?
• Did representative tasks pass their predeclared acceptance rules?
• Did prohibited actions fail safely?
• Did a human-controlled merge boundary remain effective?
• Was rollback tested against the actual release path?
• Is one person accountable for accepting the task class?

Checking every item does not certify a coding agent. It only records that this bounded pilot has the minimum evidence needed for a release decision. A different repository, task class or runtime needs its own decision.

Use this rollout checklist

Treat rollout as a sequence of increasingly consequential releases.

1. Name the owners. The engineering owner defines the task class and acceptance rule. The systems owner controls credentials, runners, branch policy and production release.
2. Choose the cohort. Select one repository and one task class with enough recent examples to expose ordinary and exception work. Preserve excluded tasks as excluded.
3. Pin the runtime. Record the exact repository state, agent client, model, settings, instructions, tools, sandbox, network, credentials and evaluator.
4. Apply least authority. Start read-only where possible. Scope writes to the working branch and required systems. Put merge and deployment behind separate controls.
5. Run the task set. Include representative, exception and prohibited cases. Retain full receipts and stop on an unexplained external effect.
6. Reconcile every outcome. Classify accepted, rejected, repaired, uncertain and divergent tasks. Do not count a submission, receipt or open pull request as accepted work.
7. Join the economics. Combine observed seat, model and infrastructure cost with modelled review and rework. Divide by reconciled accepted tasks.
8. Test rollback. Revert the agent configuration and affected release path. Confirm that ordinary engineering work can continue without the agent.
9. Release narrowly. Keep the same repository, task class and owners for the first live window. Sample accepted work and review failures daily.
10. Expand by evidence. Add one repository, task class or authority boundary at a time. Re-run the gate whenever the runtime tuple changes.

Know what the worksheet cannot decide

The calculator does not measure code quality, security, developer satisfaction, cycle-time improvement or business value. It cannot tell whether the accepted-task rule is weak, whether reviewers are rubber-stamping output or whether deferred defects moved beyond the pilot window.

Use the result as one operating view. Pair it with escaped-defect review, denied-action probes, reviewer sampling and a post-release observation window. Keep observed AI and infrastructure cost distinct from modelled human cost, and keep both separate from business value.

A vendor dashboard may answer how often a feature was used. This worksheet answers a narrower management question: for one controlled cohort, what did accepted work cost, and is the retained release evidence complete enough to proceed?