Measure Coding-Agent Cost at the Task Boundary

Three useful reports, three different grains

GitHub has materially improved the observability available to Copilot administrators. Its AI usage report now breaks each model's consumption into input, output, cache-read, and cache-write tokens beside AI credits. The billing reference says those records are grouped by date, model, and username. That is enough to explain where a charge came from at the account level. It is not a task ledger. (Per-model token release, billing reports reference)

The Copilot usage metrics API now exposes recognized third-party agent apps through totals_by_3rd_party_agent. A stable agent_id supports comparisons across reporting periods. The nested interaction count records user-initiated job starts. Aggregated enterprise and organization reports also include session counts. GitHub warns that this nested count is not the same as the top-level interaction field, that unidentified activity is omitted, and that multiple integrations for one agent are combined. (Agent-app release, field definitions)

Repository and aggregate reports separately describe daily pull-request activity: creation, review, merge, applied suggestions, and median time to merge, with dedicated fields for Copilot-authored and Copilot-reviewed work. The same documentation notes that some review events can count on multiple days and that organization and enterprise totals can differ because of user deduplication and attribution timing.

These are useful views. They answer different questions:

The published fields do not share a coding-agent task key. Joining them by day, username, model, or repository would allocate costs by assumption. That is my inference from the documented schemas, not a limitation GitHub states in those words.

Mint the join key before the agent starts

The control point is job submission. Create one immutable task_id before dispatch, then propagate it through every record that can change the economics decision:

{
  "task_id": "task-2026-08-25-0042",
  "submission": {
    "agent_id": "agent-app-a",
    "repository": "owner/service",
    "request_kind": "issue-to-pull-request"
  },
  "usage_receipts": [
    {
      "provider_event_id": "<immutable receipt id>",
      "model": "<model id>",
      "actual_cost_usd": "<observed amount>",
      "input_tokens": "<observed count>",
      "output_tokens": "<observed count>"
    }
  ],
  "human_work": [
    {
      "kind": "review | rework",
      "minutes": "<observed minutes>",
      "source_reference": "<review log>"
    }
  ],
  "terminal_outcome": {
    "value": "accepted | rejected | abandoned",
    "evidence": "<pull request or decision record>"
  }
}

Do not derive the task ID from a mutable title, branch name, or issue text. Those values are useful attributes. They are poor identities. Generate the ID at the admission boundary and retain it in the scheduler, runtime trace, provider receipt, pull-request metadata, review record, and final decision.

The propagation path should be testable:

submission
  → scheduler envelope
  → runtime and tool traces
  → usage receipts
  → pull request or other work product
  → review and rework log
  → terminal decision

If a surface cannot carry the task ID directly, retain a signed mapping at the boundary that creates its native identifier. For example, map one provider run ID to one internal task ID when the run is created. Do not reconstruct that mapping later from timestamps.

Separate observed cost, modeled cost, and value

A useful ledger refuses to collapse three different claims.

Observed AI cost comes from task-linked billing or runtime receipts. Record the provider event, model, token dimensions, currency, and actual billed amount. If the provider only supplies daily aggregate spend, leave task cost unknown. Do not distribute it evenly across tasks.

Modeled human cost comes from observed review and rework minutes multiplied by an explicit labor-rate assumption. Store the minutes and the rate separately. A salary selector or loaded hourly rate is a model input, not an observed payment.

Business value requires a buyer-defined outcome and a defensible comparison. An accepted pull request may be a delivery outcome. It is not automatically profit, time saved, or incremental output. Lines changed and pull requests merged are also vulnerable to gaming and task-mix changes.

GitHub describes its own impact dashboard as directional. The new potential-return section compares average Copilot cost, modeled payroll share, and pull-request output across adoption cohorts. GitHub explicitly says AI-credit cost is estimated, compensation comes from a selector, and the figures should not be treated as exact financial results. (ROI dashboard release, dashboard documentation)

That is the right epistemic label for a cohort view. A task ledger serves a narrower purpose: it makes the numerator, denominator, missing records, and outcome evidence auditable before anyone attempts a return model.

Count rejected and abandoned work

A success-only dataset makes agent economics look better by construction. Every admitted task needs one terminal state:

• accepted: the work met the workflow's stated acceptance boundary;
• rejected: a reviewer made an explicit decision not to accept it;
• abandoned: the task stopped without a reviewable accepted result.

Keep retries under the original task when they pursue the same requested outcome. Record each run and usage receipt separately. A second attempt should increase the task's observed cost rather than appear as a fresh successful task with the failed attempt erased.

Define denominators before computing ratios:

observed AI cost per admitted task
  = task-linked observed AI cost / admitted tasks

observed AI cost per accepted task
  = task-linked observed AI cost for complete tasks / accepted tasks

acceptance rate
  = accepted tasks / tasks with a terminal decision

review burden per accepted task
  = observed review + rework minutes / accepted tasks

The second metric must include spend on rejected and abandoned tasks in its numerator if the question is what the accepted portfolio cost. Excluding failure spend is a reporting choice, not an optimization.

Make incompleteness visible

Missing records are a measurement result. Report them beside the headline metric:

• tasks with no task-linked usage receipt;
• usage receipts with no admitted task;
• tasks with no terminal decision;
• review or rework records without source evidence;
• provider activity that could not be identified;
• windows where a telemetry source changed definition or coverage.

GitHub's own metrics documentation contains examples of why this matters. Recognized agent activity arrays can be omitted when no identifiable activity is available. Users discovered only through server-side telemetry can appear in active-user totals without appearing in dimensional breakdowns. Team reports exclude teams with fewer than five seated users, while users who belong to multiple teams appear in every applicable team aggregate. (Usage metrics fields, team metrics release)

These are documented aggregation rules, not defects. They mean that a serious economics readout must preserve source scope and missingness instead of forcing every dashboard tile to reconcile.

A validator for the ledger contract

I built a dependency-free Node reference under the same rule. It accepts newline-delimited events for task start, usage, human work, and terminal outcome. It rejects:

1. usage without a known task;
2. duplicate event identifiers;
3. usage without a source receipt;
4. negative cost;
5. multiple terminal outcomes;
6. usage after the terminal decision;
7. malformed timestamps;
8. events outside the reporting window.

It reports incomplete tasks as gaps rather than inventing values. The included test suite passes nine cases, including a structurally valid task that is deliberately missing both usage and outcome.

The output labels its basis:

{
  "observed_ai_cost": "sum of supplied task-linked actual_cost_usd values",
  "modeled_human_cost": "minutes × supplied hourly-rate modeling input",
  "value_claim": "none; acceptance is an outcome label, not a financial return"
}

The example data is synthetic. Its token counts, prices, labor rate, and outcomes are schema fixtures, not a DVNC benchmark or client result. The validator proves record shape and chronology. It does not authenticate provider receipts, verify payroll assumptions, inspect pull requests, or establish business value.

The rollout gate

Before using cost claims to widen a coding-agent rollout, require a short evidence statement:

For this reporting window, every admitted task received one immutable task ID.
Task-linked usage coverage was measured rather than inferred.
Rejected and abandoned work remained in the denominator.
Human review and rework were observed separately from modeled labor cost.
Every reported terminal outcome retained its source evidence.
No financial return was claimed without a named value metric and comparison design.

Portfolio dashboards still matter. They show adoption, model mix, credit consumption, and repository throughput at a scale a task ledger should not replace. The ledger closes a different boundary: it makes one task's cost and outcome traceable without pretending that adjacent aggregates are causal evidence.