21 Tests for an AI Incident-Triage Correlation Gate

The reference controller below passes 21 local cases. It is dependency-free Node.js, not a production detector or a claim about customer data. Its job is narrower: prevent a triage agent from turning plausible pairwise links into an unsupported incident group.

The transitive bridge is the failure

Connected components are unsafe when correlation compatibility is not transitive. Consider three alerts under a six-minute window:

• A occurred at minute 0.
• B occurred at minute 5.
• C occurred at minute 10.

A can correlate with B. B can correlate with C. A cannot correlate with C because they are ten minutes apart. A union-find or graph connected-component pass still puts all three in one group through B.

That changes more than the notification count. The group may now carry the wrong entity evidence, incident severity proposal, approval request and destination record. If an agent later proposes containment against that record, the false merge has crossed from noisy telemetry into action authority.

NIST's final incident-response recommendations place response inside Cybersecurity Framework 2.0 risk management. The grouping policy is therefore part of the response contract, not an invisible preprocessing trick.

Make the first policy deliberately narrow

The first grouping rule should prefer false separation over false merger. Start with exact agreement on fields that already have accountable meaning:

• tenant or customer boundary;
• service or owned system;
• entity type and entity identifier;
• alert-rule version;
• a bounded occurrence-time window.

This will split some alerts a human later joins. That is acceptable in a first release. A false separation creates two packets for review. A false merger can attach unrelated evidence and authority to one record.

Amazon CloudWatch supports metric and composite alarms, and its alarm documentation says composite alarms use rule expressions over underlying alarm states. AWS's composite-alarm guide shows Boolean conditions such as two alarms being active while a deployment alarm is not active. That is explicit, versionable aggregation. It is different from asking a model whether two free-form summaries feel related.

CloudWatch also separates evaluation from action. Its action-suppression controls can prevent actions during maintenance, deployments or an active investigation without deleting the alarm configuration. Preserve the same distinction in an agent: correlation may produce a proposal while action remains stopped.

Require compatibility with every group member

The safe reference sorts events deterministically, then lets a new event join a group only when it is compatible with every existing member. The method is deliberately conservative and quadratic within a candidate group. That cost is reasonable for a bounded pilot cohort and easier to inspect than a learned similarity threshold.

export function compareEvents(a, b, { maxGapMs }) {
  for (const key of [
    "tenantId", "service", "entityType", "entityId", "ruleVersion"
  ]) {
    if (a[key] !== b[key]) return { eligible: false, reason: `${key} differs` };
  }

  const gap = Math.abs(Date.parse(a.occurredAt) - Date.parse(b.occurredAt));
  if (gap > maxGapMs) {
    return { eligible: false, reason: "events exceed the correlation window" };
  }

  return { eligible: true };
}

for (const event of sortedEvents) {
  const group = groups.find((candidate) =>
    candidate.events.every((member) => compareEvents(event, member, policy).eligible)
  );

  if (group) group.events.push(event);
  else groups.push({ events: [event] });
}

The complete retained artifact also validates required fields, rejects invalid timestamps, removes duplicate source event IDs and holds malformed events outside every group. Input order does not change the result because sorting uses occurrence time and then source ID.

OpenTelemetry's event semantic conventions make the evidence split useful. An EventRecord has an event name, an occurrence timestamp and occurrence-specific attributes. Event names should identify the event structure rather than carry dynamic identifiers. The document is still marked Development, so pin the convention version instead of treating current field names as permanent.

The OpenTelemetry logs data model also distinguishes Resource, which is fixed for a source, from Attributes that vary per occurrence. Do not flatten that distinction into one generated incident summary. The group needs both source identity and occurrence evidence.

Bind approval to exact membership

Approval should cover a specific correlation proposal, not a mutable incident idea. The reference hashes the policy version and sorted event IDs into a proposal key:

const material = `${policyVersion}\n${[...eventIds].sort().join("\n")}`;
const proposalKey = createHash("sha256").update(material).digest("hex");

Adding or removing an event changes the key. Updating the policy version changes the key. Either change invalidates an approval for the earlier proposal.

The key is not proof that the grouping is correct. It is a stable name for exactly what the reviewer saw. Retain the underlying event references, correlation reasons, policy version, reviewer identity, approval time and destination intent beside it.

This is the engineering edge behind the incident-response cost and rollout worksheet. The resource defines accepted enrichment packets, reconciled incident records and acknowledged handoffs. Exact proposal membership makes those outcomes countable without pretending that alerts touched are business acceptance.

Reconcile the incident-system write

A write receipt is not the terminal. The controller keeps four distinct states:

1. Not authorized: the approved proposal key does not match the current key.
2. Not accepted: no destination receipt exists.
3. Uncertain or divergent: the destination cannot be read, or its state contradicts the accepted write.
4. Observed: the authoritative record returns the same proposal key and event digest.

export function evaluateIncidentWrite({ proposed, approved, receipt, observed }) {
  if (!approved || proposed !== approved) return { status: "not-authorized" };
  if (!receipt) return { status: "not-accepted" };
  if (!observed) return { status: "uncertain" };
  if (
    observed.proposalKey !== proposed ||
    observed.eventDigest !== receipt.eventDigest
  ) return { status: "divergent" };
  return { status: "observed" };
}

Do not retry an uncertain write with a new idempotency key. Reconcile the existing intent first. The broader tool receipt and outcome method explains why provider acceptance and authoritative state are separate evidence.

Release against the 21 cases

The local gate passes 21 of 21 cases:

• exact bounded correlation;
• tenant, service, entity and rule-version mismatch;
• events outside the time window;
• missing evidence and invalid timestamps;
• duplicate source IDs;
• the A-B-C transitive bridge;
• deterministic output under reversed input order;
• cross-tenant batch separation;
• membership-sensitive and policy-sensitive proposal keys;
• exact approval matching;
• missing receipt, uncertain read-back, divergence and observation;
• invalid policy windows.

These tests prove controller behavior against synthetic fixtures. They do not establish detection quality, a useful correlation window, incident reduction or operator acceptance. A real release still needs representative local alerts, labelled false-merge and false-separation cases, destination-system reconciliation and an incident commander who owns the stop decision.

The first metric should be reviewed false merges per proposed group, not alert reduction. Noise reduction is valuable only after the grouping preserves the incident boundary. Optimize the wrong denominator and the agent becomes a fast way to create convincing, incorrect records.

Let the model propose context, not rewrite identity

The model still has useful work after the deterministic boundary. It can summarize the shared symptoms, retrieve the current playbook, identify missing evidence, propose likely owners and explain why a group deserves review. Those outputs should refer to the immutable source events and proposal key. They should not silently change group membership.

Treat topology and semantic similarity as candidate evidence. A deployment event on the same service, a shared trace identifier or an upstream dependency can justify a richer correlation proposal. Add each signal as a versioned field with a testable rule. Do not replace exact tenant and entity boundaries with one embedding threshold.

Late events need a separate path. If an incident record has already been approved and written, a newly arrived event creates a new proposal key. The agent may suggest attaching it to the existing record, but the earlier approval did not cover that membership. Preserve the original group, record the late-event proposal and require a new decision. This keeps the evidence history intact instead of rewriting what the incident commander previously accepted.

The same rule applies when the correlation policy changes. Replaying old events under correlation-v2 may produce a useful comparison, but it must not mutate a correlation-v1 record in place. Store the old proposal, the new proposal, the exact difference and the reviewer outcome. A version change is evidence, not permission.

Start richer grouping only after the narrow gate produces labelled disagreements. Reviewers can mark false separations and false merges, then the team can decide whether a new topology field or a longer window improves the acceptance set. That creates an evaluation corpus from the operator's actual decision instead of training the system to imitate its own earlier summaries.

Frequently asked questions