Model Routing for Production AI Apps

The Production Rule: Route The Policy Yourself

The winning model-routing architecture is a thin app-owned policy over one or more managed routing tools. Let Microsoft Foundry, OpenRouter, or Amazon Bedrock make a local model choice when they are useful, but do not outsource the question that matters most: which models are allowed for this task, this tenant, this data class, this budget, and this incident state.

That distinction matters because managed routers optimize inside their own boundary. Microsoft Foundry model router is a deployable chat model that routes prompts in real time to the most suitable underlying LLM, and it selects an underlying model for each request based on routing settings. OpenRouter Auto Router uses openrouter/auto, is powered by NotDiamond, and returns response metadata showing which model was selected. Amazon Bedrock intelligent prompt routing provides a single serverless endpoint, but it routes between foundation models within the same model family.

Those are useful capabilities. They are not your production policy.

Your production policy should own the decision set before a request touches a router:

• task_class: classify the request as support answer, code edit, extraction, summarization, planning, agent tool-use, or another product-owned class.
• data_class: decide whether the request can leave a region, provider family, or compliance boundary.
• allowed_models: constrain the router to models your team has evaluated for this task.
• fallback_path: decide when to retry, downgrade, escalate to a stronger model, or send the request to human review.
• audit_record: log the policy input, chosen router, selected model, cost bucket, latency bucket, eval result, and fallback reason.

If a vendor router makes the final model choice, that choice still becomes an observable event in your system. The app is accountable for the answer, not the router.

What Managed Routers Actually Give You

Managed routers are best treated as routing primitives: they reduce operational work inside a bounded scope, but each one has different control and visibility.

Microsoft Foundry is the most concrete signal that cloud vendors now treat model routing as a first-class deployment object. Its current model router version is 2025-11-18, and the May 2026 update says the router supported 28 models while adding gpt-5.4, gpt-5.4-mini, gpt-5.4-nano, gpt-5.3-chat, gpt-5.5, claude-opus-4-7, and grok-4.1-fast-reasoning. It also supports model subsets, built-in automatic failover for the default deployment, and deployment in East US 2 and Sweden Central for Global Standard and Data Zone Standard deployment types.

The catch is not hidden. Foundry says the effective context window is limited by the smallest underlying model, and larger prompts only work reliably when your model subset supports the required context. It also says image inputs are accepted for vision-enabled chats, but routing decisions are based on text input only, and audio input is not processed. For Claude models such as claude-opus-4-7, you must first deploy the Claude model to your Foundry resource before the model router can invoke it.

OpenRouter gives a different tradeoff. Auto Router can pin both the selected model and provider for subsequent requests in the same conversation, and an explicit session_id is recommended for multi-turn conversations and agent workflows where consistent routing is needed from the start. The sticky-routing cache expires after 5 minutes of inactivity. You can restrict selectable models with allowed_models patterns, and cost_quality_tradeoff is an integer from 0 to 10, where 0 favors pure quality and 10 maximizes cost optimization. The default is 7. There is no additional Auto Router fee; you pay the standard rate for whichever model is selected.

Amazon Bedrock's router is narrower but useful when the deployment stays inside Bedrock. Bedrock predicts response quality for each model on each request and routes to the model with the best response quality to optimize quality and cost. AWS says Intelligent Prompt Routing can reduce costs by up to 30% without compromising accuracy. The practical limit is scope: configured prompt routers require exactly two models within the same family, and the service is only optimized for English prompts.

The Routing Policy That Survives Production Traffic

A production router starts as a policy table, not a model-ranking science project. The first version should be boring enough that an on-call engineer can explain a bad route from logs.

The point is not to avoid dynamic routing. It is to make dynamic routing reversible. AWS describes static routing and dynamic routing as the two main approaches, and describes LLM-assisted routing, semantic routing, and hybrid routing as common dynamic routing patterns. It also notes that LLM-assisted routing can add cost and latency, while semantic routing adds vector database and embedding-model complexity. That is the production tradeoff: a smarter router is only better when its overhead is lower than the waste or failure it removes.

For most SaaS products, the first working policy is hybrid:

• Static routing for obvious product surfaces, such as "CSV extraction always uses the extraction route."
• Semantic routing for broad intent classes where phrasing varies.
• Managed routing inside the selected provider boundary.
• Human approval for high-risk tool actions, account changes, irreversible writes, and user-visible decisions with weak eval confidence.

If your model stack already includes agents, connect this policy to the runtime rather than burying it in prompt text. The same production concern appears in agent framework choices: state, tracing, approvals, and failure handling matter more than the demo path.

A Minimal App-Owned Router Shape

The router should be explicit enough to diff in code review and small enough to keep out of prompt templates. This shape is intentionally plain TypeScript: a policy function returns a route plan, then provider adapters execute it.

type TaskClass =
  | "short_answer"
  | "long_context_synthesis"
  | "structured_extraction"
  | "code_change"
  | "agent_tool_use"
  | "high_risk_decision";

type DataClass = "public" | "tenant_private" | "region_bound" | "restricted";

type RoutePlan = {
  router: "foundry-model-router" | "openrouter-auto" | "bedrock-ipr" | "direct";
  allowedModels: string[];
  mode: "quality" | "balanced" | "cost";
  fallback: "retry_same_router" | "escalate_model" | "human_review";
  auditTags: string[];
};

export function planModelRoute(input: {
  task: TaskClass;
  dataClass: DataClass;
  tenantTier: "standard" | "regulated" | "enterprise";
  needsTools: boolean;
}): RoutePlan {
  if (input.dataClass === "restricted" || input.task === "high_risk_decision") {
    return {
      router: "direct",
      allowedModels: ["approved-frontier-review-model"],
      mode: "quality",
      fallback: "human_review",
      auditTags: ["approval_required", "no_auto_downgrade"],
    };
  }

  if (input.tenantTier === "regulated" || input.dataClass === "region_bound") {
    return {
      router: "foundry-model-router",
      allowedModels: ["foundry-approved-subset"],
      mode: "balanced",
      fallback: "escalate_model",
      auditTags: ["region_policy", "subset_enforced"],
    };
  }

  if (input.task === "short_answer" || input.task === "structured_extraction") {
    return {
      router: "openrouter-auto",
      allowedModels: ["anthropic/*", "openai/*", "google/*"],
      mode: "cost",
      fallback: "retry_same_router",
      auditTags: ["low_risk", "cost_preferred"],
    };
  }

  return {
    router: "direct",
    allowedModels: ["approved-reasoning-model"],
    mode: "quality",
    fallback: input.needsTools ? "human_review" : "escalate_model",
    auditTags: ["quality_preferred"],
  };
}

This is not enough by itself. The adapter still needs to map mode into vendor-specific controls. In Foundry, that means Quality, Cost, or Balanced, where Balanced is the default and considers underlying models within a 1% to 2% quality range compared with the highest-quality model for the prompt before choosing the most cost-effective model. In OpenRouter, that means setting the Auto Router cost_quality_tradeoff and passing session_id for agent workflows or multi-turn chats that must stay consistent. In Bedrock, that means selecting the model family, choosing exactly two models when configuring a prompt router, and setting routing criteria such as responseQualityDifference.

The audit record is where this becomes production infrastructure:

{
  "policy_version": "model-routing-current",
  "task_class": "long_context_synthesis",
  "data_class": "tenant_private",
  "router": "foundry-model-router",
  "allowed_models": ["foundry-approved-subset"],
  "mode": "balanced",
  "selected_model": "recorded_from_provider_metadata",
  "fallback": "escalate_model",
  "eval_suite": "synthesis-citation-regression",
  "eval_result": "pass"
}

Do not log secrets, raw private prompts, or direct personal data by default. Store prompt hashes, redacted spans, retrieval document IDs, model IDs, policy versions, and eval labels. When debugging requires prompt inspection, gate access separately and expire it.

What Breaks First

Context and model eligibility break before the router abstraction. Foundry states that the effective context window is limited by the smallest underlying model, and that larger API calls only succeed if routed to a model that supports the larger context. That means "the router supports the model" is not the same as "the route supports this request."

Prompt-cache behavior is the next break. OpenRouter pins the selected model and provider for a conversation to keep behavior consistent and improve prompt-cache hits, but that pin expires after 5 minutes of inactivity. If your agent expects a consistent model across a long workflow, pass session_id and log when stickiness changes.

Quality drift is the quiet failure. Microsoft says model router version 2025-11-18 is actively maintained and can receive new underlying models and features without changing the version identifier. Older versions 2025-08-07 and 2025-05-19 are frozen. Auto-update is useful, but every model-set change can move cost, quality, latency, refusal behavior, and tool-call reliability. Treat model-router updates like dependency updates: replay evals before and after.

Fallbacks can also hide incidents. Foundry's default deployment includes built-in automatic failover with no additional configuration. That keeps user requests moving, which is good. It can also mask provider instability unless you log the selected model and fallback path. A dashboard should show fallback rate, route distribution, eval pass rate, cost per successful completion, and human-review escalation rate by task class.

The Decision Rule

Use managed routing for bounded optimization, and use an app-owned policy for production accountability.

Choose Microsoft Foundry model router when the workload stays inside Azure Foundry, your region and deployment types fit East US 2 or Sweden Central, and the model subset can enforce context, compliance, and cost boundaries. Choose OpenRouter Auto Router when you want broad provider access, selected-model metadata, model allowlists, and a configurable cost-quality tradeoff behind one API. Choose Amazon Bedrock Intelligent Prompt Routing when you are inside Bedrock and can accept same-family routing with the current service constraints. Build your own router when task class, data class, tenant policy, eval score, tool permissions, or human approval determine the route.

The article surface is model-stack architecture, but the operational owner is usually the same control layer that owns prompts, evals, traces, and budgets. Routing is not a one-time provider choice. It is a production policy that changes as models, prices, workloads, and incidents change.