ROUTE404: The Economics of Intelligent Inference Routing

Abstract & Protocol Thesis

AI inference is a fragmented allocation problem. Models differ in token prices, capabilities, service availability and response-time distributions. An advertised intelligence benchmark does not establish suitability for every request; a low token price does not establish low cost per usable answer. Efficient routing requires a model of the work being requested and the operational constraints under which it must finish.

ROUTE404 selects an operator-allowlisted model through OpenRouter, dispatches a single user message and returns an inspectable execution trace. Its present implementation uses transparent heuristics, catalog-derived token prices and process-local latency observations. Learned quality prediction, competitive procurement, cryptographic execution proofs and onchain settlement are not implemented.

The Model Selection Problem

The research vector separates demand from supply. Complexity estimates reasoning difficulty, latency tolerance specifies a deadline, and required capabilities determine suitable supply. The production API accepts prompt, mode, optional model and maxTokens. It does not accept complexity, deadlines, spending ceilings or a capability set; AUTO derives one task label from prompt text.

Normalized cost and latency are dimensionless. Fixed reference budgets could make scores comparable across candidate sets; request-specific candidate maxima instead express relative preference. Candidate-dependent normalization can change rankings when alternatives enter or leave. No calibrated quality model or feasibility engine exists in production.

The coefficients imply a strong ordering: every capability-matched model scores at most 0.35, while every mismatched model scores at least 0.65. Therefore, if any eligible model declares the classified capability, AUTO always selects a matched model, regardless of the other candidates’ price or latency. Cost and latency trade off within that capability tier; when no candidate matches, the shared mismatch term does not differentiate them.

Classification first tests /prove|reason|analy[sz]|complex|math/i and returns reasoning; otherwise /code|function|typescript|python|debug|sql/i returns code; otherwise general. Reasoning wins when both expressions match. These are substring tests, not semantic understanding: incidental words, quoted text, negation and multilingual requests may be misclassified.

Current eligibility requires a configured ID in the gateway catalog, finite nonnegative input/output prices and a zero request fee. Configuration requires 2–12 distinct IDs, each with at least one capability from general, code and reasoning; at least two eligible entries must remain. Context limits, credits, endpoint health and actual task support are not independently checked.

AUTO minimizes its fixed score. CHEAP minimizes estimated cost directly. FAST minimizes measured EWMA latency, treating unmeasured entries as positive infinity. ADVANCED performs exact ID lookup. These are separate policies, not a shared adjustable-weight solver. Equal primary scores fall back to lower estimated cost; remaining ties preserve configuration order through stable sorting. FAST with no measurements chooses the lowest estimated cost.

Missing latency is explicit uncertainty, but confidence intervals are absent. Successful requests to selected models generate the observations; unselected and failing models contribute no samples. This selection bias can favor previously measured candidates. A deterministic ranking is not a probabilistic utility estimate.

Inference Economics

Input tokens represent supplied context; output tokens represent generated content. Their prices can differ substantially, making a single headline rate an incomplete basis for comparison. With explicitly illustrative prices of $0.40 per million input and $1.60 per million output tokens, 1,200 input and 800 output tokens cost $0.00048 + $0.00128 = $0.00176. This is arithmetic, not a provider quote or completed execution.

maxTokens is a ceiling, not an output forecast. CHEAP uses that ceiling and estimates input from UTF-16 length divided by three; this can alter model rankings. The input estimate is not a tokenizer-based upper bound. Language, code and Unicode content change tokenization and can produce errors in either direction.

The server fetches OpenRouter catalog prices with an eight-second timeout and caches them per process for five minutes. Execution cost multiplies reported prompt_tokens and completion_tokens by selected-model catalog rates. Both usage fields must be nonnegative integers; otherwise both token counts and estimated cost are null. Zero and unknown remain distinct.

This estimate is not invoice reconciliation. It excludes retry-attempt charges, price-tier overrides, cache-specific rates, taxes, ancillary fees and provider adjustments. Nonzero per-request fees cause exclusion from the eligible catalog. Pricing comes from the selected ID even when the provider reports a different resolved model name. The trace has no immutable quote timestamp or downstream endpoint identifier.

Under the restrictive assumptions of independent attempts, constant cost c and constant usable-answer probability s, unbounded retry-until-success costs c/s in expectation. Real failures are correlated, output quality is task-dependent and retries are bounded. With at most two equally priced attempts, simplified expected attempt spend is c[1 + P(retry)], before partial or unreported billing.

Budget-aware routing would reserve spend and reconcile after completion. The 200-request daily cap is a count limit, not a dollar ceiling or token ledger. Realized savings require comparable quality, success and latency constraints plus a measured baseline. ROUTE404 does not claim a measured savings percentage.

Latency Modeling

Percentiles require a sample count, window and sampling rule. P99 from a small sample is unstable. Successful-only samples omit timeouts; reliable experiments disclose failures and censored requests rather than removing them from the performance story.

The timer begins after catalog resolution and model selection, immediately before the completion loop. It stops after parsing and validating successful response content and is rounded to integer milliseconds. It includes any allowed retry and its 300 ms pause. It excludes browser upload/delivery, limiter lookup, catalog fetch and server work before dispatch. User-perceived latency may therefore exceed latencyMs.

Each completion fetch has a 20-second timeout. Only an explicit HTTP 429 or 503 permits one retry, for at most two attempts. Ambiguous network failures are not retried. There is no cross-model fallback; OpenRouter receives provider.allow_fallbacks=false. The route declares maxDuration=60 and the UI aborts its fetch after 55 seconds. These separate limits are not a guaranteed end-to-end deadline.

Observations are a module-level Map, not globally synchronized storage. Cold starts, deployments, regions and independently scaled instances have different or empty histories. Failed requests do not update the EWMA. FAST cold starts use cost; AUTO assigns unknown latency a penalty without inventing milliseconds.

Queueing makes response time depend on load, concurrency and service-time variance. EWMA assumes recent successful durations remain informative; demand shifts, longer prompts, output variance and provider queues can violate that assumption. No queue depth, workload-conditioned latency, provider availability probe or separate network/service-time estimate is implemented.

Routing Policies & Decision Theory

Assume a code task, 3,000 UTF-16 code units, estimated 1,000 input tokens and a 500-token output budget. All candidates are catalog-eligible. A declares only general; B and C declare code. Rates are USD per million tokens. Durations are hypothetical existing EWMA observations.

AUTO chooses B; CHEAP chooses A; FAST chooses C; ADVANCED chooses the explicitly requested eligible ID. A pays the capability-mismatch penalty in AUTO. C is fastest but its higher estimated price outweighs that speed advantage under AUTO’s coefficients. Manual selection intentionally does not apply automatic ranking.

This result does not establish that B will answer correctly. If A rarely produces usable code, its cost per usable completion can exceed B’s. Conversely, overly broad capability labels erase useful differentiation. Decision-theoretic quality requires a task-relevant loss function and evaluated outcomes; keyword labels are a transparent approximation.

Budgeted Bayesian selection, contextual bandits and constrained regret minimization are possible extensions. They require exploration controls, spending limits, reproducible evaluations and a way to distinguish transient service failures from persistent model-task mismatch. No learning loop runs in the deployed router.

Privacy-Aware Inference

User browser → Next.js server router → OpenRouter gateway → downstream model provider
User browser ← Next.js server router ← OpenRouter gateway ← model response
Wallet / chain RPC: separate optional browser-side interaction; no prompt payload

The application server, gateway and downstream inference service are separate trust boundaries. The router processes prompt content and forwards it to OpenRouter; the downstream provider receives content needed for execution. HTTPS protects transport where supported by deployed endpoints. It does not hide plaintext from processors or establish end-to-end encryption between the user and a model.

OPENROUTER_API_KEY is read exclusively by server code. The browser sends no provider credential. Server secrets belong in protected environment variables; NEXT_PUBLIC_ configuration is public and must never contain private provider keys. A browser-facing chain RPC URL may contain a public-facing provider identifier, so its restrictions must match that exposure.

Application code does not log raw prompts/responses, send them to analytics or persist prompt history in browser storage. Up to 20 successful response records remain in React state in the current tab. Clear Session removes these records, the input and displayed error. Reload discards that application history. JSON export contains traces only, not prompts or response text. Documentation code copying is user-triggered and uses the operating-system clipboard.

These controls do not erase external records. Hosting infrastructure may retain IPs, timing and request metadata. Providers apply their own retention policies. Browser extensions, developer tools, screenshots, clipboard history and downloaded exports create additional disclosure paths. Clearing React state is not secure memory erasure or deletion of provider/host logs.

The production limiter derives a keyed SHA-256 HMAC from ingress IP and RATE_LIMIT_SALT and stores counters under that pseudonymous key. HMAC is not anonymization: the operator holds the key and infrastructure observes originating requests. Counters correlate activity during a salt’s lifetime; NAT can combine different people behind one identity.

Inference has no wallet-address field and requires no connection. Application code does not forward wallet identity in the model request. Identifying prompt content and network metadata can nevertheless correlate wallet and service activity. No complete anonymity, zero-knowledge computation, trusted execution environment or confidential model execution is claimed.

Execution Trace & Observability

Auditing requires separating router decisions, provider-reported metadata and local measurements. The API emits content plus trace only after usable success. Failures return an error envelope and do not create a success trace or update observed latency. JSON export contains only traces currently retained in tab memory.

{
  "access": "HOLDER",
  "content": "Illustrative response: routing trades capability fit against estimated cost and observed latency.",
  "trace": {
    "requestId": "a69e1760-581d-49e0-bb63-f8d55d649937",
    "providerRequestId": "illustrative-provider-id",
    "model": "openai/gpt-4.1-mini",
    "selectedModel": "openai/gpt-4.1-mini",
    "mode": "AUTO",
    "reason": "Task: general; capability 65%, estimated cost 25%, observed latency 10%",
    "latencyMs": 840,
    "inputTokens": 120,
    "outputTokens": 80,
    "estimatedCostUsd": 0.000176,
    "pricing": {
      "inputPerToken": 4e-7,
      "outputPerToken": 0.0000016,
      "source": "https://openrouter.ai/api/v1/models"
    },
    "provider": "OpenRouter",
    "providerStatus": 200,
    "timestamp": "2026-10-27T23:00:00.000Z"
  }
}

The schema-valid example is a fixture, not a live execution. Its 120 input and 80 output tokens yield 120×0.0000004 + 80×0.0000016 = $0.000176. Its duration and identifiers are invented. The API does not sign records, hash prompts, expose attempt counts, include a pricing observation timestamp or persist an append-only execution ledger.

A trace explains what this server reported selecting and calculating. It cannot prove model execution, answer correctness, unmodified server behavior, absence of third-party retention or a completed payment. Cryptographic attestations and verifiable receipts require independent protocol design and threat analysis.

Web3 & Robinhood Chain Architecture

Wallet-native interfaces can place inference within an ecosystem of accounts, assets and programmable settlement. That context does not require model computation onchain. The deployed inference path is entirely offchain; wallet controls are separate from selection and provider billing.

The wallet control supports an injected browser wallet, connect/disconnect and a request to switch to chain 4663 when necessary. Unavailable/rejected wallet requests display a safe error. There is no WalletConnect relay, backend signature challenge, account authorization or balance-based inference gate. A wallet connection is not authentication. Real wallet approval and network switching remain unverified in the release audit.

Future accounting could meter usage offchain and settle authorized balances onchain. It would require replay-resistant authorization, spend reservation, reconciliation, refunds and failure semantics. Even receipt hashes can reveal correlations or enable dictionary attacks on predictable content; hashing a prompt is not a privacy proof. No receipt is written today.

Robinhood documentation distinguishes mainnet 4663 from testnet 46630. This application targets mainnet. The official public RPC is rate-limited; NEXT_PUBLIC_ROBINHOOD_RPC_URL can select a production provider appropriate to operational requirements.

Reliability & Security

POST processing checks browser origin, content type and rate limiting before body/schema validation. If Origin exists, it must parse as HTTP/HTTPS with a host matching incoming Host (or internal URL host when Host is absent). Requests without Origin remain allowed for direct API clients. This is a browser policy, not an API credential or protection against scripted requests with constructed headers.

On Vercel, VERCEL=1 requires Redis URL, Redis token and RATE_LIMIT_SALT. Missing configuration throws CONFIGURATION_REQUIRED with HTTP 503. Redis dependency failures also fail closed through the generic 503 path. There is no process-local production fallback. Outside Vercel, local mode shares one identity, a process-local 10/minute counter and a development salt; it is not equivalent multi-instance protection. Other hosts require appropriate ingress identity and production limiter configuration.

Quota evaluation precedes JSON validation and provider-key checks, so malformed or configuration-blocked requests may consume quota. A denied per-IP request short-circuits the global limiter. HTTP 429 always carries Retry-After:60, including daily-cap and provider-limit results; that is an advisory delay, not an exact reset timestamp.

One retry follows explicit provider 429/503 with a fixed 300 ms pause; the provider’s Retry-After is not used. A final provider 429 maps to application 429; other non-success responses map to 502. Unexpected exceptions, network failures and timeouts map to sanitized 503. Invalid provider content maps to 502 PROVIDER_RESPONSE_INVALID. No automatic cross-model recovery is performed.

Inference is public, not token-gated, wallet-authenticated or keyed per user. Count limits and output bounds reduce abuse but do not eliminate distributed attacks, budget exhaustion or billing risk. Host access restrictions and provider spend caps are operational options; this document does not claim those external settings are enabled.

API & Developer Reference

ADVANCED with an absent/unlisted model passes the base object schema but fails selection through generic HTTP 503 once preceding configuration checks pass. It does not currently return a specialized validation 400. Correct request syntax is not proof of configured credentials, funded credit or successful inference.

{
  "prompt": "Explain the tradeoff between inference cost and latency.",
  "mode": "CHEAP",
  "maxTokens": 256
}

type Mode = "AUTO" | "FAST" | "CHEAP" | "ADVANCED";
type RouteInput = { prompt: string; mode: Mode; model?: string; maxTokens?: number };
type Trace = {
  requestId: string; providerRequestId?: string | null;
  model: string; selectedModel: string; mode: Mode; reason: string;
  latencyMs?: number; inputTokens?: number | null; outputTokens?: number | null;
  estimatedCostUsd?: number | null;
  pricing?: { inputPerToken: number; outputPerToken: number; source: string };
  provider: "OpenRouter"; providerStatus: number; timestamp: string;
};
type RouteResult = { content: string; trace: Trace; access: "PUBLIC" | "HOLDER" };
type RouteFailure = { error: string; requestId: string; code?: string };
export async function route(input: RouteInput): Promise<RouteResult> {
  const response = await fetch("/api/route", {
    method: "POST", headers: { "Content-Type": "application/json" },
    body: JSON.stringify(input), signal: AbortSignal.timeout(55_000),
  });
  const body: RouteResult | RouteFailure = await response.json();
  if (!response.ok || "error" in body) {
    const failure = body as RouteFailure;
    throw new Error(failure.error + " [" + failure.requestId + "]");
  }
  return body;
}
// Same-origin application; AUTO requires a verified holder session.
// No provider key or wallet address belongs in this request.
const result = await route({prompt: "Explain inference routing.", mode: "AUTO"});
console.log(result.trace.requestId);

type Catalog = {
  configured: boolean;
  models: Array<{ id: string; capabilities: string[];
    inputPrice: number; outputPrice: number; latency: number | null }>;
  message?: string;
};
export async function listModels(): Promise<Catalog> {
  const response = await fetch("/api/models", {cache: "no-store"});
  if (!response.ok) throw new Error("Catalog request failed");
  return response.json();
}
const catalog = await listModels();
if (catalog.configured && catalog.models.length >= 2) {
  // Use catalog.models[0].id as model in an ADVANCED request.
}

GET /api/models returns {configured, models} on catalog success. Model entries contain id, capabilities, inputPrice, outputPrice and latency. Prices are USD per token; latency is EWMA milliseconds or null. configured tests environment-value presence, not validity, funded credit or provider health. Catalog exceptions still yield HTTP 200 with configured:false, models:[] and a message. Clients must inspect the body.

{
  "error": "Configure shared production rate limiting on the server.",
  "code": "CONFIGURATION_REQUIRED",
  "requestId": "5d6d9a31-992d-49eb-9987-4912cc27ad34"
}

POST sets Cache-Control:no-store and X-Request-ID. HTTP 429 adds Retry-After:60. GET responses also disable browser caching, independently of the five-minute server catalog cache. Configuration failures can precede malformed-input errors because quota checks run before body parsing.

The audited release configured openai/gpt-4.1-mini with general/code and anthropic/claude-sonnet-5.5 with general/code/reasoning. These are catalog IDs and operator declarations, not measured performance or guarantees of credit. Retrieve /api/models for current eligibility. Never put provider or Redis secrets in public prefixes, examples or Git history.

Limitations & Research Directions

Quality is the central measurement gap. Capabilities are operator declarations, classification is lexical, and no evaluator or ground-truth success signal calibrates decisions. Credible quality estimation requires task-stratified evaluation, uncertainty reporting, disagreement analysis and distribution-shift monitoring.

A single gateway creates correlated dependency risk. Cached prices can diverge from charges; endpoint rates can differ from headline catalog prices. Future accounting should reconcile billed usage, aliases and price schedules without retaining sensitive content unnecessarily.

Latency samples are sparse, selected, successful-only and process-local. They cannot substantiate global percentiles or causal model comparisons. Research instrumentation should disclose sample counts, windows, failures and censoring, with a retention policy and privacy review before cross-request telemetry is enabled.

Potential extensions include token-aware estimates, task-conditioned latency regression, bounded exploration, durable observations, spend reservations, health gates and calibrated loss predictions. Each adds failure modes: exploration spends money, telemetry reveals behavior, evaluators can be gamed and dynamic routing can complicate reproducibility.

Privacy-aware eligibility would require a verified data-processing policy registry and enforceable operational commitments. Confidential hardware, encrypted computation, zero-knowledge proofs and onchain receipts are distinct mechanisms with different trust assumptions. None is implemented here.

Conclusion

Inference routing is an allocation discipline: specify the workload, declare available supply, measure the operational boundary and expose selection assumptions. ROUTE404 currently provides a small, inspectable implementation of that discipline, not a guarantee of optimal allocation under unknown quality or unobserved latency.

An objective becomes operationally meaningful only when its variables are trustworthy and its constraints are enforced. Transparent heuristics are a defensible starting point when limitations are explicit. Empirical quality, reliable accounting and privacy-aware controls are necessary to advance beyond that starting point.

The future of inference is not defined by access to a single model, but by the efficiency with which intelligence is allocated.