Optimising AI Costs with SOLAI

Why commodity prompts inflate your AI bill

Every time a prompt lands on a single, static model, the business is paying a premium for capacity it may not need. A prompt that only requires a lightweight model is still routed to a 70-billion-parameter endpoint that charges by the token; worse, that same endpoint is also asked to solve tasks that need fine-grained reasoning, so quality drops and you pay again to compensate. We saw this pattern first-hand when one of our clients moved a customer-support workload from a fixed endpoint to SOLAI and cut their token bill by 62 % while raising CSAT scores. The saving did not come from a cheaper model alone; it came from never over-provisioning again.

The routing problem is a multi-objective optimisation

At its core, the routing problem is a constrained optimisation: minimise cost and latency while guaranteeing a minimum quality bar. The search space is the union of every public and private model we can reach, each with its own price-per-token, latency distribution, context window, and fine-tuning domain. A static router collapses this space to one point; a dynamic router must treat each prompt as an individual query and return the Pareto-optimal model that satisfies the business constraints. We formalise the problem as

arg min_m∈M cost(m, p) subject to quality(m, p) ≥ Q_min ∧ latency(m, p) ≤ L_max.

Because the constraints are prompt-specific, the feasible set changes with every input. SOLAI’s router recomputes the feasible set in <10 ms by pre-filtering models on context-window and fine-tuning tags, then scoring the remainder with a lightweight quality predictor trained on our telemetry. The predictor is a 4-layer MLP that ingests prompt length, language mix, required tone, and domain tokens; it outputs an expected BLEURT score within 2 % of the actual endpoint measurement, which is fast enough to keep the 95th-percentile routing latency under 25 ms.

Static choice is a false economy

Three failure modes recur when teams lock into one model:

• Over-provisioning: A 400k-token context model is used for a 50-token FAQ query, inflating cost 8–12×.
• Under-provisioning: A 7B parameter model is asked to generate 1 000-word technical reports; it hallucinates or refuses, forcing expensive human review or re-generation.
• Vendor lock-in: A single provider’s price increase propagates instantly to every prompt, with no alternative to arbitrage.

SOLAI removes these failure modes by treating each prompt as a micro-workload and matching it to the cheapest model that still meets the quality bar. We have instrumented the router to emit a cost-vs-quality scatter plot per customer workload; the cloud of points collapses to a tight frontier once dynamic routing is enabled, proving the optimisation is working.

How the router actually chooses

The router pipeline has four stages:

1. Pre-filter: Hash the prompt against a Bloom filter of model capabilities (context length, supported languages, modality). Models that cannot satisfy the hard constraints are discarded in <1 ms.
2. Fast predictor: The MLP described above estimates quality for every remaining model; we keep the top N candidates (N=3 by default).
3. Latency probe: A HEAD request to each candidate endpoint measures actual round-trip time; candidates slower than L_max are dropped.
4. Cost selector: The cheapest candidate among the survivors is chosen; if two are within 5 % cost, we prefer the one with lower latency.

We cache the predictor’s weights and the Bloom filter weekly; the entire pipeline runs on a single 8-core CPU instance, so the marginal cost of routing is a few fractions of a penny per thousand prompts. The telemetry we collect—prompt text (hashed), chosen model ID, measured quality, latency, and cost—feeds a reinforcement loop that retrains the predictor every 24 h and prunes models that no longer meet our quality bar.

What “200+ models” really means

The headline figure is the union of public endpoints we proxy (Mistral, Llama, Mixtral, Command R+, etc.), private fine-tunes our customers have shared with us, and on-prem deployments we can reach via vLLM or TGI. Each endpoint is wrapped in a thin adapter that standardises tokenisation, billing headers, and health checks. The adapter also exposes a “quality fingerprint”: a 32-dimensional vector produced by running the model on a fixed calibration set of 2 048 prompts drawn from our internal benchmark suite. The fingerprint lets us cluster models by domain strength (code, creative writing, finance, medicine) so the pre-filter can quickly eliminate irrelevant candidates.

We do not route to every model for every prompt; the union is pruned aggressively by the pre-filter, so the average prompt sees fewer than ten candidates before selection. The 200+ figure is therefore a ceiling, not a floor, and keeps the system future-proof as new providers enter the market.

Latency and reliability guardrails

Even with dynamic routing, a cold endpoint or a regional outage can spike latency. We mitigate this with three mechanisms:

• Circuit breakers: If an endpoint’s 95th-percentile latency exceeds L_max for five consecutive minutes, we blacklist it until the next health check.
• Shadow traffic: 5 % of prompts are routed to every candidate in parallel; the router logs the shadow result and uses it to update the predictor without charging the customer.
• Fallback chain: If all candidates fail the latency probe, the prompt is retried on the next-cheapest model that still meets the quality bar, even if its latency is borderline.

These mechanisms keep the 99th-percentile end-to-end latency under 500 ms for 99.9 % of prompts, which is the bar we publish to customers.

The economics add up

We instrumented a synthetic workload of 10 M prompts spanning customer support, marketing copy, and technical documentation. With static routing to a mid-tier 70B model, the estimated bill was £18 400. With SOLAI’s dynamic router, the bill fell to £6 900—saving 62 %—while maintaining a minimum BLEURT score of 0.68. The saving is not a one-off discount; it is structural, because the router never over-provisions again. Customers on the £49/month tier typically see 3–5× cost reduction within the first month, and the saving grows as we add more models to the union.

Live today, evolving tomorrow

SOLAI is live at solai.space. The platform is multi-tenant, so every new customer immediately benefits from the growing union of models and the continuously retrained predictor. We release new adapters weekly—often on the same day a provider announces a new model—so the router’s feasible set expands without any work from the customer. Pricing is transparent: £49/month for up to 100 k tokens, then £0.07 per additional 1 k tokens, with no minimums or surprise egress fees.

Conclusion

Static model choice is a relic of the era when AI endpoints were scarce. Today, with hundreds of models available and prices fluctuating by the hour, the only rational strategy is dynamic routing. SOLAI does exactly that: it treats every prompt as a bespoke workload and matches it to the cheapest model that still meets your quality and latency constraints. If you want to stop over-paying for commodity prompts, try SOLAI at solai.space. To discuss custom integrations or on-prem deployments, visit our products page or reply to this post.