Scaling From a Long-Running Simple Solution

When growth forces a system that has quietly worked for years to change, the safe order is observability first, a second environment second, testing third, and formal governance last — because each stage is what makes the next one possible without a risky rewrite.

Added 3 Sep 2026 12 min read Updated 3 Sep 2026

#scaling #observability #technical-debt #production-systems #environments #testing-strategy #governance #legacy-modernization

Learn this your way

Read Guided course

A script, a single Lambda function, a direct call to a hosted model — something simple, built by one or two people, running in the one environment that has always just been “production.” No CI/CD beyond a manual deploy. No formal test suite beyond eyeballing the output. No governance process, because there was never a second person to govern. And it has worked, quietly, for months or years.

That is a real result. Most software projects never reach “worked in production for years.” It did not happen by accident: the system was small enough that one person could hold its entire behavior in their head, and that is a legitimate way to build something that lasts. The problem is not that this was done wrong. The problem is that every risk which felt safe to defer at that size — no rollback path, no way to test a change before real users see it, no record of what the system is allowed to do with people’s data — was deferred, not eliminated. Growth is what presents the bill: more users, more request volume, more data, more business exposure, all landing on infrastructure and habits that were sized for something much smaller.

The question this guide answers is not whether to add rigor. It’s what to add first, second, and third, given that engineering time is limited and the system cannot simply be taken down while it’s rebuilt. Get the order wrong and you either freeze feature work for months trying to do everything at once, or you add the wrong thing first — a compliance framework for a system you have no way to verify actually behaves the way the framework claims, for instance — and end up with paperwork instead of reduced risk.

This is a retrofit, not a rebuild

This wiki already covers how to build an AI product from nothing: From Zero to Production walks the demo-to-MVP-to-ecosystem progression, and Localhost to Production: Deployment Stages explains why you need separate environments before you ever ship. Those guides assume you are choosing the order things get built in. The Production Readiness Checklist assumes something similar: that you’re working toward a launch and verifying you’re ready for it.

None of that is this situation. You already launched — probably more than once, via whatever manual process has always worked. Real users depend on the system right now, today, while you read this. The constraint isn’t “what order should I build things in,” it’s “how do I introduce safety into something that already works, without breaking the thing that works, and without the luxury of stopping to do it properly.” That constraint is what determines the order argued for below — it is not the same order you’d choose starting from a blank repository.

What growth actually breaks first

Before the sequencing argument, it’s worth being concrete about what “growth is forcing the issue” actually looks like in a simple system. It is rarely a clean, gradual slowdown. It’s usually one of a small number of specific mechanisms hitting a hard limit at once.

Symptom under load	What’s actually failing	Cheap first mitigation
Requests time out in bursts, not steadily	A database connection pool sized for the old traffic level is exhausted; new requests queue behind connections nothing is releasing	Raise the pool size, add a request timeout so failures fail fast instead of piling up behind each other
One slow dependency drags down the whole app	A synchronous request path with no queue means every request thread blocks on a downstream call — a rate-limited third-party API, a slow model endpoint — so one degraded dependency throttles everything behind it	Move the slow call off the request path into a job queue , or wrap it in a circuit breaker so a failing dependency degrades instead of cascading
Memory climbs until a restart is required	An in-memory cache that was harmless at low traffic has no eviction policy or size bound, and grows with request diversity	Bound it with an eviction policy and a max size; move it out-of-process if state needs to survive a restart
A fix that worked locally breaks in production, unpredictably	There is nowhere to run the change under production-like conditions before real users see it	This is the second-environment problem — see below

These are the mechanics of scaling itself: connection limits, request concurrency, cache growth, serving capacity. Scaling AI Infrastructure covers the infrastructure side in depth — vertical and horizontal scaling of serving instances, batching, autoscaling policies, and cost control at scale. That guide assumes you can already see what’s breaking and have somewhere to test a fix. The rest of this page is about getting to that position first.

The debt already on the books

A system that has run a long time under a single-environment, single-maintainer model has accumulated technical debt — just not always the kind that shows up in a code review. Managing Technical Debt in ML Systems covers data debt, pipeline debt, and configuration debt in depth; the specific shape that matters here is operational debt: config baked into code instead of separated from it, a deploy process only one person remembers, no record of what a production incident looked like the last three times it happened.

Not all of it needs paying down before you start. Some of it is the work of the stages below, not extra work competing with them:

Debt	When to address it	Why
Config and secrets hardcoded in application code	Now, as part of building the second environment	You cannot stand up a second environment without separating config from code — this isn’t preparatory work, it is the environment work. See Twelve-Factor AI on config as a first-class, environment-specific concern
Ad hoc `print`/console logging instead of structured logs	Now, as part of adding observability	Same logic: this is most of what “adding observability” means in a system that has none
An undocumented, single-person deploy process	Now, alongside the second environment	Having two places to deploy to is what forces the process to get written down
Data with no schema versioning	Only once a second environment needs its own dataset	Deferrable if environments can share a masked copy of production data; becomes urgent once you need real test data management
A monolithic codebase with no internal seams	Tolerate for now	Refactoring for testability is real work — let Testing Strategy by Maturity Stage tell you which seams the first tests actually need, rather than refactoring speculatively ahead of that

The pattern: debt that blocks the next stage gets paid down as part of reaching that stage. Debt that doesn’t block anything yet gets tolerated and revisited later, with a specific trigger for revisiting it rather than an open-ended promise to “clean it up eventually.”

The order, and why it’s this order

Each stage is what makes the next one safe to attempt — not an arbitrary checklist order.

The retrofit order of operations: observability first because you cannot safely change what you cannot see, a second environment second because it is what makes every later change testable at all, automated testing third because it now has somewhere safe to run, and governance formalization last, sized to actual scale and exposure rather than added preemptively.

1. Observability first, because you cannot safely change what you cannot see

This is the cheapest, lowest-risk thing you can add to a system that already works. It is purely additive — instrumentation does not change what the system does, only what you know about what it’s doing — and it is the one prerequisite every later stage depends on. You cannot tell whether a second environment behaves like production without a baseline to compare it against. You cannot tell whether a new test caught a real regression without knowing what “normal” looks like. Google’s SRE practice puts this plainly: monitoring is what lets you tell whether a system is working, and what it’s doing, before you can reason about changing it at all [1].

Concretely, this means structured logging on the request path, a small number of metrics that matter (request rate, error rate, latency, and for AI-specific paths, token cost and model latency), and alerts on the few that would actually wake someone up. Full-Stack Observability for AI Systems covers the complete stack — traces, evaluations, metrics, alerting — in depth; you do not need all of it on day one, but that guide is where to go once the basics are in.

Be ready for what you find. A system that has run for years with no monitoring has often been quietly degrading in ways nobody noticed — this is exactly what Detecting and Handling Model Drift covers, and it is common, not a sign anything was done wrong. If observability surfaces a real problem, the Incident Response Playbook covers how to triage and respond to it without turning the first real signal you’ve ever had into a crisis.

2. A second environment second, because it’s what makes every later change safe to try

Once you can see what the system is doing, the next constraint is that you have nowhere to try anything without it happening to real users. Every subsequent stage — a test suite, a governance control, even an ordinary bug fix — needs somewhere to run before it reaches production. Without that, “testing” a change means shipping it and watching, which is not testing, it’s just watching things happen to your users in real time.

Introducing Environments to a Single-Environment System covers exactly how to do this: what a minimum viable second environment needs to include, what to duplicate versus share with production, and how to handle data without either exposing real user data or testing against something so unrealistic it proves nothing. It does not need to be a full staging replica on day one — it needs to be good enough to catch the failure patterns in the table above. Once you have more than one environment, Working with Multiple Environments is where you learn to run them well — build once and promote the same artifact, configure per environment instead of rebuilding, tag resources so the bill and the dashboards make sense. That guide assumes the environments already exist; use it after this stage, not instead of it.

3. Testing third, once there’s somewhere safe to run it

This wiki has extensive depth on AI testing — eighteen separate guides covering everything from unit tests to chaos testing to RAG-specific evaluation. The instinct to reach for that depth first, before observability or a second environment, is understandable but backwards: a test suite needs somewhere to execute that isn’t production, and it needs a baseline (from stage 1) to know whether a failure is the test catching something real or the test itself being wrong.

Testing Strategy by Maturity Stage is the routing guide for this: given where your system actually is, which of the eighteen testing guides to reach for first, second, and third, rather than trying to adopt all of them simultaneously. For AI-specific systems, Testing Non-Deterministic Systems covers how “passing” has to be redefined when the same input can legitimately produce different outputs — the maturity-stage guide will route you there when it’s relevant to your system.

4. Governance last, sized to actual exposure — not skipped, and not first

Governance and compliance formalization comes last in this order, and that placement is easy to misread as “least important.” It isn’t. It’s sequenced last because a risk assessment, a model card, or an audit trail is only as trustworthy as the observability and tests that back its claims. A governance document written before you can verify what the system actually does is paperwork, not risk reduction — it asserts controls you have no way to confirm are enforced.

Governance Thresholds as You Scale covers exactly when each obligation actually kicks in — by user count, data category, jurisdiction, and risk classification — so you formalize because a real threshold was crossed, not preemptively and not so late you’re documenting after an incident or a regulator’s question rather than before it. Once a threshold is crossed, AI Risk Assessment Guide is typically the first concrete artifact to produce.

This is also where the order can legitimately bend, and it’s worth being honest about that rather than treating the sequence as absolute. If the “business exposure” forcing this whole retrofit is itself the legal trigger — you started processing health data, a new customer contract requires a security audit, you crossed into a regulated category — that specific obligation can jump the queue. The governance-thresholds guide is where that judgment call gets made explicitly, not this one.

Sequencing this with limited engineering time

None of the four stages needs to be finished before the next one starts — they need to be far enough along that the next stage’s first real step is possible. You don’t need full observability coverage before starting on a second environment, just enough to tell whether the new environment behaves like production. You don’t need a fully mirrored second environment before writing the first test, just enough that the test’s result means something.

In practice this argues for a fixed, ongoing allocation rather than a stop-everything project: the same logic Managing Technical Debt in ML Systems recommends for debt cleanup generally — a consistent slice of engineering time each sprint, rather than a big-bang effort that competes with feature work for approval and loses. Spend the first one to two weeks almost entirely on observability, because it’s cheap and because everything after it is more valuable with a baseline in place. After that, let the stages overlap: environment work can start once observability is minimally useful, testing can start on the first environment-safe path while the rest of the environment work continues, and governance conversations can begin in parallel once there’s something real to point at, even before every threshold is formally crossed.

Sources

Beyer, B., Jones, C., Petoff, J., and Murphy, N.R. (eds.), Site Reliability Engineering: How Google Runs Production Systems, “Monitoring Distributed Systems”: https://sre.google/sre-book/monitoring-distributed-systems/
Sculley, D., et al., “Hidden Technical Debt in Machine Learning Systems,” Advances in Neural Information Processing Systems 28 (NeurIPS 2015): https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems