Digital Engineering/AI Product Assurance/Production Readiness & Hardening
From working software to software you can operate
Production readiness is the distance between software that runs and software that keeps running. We assess what you have, then build what is missing: environments, deployment and rollback, searchable monitoring, alerts worth waking for, tested backups and an incident process three people can follow.
The build finished. The operations never started.
There is a moment, a few weeks after the first real customers arrive, when a founder realises the product is no longer a project. It is a service. Somebody depends on it being there on a Tuesday morning, and that somebody will not be sympathetic about how quickly it was built. What we find at that moment is consistent.
There is one environment, and customers are in it.
Changes are tested by shipping them. There is no staging system, or one that drifted so far from production that testing against it proves nothing.
Configuration lives in people’s heads and people’s terminals.
Environment variables were typed into a hosting dashboard by hand, once, and nobody recorded what they were for. A shared document holds the database password, the payment key and a token rotated in March but not there.
Deployment is a person.
One laptop, one terminal, one set of muscle memory. No record of which version is live, no way to see what changed between Thursday and Friday, and no rollback beyond redeploying an older commit and hoping the database still agrees. Migrations run forwards; nothing runs backwards.
You learn about failures from customers.
Logs are written somewhere. They cannot be searched across a time range, correlated to a request, or retained past a few hours. Errors are collected nowhere, and nothing is configured to alert anybody.
Backups are assumed.
The provider takes snapshots. Nobody has restored one, so nobody knows how long it takes or what it contains.
None of this is negligence. It is work that was never scheduled, because building the product was the only thing on the list.
What production readiness actually means
Readiness is not a certificate and it is not a feeling of confidence. It is a set of specific properties your system either has or does not have, each demonstrable rather than asserted.
You can change the system safely.
A second environment resembles production closely enough that a test there means something, a deployment path anybody can run, and a rollback performed deliberately once rather than in a panic.
You know what is happening inside it.
Logs are structured, centralised and searchable across a useful retention window. Errors carry the context needed to reproduce them. Where a request crosses several services, you can follow it. Request rate, error rate, latency, queue depth and saturation are recorded continuously, not reconstructed afterwards.
You are told before your customers tell you.
This is the principle we care most about. Monitoring exists so that the first person who knows about a problem works for you. If your fastest detection route is a support email, the rest of your operations is decoration.
You can get your data back.
Backups run on a schedule you chose deliberately, stored where a compromise of your main account cannot reach them, and a restore has been performed this quarter by someone who timed it.
You know what to do when it breaks.
A written runbook, a severity scale, an agreed answer to who declares an incident, and a review afterwards examining the system rather than the person.
It degrades rather than collapses.
Health checks are honest. Every outbound call carries a timeout. A slow dependency slows one feature instead of exhausting the connection pool and taking the product with it.
Readiness sits within two dimensions of The Noseberry Product Assurance Framework: Cloud & Delivery and Engineering Governance, with Scalability & Performance contributing wherever capacity is the constraint.
Read the assessment methodologyWho this is for
Founders whose product has become somebody’s dependency.
You have paying customers, you are the entire operations function, and you cannot be away from a signal for a few hours without a low-level anxiety about it.
Teams onboarding their first customer with a contract attached.
The terms mention availability, support response times or data retention, and somebody needs to sign that page honestly.
Small teams that inherited an AI-built product.
The application code is understandable enough. What is missing is everything around it: environments, pipelines, telemetry, recovery. None of it was anybody’s job.
Teams about to hire their first engineer.
You want the new person to deploy in week one and to be woken in month two without that being a cruelty.
Teams whose cloud bill has outgrown their usage.
Cost is a readiness dimension, and a bill growing faster than customers usually means something is operationally wrong rather than expensive.
When people call us
The enquiry almost always follows an event, and the event is usually one of five.
There was an outage.
Not a blip. A real one, measured in hours, with customers emailing and a founder guessing at causes with no telemetry to guess from. It came back, nobody is confident why, and the same conditions could recur tomorrow. This call usually arrives within days.
A contract arrived with an SLA in it.
A customer wants availability commitments, a support response window and an escalation path. Signing that as things stand would be a promise made on hope, and the deadline is real.
The first enterprise customer is being onboarded.
Their vendor-risk process asks about disaster recovery, change management, monitoring and incident handling. The questions are ordinary. The answers do not exist in a form anybody can send.
A first engineer is arriving.
Handing someone a codebase is straightforward. Handing them a production system with no runbook, no staging environment and no deployment process is how a good hire becomes an unhappy one by month three.
The founder wants a holiday.
This sounds lighter than the others and is not. It is the clearest statement of the problem: the product cannot currently be operated by anyone else, so the business has a single point of failure who also needs to sleep.
Reliability is a practice, not a purchase
Almost every guide to production readiness was written by people operating at a scale you do not have. It assumes a platform team, an SRE function, an error budget policy and a tooling budget with a comma in it. Imported into a company of three, that advice either stalls, because nobody has a free fortnight to start it, or succeeds and then quietly eats the engineering capacity you were counting on to build the product.
The mistake is not ambition. Hyperscaler practice was designed to solve a coordination problem between forty engineers who cannot all hold the system in their heads. You do not have a coordination problem. You have a continuity problem: a few people carrying a large amount of undocumented context, with no defined route back when something goes wrong.
So the right question is not “what does a mature engineering organisation do”. It is: what would we need if the person who built this were unreachable for four days?
Answered honestly, that question produces a short list. A second environment that genuinely resembles production. A deployment anybody can trigger and a rollback somebody has actually used. One searchable place where errors land, retained long enough to investigate last week’s complaint. Five to eight alerts the team believes. A restore performed this quarter by someone who wrote down how long it took. A runbook covering the six things most likely to break.
It also produces a list of things to leave out, and that matters just as much. Container orchestration before there is anything to orchestrate. Multi-region before a single region is dependable. Distributed tracing across three services sharing one database. A formal on-call rota with escalation tiers when the whole company would fit in a taxi. Availability targets to several decimal places that nobody is measuring.
Every operational control carries a running cost, and the cost is rarely the invoice. It is attention, which is the scarcest resource a small team has. An alert that fires falsely twice a week is worse than no alert, because it teaches people to ignore the channel where the real one will appear.
Half of this work is choosing what to leave out, and writing down why, so the next engineer does not reintroduce it out of habit. Reliability is a practice you keep, not a state you buy.
How the engagement runs
This is the one service in the cluster where we do the work rather than describe it. The assessment is separable, but most clients continue into hardening.
Scope and access
day 0
A call covering what the product does, who depends on it, what has already failed and what commercial pressure sits behind the request. We agree the environments in scope. NDA first, access second.
Readiness assessment
days 1–5
We review environments and configuration handling, infrastructure definition, the deployment and release path, migration practice, telemetry, alerting, backup configuration and cost behaviour. Where it is safe, we test rather than read: we attempt a restore into an isolated environment, trace a request end to end, and check that an alert fires when its condition is true.
The readiness plan
days 5–7
Findings classified with an availability lens, a target state right-sized to your team, and a sequenced plan. Sequencing is deliberate: recovery first, because it limits the damage of everything else; then visibility, because you cannot fix what you cannot see; then prevention. You receive this whether or not we go further. Where the application code itself needs remediation before hardening can hold, that work runs as a Vibe Code Cleanup alongside.
Hardening execution
typically 3–6 weeks
We build it. Environments and configuration management, a deployment pipeline with a tested rollback, reversible migrations, structured logging and error tracking, a small set of symptom-based alerts, backup schedules with a restore drill you take part in, health checks, timeouts and back-pressure, and the incident process. Work lands as pull requests reviewed by your team. The product stays live and feature delivery continues alongside.
Drills and handover
final week
We run a restore drill and a simulated incident with your team at the keyboard. We hand over the operational documentation, walk it line by line, and revoke our own access. A review thirty days later checks what fired, what was ignored and what needs tuning.
The production readiness checklist
The assessment, as statements either true of your system or not. The full version, with the evidence behind each, is available on request.
Environments and configuration
- A non-production environment resembles production closely enough for a test there to predict live behaviour.
- Configuration is defined in version control, and no secret sits in a document, chat message or repository.
Deployment and rollback
- Any team member can deploy without a particular laptop, and every release records what changed and who released it.
- A rollback path exists and has been used deliberately once, including for migrations that cannot be reversed.
Observability
- Logs are structured, centralised and searchable across a window long enough to investigate last week’s complaint.
- Errors carry enough context to reproduce them, and key service measurements are recorded continuously.
Alerting and on-call
- Every alert that fires is one a human can act on, and the team knows who looks first this week.
- Alerts describe symptoms a customer would notice rather than internal causes, and none fires routinely and gets ignored.
Backup and restore
- Backups sit where a compromise of the primary account cannot delete them, on a schedule chosen against a stated tolerance for data loss.
- A restore has been performed and timed in the last three months, by someone other than whoever configured it.
Incident response
- A written runbook covers the likeliest failures, and a severity scale defines who is woken and when.
- Incidents get a blameless review producing a few tracked actions rather than a document nobody reopens.
Resilience and cost
- Health checks reflect real dependency state, outbound calls carry timeouts, and queues apply back-pressure rather than filling silently.
- Capacity headroom is known against measured load, and spend is attributable to features, so a rising bill is explainable.
See where you stand
Request the full production readiness checklist, with the evidence we look for behind each statement, and use it to mark your own system before we speak.
What you receive
A readiness assessment report
covering environments, delivery, observability, alerting, recovery, incident practice, resilience and operational cost, with a status for each and the evidence behind it.
A findings register
in which every item is classified Critical, High, Medium or Optimisation, with the availability or data consequence stated plainly and the effort to close it.
The 3am Handbook
The named operational deliverable of this engagement, written for a tired engineer with no context. How the system is composed, what depends on what, how to deploy and roll back, how to read the dashboards, what each alert means and what to do about it, how to restore the database, who to contact at every third-party provider, and the six likeliest failures with diagnosis steps.
A working deployment pipeline
with environment separation, managed configuration, a tested rollback and a release record.
Instrumentation in place
Structured logging, centralised search, error tracking, service dashboards, and tracing where the architecture makes it worth the cost.
An alert set you trust
Deliberately small, tuned against your real traffic rather than shipped at defaults.
A verified restore
With the elapsed time recorded and the procedure documented, plus recovery objectives in terms a founder can decide on: how much data you can afford to lose, and how long you can afford to be down.
An incident process
covering severity levels, declaration, communication and blameless review, sized for the people you have.
How findings are classified
Findings carry the same four classes used across the cluster, read through an availability and recoverability lens.
Classification
What it means
Expected response
Data could be lost unrecoverably, or an ordinary failure takes the product down with no route back.
Close before the next release.
A failure you should expect within months, such as a bad deploy, a dependency outage or a traffic spike, causes a long or improvised recovery.
Close inside the engagement.
Recovery is possible but slow, manual, or dependent on one person.
Planned work, sequenced after the above.
Operational cost, noise reduction or convenience, no availability consequence.
Do it when the area is open anyway.
The test for a Critical finding is not aesthetic. If this happened at three on a Saturday morning, could you get back to a known good state without the person who built it? Where the answer is no, the finding is Critical however unlikely the trigger.
Working inside your infrastructure
An audit reads. This engagement changes things, and that needs more access than any other service in the cluster. How it is bounded is part of the offer.
What we need
Repository access with the ability to open pull requests, and cloud or platform access with permission to create the resources named in the agreed plan. Where infrastructure is defined in code for the first time, that includes provisioning rights in the accounts concerned.
What we do not take
Standing administrative access to production. Elevated permissions are requested for a stated task, granted for a defined window, and expire.
Named engineers only.
Individual accounts under your identity provider, never a shared credential, never an account that outlives the engagement. You keep the audit log, and everything we did is in it.
Everything lands as a reviewable change.
Infrastructure and pipeline work arrives as pull requests, in increments small enough for your team to read. Nothing is configured in a console without a matching definition in code, because undocumented console changes are what we are here to remove.
Customer data stays where it is.
Restore drills run into an isolated environment, using masked or synthetic data unless you direct otherwise in writing.
Handover closes it.
Our accounts are removed and any credential we touched is rotated by you.
What changes afterwards
You find out first.
The most immediate difference clients report is how they hear about problems. A message arrives from monitoring before impact is widespread, and the support email that used to be the detection mechanism becomes a confirmation.
Deploying stops being an event.
When a release can be rolled back in a minute by anyone on the team, the weight comes off shipping. Teams releasing fortnightly because releases were frightening tend to move to several times a week, a velocity gain bought with operations work.
You can answer the vendor questionnaire.
Disaster recovery, change management, monitoring and incident handling become documented facts with dates attached rather than intentions.
Somebody else can hold it.
A new engineer can deploy in their first week and take first response in their second, because the handbook tells them what to do instead of requiring them to interrupt the founder.
The bill becomes explainable.
Cost attribution usually surfaces a resource sized for a load that never arrived.
You can be unreachable.
The point at which a founder can be on a plane for nine hours without checking anything is the milestone most people are buying.
Why Noseberry
We operate systems, we do not only review them.
The engineers on this engagement have carried pagers and run recoveries under real pressure. That is why the alert set we leave behind is small, and why the runbook is written for someone tired rather than someone curious.
We size the work to your team, not to a reference architecture.
The plan for a company of three looks different from one for a company of forty, and we will argue against controls you do not need even when you have asked for them by name.
Assessment and execution in one engagement.
You are not handed a list of gaps and left to find someone to close them. The people who identified the finding implement the fix, which removes a round of context transfer and the misunderstandings it brings.
Your team holds the keyboard for the drills.
A restore only we can perform has not solved your problem.
One framework across the cluster. Findings from an AI Code Audit or an Architecture & Scalability Review carry into this work without re-diagnosis, and a reassessment later measures the same things.
Frequently asked questions
First, get visibility, because you are currently guessing. Add error tracking and centralised logging today; on a small application that is an afternoon’s work and it usually identifies the pattern within a day of real traffic. Second, check the boring causes before the interesting ones: memory limits, connection pool exhaustion, a missing timeout on an outbound call, an unindexed query under load. Third, make sure you can roll back, so the next occurrence is a two-minute recovery rather than debugging in front of customers.
Almost certainly not. Kubernetes solves the problem of scheduling many services across many machines, for a team large enough to operate the cluster. With one application, a database and three engineers it adds an operational surface that consumes more attention than it returns. A managed application platform, defined in code, with health checks and a tested rollback, serves you better and is far easier to hand to a new hire. If you outgrow it later, the readiness work carries over.
Not a number you have not measured. We do not promise availability outcomes, and we would be cautious of any supplier who does, because availability depends on your architecture, your dependencies and your own operational decisions long after we leave. What we do is help you measure current availability honestly over a meaningful period, identify what limits it, and support you in framing terms you can meet. In early contracts, response and communication commitments are usually more defensible than a percentage nobody monitors.
The assessment runs about a week. Execution is typically three to six weeks for a single product on a managed platform, longer where infrastructure is being defined in code for the first time or migration practice needs rebuilding. We sequence so that value lands early: recovery capability and monitoring are usually in place inside the first fortnight, which shortens the highest-risk period well before the engagement finishes. Feature delivery continues alongside the work.
Quoted rather than published, because the range is genuinely wide. A product on a managed platform with reasonable foundations needs a fraction of the effort of one with a single hand-configured environment, no infrastructure definition and irreversible migrations. After a scoping call you get a fixed price for the assessment and a banded estimate for execution, converted to a fixed price once the scope is known. You are never asked to commit to execution before you have seen the findings.
For this engagement, yes, and more than for an audit, because we are changing how production is built and operated. That access is bounded: named individual accounts under your identity provider, elevated permissions requested for a stated task and expiring afterwards, no standing administrative rights, and every change landing as a reviewable pull request rather than an untracked console edit. Restore drills use isolated environments and masked data. On handover our accounts are removed and credentials rotated by your team.
By restoring one and timing it. An untested backup is not a backup; it is an assumption with a monthly invoice attached. We have found snapshots that excluded the schema, retention windows shorter than the client believed, backups held in the same account as the primary database, and restore credentials that had expired unnoticed. During the engagement we perform a restore into an isolated environment, record how long it takes and document the procedure. Then your team performs one.
Less than a self-hosted estate, and more than most people assume. A managed platform handles machines, patching and often database snapshots. It does not give you a second environment, a tested rollback, reversible migrations, searchable logs beyond a short retention window, meaningful alerting, an incident process or operational documentation. It also will not tell you when your application is failing while returning a valid response. Managed platforms remove infrastructure work, not operational practice, and that is where outages come from.
You are, and that is the point. We are not a managed service and this engagement does not end with us holding your pager. What it produces is a system a normal person on your team can hold: a small alert set that fires for real conditions, a runbook covering the likely failures, a rollback that works and a restore that has been rehearsed. Where a team genuinely needs external cover, we will help you specify it.
Not necessarily. If your concern is operational, such as outages, deployment, recovery or an SLA on the desk, start here. If the application code itself is suspect, or nobody has ever assessed it, the AI Code Audit is the better first step and its findings feed directly into this work. Where load is the problem rather than reliability, the Architecture & Scalability Review answers the more useful question. Tell us what actually went wrong and we will say which applies.
Stop being the monitoring system
Tell us what broke, what you are being asked to sign, or what you are afraid of. We will come back within two working days with a scope, a price for the assessment and a straight view of how much hardening you need, including when the answer is less than you expected.
Related resources
Go deeper with our thinking
From Spreadsheets to a Scalable Platform – building Canada’s Most Tenant- First Coliving System
Read case study
How to Build a SaaS Product
Read guide
The MVP Development Guide
Read guide
Building a Digital Engineering Roadmap That Aligns With Business Goals
Read article
Engineering IoT Products: Key Challenges and Best Practices
Read articleNot sure where you stand?
Take a free two-minute readiness scorecard built for your industry.