Designing Azure for Reliability: Five Principles That Survive an Incident

Most reliability conversations I get pulled into open with a product name. Someone bought Site Recovery. Someone flipped on geo-redundant storage. The question on the table is whether that counts as being reliable.

It doesn’t.

Five decisions actually determine whether an Azure workload degrades gracefully or falls over in a way you first hear about from a user at 02:00. Design for business requirements. Design for resilience. Design for recovery. Design for operations. Keep it simple. That’s the whole list, and none of the five is a purchase.

Reliability is not uptime you measure afterward. It’s a stack of decisions made before the incident, and each one leaves a specific artifact behind in the environment. That artifact is the point. It’s what tells you the decision was made at all.

These map to the Reliability pillar of the Microsoft Azure Well-Architected Framework. They’re also, whether the pillar says so or not, exactly what a DORA or NIS2 examiner is testing when they ask you to prove business continuity. What follows gets Azure-specific for each one: named services, the settings that matter, and the artifact it should produce. This is one of five companion walkthroughs under the architecture principles a real Azure assessment measures against.

It starts with a number, not a service (R01)

The first principle is to gather requirements around what the workload is actually for. The user experience. The data. The workflows. The things unique to that workload and nobody else’s. Expressed as quantified targets, not adjectives.

“Highly available” is not a target. Fifteen minutes is.

In Azure terms that means an RTO and an RPO for every workload, plus the latency and throughput numbers that define “healthy.” It also means knowing the platform’s own limits: the service quotas, regional caps, and capacity constraints of whatever Azure services you’re leaning on. Design a workload for a 15-minute RTO on a service whose regional failover takes longer than that, and you have a requirement the architecture physically cannot meet. The only way you ever find out is by having written the requirement down in the first place.

The finding is almost always the same. The targets live in people’s heads, not in a register. Eleven workloads, not one documented RTO. Redundancy decisions made in a corridor, which means nobody can evaluate them, defend them, or show them to an auditor.

So try this. Pick your most important workload and say its RTO out loud, in minutes, without opening anything. If you hesitated, that’s the finding, and it’s the first reliability finding in nearly every environment I look at.

The artifact this principle owes you is a recovery-objective register mapped to workloads. Nothing fancy. A table.

Resilience and recovery are different jobs (R02, R03)

Resilience keeps the workload serving while a fault is happening. Recovery brings it back after a fault has already put it on the floor. Separate architecture, separate services, separate discipline. Most Azure environments build one of them and quietly assume the coverage carries over.

It does not.

Resilience on Azure comes from availability zones, redundancy across the application tiers, and patterns that stop a fault spreading: retry with backoff, circuit breakers, bulkheads, graceful degradation. The work is sorting the critical-path components from the ones that can limp along degraded, then hardening the critical ones with redundancy and scale-out. The classic finding is a database sitting on zoneRedundant=false because that’s the portal default, running a workload the business would absolutely call critical. Azure SQL Database, Azure Cosmos DB, zone-redundant storage: they all support zone redundancy. The gap is almost never capability. It’s a default nobody looked at.

Recovery is a different toolbox. Azure Site Recovery for compute failover. Azure Backup for data. Geo-redundant storage for cross-region durability. Self-healing for the stateless pieces. And it carries a discipline resilience doesn’t: structured, tested, documented recovery plans tied to the objectives you negotiated, plus the ability to repair stateful data inside the RPO. The Well-Architected reliability pillar walkthrough goes deep on the failover findings.

The short version fits on a sticky note. Replication health showing green is not recovery capability. A test failover that succeeded on a date you can name is.

“But we’ve got zone redundancy and we take backups. Isn’t that the same thing?”

No, and the gap between them is where the bad afternoons live. Zone redundancy is a resilience control: it keeps the thing serving. A backup is a recovery input, and an input is not a capability until someone has restored from it and written down when. One of those protects you during the fault. The other protects you after it. Buying one and assuming the other is the single most common reliability mistake I see, and it’s invisible until the day it isn’t.

I should be honest here. Early in my career I wrote a recovery runbook, never tested it, and then quoted it back to a customer with a completely straight face. It read beautifully. I have no idea whether it worked.

The people who get paged (R04)

Shift left. Anticipate the failure conditions while you’re still writing the thing, instead of meeting them for the first time in production. Build systems you can actually see into, so a degradation shows up on a dashboard before it shows up in a support queue.

That’s the fourth principle in one breath: treat operability as a design concern. Test failures early and often. Give the team shared visibility of what every dependency is doing. Learn from the incidents you do have.

On Azure it looks concrete. Azure Monitor and Application Insights wired in from day one, telemetry correlated, and a defined health model: a composite signal that reports whether the workload is healthy, not whether some CPU is busy. Alerts that are prioritised and actionable, tied to that health model, not raw thresholds everyone has learned to ignore. And failure-mode analysis. Walk each dependency, ask what happens when it dies, write the answer down instead of leaving it in one engineer’s head.

This is where reliability and operational excellence shake hands. Reliability asks whether you can see the degradation. Operational excellence asks whether you can respond. Both live in the operational excellence principles.

The reliability-specific finding here is the missing health model. Azure Monitor deployed, dutifully reporting infrastructure metrics, and no business-level health signal anywhere, because nobody ever sat down and defined one.

Fewer moving parts is a reliability decision (R05)

Every component on the critical path is a failure mode. That’s the whole argument.

The fifth principle is a warning against over-engineering, because each thing you add widens the surface for misconfiguration, weird interactions, and 3 a.m. operator error. Keep the critical path lean.

There’s a counterweight, and it’s real. Oversimplify and you hand yourself single points of failure. The rule is not “fewer components, always.” It’s “just enough.” On Azure that pushes you toward platform capabilities and prebuilt assets over hand-rolled equivalents: managed services that patch and fail over themselves, Azure Policy enforcing standards through automated validation instead of a human reading configs, established design patterns instead of your own. Write just enough code. Let the platform do the parts it already does reliably.

This finding hides well, because it doesn’t look like a problem. A workload carrying three caching layers, a custom retry framework, and a message queue in front of a load the database could have served directly. Each piece had a good reason on the day it went in. Stacked together they form a critical path with more ways to fail than the business value justifies.

The simplicity principle is your licence to ask the awkward question. Would taking something out make this more reliable? More often than people expect, the answer is yes.

The evidence a reliable architecture leaves behind

Build it right and the evidence assembles itself. A documented RTO/RPO register. Zone-redundant resources you can see in Azure Resource Graph. Geo-redundant storage on the critical data. Dated test-failover and test-restore logs. A defined workload health model. Same objects the posture is made of, same objects the examiner asks for. There is no second set of books.

The regulatory mapping isn’t a stretch either. DORA Article 12 wants backup policies, restoration procedures, and periodic testing of both, so your Azure Backup configuration plus the dated test-restore logs are the evidence. NIS2 Article 21(2)(c) wants business continuity and disaster recovery measures, so your recovery-objective register, your zone and region redundancy, and your exercised Site Recovery plans are the evidence. A well-architected environment throws this off as a byproduct of being built correctly. You are not assembling a binder for the audit. The configuration is the binder.

What eventually annoyed me into building something was the sameness. Different tenant, different industry, same three findings: the single-zone database, the missing geo-redundancy on data that mattered, the Site Recovery plan with no test history. And every time I dug them out by hand before I could even start the conversation worth having. So Platform Architecture Authority reads the estate through Resource Graph and puts that list in front of me instead. Which workload genuinely warrants zone redundancy is still a judgment call, and it stays with a senior architect. The tool just makes sure we’re arguing about the right resources.

The reliability posture that holds

A reliability posture is not the absence of outages. Outages happen.

It’s the documented, tested, evidenced ability to recover from them on a schedule you control instead of one an incident chooses for you. The five principles are how you build that ability on Azure: requirements, resilience, recovery, operations, simplicity. The artifacts they leave behind are how you prove it.

Build them in order. Requirements first, because everything after is measured against them.

What to take away

• Reliability starts with a documented target, not infrastructure. Write the RTO and RPO per workload first. Without them you can’t tell over-engineering from under-engineering.
• Resilience and recovery are not the same thing. One keeps the workload serving through a fault, the other brings it back after one. Azure gives you distinct services for each, and most environments buy one and quietly assume it covers the other.
• An untested recovery plan is a hypothesis. Azure Site Recovery and Azure Backup both test without disruption. The dated log is what DORA Article 12 actually wants.
• Simplicity is a reliability principle, not a nicety. Every component on the critical path is a failure mode. Sometimes the most reliable change is deleting one.
• Every principle leaves evidence. Documented objectives, zone-redundant resources, dated test-failover logs, a health model. That evidence is both the posture and the audit artifact.

FAQ

What is the difference between RTO and RPO? RTO (Recovery Time Objective) is the maximum acceptable time a workload can be unavailable after a failure. RPO (Recovery Point Objective) is the maximum acceptable data loss, measured in time. RTO drives failover and resilience design; RPO drives backup and replication frequency. The first reliability principle requires both documented per workload.

Do availability zones remove the need for a second Azure region? Availability zones protect against a datacentre-level failure within a single Azure region and satisfy most reliability requirements. A second region, via region pairing and geo-redundant storage, protects against a full regional outage. Whether you need it is determined by the documented RTO and RPO for the workload, which is why those targets come first.

How often must I test failover under DORA? DORA Article 12 requires backup and restoration procedures to be tested periodically without prescribing a fixed interval. Annual test failover and test restore is a defensible baseline for most mid-market workloads; critical workloads warrant more frequent exercises. The evidence that matters is a dated log showing the test occurred and succeeded.

Can a system be too reliable? Yes. Reliability beyond the documented business requirement is spend with no return: geo-redundant storage on disposable data, a multi-region active-active design for a workload with a four-hour RTO. The first and fifth principles guard against this. Requirements set the target, and simplicity keeps you from exceeding it without reason.