The Azure Well-Architected Reliability Pillar: What Most Environments Are Missing
Reliability assessments of Azure estates keep landing on the same three things: no documented RTO or RPO, single points of failure sitting in the data tier, and failover nobody has ever tested.
Marc Dekeyser |
The Azure Well-Architected Reliability Pillar: What Most Environments Are Missing
TL;DR: the reliability pillar almost never fails on infrastructure. It fails because nobody wrote down what “recovered” is supposed to mean.
I have lost count of how many reliability sections I have written that say the same thing. The recovery commitments were never written down, so there is nothing to measure against. And the pillar does not ask whether your workload runs on a good day. It asks whether it meets its availability and recovery commitments when something breaks.
That is the actual gap. Not the infrastructure.
Most mid-market Azure environments I look at were built by competent people who kept the lights on and shipped features. Nothing wrong with them. But “it hasn’t gone down yet” is not a reliability posture. It is an absence of evidence, and those are different things. The pillar asks you to make the implicit explicit, and that is where every finding comes from.
What the five design principles actually require
Five principles. Each one turns into a concrete artifact an assessment goes looking for. They sound abstract on the Microsoft docs page. The findings are not abstract at all.
Start with designing for business requirements. That means a documented RTO and RPO for each workload, agreed with the business, not assumed by the platform team. This is the one that fails most often. Recovery Time Objective is how long you can be down. Recovery Point Objective is how much data you can afford to lose. Leave them undocumented and every downstream decision (backup frequency, replication tier, failover design) is guesswork wearing an architecture costume.
Designing for resilience means the workload survives the failure of a component without failing as a whole. In practice I go hunting for redundancy at each tier, and for the single points of failure nobody put there on purpose.
Designing for recovery means you can restore service after a failure you did not design out, and you have proven it. That is where Azure Backup and Azure Site Recovery get poked at.
Keep it simple is a reliability control in its own right, which surprises people. Complexity you did not earn against a requirement is a risk. Over-engineered failover that nobody understands fails during the incident, not before it.
Designing for operations means the workload is observable enough that you know it is degrading before it fails. That runs straight into health modelling, which I will come back to.
Where the single points of failure actually live
Compute redundancy usually gets handled. Teams understand that one virtual machine is a risk; that lesson landed years ago. The data tier is where the gap sits, because the failure mode is quieter and the default configuration looks like it works.
And none of it shows up in an Architecture Decision Record. Ever.
Here is the pattern I keep meeting. An Azure SQL Database on the General Purpose tier without zone redundancy. A storage account set to Locally Redundant Storage (LRS), copies held in one datacentre, backing a workload the business calls critical. An Azure Cache for Redis on the Basic tier: a single node, no replication.
None of those are wrong by default. All of them are single points of failure nobody chose.
The table maps each design principle to the finding it produces and the configuration setting sitting behind it.
| Design principle | Finding in nearly every assessment | The setting behind it |
|---|---|---|
| Design for business requirements | No documented RTO/RPO per workload | Absent, no artifact exists |
| Design for resilience | Single-zone database in a critical path | zoneRedundant set to false on Azure SQL |
| Design for resilience | Storage with no regional redundancy | Storage account SKU set to Standard_LRS |
| Design for recovery | Backups configured but never test-restored | Azure Backup recovery points present, no restore log |
| Design for recovery | Failover never exercised | Azure Site Recovery plan with no test failover history |
One thing ties all five together. These are configuration defaults that were never revisited. The team that built the environment optimised for delivery, not for auditing its own resilience, which is a perfectly rational thing to do when you have a deadline. Running an environment and auditing one are different jobs. That difference is the entire reason a structured review finds things the operators genuinely could not see.
Why availability zones get skipped
Zonal deployment is a provisioning-time decision. Miss it, and enabling zone redundancy later usually means a redeploy nobody wants to schedule. That is the whole mechanism behind the finding.
An availability zone is a physically separate datacentre inside an Azure region, with its own power, cooling, and networking. Spread a workload across zones and a datacentre-level failure stops taking the workload down with it, for marginal cost. The regions Dutch and EU organisations tend to land in, West Europe and North Europe, support them.
So the finding is almost never that zones were not available. It is that the resource shipped with zone redundancy off, because that was the path of least resistance in the portal or the template, and nobody ever wrote the decision down as an ADR.
Region pairing is the next layer up. Azure pairs regions (West Europe with North Europe) for geo-redundant services and coordinated platform updates. Geo-Redundant Storage (GRS) replicates to the paired region. I check whether critical data stores use GRS, or RA-GRS where read access during an outage actually matters. The common finding is LRS sitting under data the business would call unrecoverable if the primary region went away. Worth cross-linking with the cost optimization pillar, because the inverse turns up just as often: GRS being paid for on data that is genuinely disposable.
What health modelling and failure-mode analysis surface
Can your team tell a degraded workload from a healthy one before your users do? In most environments I look at, no.
Monitoring reports component status, not workload health. A health model defines what “healthy” means at the workload level. Not “the VM is up” but “the application is serving requests inside its latency budget.” Those are not the same sentence and the gap between them is where incidents live.
The finding repeats itself. Azure Monitor and Application Insights are deployed. They report CPU, memory, instance availability. What they do not report is a composite health signal tied to the business function, because nobody ever defined one. Failure-mode analysis, the boring exercise of systematically asking “what happens when this component fails?”, is almost never written down. The dependencies live in people’s heads.
When the dependency fails, the knowledge fails with it.
This links to operational excellence: reliability asks whether you can see degradation, operational excellence whether you can respond to it.
How untested failover fails
The configuration looks correct. That is the trap. Azure Site Recovery replicates virtual machines to a secondary region, the replication health shows green, everyone assumes recovery works. It frequently does not, for reasons that stay invisible until somebody runs a test failover.
The recovered VMs come up, but the network security group rules in the secondary region do not match. Or DNS still points at the primary. Or the application references a Key Vault with no private endpoint in the failover region. Or the recovery plan boots the database after the application tier, and the application dies on startup because its dependency is not there yet.
Every one of those is discoverable only by running a test failover. Site Recovery supports that without touching production. Most teams have never done it once.
Azure Backup has the twin finding. Recovery points exist. Nobody has run a test restore.
A backup you have never restored from is a hypothesis, not a recovery capability.
“But our backups are green in the portal, so we’re fine, right?” That is the objection I get most, usually in a slightly wounded tone, and I understand it. Green means the backup job completed. It says nothing about whether the restored thing boots, whether it comes back inside your RTO, or whether the person who knows the runbook still works there. DORA Article 12 is explicit on this: backup and restoration procedures must be tested periodically. “We have backups” does not satisfy it. “We test-restored on this date and it succeeded” does.
The evidence a reliable configuration produces
It shows up by construction: a documented RTO/RPO register, zone-redundant resources visible in Azure Resource Graph, GRS on critical data, dated test-failover and test-restore logs, a defined health model. That is what turns a reliability claim into a position you can defend in front of a DORA or NIS2 auditor without sweating.
The mapping is direct. DORA Article 12 wants backup policies, restoration procedures, and periodic testing of both, so your Azure Backup configuration and your test-restore logs are the evidence. NIS2 Article 21(2)(c) wants business continuity and disaster recovery measures, so your documented recovery objectives, your zone and region redundancy, and your exercised failover plans are the evidence.
A well-architected environment produces this material as a byproduct. You are not assembling evidence for the audit. The configuration is the evidence.
Somewhere around the fifteenth time I typed “no documented RTO or RPO” into an assessment report, I accepted that I was the bottleneck and not the finding. So I built Platform Architecture Authority to read the estate through Azure Resource Graph and surface the boring half for me: the single-zone databases, the LRS storage under critical workloads, the Site Recovery plans with no test history. It is read-only, and it writes the remediation as infrastructure-as-code that you review and apply on your own schedule. It still cannot tell you which workloads genuinely deserve zone redundancy. That judgment is the part worth paying a human for, and I would rather spend my hours there than on the inventory.
The short version
- No documented RTO or RPO is the most common finding I write. You cannot design redundancy against a target that does not exist.
- The single points of failure that matter sit in the data tier (single-zone database, storage with no geo-redundancy) and were defaults, not decisions.
- Availability zones are among the cheapest reliability controls Azure offers and the most often skipped, because nobody selected zonal deployment at provisioning time.
- Failover you have never tested is not a recovery plan. Untested Site Recovery configs fail in ways nobody anticipated.
- All of it maps to DORA Article 12 (backup and recovery) and NIS2 Article 21(2)(c) (business continuity).
A reliability posture is not the absence of outages. It is the documented, tested, evidenced ability to recover from them on a schedule you control rather than one an incident picks for you.
FAQ
What is the difference between RTO and RPO in the reliability pillar? RTO (Recovery Time Objective) is the maximum acceptable time a workload can be unavailable after a failure. RPO (Recovery Point Objective) is the maximum acceptable amount of data loss, measured in time. RTO drives failover design; RPO drives backup and replication frequency. The reliability pillar requires both, documented per workload.
Are availability zones enough, or do I need a second region? Availability zones protect against a datacentre-level failure within one Azure region and cover most reliability requirements. A second region, via region pairing and geo-redundant storage, protects against a full regional outage. Whether you need it depends on the documented RTO and RPO for the workload, which is why those come first.
How often should failover be tested under DORA? DORA Article 12 requires backup and restoration procedures to be tested periodically, without prescribing a fixed interval. Annual test failover and test restore is a defensible baseline for most mid-market workloads; critical workloads warrant more frequent exercises. The evidence that matters is a dated log showing the test occurred and succeeded.
