Operational Excellence by Design on Azure

A workload runs without heroics, or it doesn’t. There is no third state.

Four principles decide which one you end up with: define all infrastructure as code, automate every repeatable process, let components react to events instead of calling each other synchronously, and build observability in from the start. That’s the Operational Excellence pillar of the Well-Architected Framework with the framing taken off.

Get them right and running an Azure environment is a discipline you can hand to someone else. Get them wrong and it’s a sequence of manual interventions that holds up beautifully until the one person who knows the steps takes a week off.

I assess Azure estates for a living, and this is the pillar where the distance between the architecture document and the running environment is widest. Not because anybody disagrees with the principles. Because none of them feel urgent until the morning they do.

This is one of five companion walkthroughs under the architecture principles a real Azure assessment measures against.

TL;DR: the other three principles all stand on infrastructure as code. Build that one first.

Infrastructure as code is the foundation (C03)

You can’t automate an environment you assembled by hand. You can’t observe it either, and you certainly can’t recover it.

That’s the whole case. Every other operational principle assumes a consistent, reproducible environment, and clicking through the portal does not produce one. So the rule is that all infrastructure gets defined and managed through code. Version-controlled, deployed the same way to every environment, no exceptions carved out for the urgent thing.

On Azure that means Bicep or Terraform as the declarative source of truth, deployed through a pipeline in Azure DevOps or GitHub Actions, with Azure Policy enforcing standards on anything that tries to deploy around the pipeline. Drift stops. The snowflake resources that exist only in production and nowhere in source control stop appearing. You get to test infrastructure changes before they reach production, and you get your disaster-recovery and change-control documentation as a byproduct, because the code is the documentation.

Here’s the finding I keep writing up. The portal-created resource. A virtual machine, a storage account, a network security group rule that somebody added through the Azure portal during an incident and never codified. It works.

It’s also invisible to source control, untested, unreproducible, and in a region failover it will not come back, because nothing in your deployment knows it exists.

(I have done this. At two in the morning, mid-incident, promising myself I’d codify it properly the next day. Reader, I did not.)

“But the important stuff is all in Bicep, it’s just a few small bits that aren’t.” Right. Go and count them. In every assessment I’ve run, the person who says that is describing the resources they remember creating, not the ones the estate actually contains, and the gap between those two numbers is the finding. If you want a cheap version of this test, list your production resource groups and ask which repo deploys each one. The silences are informative.

The deeper treatment is in the Well-Architected operational excellence pillar walkthrough.

Automate every repeatable process (O01)

Manual steps are where operational error lives. Not because engineers are careless, but because a human executing a sequence under pressure makes mistakes that deterministic automation doesn’t. So the principle is blunt: if a process repeats, automate it.

The trade is real and worth stating. You invest in tooling and skills, and you take on a dependency on automation systems that now have to be reliable themselves. In exchange you get consistency and scale that hand-operation never reaches.

On Azure the surface area is wide. Deployments through CI/CD pipelines, not portal clicks. Configuration through Azure Policy and desired-state tooling, not per-resource adjustment. Routine operational work (certificate rotation, scaling actions, scheduled clean-up) through Azure Automation runbooks and Logic Apps, so it stops eating engineer time and stops introducing variance. The payoff the principle names outright is a smaller operational team running a larger estate, because the team designs and maintains the automation instead of performing the operations by hand.

The finding here is the runbook that’s really a wiki page. A carefully documented sequence of manual steps a human follows during a deployment or an incident. Accurate, well-written, reviewed, and a source of error every single time it runs.

Documentation is not automation. It’s a description of the thing you didn’t automate.

The test takes two questions. Does this process repeat? Is it automated? Where the answer is yes and then no, the finding writes itself.

Event-driven design, and what it costs (O02)

Let components talk through events rather than direct synchronous calls and three things follow: loose coupling, independent scaling, and failure isolation. One component’s outage stops cascading into its neighbours.

On Azure the building blocks are specific. Azure Event Grid for reactive event routing. Azure Service Bus for reliable message queuing and decoupling. Azure Event Hubs for high-throughput streaming. Build a workload this way and each component scales to its own demand, and a slow consumer backs up its own queue instead of blocking the producer. A degraded component degrades, rather than taking the whole synchronous call chain down with it.

Now the cost, because the assessment names it and so will I. Event-driven systems are harder to debug and harder to reason about for transaction management. A request that flows through three queues and two functions is more resilient and less traceable than a single synchronous call.

Which is exactly why observability isn’t optional here. The loose coupling that buys you resilience also hides the request path. Without distributed tracing you have traded an incident you can debug for one you can’t, and you will discover this at the worst possible moment.

Observability goes in at build time (O04)

Bolt monitoring on after something breaks and you’re paying retrofit prices for telemetry the system was never designed to emit. You can’t operate what you can’t see, and you can’t recover it either.

So observability goes in from inception. Monitoring, logging and tracing built into every system, so it emits enough to explain its own behaviour, performance and health without anyone instrumenting it under duress.

That means a standardised approach to metrics, logs and traces. It also means the part teams skip: a health model that reports workload health rather than raw component status. It costs more up front and it repays that by turning reactive troubleshooting into proactive detection. You see the degradation before your users feel it, and the same data feeds continuous, evidence-based improvement instead of quarterly guesswork.

On Azure this is Azure Monitor, Application Insights and Log Analytics, instrumented as part of the workload rather than added the day something breaks.

There’s a compliance angle the operational principle carries quietly. Log Analytics workspace retention gets set too short to satisfy NIS2 or DORA evidence requirements more often than not. The telemetry exists, it does its operational job, and then it ages out before it can serve as an audit record. Designed-in observability means retention set deliberately against both the operational need and the regulatory one.

The finding is always the same shape. Monitoring deployed and dutifully reporting CPU and memory, no workload-level health signal anywhere, and a retention period nobody ever mapped to a requirement.

The evidence falls out of doing it right

Run an environment to these four principles and you don’t assemble evidence for a review. You point at it.

Version-controlled infrastructure-as-code repositories, pipeline deployment logs, Azure Policy assignments in enforcement mode, automation runbooks, an observability stack with deliberate retention. Every one of those is operational maturity and audit evidence at the same time. Same artifact, two angles.

The infrastructure-as-code repository is your disaster-recovery documentation and your change-control evidence. The Azure Policy assignments are the codified standards an examiner asks to see. The Log Analytics retention configuration is the audit trail for NIS2 and DORA. Operating the environment correctly generates the evidence continuously, so there’s nothing to reconstruct when someone comes asking.

That’s the through-line. The discipline that makes an environment runnable is the discipline that makes it auditable.

I got tired of writing the same four findings by hand, in the same order, for a different logo every few weeks. So I built Platform Architecture Authority to read the estate through Azure Resource Graph and produce the findings I was going to write anyway. It still takes a senior engineer to decide which of them is worth doing first. It just means that decision starts from the estate rather than from whoever in the room speaks with the most confidence.

The environment that runs itself

Operational excellence is the environment designed so that fewer incidents need a hero at all. Automated, event-driven where it counts, observable by construction, defined entirely in code.

The four compound. Infrastructure as code makes automation possible. Automation makes scale possible. Observability makes event-driven design safe to operate. And all four turn your next audit into pointing at the configuration instead of reconstructing it from memory.

Build the foundation first. Everything operational stands on the code.

FAQ

Should I use Bicep or Terraform for infrastructure as code on Azure? Both satisfy the principle. Bicep is Azure-native, tracks new Azure features quickly, and integrates tightly with Azure Resource Manager. Terraform is multi-cloud with a broader ecosystem. Azure-only estate? Bicep reduces friction. Multi-cloud, or an existing Terraform investment? Terraform avoids fragmentation. The principle cares that infrastructure is codified, not which tool does the codifying.

Is event-driven architecture always the right choice? No. It buys resilience and independent scaling at the cost of debugging difficulty and harder transaction management. For simple, low-volume, strongly-consistent workloads, a synchronous design is simpler and more reliable. Use event-driven patterns where loose coupling and scale create value, not everywhere by default.

What is the difference between monitoring and observability? Monitoring reports known signals (CPU, memory, instance availability) against predefined thresholds. Observability is the property of a system that emits enough metrics, logs and traces to answer questions you didn’t anticipate, including why a workload is degraded. The fourth operational principle requires observability. Monitoring is one component of it.

How does operational excellence relate to compliance? Directly. Infrastructure-as-code repositories, Azure Policy enforcement, automation logs and Log Analytics retention are the evidence NIS2 and DORA examiners ask for. The disciplines that make an environment runnable without heroics are the same ones that make it auditable, which is why operationally mature environments rarely scramble before a review.