DORA Metrics: 4 Ways to Measure Software Delivery Performance

Introduction

DORA metrics give a high-level view of a team's performance, allowing to assess how well the team balances speed and stability and spot areas for improvement. 

That allows engineering leaders to continuously streamline processes and increase the speed of delivering customer value, which is crucial for a product to remain competitive.

Let’s break these metrics down and analyze how you can implement them easily.

What are DORA metrics?

DORA DevOps metrics definition

DORA metrics are four indicators used to calculate DevOps team efficiency. They measure a team’s performance and provide a reference point for improvements. They also help to make data-driven decisions.

The metrics were invented by an organization gathered by Google and called DevOps Research and Assessment (DORA). They surveyed thousands of development teams from various industries to understand what differentiates a high-performing team from the others. Establishing DORA metrics is the result of this research. 

‍

‍

DORA metrics were explained for the first time in the DORA organization’s report ﹣ Accelerate: State of DevOps 2018. The report is released on a regular basis ﹣ the newest version was released in 2021.

DORA metrics core objectives

<span class="colorbox1" fs-test-element="box1"><p>Key takeaway:</p><p>DORA metrics show what level of performance is needed to achieve desired business objectives.</p></span>

The metrics reflect key areas that influence performance and equip engineers with detailed insights. 

By measuring the velocity of development and stability of deployment and the product itself, teams are motivated to improve their results during subsequent iterations. And improving them leads to better business outcomes.

DORA uses its metrics to identify Elite, High, Medium, and Low performing teams. They claim that Elite teams are twice as likely to meet or exceed the performance goals of the organization.

Benefits of implementing DORA metrics

DORA metrics make the process of software delivery more transparent and understandable, breaking it into pieces.

DORA metrics help teams to:

• spot the areas for improvement when it comes to the delivery process,
• understand what actions to take to streamline the process,
• deliver quicker, ensuring higher quality,
• ensure stability and accountability of a product
• make decisions based on data, not assumptions,
• spot problems like the lack of people or inefficiency of testing,
• deliver more value to clients by ensuring fast iterations and better responsiveness to feedback,
• increase the business value of a product faster,
• reduce waste.

<span class="colorbox1" fs-test-element="box1"><p>Note:</p><p>Implementation of DORA metrics and the whole DevOps culture is also one of the indicators of the best dev shops. Pay attention to that aspect while choosing an external software development company to work with.</p></span>

What are the 4 DORA metrics?

1. Deployment Frequency
2. Lead Time for Changes
3. Time to Restore Service
4. Change Failure Rate

At the highest level, Deployment Frequency and Lead Time for Changes measure velocity, while Change Failure Rate and Time to Restore Service ﹣ stability.

Deployment Frequency

The Deployment Frequency metric refers to the frequency of successful software releases. It measures how often a company successfully deploys code to production for a particular application. 

Deployment Frequency depicts the consistency and speed of software delivery. It determines whether a team is meeting goals of continuous delivery. 

Smaller and more frequent deployments are the target, but the most effective number of deployments will differ from product to product. Delivering code quickly to production comes from shipping as many times as possible while changing the batch size to be as small as possible.

<span class="colorbox1" fs-test-element="box1"><p><strong>Key takeaway:</strong></p><p>Deploying often allows the team to constantly improve the product, and spot issues easier.</p></span>

Benchmarks

How to calculate Deployment Frequency?

Various tools measure Deployment Frequency by calculating how often a team completes a deployment to production and releases code to end-users.

To measure the frequency, calculate the median number of days per week with at least one successful deployment.

Why should you measure Deployment Frequency?

Deployment Frequency depicts how well your current process is functioning. 

By spotting specific periods when deployment is delayed, you can identify problems in a workflow ﹣ unnecessary steps or issues with tools. You can also recognize problems like staff shortage or the need for longer testing time. 

Deployment Frequency also allows for comparing deployment speed over time and mapping your team’s velocity to deliver.

Causes of low Deployment Frequency

There may be several reasons for low Deployment Frequency, often related to the process and tools:

• deploying only very large changes to production,
• having unnecessary complex routes to production,
• not using the right tools,
• shortage of people,
• various blocks and dependencies in the deployment process.

How to improve Deployment Frequency?

To increase your team’s ability to deliver, try to:

• implement CD/CI best practices to increase the deployment speed,
• ship smaller changes,
• introduce automated processes when it comes to testing and validating new code,
• reduce the time span between error recovery time and delivery.

Lead Time for Changes

Lead Time for Changes indicates how long it takes to go from code committed to code successfully running in production. Along with Deployment Frequency, it measures the velocity of software delivery. 

Lead Time for Changes allows us to understand what the DevOps team cycle time looks like, and how the team is handling an increased number of requests. 

The team should strive for the low Lead Time for Changes ﹣ the lower it is, the more efficient a team is in deploying code. It means that a team is able to respond to feedback faster and correct the course rapidly. Defects are most likely being fixed quickly.

Benchmarks

How do you calculate Lead Time for Changes?

To measure Lead Time for Changes, you need two pieces of data:

• the exact time of the commit
• the exact time of the deployment

Then, use the average time as an indicator of overall performance. Lead Time for Changes is an approximation.

Why should you measure Lead Time for Changes?

Lead Time for Changes is an indicator of how quickly a team responds to needs and fixes. It represents the efficiency of the process, code complexity, and team’s capacity. 

The metric helps to understand how long it takes to get changes to production. When Lead Time for Changes is too high, it shows inefficiencies and bottlenecks in the process. A low one shows that a team reacts to feedback and changes quickly

<span class="colorbox1" fs-test-element="box1"><p><strong>Key takeaway:</strong></p><p>The team’s goal should be to reduce Lead Time for Changes and react to issues in a timely manner.</p></span>

This metric is also essential while working with clients, who prefer to work with a team that responds to urgent bug fixes within hours. Making changes quickly keeps clients satisfied.

Causes of high Lead Time for Changes

High Lead Time for changes is most often caused by inefficient processes and introducing too big changes to production. Other reasons are:

• poor Definition of Ready,
• manual testing,
• legacy code,
• blocks and dependencies,
• inefficient development process,
• complex routes to production,
• huge changes introduced to code.

How to reduce Lead Time for Changes?

To reduce Lead Time for Changes, try to:

• work with smaller iterations ﹣ smaller changes will help developers to get feedback faster and resolve issues quicker,
• use automated tests at every stage of CI/CD pipeline to spot issues earlier,
• use automated code reviews to improve quality and save time,
• integrate testing into the development process,
• eliminate bottlenecks.

Time to Restore Service

Mean Time to Recovery, Time to Recover, or Time to Restore Service. This metric goes by many names. But, what’s most important, it measures how long it generally takes to restore service when a service incident or a defect that impacts users occurs. 

Simply speaking, it’s the time it takes for a service to bounce back from a failure.

<span class="colorbox1" fs-test-element="box1"><p><strong>Key takeaway:</strong></p><p>Failures can’t be avoided, so what makes a difference is how quickly a system can be restored or recovered.</p></span>

Benchmarks

‍

How to calculate Time to Restore Service?

Time to Restore is calculated by tracking the average time between a bug report and the moment the fix is deployed.

The formula for that could look like this:

average(incident resolved timestamp ﹣ incident created timestamp).

Why should you measure Time to Recover?

Time to Restore Service indicates a team’s response time and development process efficiency. 

When Time to Restore Service is too high, it may be revealing an inefficient process, lack of people, or an inadequate team structure. When it’s low, it shows that a team responds and solves problems quickly. Low Time to Restore Service ensures availability and correct functioning of the software.

The team’s goal should be to reduce Time to Recover. Then, when there’s an incident, a team can fix it in a timely manner, so the availability of software isn't compromised.

Causes of high Time to Restore Service

There are various aspects that can increase Time to Restore Service:

• manual testing,
• inadequate team structure,
• lack of people,
• inefficient process,
• changes in the organizational structure,
• inefficient incident management process,
• blocks and dependencies,
• not using the right tools.

How to reduce Time to Restore Service?

To reduce Time to Restore Service, do the following:

• use automated tests at every stage of CI/CD pipeline to reduce delivery times and help to recover quicker,
• improve the organizational structure (the right people should be able to identify the problem’s root cause),
• document the incident management process,
• train team members on how to react in case of incidents,
• automate several steps of the incident management process,
• assign responsible people for steps that need to be performed manually,
• try out Feature Flags.

Change Failure Rate

Change Failure Rate measures the percentage of deployments causing failure in production ﹣ the code that then resulted in incidents, rollbacks, or other failures. 

<span class="colorbox1" fs-test-element="box1"><p><strong>Key takeaway:</strong></p><p>Change Failure Rate is a measure of stability and quality of software delivery.</p></span>

Change in Failure Rate is inspired by Lean principles. Over time the metric provides insights on how much time is spent on fixing bugs versus delivering new code.

Benchmarks

How to calculate Change Failure Rate?

Change in Failure Rate is calculated by counting the number of deployment failures and dividing it by the total number of deployments. 

Why should you measure Change Failure Rate?

Change Failure Rate shows how well a team guarantees the security of changes made into code and how it manages deployments. It’s an indicator of a team's capabilities and process efficiency. 

<span class="colorbox1" fs-test-element="box1"><p><strong>Key takeaway:</strong></p><p>The metric also shows how much developer’s time is devoted to tasks that don’t contribute to business value.</p></span>

A high Change Failure Rate indicates problems in the development process or lack of testing before deployment. 

A low Change Failure Rate shows that a team identifies infrastructure errors and bugs before the code is deployed. It’s a sign of a sound deployment process and delivering high-quality software.

The DevOps team’s goal should be to reduce Change Failure Rate to ensure software’s availability and correct functioning. 

Causes of high Change Failure Rate

Behind the high Change Failure Rate are often inefficiencies in testing. It is also caused by:

• manual testing,
• lack of testing before deployment,
• problems in the process,
• relying on a manual process,
• poor quality code that leads to unexpected errors,
• difficult maintainability and introduction of new code,
• not reproducible infrastructure changes.

How to reduce Change Failure Rate?

If your Change Failure Rate is high, make sure to:

• work with smaller iterations ﹣ smaller changes are easier to test and less likely to break,
• use automated testing at every stage of CI/CD process to catch issues earlier,
• test the code extensively,
• use automated code reviews to improve code quality,
• engage developers in production so that they understand the impact of changes and failures,
• create a critical feedback loop,
• mission-critical configurations and infrastructure should be reproducible.

<span class="colorbox1" fs-test-element="box1"><p>To many permissions? Discover how the CASL library can simplify your web application's permission management with its intuitive and flexible approach.</p></span>

How to implement DORA DevOps metrics in your process

DORA metrics implementation

• Start with choosing a tool or creating a DIY solution.
• Don’t focus on metrics too much. Start gathering data, track the metrics for the first period, and then analyze what you have to improve.
• While setting the improvement goals, focus on your product and its growth, as well as the growth of your team and improvement of the processes.
• Treat DORA metrics as a set of indicators of what you can do to make a positive impact on the product and its business results.

<span class="colorbox1" fs-test-element="box1"><p><strong>Key takeaway:</strong></p><p>Improving the metrics shouldn’t be the goal by itself ﹣ the ultimate goal teams need to focus on is to improve the way they deliver a product to increase its value faster and in a stable way.</p></span>

Tools to implement DORA

There are various tools that help to implement DORA metrics:

• Faros,
• Sleuth,
• Haystack,
• Pulse.

To help teams generate these metrics, DORA created the Four Keys open source project. It automatically sets up a data ingestion pipeline, getting data from teams’ Github or Gitlab repos through Google Cloud and Google Data Studio.

The Four Keys project aggregates data and compiles it into a dashboard using four key metrics. You can track the progress over time without the need of using extra tools or creating solutions on your own.

Challenges of DORA assessment

For some companies, DORA metrics will become a starting point, which then needs to be customized to fit their project.

During the implementation, pay attention to:

• Decentralized data ﹣ data can be scattered in different sources.
• Data transformation ﹣ data needs to be combined into calculable units.

One of the biggest problems is also the assessment speed versus stability. To avoid mistakes, you always need to put singular metrics in context. A high Deployment Frequency doesn't say anything about the quality of a product. To assess the quality, a Change Failure Rate will be a good indicator.

Take your DevOps performance to the next level

It’s not only about becoming an elite performer. It’s about delivering better business value faster.

By measuring and tracking DORA metrics and trends, teams can make better, more informed decisions about what can be improved, and understand how to do that. When DORA metrics improve, a team can be sure that they’re making good choices and delivering more value to customers and users of a product.

As an engineering leader, you need to empower your team with tools to achieve that.

DORA metrics can help you boost your DevOps performance by increasing velocity and stability, but your team can build even better products after implementing the BizDevOps approach. BizDevOps smashes the walls not only between developers and operations but also the business. It allows teams to make more informed decisions, speeds up the feedback loop, and results in building products that meet the business needs more efficiently.

<span class="colorbox1" fs-test-element="box1"><p><strong>Need help?</strong></p><p>You don’t have to figure it all out on your own. Let’s chat about DORA metrics implementation and accelerate your growth together.</p></span>

Resources:

• https://cloud.google.com/blog/products/devops-sre/announcing-dora-2021-accelerate-state-of-devops-report