Table of contents
Although there are several differences to note when discussing Agile metrics vs. Lean metrics, ultimately, the roles that these metrics play are more similar than not. In this article, we’ll share how using Lean and Agile metrics mindfully can help Agile Release Trains (ARTs) deliver more effectively, drive continuous improvement, and sustain their Agile programs.
The 7 Must-Haves for Achieving Scaling Agile Success
Get the Must-Haves and Pro-Tips Needed to Scale Agile
View the eBook • The 7 Must-Haves for Achieving Scaling Agile SuccessGetting Started with Lean
Lean is a mindset that helps you make smarter decisions about how to invest your time, energy, and money.
View the eBook • Getting Started with Lean
What Are Agile Metrics?
First: What are Agile metrics? To understand the answer to this question, it’s helpful to review the origins of Agile. Agile first took root a couple of decades ago, around the same time that the discipline of software development was really taking shape.
Software development teams needed a better way for managing the unique challenges associated with their work. Existing workflows, such as those for manufacturing physical goods, didn’t lend themselves well to the inherently complex nature of software development.
For one thing, software development is iterative. It is cyclical in nature, not linear like many of the workflows found in other fields. It also is inherently complex, requiring several different types of work (planning, development, testing, break / fix work, etc.), each with their own workflows, all done by the same team.
Several methodologies preceded what we now know as Agile, including XP, Scrum, and others. The Agile that is the foundation for today’s Agile metrics incorporates many of the elements and key ideas of those methodologies. Although the methodology was specifically designed for managing the software development process, it has been adopted in virtually every discipline.
Agile was originally developed as a method for teams, without much guidance as to how to scale Agile practices across an organization. Agile program management is the term for the methods that have been developed to successfully practice Agile at scale.
At the heart of Agile is the drive to develop a better product through a better process. One of the hallmarks of Agile is the use of time-boxed iterations, or sprints, usually two weeks long, during which teams execute work that they have prioritized and to which they have committed. Working in sprints allows Agile teams to strike a balance between focused execution and real-time planning and decision making.
Agile metrics provide insight into the performance of a team or group of teams, measured and analyzed in fixed iterations (sprints). They are meant to help teams analyze their productivity and effectiveness through different stages of their software development (or, more broadly, development) lifecycle.
Agile metrics are not meant to replace business metrics, but rather to be used in conjunction with them. Business metrics measure whether the solution is meeting the market need, and Agile metrics measure how well the team can deliver those solutions.
What Are Lean Metrics?
Lean metrics, similarly, are meant to be used in conjunction with business metrics to provide a holistic view of system health. Lean metrics measure the efficiency of the system and are usually used by teams practicing any Lean methodology, including but not limited to Kanban, Lean, and others.
Understanding the history of Lean is also helpful for understanding how Lean metrics are meant to be used. The practices that we now think of Lean first took shape on the shop floors of auto manufacturers in Japan in the early twentieth century. In a quest to dominate market share and consistently produce a high-quality product, manufacturers were looking for ways to reduce (both physical and process) waste, increase efficiency, minimize variability, and create more value with fewer resources.
This gave rise to the Toyota Production System, which was iterated upon, and became known as Lean manufacturing. In the later part of the twentieth century, many software development teams began adopting Lean principles and practices, and the influence of Lean spread to other teams as well.
Agile Metrics vs. Lean Metrics
Since they share many of the same key concepts, goals, and even terminology, it’s not surprising that the lines are often blurred between Agile and Lean. There’s no rule that says you cannot use both Lean and Agile metrics – but it can be helpful to understand the differences between Agile metrics vs. Lean metrics so you can use them in context.
The major difference between Agile and Lean metrics is that many Agile metrics are meant to analyze performance during time-boxed iterations, whereas Lean metrics are meant for measuring performance in a system working towards continuous flow.
| Agile Metrics | Lean Metrics |
|---|---|
| Meant to be used in the context of continuous flow. | Meant to be used in the context of continuous flow. |
Agile Metrics vs. Lean Metrics vs. Business Metrics
As we’ve shared, neither Agile metrics nor Lean metrics are meant to replace business metrics. Business metrics, such as sales revenue, gross margins, net promoter scores, etc. help a business assess how well it is meeting the market need.
Business metrics are typically measured and reported on an organization-wide level. They are often used to orient teams toward specific goals and used by leaders to assess how well teams are tracking to achieve those goals.
| Agile Metrics and Lean Metrics | Business Metrics |
|---|---|
| Help teams, and teams of teams, analyze their own performance. Typically measured and reported on a team or program level. | Help a business assess how well it is meeting the market need. Typically measured and reported on an organization-wide level. |
Lean and Agile metrics are meant to be used by teams, or teams of teams, to assess their own performance. They should not be used to pit teams against each other. Unlike business metrics, they should not be used by leaders to “check up on” teams.
Like with business metrics, it’s important to always analyze Lean and Agile metrics in context, with an understanding of when, where, and why the data is being collected and shared.
Types of Agile Metrics
Delivery Metrics: Release Frequency and Delivery Speed
| What It Is | What It Measures | What It Tells Us | Why It Matters to ARTs |
|---|---|---|---|
| At the end of each sprint, Agile teams should deliver a working product to production. In software, this might be a feature or a bug fix. In other disciplines, it might be a marketing campaign, website update, or new product design. | Release frequency measures how often Agile teams can actually deliver completed work at the end of each sprint. Delivery speed measures how long it takes to complete a specific type of work. | Measuring release frequency and delivery speed can help teams understand how well they’re planning and executing during sprints. | For Agile Release Teams, these Agile metrics can inform better, more accurate planning and capacity management. |
Velocity
| What It Is | What It Measures | What It Tells Us | Why It Matters to ARTs |
|---|---|---|---|
| Velocity is a measure of the average amount of work a team completes during a sprint, measured in either story points or hours. | Velocity measures how many work items can be completed during a specific period of time, such as story points per iteration. | Velocity Agile metrics are most helpful when teams analyze them over longer time periods. New teams can expect to see a steady increase in velocity as the team optimizes its work process. Existing teams can track their velocity to ensure consistent performance over time and can confirm whether a particular process change made improvements or not. | Velocity is a useful benchmark for Agile teams to gauge how consistently and effectively they are planning, balancing, and executing their work. It is also incredibly helpful for forecasting, because over a period of time, teams can begin to predict how quickly they can work through their backlogs. |
Sprint Burndown
| What It Is | What It Measures | What It Tells Us | Why It Matters to ARTs |
|---|---|---|---|
| At the beginning of each sprint, Agile teams forecast how much work they plan to complete during that sprint. A sprint burndown report tracks the completion of work throughout the sprint. | The x-axis represents time, and the y-axis refers to the amount of work left to complete, measured in either story points or hours. The goal is to have all the forecasted work completed by the end of the sprint. | A sprint burndown report can be used before, during, and after a sprint to measure how the team is progressing toward its work completion goals. | Predictability is key for agility, and using Agile metrics like sprint burndown reports (in addition to epic and release burndown reports), can help teams plan, execute, and deliver increments of value with more predictability. |
Epic and Release Burndown
| What It Is | What It Measures | What It Tells Us | Why It Matters to ARTs |
|---|---|---|---|
| For tracking progress on larger bodies of work, Agile organizations use epic and release burndown reports. These reports then track the completion of work throughout a series of sprints. | Measuring burndown at this level can help Agile teams (and ARTs) manage scope creep – the addition of new requirements after the scope of the epic or release was already defined. | If an ART is consistently meeting its forecast, this makes a compelling case for Agile in the organization. If not, these Agile metrics can help to inform capacity planning in the future. | Tracking epic and release burndown can help ARTs understand how accurately they are forecasting – a key part of predictability. It can also help to reinforce the idea of not committing to unplanned work within an epic, which can further encourage teams to sustain their agility.reinforce the idea of not committing to unplanned work within an epic, which can further encourage teams to sustain their agility. |
Types of Lean Metrics
Remember, in contrast to Agile metrics, Lean metrics measure performance in a system working towards continuous flow.
Cycle Time
| What It Is | What It Measures | What It Tells Us | Why It Matters to ARTs |
|---|---|---|---|
| A measure of how long work takes to get from point A to point B within the team’s process. | The time it takes for a work item to move through two specific points in a process. In Kanban, the time a card enters one specified lane to the time it enters another specified lane (which lane depends on what part of the team’s process they’re trying to measure). | How long certain portions of the process might take; therefore, giving insight into where work gets stuck in the team’s process and how to improve flow. Both lead time and cycle time help teams understand how long work takes to flow through their workflows, but they measure different segments of the process. While lead time measures the total time a work item spends in the system, cycle time measures how long it takes a work item to get from point A to point B. | Depending on the economic value teams choose to measure, both cycle time and lead time can be applied directly to the team. For example, if the team wants to improve the delivery capabilities of the software development team, cycle time measurement can track the time it takes a work item to go from the commitment point to deployment. Measuring cycle time is a great place to start because it provides actionable insights no matter how small the data set. Measuring cycle time is an efficient and flexible way to improve a team’s processes because the results of changes are discernable almost immediately, allowing them to make any further adjustments right away. The end goal is to have a consistent and short cycle time, regardless of the type of work (new feature, technical debt, etc.). |
Lead Time
| What It Is | What It Measures | What It Tells Us | Why It Matters to ARTs |
|---|---|---|---|
| Lead time measures the total time it takes for work to move through the value stream, from the moment the work is requested to the time it’s delivered. Lead time measures duration from beginning to end. This includes process time, as well as time that work spends sitting in queues, or wait states. | The time it takes for a work item to move through the entire process, from start to finish. In Kanban— the time a card is created to the time moves into the “Done” column (reported on lead time reports, cycle time reports). | The perceived time that a piece of work took in the eyes of the customer (internal or external). | By tracking lead time metrics over a set period of time, teams can determine the impact of any changes you make – whether they help deliver value faster or get in the way of delivering value to customers. Lead time helps teams become more predictable by quantifying the probability of the percentage of work that will get done in a particular time frame. |
Cumulative Flow
| What It Is | What It Measures | What It Tells Us | Why It Matters to ARTs |
|---|---|---|---|
| A cumulative flow diagram (CFD) is a graphical representation of the team’s work in process (WIP) as it flows through the Kanban system. | In a cumulative flow diagram, each band of color represents a lane on the board, and the width (vertically) of that lane represents the number of cards in that lane. When plotted over time (the horizontal axis), team can see how the WIP in that lane changes. When WIP is decreasing, the lane narrows; however, when WIP is increasing, it starts to “bulge.” Drawing a vertical line from the edge of the “done” lane to the edge of the topmost lane will give teams the total WIP at that point in time. | A CFD provides quick, visual insight into the overall WIP in a system and how that WIP is flowing through the system. It can be used to calculate burn-up trajectories for work, as well as to provide a quick visual identification of where process bottlenecks may be forming. The metric helps teams understand the state of current work and what might need to be done to speed up the pace of delivery. | A cumulative flow diagram helps teams ensure that the flow of work across the team (or team of teams) is consistent. It makes it easy to identify bottlenecks, especially when used in conjunction with WIP limits. |
Advancing Beyond Factory Metrics
Both Lean and Agile metrics are tools meant to quantify team performance and provide actionable insights for how to improve. But of course, just as a hammer can be used to build a treehouse or destroy a glass vase, these tools have to be used intentionally and correctly in order to achieve their intended purposes.
Before diving into collecting data and implementing changes, it’s important for teams to understand why these metrics matter, and how they can help drive continuous improvement. In traditional corporate cultures, metrics have been and are often still used to closely monitor team performance in ways that are detrimental to true innovation.
It’s reasonable to assume that there will be people on every team who will be resistant to the idea of collecting, analyzing, and discussing Lean and Agile metrics. It’s important to reinforce the idea that these metrics are only meant for teams (and teams of teams) to improve their own performance, because this is the context in which they are truly useful.
It’s also important to not let the metrics themselves become the goal: The goal is to consistently deliver value, while increasing the predictability and health of the system. The goal is not to have the fastest cycle time or most accurate sprint burndown – these are the means to the end. But if the end goal is unclear, then these “achievements” are in vain.
In other words, it’s not just about doing things – it’s about doing the right things. All Lean and Agile metrics should be rooted in the product roadmap. For each initiative on the roadmap, there should be several key performance indicators (KPIs) that map to the program’s goals.
There must also be success criteria for each requirement, such as adoption rate by end-users or percentage of code covered by automated tests. These success criteria feed into the program’s Agile metrics. The more teams learn, the better they can adapt and evolve.
The first step is to define desired outcomes (OKRs) and select the right metrics to measure progress toward those goals. The next step is to make it easy for teams to collect and analyze that data.
Lean and Agile metrics should be reviewed frequently and discussed often. Over time, this data will accumulate, and teams will be able to use it not only improve upon past performance, but more accurately predict how changes now will impact future performance.
