What Great Engineering Teams Measure—and What They Don’t

Measuring developer productivity has never been more critical—or more complicated. As engineering teams are asked to "do more with less," it's essential to evaluate how productivity is defined, tracked, and improved. That was the core focus of a recent roundtable webinar featuring the following engineering leaders from Daisy Health, Keebo, and formerly Smartbear, as well as Ambassador, who hosted the talk. The discussion provided valuable insights into the evolving definition of productivity and the real-world metrics that matter most. It also revealed just how much variety there is in the nature and number of metrics and how they’re captured. Factors that seemed to inform the preferred analytics approach most include industry, organization size, business goals, and leadership styles. Despite the overall variation in approach, we did identify some key metrics that developer teams will want to consider using and applying immediately and a perspective on the impact of AI on the future of DevProd metrics. Out with the Old: Why Traditional Metrics of LoC & Velocity Miss the Mark Many engineering leaders have grown increasingly skeptical of “legacy” productivity metrics, which often fail to capture the complexity and nuance of modern software development. One commonly criticized metric was lines of code (LoC). Lou Sacco, CTO at Daisy Health, shared an anecdote about a manager proudly reporting 200,000 LoC in a quarter—only to later realize much of it was autogenerated and possibly duplicated. Our panelists agreed that more code does not equate to more value. In fact, the ability to remove or simplify code is often a hallmark of great engineering. “The main takeaway for me is that productivity isn't about counting keystrokes, counting lines of code, or velocity. It's about removing friction. And I really try to harken back a focus on the interactions and people over processes and tools,” shared Sacco. “Do the things that help promote trust and keep teams happy.” Similarly, velocity came under scrutiny. While it's a common metric that many engineering leaders have used in the past, it’s no longer making the mark. Mary Moore Simmons, vice president of engineering at Keebo, shared, “While velocity can be helpful for sprint planning, it’s not a good measure of individual or team productivity. Instead, I think it’s important to focus on developer pain points and reducing friction instead of simply chasing story points.” What about everyone’s favorite, the DORA framework? Even widely adopted DORA metrics like deployment frequency and mean time to recovery (MTTR), have also come into question. Raleigh Schickel, previous director of engineering at Nirvana, pointed out that these can be gamed easily, leading to artificial improvements without delivering real business value. For example, splitting a unit test into 15 small commits might look good on paper but does little for overall progress. Ok, so what ARE the meaningful measurements then? Consider Cloud Costs, Predictability, Failure Lead Time, & Merge/PR Review Time Rather than rely on shallow indicators, the panel advocated for a more nuanced, context-aware approach to measurement. “One of my favorites to track is cloud costs, using them as a proxy for engineering efficiency and ROI,” shared Moore Simmons. “Knowing how much it costs to run and scale your applications is just as important as tracking how quickly you deliver them.” Schickel introduced the idea of predictability as a hybrid metric combining planning and execution. By comparing planned work to completed work each sprint, teams can identify where coordination or estimation is breaking down. “It’s one of my favorite ones that's hard to pin down as just ‘one metric,’ but predictability is your strongest indicator. It's not about assigning blame but about fostering accountability and improving delivery accuracy.” Sacco focused on merge and PR review times as signals of team health. He noted, “if pull requests sit idle for too long, it might indicate blockers, poor collaboration, or lack of engagement. These metrics, while operational, can reveal deeper insights into team dynamics.” For Ambassador, Kenn Hussey proposed a forward-thinking concept that he favors and that might be new to other developer leaders. “Don’t forget failure lead time. I like to think of this as the time it takes to discover a problem after it's introduced. By identifying bugs or misalignments earlier in the development lifecycle, teams can drastically reduce the cost and complexity of fixes.” Tracking Without Micromanaging “Don’t forget, engineering IS a team sport,” shared Moore Simmons. One of the most frustrating things that arises for engineering teams when discussing developer productivity metrics is that developers hate being micromanaged, and oftentimes being tied to a few specific numbers can make that pressure feel exponential. However, leaders need to have metrics to measure

Apr 16, 2025 - 07:28

What Great Engineering Teams Measure—and What They Don’t

Measuring developer productivity has never been more critical—or more complicated. As engineering teams are asked to "do more with less," it's essential to evaluate how productivity is defined, tracked, and improved. That was the core focus of a recent roundtable webinar featuring the following engineering leaders from Daisy Health, Keebo, and formerly Smartbear, as well as Ambassador, who hosted the talk.

The discussion provided valuable insights into the evolving definition of productivity and the real-world metrics that matter most. It also revealed just how much variety there is in the nature and number of metrics and how they’re captured. Factors that seemed to inform the preferred analytics approach most include industry, organization size, business goals, and leadership styles.

Despite the overall variation in approach, we did identify some key metrics that developer teams will want to consider using and applying immediately and a perspective on the impact of AI on the future of DevProd metrics.

Out with the Old: Why Traditional Metrics of LoC & Velocity Miss the Mark

Many engineering leaders have grown increasingly skeptical of “legacy” productivity metrics, which often fail to capture the complexity and nuance of modern software development. One commonly criticized metric was lines of code (LoC). Lou Sacco, CTO at Daisy Health, shared an anecdote about a manager proudly reporting 200,000 LoC in a quarter—only to later realize much of it was autogenerated and possibly duplicated. Our panelists agreed that more code does not equate to more value. In fact, the ability to remove or simplify code is often a hallmark of great engineering.

“The main takeaway for me is that productivity isn't about counting keystrokes, counting lines of code, or velocity. It's about removing friction. And I really try to harken back a focus on the interactions and people over processes and tools,” shared Sacco. “Do the things that help promote trust and keep teams happy.”

Similarly, velocity came under scrutiny. While it's a common metric that many engineering leaders have used in the past, it’s no longer making the mark.

Mary Moore Simmons, vice president of engineering at Keebo, shared, “While velocity can be helpful for sprint planning, it’s not a good measure of individual or team productivity. Instead, I think it’s important to focus on developer pain points and reducing friction instead of simply chasing story points.”

What about everyone’s favorite, the DORA framework? Even widely adopted DORA metrics like deployment frequency and mean time to recovery (MTTR), have also come into question. Raleigh Schickel, previous director of engineering at Nirvana, pointed out that these can be gamed easily, leading to artificial improvements without delivering real business value. For example, splitting a unit test into 15 small commits might look good on paper but does little for overall progress.

Ok, so what ARE the meaningful measurements then?

**Consider Cloud Costs, Predictability, Failure Lead Time, & Merge/PR Review Time
**Rather than rely on shallow indicators, the panel advocated for a more nuanced, context-aware approach to measurement.

“One of my favorites to track is cloud costs, using them as a proxy for engineering efficiency and ROI,” shared Moore Simmons. “Knowing how much it costs to run and scale your applications is just as important as tracking how quickly you deliver them.”

Schickel introduced the idea of predictability as a hybrid metric combining planning and execution. By comparing planned work to completed work each sprint, teams can identify where coordination or estimation is breaking down.

“It’s one of my favorite ones that's hard to pin down as just ‘one metric,’ but predictability is your strongest indicator. It's not about assigning blame but about fostering accountability and improving delivery accuracy.”

Sacco focused on merge and PR review times as signals of team health. He noted, “if pull requests sit idle for too long, it might indicate blockers, poor collaboration, or lack of engagement. These metrics, while operational, can reveal deeper insights into team dynamics.”

For Ambassador, Kenn Hussey proposed a forward-thinking concept that he favors and that might be new to other developer leaders.

“Don’t forget failure lead time. I like to think of this as the time it takes to discover a problem after it's introduced. By identifying bugs or misalignments earlier in the development lifecycle, teams can drastically reduce the cost and complexity of fixes.”

Tracking Without Micromanaging

“Don’t forget, engineering IS a team sport,” shared Moore Simmons.

One of the most frustrating things that arises for engineering teams when discussing developer productivity metrics is that developers hate being micromanaged, and oftentimes being tied to a few specific numbers can make that pressure feel exponential. However, leaders need to have metrics to measure against, so there needs to be a happy balance. Answering the age-old question of how to track performance without micromanaging is key.

To collect useful metrics without veering into micromanagement, Moore Simmons emphasized the importance of viewing engineering as a team sport. Rather than evaluate individuals, she tracks metrics at the team level, celebrating collective wins and sharing responsibility for setbacks.

“Also, it’s worth noting that focusing on individual metrics can discourage senior engineers from mentoring junior teammates. Instead, I think engineering leaders should foster environments that reward collaboration, not competition. Metrics should be used to diagnose and improve systems, not to punish developers,” shared Shickel.

Hussey also made the great point that the best metrics are those that naturally emerge from the work itself.

“If your team constantly updates ticket statuses or inflates metrics for the sake of reporting, you're missing the point. Productivity measurement should be a byproduct of healthy workflows, not a burden,” shared Hussey.

So what does a top-performing engineering team look like? It’s not just the star metrics, it’s a couple of other key components that tug on the developer experience side of things. Our four leaders concluded that the common traits of high-performing engineering teams include:

Predictability: As we mentioned earlier, consider making predictability a metric on your spreadsheet. Teams that consistently deliver on their commitments are a great indicator of long-term success.
Autonomy: Aim to empower your engineers to make decisions and solve problems without micromanagement or the need to consult the higher-ups.
Balance: Avoid overwork or "heroic efforts" that signal deeper process issues. If your developers are working overtime or on the weekends to make a release happen or fix a bug, it’s not always a good thing, nor is it a sign of your team’s health.
Growth Mindset: **Ensure you’ve crafted a culture of continuous learning and constructive feedback. There is room for this to happen organically when you’re not having your developers work egregiously overtime or trying to perform some of those heroic efforts mentioned above.
**Happiness: Not just a feel-good metric—happy teams tend to be more creative, resilient, and productive. You can also think of this as your “developer experience” category. And if you’re doing it right, this should be an outcome that results from nailing the rest of these components correctly.
“I get that tracking developer happiness and satisfaction may seem intangible, but it’s crucial for long-term success. Teams that are constantly in "crisis mode" can’t reflect, learn, or grow,” shares Moore Simmons.

Sacco also added to that saying, “Try to reduce the reliance on so-called ‘superhero engineers,’ who burn themselves out to meet deadlines. In the end, it won’t help you and it won’t help them succeed long term.”

AI on DevProd Metrics: Hype or Help?

This conversation wouldn’t have been possible with the AI elephant in the room and no modern discussion of engineering productivity would be complete without a mention of how AI is going to impact developer productivity metrics moving forward. Though, like any good panel–opinions varied widely.

“I’m a little more of a skeptic when it comes to AI, once a month, I’ll try to use it for something and I can cite numerous examples where AI-generated code didn’t function as expected and required extensive rework. As for where it stands right now, I believe human validation remains essential,” shared Shickel.

On the other hand, Sacco and Moore Simmons see tremendous potential. At Daisy Health, Sacco has introduced tools like CodeRabbit AI for automated code reviews and GitStream for auto-merging safe changes.

“I’m more of an optimist when it comes to AI. These tools reduce context switching and free up developers to focus on high-value tasks. As a startup it definitely helps us streamline,” shared Sacco.

Moore Simmons compared AI's potential to the shift from paper-based to digital workflows: it doesn’t eliminate jobs, but it changes what the job looks like and allows for greater output and innovation to happen so that developers can focus on the more interesting parts of their jobs.

Hussey noted that is one such area where positive AI enhancement is happening. Without AI, API developer teams are left to manually mock, create documentation, and debug API errors. With AI, API development becomes a streamlined and efficient process. Tools like , for example, combine the power of AI with expertise in and tools to offer a cloud and CLI-accessible platform that simplifies and accelerates API development.

Overall the consensus was that AI can and will greatly accelerate lower-level tasks like , but core skills like debugging, architecting, and critical thinking remain irreplaceable (for now at least). In fact, as AI becomes more common, these skills will be even more essential to keeping our collective knowledge as a developer society where it needs to be.

However, back to the metrics side of things, as AI continues to evolve the definition of productivity will likely shift from activity-based metrics to outcome-based metrics. Instead of asking how fast code is shipped, leaders will ask:

Are we building the right features?
Are our systems resilient, scalable, and secure?
Are our engineers happy and engaged?
Are we achieving measurable business impact?
And as for outdated metrics like lines of code, with the addition of AI, those become even more obsolete of an indicator than they already are.

DevProd Metrics Should Guide Alignment

All panelists agreed that the future of engineering leadership lies in bridging the gap between technical execution and business value. The most important takeaway is that your developer productivity metrics should guide this alignment, not distract from it.

While legacy metrics like LoC and velocity may still have a place for now, they must be supplemented—or replaced—by more meaningful measures that reflect real value. Engineering leaders are embracing a more holistic approach, one that balances speed with sustainability, autonomy with accountability, and innovation with impact.

Ultimately, productivity isn't about doing more in a vacuum. It's about doing what matters most—together, effectively, and with purpose