Summary of the Book

This book by Forsgren, Humble and Kim describes and summarizes the results and conclusions of four years of research by the authors in an attempt to find a way to evaluate modern software delivery performance of organizations by using measurable indicators. It asks an important question - how can managers measure the performance of their teams objectively? This is an important skill because only once we can evaluate performance in a trustworthy manner can we then determine whether changes to our habits and processes are having a positive effect. The book is divided into three parts:

1. What We Found - the teams' conclusions.
2. The Research - a description of the research methods.
3. Transformation - a guide on how to transform your organization based on the findings.

Although the research methods section is important and interesting, the first and third sections are what have provided me with the most lasting value.

Book Findings - Key Takeaways

The overarching theme of the conclusions is that organizations with high software delivery performance have measurably higher overall organizational performance. In other words, it affects the bottom line of the business. The next subsections describe, in bullet form, what I consider the most valuable takeaways from this book.

Key Takeaway 1 - How to Measure Performance

• Previous industry attempts to measure performance have been flawed.
  • Velocity as a performance metric is flawed as teams will inevitably game their velocity and teams operate under different contexts and have different estimation patterns and therefore cannot be compared directly.
  • Utilization as a performance metric is flawed, as utilization approaches 100%, delivery lead times actually drop as there is no capacity to absorb unplanned work.
• Good measurements should focus on outcomes rather than outputs.
• Outcomes of focus should be global across teams and roles.
  • Antithesis: rewarding developers for speed/quantity while rewarding quality analysts for stability leads to developers throwing poor code "over the wall" and QA teams to develop painful change management processes that stymie change.
• Authors settle on 4 metrics, two that measure tempo, and two that measure stability:
  • Lead Time - Time between commitment to deliver an item and its delivery.
  • Deployment Frequency - This is a proxy to the measurement of "batch size" from traditional physical product delivery research which found that producing smaller batch sizes reduced cycle times, accelerating feedback and reducing risk.
  • Mean Time to Restore - Time between identification of a production incident and restoration of service.
  • Change Fail Percentage - Percentage of changes delivered to customers that result in a production incident.
• In modern systems, failure is inevitable, so the real question becomes how fast service can be restored.

Key Takeaway 2 - Architecture

• Poor architecture of software and services can be a barrier to increasing both tempo and stability.
• High performance is possible with all types of systems (web-based, mainframe, firmware etc.) so long as the systems are loosely coupled.
• Loose coupling enables teams to test easily and deploy individual components quickly.
• If architecture is loosely coupled, it means:
  • Components can be changed and validated independently of one another.
  • Changes can be made to components without necessitating changes to others.
  • Testing on any given component can be done without requiring an integrated environment.
  • Deployment of components can happen independently of the other components they depend on.
• To summarize, loose coupling means that change produces the minimum possible downstream effects.

Key Takeaway 3 - "Shift Left" on Security and Quality

• Many developers are ignorant of common security risks (not a disparagement, just a fact).
• Security and quality checks should be built in to all phases of the development lifecycle, not a separate activity downstream of development.
• Two main aspects of "shift left":
  • 1. Security and quality reviews should be conducted for ALL features and ALL changes.
  • 2. Security and quality should be integrated into the entire software delivery lifecycle, and responsibility shared equally amongst all departments and roles.
• "Making sure the people building the solutions are doing the right thing is easier than checking at the end and delaying deployment for security and compliance activities".

Key Takeaway 4 - Deployment Pain

• There is a high correlation between deployment pain and key outcomes (more pain is correlated to hitting fewer outcomes).
• If deployments need to be performed outside normal business hours, it is a sign of poor architecture.
• It is entirely possible, given the proper investment, to build large scale enterprise systems with fully automated deployments and zero downtime.
• Most deployment problems are caused by a brittle, complex deployment process.
• Probability of deployment failures rises dramatically with each manual change required in the target environment as part of the deployment process.
• Manual configuration leads to configuration drift (deviations in config) between environments.
• To reduce deployment pain:
  • Build systems designed to be easily deployable into multiple environments.
  • Detect and tolerate failures in environments (beware unsafe code!).
  • Ensure the state of the production system (except its data) can be automatically reproduced entirely with the information available in version control.
  • Make the deployment process as simple as possible.

Key Takeaway 5 - Measuring and Changing Culture

• It is possible to quantify and measure culture using Likert-type questions.
• Healthier cultures are correlated to higher software delivery performance.
• Organizations exist on a continuum that defines three identifiable phases:
  • Pathological - Organizations characterized by high levels of fear and threat. People hoard information or withhold it for political reasons, or distort it to make themselves look better. Individuals "step over others" to succeed.
  • Bureaucratic - Organizations in which departments protect themselves. Departments may share information internally but try to maintain their "turf" and insist on their own rules and ways of doing things.
  • Generative - Organizations that focus on the mission. Everything is second to good organizational performance. Information is shared freely. Time is made for innovation. Failure is not only tolerated but encouraged. Individuals "lift each other up" to succeed.

Final Thoughts - How These Ideas Track with My Experience

1 - Measuring and Incentivizing Performance

The biggest lesson I have learned is that quality must be the number one focus of any leader, and sometimes that might mean making hard decisions and telling those above yourself in the org chart "no". It can even mean denying incentives such as time-based delivery bonuses (another lesson I've learned is that time based incentives are a mistake). It can be difficult at times not to be distracted from the mission of quality. Customers and initiative sponsors want results and get impatient, set deadlines and pressure teams to produce results as quickly as possible. Although it is important for teams to produce quantity (we need to maximize value to our customers, beat competitors to market etc.), in my experience it is vastly more important to produce quality. If something is late but works, that is always better received than something on time but broken. I've learned that lesson the hard way and will now always use the quality/stability metrics of Mean Time to Restore and Change Fail Percentage as my north star.

2 - Loosely Coupled Architecture

I've always been interested in unit testing and producing the smallest, atomic units of code possible so that they can be easily and properly tested. After reading this book I see the connection between that and loosely coupled architecture. I've still never personally seen a "mature" codebase that was perfectly loosely coupled. In fact, most architectures I've seen contain terribly tightly coupled components and systems. I have no data to back it up, but I've always wondered if the reason most large systems are in such poor shape is that the "winners" that become large products are those that were able to get to market first. From what I've seen, product incubation happens in an environment where there hasn't yet been much investment and the proper resources to ensure quality do not exist. A prototype is built as quickly as possible and often doesn't need to scale to impress the first few customers. Only later when products take off and companies scale is the underlying technical debt noticed. Just my personal theory. But one fact I can confirm is that most codebases contain tightly coupled elements and it's our job as effective leaders to identify where we can loosen this coupling (and of course always build new initiatives out with loose coupling in mind).

3 - Shift Left

I've seen the "shift-left" transformation in a product company first hand. I am grateful to have had the opportunity to see it happen and to have had the support needed to do my part as a leader during that change. During my time at that company I saw us go from taking 9 months to deliver a particular release, to having the ability to release multiple times per week. A large part of that was reforming an archaic quality control process by "shifting left" on quality. In practice this meant a much higher level of collaboration between developers and quality analysts. I'm excited to see where I can apply these lessons in the future to become the primary driver of such a transformation.

4 - Deployment Pain

One of the first "side of the desk" initiatives I ever undertook was during my time in consulting and was designed to reduce deployment pain as we were experiencing it regularly. I had just assumed deployments always took until midnight and that manual steps were inevitable to at least some degree. I was working at a consulting company that had both developers and what we called "Cloud Technologists" (CTs). At this company, changes made by developers and CTs were rarely discussed during the course of the project, and when change sets were integrated and attempted to be deployed it always led to issues. One of the main issues was that configuration changes made by the CTs would break the automated tests created by the developers to test their code. One of the goals of my initiative was to create a testing framework that would be more resilient to such configuration changes. Thinking in terms of the ideas from this book, this was an example of one of the prescriptions to reduce deployment pain - detecting and tolerating failures in environments. Another one of the prescriptions for reducing deployment pain - keeping all information needed to reproduce environments in version control - is exemplified in the next section as well.

5 - Measuring and Changing Culture

Again thinking back to my days in consulting, I remember that my CT colleagues were responsible for delivering configuration changes. These configuration changes were able to be captured and stored as XML metadata. At the time I remember proposing that we start to capture all of those changes in the source control and deploy them automatically, rather than having CTs separately document and make these configuration changes manually on every deployment, as they were currently doing. I received pushback because "CTs don't want to learn git". I accepted that uncritically at the time, thinking "they do their things their way, I do my things my way" - a perfect example of Bureaucratic culture. I wasn't mature enough at the time to understand that my idea was good, and that it was in the organization's interest to implement this change. Armed with the concepts from this book I would build my case a lot more strongly in a similar situation in the future, communicating my idea with empathy, while tying my reasoning back to our shared goals. Making that change would have been an example of Generative culture. These are ideas I now keep close at hand.