Why SAFe Has Hardening

By Alex Yakyma.

We used to consider hardening effort as an extremely controversial topic. It’s been discussed multiple times in the community, never without emotions. At times the arguments can feel like full-fledged religious warfare. In SAFe, hardening is one of the three components of the HIP Sprint that is placed in the end of the PSI, and stands for Hardening, Innovation and Planning. 

The term suggests that this sprint reserves capacity for PSI planning itself and planning preparation, as well as placeholders for hackathons,  research activities for the next PSI, and, finally, the room for completing any work that was not covered during the regular sprints. Now, take a deep breath and hold your arguments for a little bit longer, as we are going on a journey that will help us understand the multiple misconceptions and misinterpretations with respect to the hardening.  If you bear with me through the rest of the article, a lot of things will become obvious and totally logical.

The most “troublesome” aspect of hardening to many is that hardening sounds overly waterfall’ish, representing an additional phase in the process covering a bunch of aspects that weren’t tackled earlier. I will surprise you with the answer to that - yes, indeed, hardening stands out as a separate timeframe that is not inherently agile. And yet, hundreds of SAFe consultants (SPCs) in the field and their numerous clients adopt hardening, a knowingly non-agile construct. Why? For a number of reasons.

First, SAFe is adopted by a number of large Fortune 500 organizations that develop both software and hardware. It may indeed be hard to do thorough testing of the navigation system on  an actual vehicle or a new air conditioning configuration system in a real building in every sprint. In this case, we fully utilize Lean Thinking and the concept of batch/lot size optimization and a balance between transaction cost (mainly reflecting different levels and aspects of system integration and testing) AND holding costs (which mostly is the growing complexity of un-validated increment of software). And in many cases, the optimum batch size for concerns like end-to-end testing on the target hardware environment turns out to be more than two weeks.

Second, somewhere in Narnia there lives a lonely agile team consisting of seven dwarfs and a Scrum Master. They develop their own product themselves. Most likely, they can move full steam ahead in a matter of two week sprints that include all aspects of software engineering. Well, the usual reality of software engineering at scale resembles more of Mordor, than Narnia. It involves tens or even hundreds of teams, millions of lines of code, and brittle product architecture that’s been around for 12 years already, because the first version was written by the company’s CEO himself. Wouldn't it be nice if they could cover all the steps in the value stream every two weeks? Yes! It would be awesome. But can they really do that at this point? Even if we deal with pure software (no hardware involved),  many aspects would absolutely disrupt sprinting. What SAFe strongly recommends is to “choose the blue pill” and get back to reality, no matter how dark and ugly it may seem. Hardening is very important, and absolutely needs to be called out explicitly, otherwise there is absolutely no chance to reduce it or even get rid of it in the future, even in the cases where removing it is feasible. Taiichi Ohno was always emphasizing the importance of clear goal setting as no improvement is possible otherwise. The goal can be even overly aggressive, according to Ohno -  that's how Toyota managed to implement such extreme practices as SMED - but it needs to be explicit and aware of the current state. That's why hardening matters. In many cases it's a temporary necessity that is absolutely critical to purposeful, continuous improvement. This perspective of hardening allows agile release trains to clearly separate definition of done (DoD) for stories on one hand and from that for the features, on the other hand. The whole matter of reducing or even fully eliminating hardening can then be explicitly interpreted as a gradual process of bridging the gap between these two DoDs.

Next, every once per PSI they have hardening activities... Well, here comes the limitations of the visual representation of knowledge and the overall desire to keep things simple as much as possible. Not always at zero cost. We place hardening effort at the end of the PSI on the Big Picture, and the initial suggestion would be to do exactly so in the beginning. But it is not necessarily the end state. As program matures over time, its holding cost and transaction cost curves change their shape. Normally, transaction cost curve flattens (which is the evidence of improved program's efficiency in integration, testing, tech writing, UAT, etc). As a result, the optimum batch size may become smaller. In such cases, we strongly encourage programs to split the hardening effort into more than one occurrence in the PSI. So, for instance, they may have one hardening chunk as a second week of sprint three and a second week of sprint 5, in a five-sprint PSI. Quite possibly, but not necessarily, their next step after a couple of PSIs would be splitting it even further and thus basically reach the point where hardening effort gets fully distributed across every sprint in the PSI. In other words, story DoD becomes equivalent to feature DoD. Overall, it is totally logical that different aspects of hardening may take a different optimum batch size. So, for example, full system integration could and should eventually be performed in every sprint (however it can be a real stretch for most organizations in the beginning). Tech writing and releasing to production – every two sprints, for instance. Full UAT and training internal users – once per PSI.

The whole unnecessary, emotional load that is so often associated with the word “hardening” fully disappears as soon as we adopt a systems thinking view, of which Lean Thinking is a great example that we use throughout the framework. As we stop being religious about agile and look at a broader picture and a full value stream, things begin to gradually add up. Achieving continuous flow of value at scale indeed requires additional constructs, like hardening effort. It may not seem highly desirable to have one, but it reflects the reality of many organizations and gives them a solid chance of substantially improving their practices and techniques in the future. Also, as we learned, hardening comprises a whole bunch of distinct aspects, of which some may be fully embraced by agile teams as a regular sprint routine --some can be done more frequently than once per PSI, and some may still require lower frequency. We see hardening as important tool, a result of “systems thinking” rather than the final goal itself.