New Ebook: Pacific Express

Recently, after working literally day and night, I finished a short ebook - "Pacific Express" - a novella about adopting Lean and Agile at Scale, Systems Thinking and... courage that those change agents in the field demonstrate everyday in order to succeed with the magic of software engineering at large scale.

Why magic? Well, it's not really magic, but it takes a whole different mindset to be able to break the bondage with the ways of working that were developed in the past but still prevail across the industry. It takes a whole different paradigm in thinking about software development process, as even those organizations that managed to adopt agility at the team level, have often nothing to operate with at the higher levels of the organization. But guess what - that is the level where most interesting things occur and we're about to explore it on a fictional example of an organization that grew too fast to sustain any decent level of productivity and starts stagnating in the core area of the business. This light read will guide us through the transformation processes they chose, their challenges and discoveries down that path, that may lead them to better performance in delivering value to the business.

Oh, did I say "fictional"? Although the organization and all characters are fictional, we will be witnessing facts and situations that occurred in real life, in the organizations that the author worked with in the past.

Enjoy!

Why SAFe Has Hardening



By Alex Yakyma.

We used to consider hardening effort as an extremely controversial topic. It’s been discussed multiple times in the community, never without emotions. At times the arguments can feel like full-fledged religious warfare. In SAFe, hardening is one of the three components of the HIP Sprint that is placed in the end of the PSI, and stands for Hardening, Innovation and Planning. 

The term suggests that this sprint reserves capacity for PSI planning itself and planning preparation, as well as placeholders for hackathons,  research activities for the next PSI, and, finally, the room for completing any work that was not covered during the regular sprints. Now, take a deep breath and hold your arguments for a little bit longer, as we are going on a journey that will help us understand the multiple misconceptions and misinterpretations with respect to the hardening.  If you bear with me through the rest of the article, a lot of things will become obvious and totally logical.

The most “troublesome” aspect of hardening to many is that hardening sounds overly waterfall’ish, representing an additional phase in the process covering a bunch of aspects that weren’t tackled earlier. I will surprise you with the answer to that - yes, indeed, hardening stands out as a separate timeframe that is not inherently agile. And yet, hundreds of SAFe consultants (SPCs) in the field and their numerous clients adopt hardening, a knowingly non-agile construct. Why? For a number of reasons.

First, SAFe is adopted by a number of large Fortune 500 organizations that develop both software and hardware. It may indeed be hard to do thorough testing of the navigation system on  an actual vehicle or a new air conditioning configuration system in a real building in every sprint. In this case, we fully utilize Lean Thinking and the concept of batch/lot size optimization and a balance between transaction cost (mainly reflecting different levels and aspects of system integration and testing) AND holding costs (which mostly is the growing complexity of un-validated increment of software). And in many cases, the optimum batch size for concerns like end-to-end testing on the target hardware environment turns out to be more than two weeks.

Second, somewhere in Narnia there lives a lonely agile team consisting of seven dwarfs and a Scrum Master. They develop their own product themselves. Most likely, they can move full steam ahead in a matter of two week sprints that include all aspects of software engineering. Well, the usual reality of software engineering at scale resembles more of Mordor, than Narnia. It involves tens or even hundreds of teams, millions of lines of code, and brittle product architecture that’s been around for 12 years already, because the first version was written by the company’s CEO himself. Wouldn't it be nice if they could cover all the steps in the value stream every two weeks? Yes! It would be awesome. But can they really do that at this point? Even if we deal with pure software (no hardware involved),  many aspects would absolutely disrupt sprinting. What SAFe strongly recommends is to “choose the blue pill” and get back to reality, no matter how dark and ugly it may seem. Hardening is very important, and absolutely needs to be called out explicitly, otherwise there is absolutely no chance to reduce it or even get rid of it in the future, even in the cases where removing it is feasible. Taiichi Ohno was always emphasizing the importance of clear goal setting as no improvement is possible otherwise. The goal can be even overly aggressive, according to Ohno -  that's how Toyota managed to implement such extreme practices as SMED - but it needs to be explicit and aware of the current state. That's why hardening matters. In many cases it's a temporary necessity that is absolutely critical to purposeful, continuous improvement. This perspective of hardening allows agile release trains to clearly separate definition of done (DoD) for stories on one hand and from that for the features, on the other hand. The whole matter of reducing or even fully eliminating hardening can then be explicitly interpreted as a gradual process of bridging the gap between these two DoDs.

Next, every once per PSI they have hardening activities... Well, here comes the limitations of the visual representation of knowledge and the overall desire to keep things simple as much as possible. Not always at zero cost. We place hardening effort at the end of the PSI on the Big Picture, and the initial suggestion would be to do exactly so in the beginning. But it is not necessarily the end state. As program matures over time, its holding cost and transaction cost curves change their shape. Normally, transaction cost curve flattens (which is the evidence of improved program's efficiency in integration, testing, tech writing, UAT, etc). As a result, the optimum batch size may become smaller. In such cases, we strongly encourage programs to split the hardening effort into more than one occurrence in the PSI. So, for instance, they may have one hardening chunk as a second week of sprint three and a second week of sprint 5, in a five-sprint PSI. Quite possibly, but not necessarily, their next step after a couple of PSIs would be splitting it even further and thus basically reach the point where hardening effort gets fully distributed across every sprint in the PSI. In other words, story DoD becomes equivalent to feature DoD. Overall, it is totally logical that different aspects of hardening may take a different optimum batch size. So, for example, full system integration could and should eventually be performed in every sprint (however it can be a real stretch for most organizations in the beginning). Tech writing and releasing to production – every two sprints, for instance. Full UAT and training internal users – once per PSI.

The whole unnecessary, emotional load that is so often associated with the word “hardening” fully disappears as soon as we adopt a systems thinking view, of which Lean Thinking is a great example that we use throughout the framework. As we stop being religious about agile and look at a broader picture and a full value stream, things begin to gradually add up. Achieving continuous flow of value at scale indeed requires additional constructs, like hardening effort. It may not seem highly desirable to have one, but it reflects the reality of many organizations and gives them a solid chance of substantially improving their practices and techniques in the future. Also, as we learned, hardening comprises a whole bunch of distinct aspects, of which some may be fully embraced by agile teams as a regular sprint routine --some can be done more frequently than once per PSI, and some may still require lower frequency. We see hardening as important tool, a result of “systems thinking” rather than the final goal itself.

Agile Engineering Assessment at Scale



By Alex Yakyma. 

I had the intention of putting together a consistent assessment of such kind for quite a while. Some topics were part of other assessments, others were just assumed.  Recently though, as I finished working on DevOps article for SAFe, I have decided to finalize this assessment as well. Especially because I could then add a good fresh summary related to the deployment process. So, here it is...


The purpose is to assess program capabilities in the following key engineering/technical areas: Architecture/Design, System Integration, Acceptance Testing, NFR Testing and Deployment. 
It makes sense to use it actually in both program and team environments. Programs need to understand where they are with respect to engineering concerns. But teams actually employ the practices and keep the work flowing, so they need to carefully track where they are as well. It is also useful to look at this assessment as a set of improvement objectives and incrementally work towards accomplishing them. 

Improving Enterprise Performance with Value Threads


By Alex Yakyma

Value Stream is one of the key concepts of Lean|Agile development at scale. A value stream is a set of steps, starting from the definition of the idea of a new functionality all the way through to the implementation and deployment to satisfy user and business goals. Another aspect of a value stream is people – the teams that contribute to the end user value. Thus, a value stream is most often the set of steps (and a number of teams) required to develop or maintain a software solution. More often than not, value streams consist of multiple collaborating agile teams.
Let’s start by summing up the key reasons why we care at all about value streams and value stream analysis. As we know from experience, a value stream perspective allows an organization to effectively focus on:
1.       Improving the process as a whole, thereby genuinely improving it.
2.       Continuously improving the average cycle time.
3.       Enhancing quality by building it into every process step.
4.       Actively creating, sharing and evolving relevant knowledge.
These are all meaningful goals that allow an organization to considerably enhance its operational performance. Surprisingly enough, however, value streams alone don’t let you improve these at scale and here’s why.
Let’s consider an agile program that works on a specific value stream, consisting of seven agile teams. The result of their work is an end-to-end solution – a large software system for which the teams advance the functionality and nonfunctional requirements, as required by the business. What can we say about such a value stream? Just as for any value stream at scale – not much. It consists of five process steps that repeat indefinitely over feature sets:
Figure 1. Value stream perspective

Figure 1 doesn’t say very much, primarily because we have tried to outline process commonalities for all seven teams and those are pretty generic and therefore, not overly useful. The process is much more complex inside and requires much more subtle treatment. In fact, such a rough perspective (as seen in figure 1) can be largely harmful to the program for the following reasons:
·         Applying Work-In-Process (WIP) limits on the entire program in a “wholesale” manner may unbalance the flow instead of stabilizing it
·         Some teams may have their unique, context-dependent steps in the process which substantially impact their performance but are not visible as a part of the value stream.
·         The specific ways in which the teams impact one another in the program usually has a dramatic impact on the overall program performance and requires special treatment.
In order to explore the internal structure of a value stream at scale, we need to take a microscopic view. What seems to be a flat surface to the naked eye may turn out to be an entire new world with its own ridges, peaks, plateaus and trenches. We will start our journey by defining the ground rules. We will be zooming in for a closer look at a value stream, but only insofar as we don’t lose the features (chunks of an end-to-end user value) from sight.
The reality of most programs suggests that the teams collaboratively create user value and that the patterns of such collaboration are context dependent (See figure 2).  
Figure 2. A value stream consisting of a number of value threads
So, features 1 and 2 require three teams to work together, feature 3 involves four teams, and feature 4 requires two teams’ input. The teams that are connected with a same color link form up a Value Threada subset of a program teams that are able to deliver a specific feature or a feature set. Value threads can be as small as one team (in case of feature teams, for example) or can be as big as the entire program (often, when it consists of component teams).
Most interactions occur within the value threads. Thus, in order to be able to improve the process at scale, we redirect our focus from an overly abstract value stream to significantly more detailed value threads. We would like to see value being created in a fast, synchronous manner within each value thread, as Figure 3 suggests:
Figure 3. Synchronous work on a feature within a value thread
Now it is obvious that the following Lean|Agile techniques make a lot of sense primarily within a value thread:
  • Applying WIP limits.
  • Backlog grooming, planning, tracking progress and demonstrating the results.
  • Measuring the feature cycle time and cumulative flow.
  • Creating and sharing knowledge.
  • Advancing in test automation, and collective code ownership
  • Collectively retrospecting and improving the process.
It is unarguably better to move towards feature oriented agile teams, where a team actually is a full value thread. In reality, however, this is far from always possible or may require a huge amount of time and effort. Uncontrollably bloated technology stacks, narrow specialization, geographical distribution, and organizational impediments – these can all present serious hurdles when moving towards feature teams. However, once value threads are made visible within an organization, it may considerably increase managerial support and willingness to make changes as they start to see the real source of existing inefficiencies.
Start by asking yourself several questions. What are the smaller constructs (value threads) within my program that are responsible for end-to-end value delivery? Are the teams aware of their peer teams, with whom they have to actively collaborate? How many sprints does it take to deliver an end-to-end chunk of functionality within such a thread? Do teams collaboratively plan and track progress within the thread? Now that you know what your target object is – you can help your organization improve. It will be a tough journey, but the prize is also worth it.
In future posts, we will explore different important aspects of the value threads and offer further practical advice in terms of applications.