Agile Engineering Assessment at Scale



By Alex Yakyma. 

I had the intention of putting together a consistent assessment of such kind for quite a while. Some topics were part of other assessments, others were just assumed.  Recently though, as I finished working on DevOps article for SAFe, I have decided to finalize this assessment as well. Especially because I could then add a good fresh summary related to the deployment process. So, here it is...


The purpose is to assess program capabilities in the following key engineering/technical areas: Architecture/Design, System Integration, Acceptance Testing, NFR Testing and Deployment. 
It makes sense to use it actually in both program and team environments. Programs need to understand where they are with respect to engineering concerns. But teams actually employ the practices and keep the work flowing, so they need to carefully track where they are as well. It is also useful to look at this assessment as a set of improvement objectives and incrementally work towards accomplishing them. 

Improving Enterprise Performance with Value Threads


By Alex Yakyma

Value Stream is one of the key concepts of Lean|Agile development at scale. A value stream is a set of steps, starting from the definition of the idea of a new functionality all the way through to the implementation and deployment to satisfy user and business goals. Another aspect of a value stream is people – the teams that contribute to the end user value. Thus, a value stream is most often the set of steps (and a number of teams) required to develop or maintain a software solution. More often than not, value streams consist of multiple collaborating agile teams.
Let’s start by summing up the key reasons why we care at all about value streams and value stream analysis. As we know from experience, a value stream perspective allows an organization to effectively focus on:
1.       Improving the process as a whole, thereby genuinely improving it.
2.       Continuously improving the average cycle time.
3.       Enhancing quality by building it into every process step.
4.       Actively creating, sharing and evolving relevant knowledge.
These are all meaningful goals that allow an organization to considerably enhance its operational performance. Surprisingly enough, however, value streams alone don’t let you improve these at scale and here’s why.
Let’s consider an agile program that works on a specific value stream, consisting of seven agile teams. The result of their work is an end-to-end solution – a large software system for which the teams advance the functionality and nonfunctional requirements, as required by the business. What can we say about such a value stream? Just as for any value stream at scale – not much. It consists of five process steps that repeat indefinitely over feature sets:
Figure 1. Value stream perspective

Figure 1 doesn’t say very much, primarily because we have tried to outline process commonalities for all seven teams and those are pretty generic and therefore, not overly useful. The process is much more complex inside and requires much more subtle treatment. In fact, such a rough perspective (as seen in figure 1) can be largely harmful to the program for the following reasons:
·         Applying Work-In-Process (WIP) limits on the entire program in a “wholesale” manner may unbalance the flow instead of stabilizing it
·         Some teams may have their unique, context-dependent steps in the process which substantially impact their performance but are not visible as a part of the value stream.
·         The specific ways in which the teams impact one another in the program usually has a dramatic impact on the overall program performance and requires special treatment.
In order to explore the internal structure of a value stream at scale, we need to take a microscopic view. What seems to be a flat surface to the naked eye may turn out to be an entire new world with its own ridges, peaks, plateaus and trenches. We will start our journey by defining the ground rules. We will be zooming in for a closer look at a value stream, but only insofar as we don’t lose the features (chunks of an end-to-end user value) from sight.
The reality of most programs suggests that the teams collaboratively create user value and that the patterns of such collaboration are context dependent (See figure 2).  
Figure 2. A value stream consisting of a number of value threads
So, features 1 and 2 require three teams to work together, feature 3 involves four teams, and feature 4 requires two teams’ input. The teams that are connected with a same color link form up a Value Thread – a subset of a program teams that are able to deliver a specific feature or a feature set. Value threads can be as small as one team (in case of feature teams, for example) or can be as big as the entire program (often, when it consists of component teams).
Most interactions occur within the value threads. Thus, in order to be able to improve the process at scale, we redirect our focus from an overly abstract value stream to significantly more detailed value threads. We would like to see value being created in a fast, synchronous manner within each value thread, as Figure 3 suggests:
Figure 3. Synchronous work on a feature within a value thread
Now it is obvious that the following Lean|Agile techniques make a lot of sense primarily within a value thread:
  • Applying WIP limits.
  • Backlog grooming, planning, tracking progress and demonstrating the results.
  • Measuring the feature cycle time and cumulative flow.
  • Creating and sharing knowledge.
  • Advancing in test automation, and collective code ownership
  • Collectively retrospecting and improving the process.
It is unarguably better to move towards feature oriented agile teams, where a team actually is a full value thread. In reality, however, this is far from always possible or may require a huge amount of time and effort. Uncontrollably bloated technology stacks, narrow specialization, geographical distribution, and organizational impediments – these can all present serious hurdles when moving towards feature teams. However, once value threads are made visible within an organization, it may considerably increase managerial support and willingness to make changes as they start to see the real source of existing inefficiencies.
Start by asking yourself several questions. What are the smaller constructs (value threads) within my program that are responsible for end-to-end value delivery? Are the teams aware of their peer teams, with whom they have to actively collaborate? How many sprints does it take to deliver an end-to-end chunk of functionality within such a thread? Do teams collaboratively plan and track progress within the thread? Now that you know what your target object is – you can help your organization improve. It will be a tough journey, but the prize is also worth it.
In future posts, we will explore different important aspects of the value threads and offer further practical advice in terms of applications.      

Legacy Mindsets in Test Automation

By Alex Yakyma. 

 In software development even these days test automation fails much more often than it succeeds. There are multiple reasons for this, but traditional thinking has a very strong impact in this regard – I’m referring to various legacy mindsets, originating from decades of the “divide and manage by function” pseudo-scientific set of beliefs that don’t withstand any practical validation, but unfortunately thousands of minds of otherwise smart people still adhere to such thinking. After one or two failed attempts, many organizations do not automate at all. Others do, but their process is not very effective even though automated test engineers manage to stay surprisingly busy. We will explore these false beliefs as well as working alternatives that evolved as a result of applying Lean thinking and Agile testing approaches inspired by Extreme Programming and its further evolution into areas such as BDD and ATDD. We will also be primarily concerned with principles that work at scale. 

Let’s consider these areas of concerns in more detail: 

Legacy Mindset #1: Automation should primarily be done by a separate automation team. 

This is one of the most severe issues with testing per se, creating endless list of problems, the most challenging of which are:
-          Developers never use these tests on a daily basis to prevent the propagation of un-validated code into the main code branch in the first place. This basically makes the feedback loop, which should be as short as possible, just the opposite, necessitating significant rework later when the tests are run. A lot of money gets wasted this way.
-          Developers and architects never bother with system testability and grow system design under completely different set of forces, making test automation an incredibly tough work.
-          Such tests never keep up with the latest requirements as developers (and possibly manual testers) move faster and are more flexible in responding to new requests and clarifications from the product owner and stakeholders.  

Agile team has to cover as many steps in the target value stream as possible. And even though something may be left outside their purview within iteration, this cannot be automated testing as it is indeed one of the key agile practices. If you cannot automate the tests for your own code, you are not a truly agile team. However, automation must be taken very seriously; many teams would be surprised at how much simpler automation actually is than they might have initially thought.
It is also important to note that the System Team can perform certain part of test automation within the program even in a very effective agile environment. However, if this is the only team creating automated tests and if the rest of the agile teams in the program do not automate, this just introduces equally poor waterfall-like behaviors with all the corresponding disadvantages, leading to a slowly moving, unreliable process. A good scenario entails just the opposite – when the system team actively collaborates with agile teams and aims to continuously give away more and more automation work to them while the system team themselves looks for the next systemic bottleneck for the program to cope with. This is part of the self-organization paradigm for the Agile Release Train as a team of teams capable of producing “the best architectures, requirements, and designs” at scale. 

Legacy Mindset #2: Automation is primarily done by specially trained automation engineers, using expensive automation tools. 

Companies waste hundreds of thousands of dollars on tools, and spend their time and effort hiring external people who can man them, essentially creating a hard dependency and largely ineffective process. These days, one can find very decent free tools with wide user communities, which are:
1.       Easier to use.
2.       Developed by agilists for agilists, so it’s no wonder it works for agile teams.
3.       Can be used by virtually anybody on the team.
Most unit testing frameworks are free for most platforms. Most BDD/ATDD tools are also free and work great for acceptance testing. Moreover, these tools are typically open source, allowing for customization whenever required. Using such tools helps to avoid multiple different bottlenecks that expensive commercial tools and employees with rare skills tend to introduce in the organization. 

Legacy Mindset #3: Testing via the User Interface. 

Do not test via the UI whenever it may be avoided. Always first look for alternatives and be ready to invest in those – it will tremendously pay off over time. Tests that operate via the UI are brittle and laborious both in terms of their creation and maintenance. Once again, a lot of money wasted. Even with such useful tools like Robot Framework or Cucumber (that leverage the re-use of “bindings”), their application to testing via the UI remains very fragile. Apart from very rare cases, following is the right thing:

Test via the programming API itself or simpler protocols such as XML or REST services, flat files, etc., but not UI. 

To enable this, the system under test should allow for testability. In very many cases, though, the system is not designed so. In this case it is time to revise key system design assumptions and to start gradually breaking hard dependencies in the overloaded UI and thus start moving towards the separation of concerns. System testability is one of the key nonfunctional requirements (just like performance or scalability or any other “-ility”) and should be considered very seriously as it ultimately enables everything else. 

Legacy Mindset #4: Manual testers cannot write automated tests.
In the post about ATDD and Exploratory Testing, I describe a scenario that enables manual test engineers to effectively create new automated tests from existing step definitions that have been created by others. While this process still requires little attention from developers (or automated test engineers), at some point, it allows manual testers to do a lot on their own without interrupting anybody else.  

Legacy Mindset #5: It is OK to write automated tests later. 

No it is not. “Building quality in” is one of the key Lean themes. It is too late and fairly pointless to write tests against the existing functionality. It requires too much effort to achieve an effect that is too marginal. 

Legacy Mindset #6: It is OK if only technical people are able to understand automated tests.

The disruptive success of new agile tools and ATDD techniques involving agile teams and their stakeholders in collaboratively defining system requirements in the form of executable examples (read: tests) is proving to be just the opposite. Tests can be written in plain English (or another natural language) and yet execute against the system and indicate what is passing and what is not. This type of tests becomes a kind of a cornerstone for conversation between business people, developers and testers. Nothing better sets the shared context. Tests in a natural language save significant effort and money by driving away ambiguities in requirements and their implementation. Gherkin-based tools (such as Cucumber, SpecFlow, JBehave, Lattuce, Behat, RSpec – for whole variety of platforms) that support business readable tests have already been widely adopted and are gaining more and more users as we speak. Other tools like Robot Framework or Fitnesse are also solving this problem from a slightly different angle.    

----------------------------
By taking an economic view and applying systems thinking, this allows organizations to critically look at their current practices and improve the quality and speed of the process. Automated testing is much more tightly connected with requirements definition, is primarily a collaborative practice used by agile teams, and relies heavily on paying continuous attention to proper design that would support testability.

Project Managers in Agile Enterprise

By Alex Yakyma.

Project managers exist in many organizations. With agile adoption throughout the industry, it turns out that most of the work tactics of the inglorious past, such as top-to-bottom planning, tracking, big up-front requirements and design, are no longer competitive. Methodologies like Scrum had clearly put project managers out of the scene, calling them "chickens" and claiming that they are not committed. Many authors explicitly call them the "overhead" in an agile organization. Given the overwhelmingly growing need for methods for scaling agility within an enterprise, the following question is becoming more and more important:

Where do project managers fit in the agile enterprise?

To be completely clear about our objective here, we are not trying to legitimize the role of the project manager or his equivalent. Nor do we aim to build a mapping of responsibilities or see how we could sneak traditional methods into the agile world just to ensure that a certain number of jobs remain secure. In fact, there are hordes of project managers who, for months, if not for years, have not been able to figure out how they fit into this new Lean | Agile paradigm. For the most part, these people should be given a chance to succeed elsewhere – agile teams only need strong and courageous enablers who are capable of challenging existing organizational rules and procedures in order to enable a fast sustainable value flow. But the question remains – if they (project managers) can't even enable themselves, then who needs them?

That being said, there are people who hold this position and are effective leaders, who, instead of impeding agility, end up being essentially helpful. I personally know such people – they embrace Lean thinking, paradigms of agile software engineering, and lead teams and programs to better alignment with business needs, a faster flow, and eventually assist the organization in its agile transformation at scale. Moreover, the actual reality of a large organization makes us think beyond the simplistic chicken and pig metaphor in order to realistically cope with external dependencies and overall organizational resistance to any change that could expose the dysfunctional aspects of its behavior. There is wide variety of areas where these people are able to help tremendously, and that is what I would like to explore in this article. But prior to starting our journey, it is necessary to indicate exactly what these people should stop doing once they embrace their new role, and that is... managing people on the teams in the context of their daily engineering activities. This has to unconditionally go away. Agile teams essentially estimate and plan their work, and they make most of tactical decisions on their own and do not require anyone to micromanage them.


There must be a conceptual shift from a manager-centric model towards a team-member centric model, where the project manager strives to find a way to enable teams in all possible aspects, instead of becoming a bottleneck for their daily work, as well as encouraging team members’ learning process and continuous improvement. Below are the areas where project managers are indeed capable of helping. It takes much more skill, expertise and courage then creating another Gantt chart or time reporting process. But this real leadership becomes quickly visible due to the fast feedback loops in the Lean | Agile enterprise, and serves as an optimum strategy to make a meaning and impact on your organization. Here are those areas where they may benefit the organization:

1. Foster organizational changes in bottleneck areas. This is the toughest one that requires real courage to challenge what has been believed to be best practices (or at least necessary ones) within your organization for years and years. Teams extremely appreciate effective help in this regard, whether it is an overly heavyweight procedure of deployment, obtaining approvals for something, receiving admin/ops/production support, or such a useless thing as time reporting, etc. The people who created these procedures in the first place are most likely ignorant in terms of lean thinking or understanding the principles of flow. Thus, what they have created for repetitive use by agile teams not only could be, but rather actually should be challenged. Most project managers simply fail in this respect, as their role never actually requires taking a chance by changing the status quo within the organization. Now, not only have they fallen outside of their own comfort zone, but they also have to challenge the status quo of others. A very simple formula works here: no courage – no result.

2. Create an environment that fosters the right team behavior. Work to eliminate individualistic performance review models. Instead, work to embrace team-centric improvement objectives, where individuals are left with a wide variety of options to manifest spontaneous situational leadership in different domains and thus, be evaluated "according to their contribution" to these team-centric objectives. Work hard with teams and communities of practice in terms of adopting collaborative agile practices such as side-by-side (and pair-) programming, specification by example, ATDD, and continuous integration. These are not just engineering techniques, but important cultural scenarios within and across teams.

3. Make key problems at scale visible and unambiguous to everyone. This should be done so that they can actualize and collaboratively improve once the problem becomes apparent. Is the problem due to too much work-in-progress across multiple teams, excessive delays on the part of ops teams, or too much rework of overly detailed design specs, or it is due to something else? People in our industry are not stupid, but they have become rather blinded by decades of theories promoting overly hierarchical methods of “managing” an organization’s productivity by maximizing utilization. Unfortunately, this has made the flow and thinking around the flow of value somewhat counter intuitive to us. This is something you’ll have to change. Make it visible to developers, testers, your fellow PMs, other departments and upper management by efficiently using BVIRs and actively engaging people in a conversation about how the work is flowing (well, unfortunately it will be mostly about how and why it doesn’t flow). This should all stimulate reflection and adaptation on their behalf, which you will also likely need to actively facilitate.

4. Help teams to embrace essential agility. Practically guide them through real challenging things such as splitting value into vertical slices, working collaboratively on those slices, or continuously refactoring code to enable future value delivery, or defining effective acceptance tests. These are all very challenging things that require a substantial shift in the team’s mentality. This is much more important at scale and much harder to achieve than simply ensuring that the team formally follows the Scrum ceremonies.

5. Educate your peers and upper management in Lean thinking. Show your peers and management how your teams resolve real problems in action, how they plan, how they resolve dependencies during execution, and how they demo. Teach them "indiscernibly" by engaging them in "Gemba kaizen" – the "real place" where all of the action actually takes place. Stimulate curiosity and encourage them to challenge existing rules and processes. Be patient as it takes time, but take every opportunity to present the results to other people in the organization and take every opportunity to involve them in the collaborative workshops for your teams – such events speak for themselves without any words.

6. Engage business stakeholders in alignment sessions. Events like program backlog grooming, release planning, release demo, etc. are pretty much useless if they are done only by the teams without synchronizing actual plans with product managers, infrastructure managers, the system/enterprise architect, and other people that have stake in the forthcoming scope. The recipe is simple: developers, testers and product owners from different teams should come together in the same room with business stakeholders.

7. Facilitate learning within and across teams. Build and facilitate different communities of practice (CoPs) for Scrum Masters, Product Owners, developers and testers, where they can exchange the proven best practices emerging in their agile team.

8. Create a balanced decentralized decision-making model. Essentially, tactical decisions need to be made quickly by the people who actually do the work and know best what should be done. Legacy mindsets and cultural scenarios within teams may largely prevent such decentralization and paralyze the flow. You should practically guide teams through the decision-making process and coach them in real scenarios whenever they stumble. Over time, this will transform them into a substantially self-organizing and self-managing team and will allow you to concentrate on identifying the next bottleneck in the organization, requiring your dedicated work as an enabler.

Rather recently, I had a talk with a bunch of project managers at one of the world’s largest organizations (unfortunately, I am not entitled to specifically name them here) regarding their role in agile enterprise, and it took quite a bit of time to go through all topics of interest. I have only summarized some of them in this post, but there are many more and they require your attention. However, to execute your agenda, you need to start small and realize that your ability for analytical thinking, your passion and unconditional courage must be combined together to do the job. Start the journey as soon as possible as there is a long way to go.