Legacy Mindsets in Test Automation

By Alex Yakyma. 

 In software development even these days test automation fails much more often than it succeeds. There are multiple reasons for this, but traditional thinking has a very strong impact in this regard – I’m referring to various legacy mindsets, originating from decades of the “divide and manage by function” pseudo-scientific set of beliefs that don’t withstand any practical validation, but unfortunately thousands of minds of otherwise smart people still adhere to such thinking. After one or two failed attempts, many organizations do not automate at all. Others do, but their process is not very effective even though automated test engineers manage to stay surprisingly busy. We will explore these false beliefs as well as working alternatives that evolved as a result of applying Lean thinking and Agile testing approaches inspired by Extreme Programming and its further evolution into areas such as BDD and ATDD. We will also be primarily concerned with principles that work at scale. 

Let’s consider these areas of concerns in more detail: 

Legacy Mindset #1: Automation should primarily be done by a separate automation team. 

This is one of the most severe issues with testing per se, creating endless list of problems, the most challenging of which are:

-          Developers never use these tests on a daily basis to prevent the propagation of un-validated code into the main code branch in the first place. This basically makes the feedback loop, which should be as short as possible, just the opposite, necessitating significant rework later when the tests are run. A lot of money gets wasted this way.

-          Developers and architects never bother with system testability and grow system design under completely different set of forces, making test automation an incredibly tough work.

-          Such tests never keep up with the latest requirements as developers (and possibly manual testers) move faster and are more flexible in responding to new requests and clarifications from the product owner and stakeholders.  

Agile team has to cover as many steps in the target value stream as possible. And even though something may be left outside their purview within iteration, this cannot be automated testing as it is indeed one of the key agile practices. If you cannot automate the tests for your own code, you are not a truly agile team. However, automation must be taken very seriously; many teams would be surprised at how much simpler automation actually is than they might have initially thought.

It is also important to note that the System Team can perform certain part of test automation within the program even in a very effective agile environment. However, if this is the only team creating automated tests and if the rest of the agile teams in the program do not automate, this just introduces equally poor waterfall-like behaviors with all the corresponding disadvantages, leading to a slowly moving, unreliable process. A good scenario entails just the opposite – when the system team actively collaborates with agile teams and aims to continuously give away more and more automation work to them while the system team themselves looks for the next systemic bottleneck for the program to cope with. This is part of the self-organization paradigm for the Agile Release Train as a team of teams capable of producing “the best architectures, requirements, and designs” at scale. 

Legacy Mindset #2: Automation is primarily done by specially trained automation engineers, using expensive automation tools. 

Companies waste hundreds of thousands of dollars on tools, and spend their time and effort hiring external people who can man them, essentially creating a hard dependency and largely ineffective process. These days, one can find very decent free tools with wide user communities, which are:

1.       Easier to use.

2.       Developed by agilists for agilists, so it’s no wonder it works for agile teams.

3.       Can be used by virtually anybody on the team.

Most unit testing frameworks are free for most platforms. Most BDD/ATDD tools are also free and work great for acceptance testing. Moreover, these tools are typically open source, allowing for customization whenever required. Using such tools helps to avoid multiple different bottlenecks that expensive commercial tools and employees with rare skills tend to introduce in the organization. 

Legacy Mindset #3: Testing via the User Interface. 

Do not test via the UI whenever it may be avoided. Always first look for alternatives and be ready to invest in those – it will tremendously pay off over time. Tests that operate via the UI are brittle and laborious both in terms of their creation and maintenance. Once again, a lot of money wasted. Even with such useful tools like Robot Framework or Cucumber (that leverage the re-use of “bindings”), their application to testing via the UI remains very fragile. Apart from very rare cases, following is the right thing:

Test via the programming API itself or simpler protocols such as XML or REST services, flat files, etc., but not UI. 

To enable this, the system under test should allow for testability. In very many cases, though, the system is not designed so. In this case it is time to revise key system design assumptions and to start gradually breaking hard dependencies in the overloaded UI and thus start moving towards the separation of concerns. System testability is one of the key nonfunctional requirements (just like performance or scalability or any other “-ility”) and should be considered very seriously as it ultimately enables everything else. 

Legacy Mindset #4: Manual testers cannot write automated tests.

In the post about ATDD and Exploratory Testing, I describe a scenario that enables manual test engineers to effectively create new automated tests from existing step definitions that have been created by others. While this process still requires little attention from developers (or automated test engineers), at some point, it allows manual testers to do a lot on their own without interrupting anybody else.  

Legacy Mindset #5: It is OK to write automated tests later. 

No it is not. “Building quality in” is one of the key Lean themes. It is too late and fairly pointless to write tests against the existing functionality. It requires too much effort to achieve an effect that is too marginal. 

Legacy Mindset #6: It is OK if only technical people are able to understand automated tests.

The disruptive success of new agile tools and ATDD techniques involving agile teams and their stakeholders in collaboratively defining system requirements in the form of executable examples (read: tests) is proving to be just the opposite. Tests can be written in plain English (or another natural language) and yet execute against the system and indicate what is passing and what is not. This type of tests becomes a kind of a cornerstone for conversation between business people, developers and testers. Nothing better sets the shared context. Tests in a natural language save significant effort and money by driving away ambiguities in requirements and their implementation. Gherkin-based tools (such as Cucumber, SpecFlow, JBehave, Lattuce, Behat, RSpec – for whole variety of platforms) that support business readable tests have already been widely adopted and are gaining more and more users as we speak. Other tools like Robot Framework or Fitnesse are also solving this problem from a slightly different angle.    

----------------------------

By taking an economic view and applying systems thinking, this allows organizations to critically look at their current practices and improve the quality and speed of the process. Automated testing is much more tightly connected with requirements definition, is primarily a collaborative practice used by agile teams, and relies heavily on paying continuous attention to proper design that would support testability.

Project Managers in Agile Enterprise

By Alex Yakyma.

Project managers exist in many organizations. With agile adoption throughout the industry, it turns out that most of the work tactics of the inglorious past, such as top-to-bottom planning, tracking, big up-front requirements and design, are no longer competitive. Methodologies like Scrum had clearly put project managers out of the scene, calling them "chickens" and claiming that they are not committed. Many authors explicitly call them the "overhead" in an agile organization. Given the overwhelmingly growing need for methods for scaling agility within an enterprise, the following question is becoming more and more important:

Where do project managers fit in the agile enterprise?

To be completely clear about our objective here, we are not trying to legitimize the role of the project manager or his equivalent. Nor do we aim to build a mapping of responsibilities or see how we could sneak traditional methods into the agile world just to ensure that a certain number of jobs remain secure. In fact, there are hordes of project managers who, for months, if not for years, have not been able to figure out how they fit into this new Lean | Agile paradigm. For the most part, these people should be given a chance to succeed elsewhere – agile teams only need strong and courageous enablers who are capable of challenging existing organizational rules and procedures in order to enable a fast sustainable value flow. But the question remains – if they (project managers) can't even enable themselves, then who needs them?

That being said, there are people who hold this position and are effective leaders, who, instead of impeding agility, end up being essentially helpful. I personally know such people – they embrace Lean thinking, paradigms of agile software engineering, and lead teams and programs to better alignment with business needs, a faster flow, and eventually assist the organization in its agile transformation at scale. Moreover, the actual reality of a large organization makes us think beyond the simplistic chicken and pig metaphor in order to realistically cope with external dependencies and overall organizational resistance to any change that could expose the dysfunctional aspects of its behavior. There is wide variety of areas where these people are able to help tremendously, and that is what I would like to explore in this article. But prior to starting our journey, it is necessary to indicate exactly what these people should stop doing once they embrace their new role, and that is... managing people on the teams in the context of their daily engineering activities. This has to unconditionally go away. Agile teams essentially estimate and plan their work, and they make most of tactical decisions on their own and do not require anyone to micromanage them.

There must be a conceptual shift from a manager-centric model towards a team-member centric model, where the project manager strives to find a way to enable teams in all possible aspects, instead of becoming a bottleneck for their daily work, as well as encouraging team members’ learning process and continuous improvement. Below are the areas where project managers are indeed capable of helping. It takes much more skill, expertise and courage then creating another Gantt chart or time reporting process. But this real leadership becomes quickly visible due to the fast feedback loops in the Lean | Agile enterprise, and serves as an optimum strategy to make a meaning and impact on your organization. Here are those areas where they may benefit the organization:

1. Foster organizational changes in bottleneck areas. This is the toughest one that requires real courage to challenge what has been believed to be best practices (or at least necessary ones) within your organization for years and years. Teams extremely appreciate effective help in this regard, whether it is an overly heavyweight procedure of deployment, obtaining approvals for something, receiving admin/ops/production support, or such a useless thing as time reporting, etc. The people who created these procedures in the first place are most likely ignorant in terms of lean thinking or understanding the principles of flow. Thus, what they have created for repetitive use by agile teams not only could be, but rather actually should be challenged. Most project managers simply fail in this respect, as their role never actually requires taking a chance by changing the status quo within the organization. Now, not only have they fallen outside of their own comfort zone, but they also have to challenge the status quo of others. A very simple formula works here: no courage – no result.

2. Create an environment that fosters the right team behavior. Work to eliminate individualistic performance review models. Instead, work to embrace team-centric improvement objectives, where individuals are left with a wide variety of options to manifest spontaneous situational leadership in different domains and thus, be evaluated "according to their contribution" to these team-centric objectives. Work hard with teams and communities of practice in terms of adopting collaborative agile practices such as side-by-side (and pair-) programming, specification by example, ATDD, and continuous integration. These are not just engineering techniques, but important cultural scenarios within and across teams.

3. Make key problems at scale visible and unambiguous to everyone. This should be done so that they can actualize and collaboratively improve once the problem becomes apparent. Is the problem due to too much work-in-progress across multiple teams, excessive delays on the part of ops teams, or too much rework of overly detailed design specs, or it is due to something else? People in our industry are not stupid, but they have become rather blinded by decades of theories promoting overly hierarchical methods of “managing” an organization’s productivity by maximizing utilization. Unfortunately, this has made the flow and thinking around the flow of value somewhat counter intuitive to us. This is something you’ll have to change. Make it visible to developers, testers, your fellow PMs, other departments and upper management by efficiently using BVIRs and actively engaging people in a conversation about how the work is flowing (well, unfortunately it will be mostly about how and why it doesn’t flow). This should all stimulate reflection and adaptation on their behalf, which you will also likely need to actively facilitate.

4. Help teams to embrace essential agility. Practically guide them through real challenging things such as splitting value into vertical slices, working collaboratively on those slices, or continuously refactoring code to enable future value delivery, or defining effective acceptance tests. These are all very challenging things that require a substantial shift in the team’s mentality. This is much more important at scale and much harder to achieve than simply ensuring that the team formally follows the Scrum ceremonies.

5. Educate your peers and upper management in Lean thinking. Show your peers and management how your teams resolve real problems in action, how they plan, how they resolve dependencies during execution, and how they demo. Teach them "indiscernibly" by engaging them in "Gemba kaizen" – the "real place" where all of the action actually takes place. Stimulate curiosity and encourage them to challenge existing rules and processes. Be patient as it takes time, but take every opportunity to present the results to other people in the organization and take every opportunity to involve them in the collaborative workshops for your teams – such events speak for themselves without any words.

6. Engage business stakeholders in alignment sessions. Events like program backlog grooming, release planning, release demo, etc. are pretty much useless if they are done only by the teams without synchronizing actual plans with product managers, infrastructure managers, the system/enterprise architect, and other people that have stake in the forthcoming scope. The recipe is simple: developers, testers and product owners from different teams should come together in the same room with business stakeholders.

7. Facilitate learning within and across teams. Build and facilitate different communities of practice (CoPs) for Scrum Masters, Product Owners, developers and testers, where they can exchange the proven best practices emerging in their agile team.

8. Create a balanced decentralized decision-making model. Essentially, tactical decisions need to be made quickly by the people who actually do the work and know best what should be done. Legacy mindsets and cultural scenarios within teams may largely prevent such decentralization and paralyze the flow. You should practically guide teams through the decision-making process and coach them in real scenarios whenever they stumble. Over time, this will transform them into a substantially self-organizing and self-managing team and will allow you to concentrate on identifying the next bottleneck in the organization, requiring your dedicated work as an enabler.

Rather recently, I had a talk with a bunch of project managers at one of the world’s largest organizations (unfortunately, I am not entitled to specifically name them here) regarding their role in agile enterprise, and it took quite a bit of time to go through all topics of interest. I have only summarized some of them in this post, but there are many more and they require your attention. However, to execute your agenda, you need to start small and realize that your ability for analytical thinking, your passion and unconditional courage must be combined together to do the job. Start the journey as soon as possible as there is a long way to go.

ATDD and Effective Exploratory Testing

By Alex Yakyma.

Acceptance Test-Driven Development heavily relies on automated scenarios that reflect the key behaviors of the system. Basically it represents the way and the team process to physically bind the requirements, expressed as acceptance tests, to the code. Besides these key acceptance tests, or in other words – examples of system functionality – teams normally also want to ensure that certain less obvious and less predictable behaviors are implemented correctly. "Less predictable" and "less obvious" does not mean that they are less probable. In fact, teams, by definition, have their "implementer bias" and may easily overlook things. That is the reason for exploratory testing – to look at the system from a new angle, to try scenarios that have not been tried before, to play with different combinations of environment and configuration settings, so on and so forth. This important effort involves both creativity on the part of testers and their hard work in running desirable scenarios under certain conditions. The process most often entails one more of the following problems:

• It consumes a lot of the testers' time. Very often they perform all or almost all operations manually, which, apart from the execution of desirable scenarios, also includes manual environment setup, data preparation as part of different upstream scenarios, etc.

• It takes developers a lot of extra time in the case that they create and maintain special harnesses for manual testing.

• The tests are not persistent. Even when the team performs some kind of automation, typically there is a considerable gap between the process of exploratory testing and the automation of those valuable new test scenarios obtained as a result of exploration. The gap is both in terms of time and skillset due to the fact that exploratory testing is often performed by manual testers while, most often, automation is done by developers or automated test engineers.

• Testers' options are mostly constrained to black box testing. System-level scenarios result from a combination of different conditional flows that occur in different parts of the code. The problem is that it is impossible to test the system by only operating at a high level – we simply experience a "combinatorial explosion" – a stratospheric number of scenarios.

 • The knowledge and context of sharing with developers is ineffective. While testers may play with certain type of system behaviors and even catch some interesting defects, there is almost no sharing of their tacit knowledge about the system with the developers, no collaborative analytical thinking, and no generalization or synthesis. There is no way to induce the specific useful implementation or design takeaways from the findings discovered during exploratory testing.

ATDD turns out to be very useful in creating certain behaviors and effects that help to address these problems. The secret is in the team's ability to quickly automate new scenarios, in the tooling, that turns out to be extremely useful, and in constant collaboration that drives ATDD process. Here is how it works.

ATDD relies on automation tools like Cucumber, SpecFlow, FIT, RobotFramework etc., which allow for defining scenarios in business language. These scenarios are then physically linked to the system by what is called "bindings" or "scenario step definitions" – little pieces of code that interpret the corresponding scenario steps, and run them against the system. Very often these steps are parameterized, so that, for example, this would be a step written in Gherkin language, which is used in tools like SpecFlow and Cucumber: "customer provides , and and whether this a permanent address". This enables testers to do exploratory testing at a whole new level: 

• Testers use the same tools as developers to "play with scenarios". The only difference is that they only play with the input parameters of an existing scenario steps, which does not require any coding. 

• When testers need a new step for exploratory testing that does not yet exist, they ask developers to create it. They often pair with developers to make sure that the right steps are built. 

 • When binding new key scenarios to the code, which happens as a part of the normal ATDD flow, testers and developers ensure that steps are properly parameterized so that testers could immediately use them for exploratory testing, apart from their primary function. Sometimes, however, teams decide to hide some hairy detail in the step bindings (usually for better readability). In such cases, testers may ask developers to simply create a "parameterized" version of the steps, which will be used primarily for exploratory testing. When the initial binding code is in place, this becomes a very quick operation. 

• By easily combining and running different steps with different input/output parameters, testers can more effectively capture unexpected outcomes. Here is the fun part: every finding can be captured as an automated test by simply replicating the set of steps with the specific set of parameters into a separate scenario file. 

 • Frequent collaboration between developers and testers flows very naturally this way. Developers immediately learn what is wrong with the system and testers learn to understand and to be able to read the binding code by looking behind the developer's shoulder and operate much better in terms of "white box" or at least "grey box" testing. There is nothing sophisticated about the binding code. In fact, it has to be fairly simple by definition - those who write automated scenarios leverage simplicity in order to prevent false positive or false negative scenarios. It is really that simple! 

After an iteration or two, this approach becomes so obvious and natural to testers that all they had normally done previously, in terms of exploratory testing, seems ridiculously inefficient to them. This approach allows them to explore more, but also to capture findings so they could be run over and over again. It helps testers externalize valuable knowledge and turn it into better software as a result of collaboration with developers.

17 Truths about PSI/Release Objectives

By Alex Yakyma.

PSI (Release) Objectives are defined in Scaled Agile Framework as a key alignment tool that teams use within the agile program. For definition and key characteristics, we refer the reader to the original article on PSI Objectives in SAFe.

PSI Objectives are actually not a trivial concept, as is, in fact, the case for anything describing complex aspects of software engineering at scale. That's why it is important that agile teams, program stakeholders, consultants and coaches understand PSI Objectives and their correct application. In this post I will try to provide simple, understandable, compact and actionable points regarding PSI/Release Objectives. Even though unordered, they are useful as they represent answers to common queries:

1. PSI Objectives are not features from the Program Backlog. They can directly correspond to features sometimes, but not necessarily always.
2. PSI Objectives are the key output of PSI/Release planning event. Every team has their objectives sheet as a result of planning.
3. Business value, assigned to each objective by business owners, should be interpreted unambiguously by both stakeholders and the team.
4. Business value has nothing to do with the effort involved. It only shows you "how much useful stuff is there" for the business.
5. PSI Objectives are kind of a "reality check". While top N Features from the Program Backlog are the idealistic plan of intent, PSI objectives reflect them in the real world: what contribution is required from different teams? What are the different aspects, such as research, infrastructure, design enablement, etc.?
6. If a team does not have Stretch Objectives, then they either honestly believe in the fixed scope game or they are made to believe in it. Either way it has nothing to do with agile development.
7. Business Owners should be able to understand all PSI Objectives and their business value; otherwise, they likely lack some members of the Business Owner team (such as system or enterprise architect, infrastructure manager etc.)
8. PSI Objectives may reflect scope that does not originate from Program Backlog – for instance, the team's technical debt, maintenance, other commitments (routine releases, patches, etc.), improvement initiatives, infrastructure and process related effort, etc.
9. PSI objectives are used by teams at every Backlog Grooming, sprint planning and sprint demo event to ensure that they are on track.
10. There's no direct correlation between feature priorities (WSJF) and the business value of the PSI objectives, which the feature spawns on different teams' objectives sheets. However, if the feature is important, then at least one of the corresponding objectives will have a relatively high business value (but not necessarily all).
11. PSI Objectives are the main communication protocol between stakeholders and teams both at PSI/Release planning, execution and Demo.
12. The same PSI Objective may create multiple business effects, all of which should be considered when assigning business value. Thinking in terms of business effects creates the foundation for alignment.
13. S.M.A.R.T. PSI Objectives are context dependent. Besides the basic requirements of S.M.A.R.T.-ness, different organizations evolve their own understanding of what the quality PSI objectives are in their own specific case. It is important to capture these context-specific characteristics as early as possible.
14. PSI Objectives are used at Scrum-of-Scrums, PO/PM meetings, and as system demos.
15. Business Value is always considered at a current moment of time. If something is only valuable in the distant future, it should get high business value in the future, but not right now.
16. Business Value does not determine the sequence in which PSI objectives are taken into development by the team. Business value only shows what to expect as a result.
17. Normally PSI Objectives may be accomplished with reduced scope. This only means that the achieved business value will be lower than the planned one, but key business expectations can still be met.    

Figure 1. PSI/Release Objectives are the primary tool for alignment within agile program

 Understanding PSI objectives is very important for business alignment. Programs and teams realize they are able to deliver value effectively, as they really come to understand what they are building and why. By having a set of clear, committed to, PSI Objectives, every team will know how to be efficient within the program. PSI Objectives enable a program to become a self-organized team of teams – a key mechanism for agility at scale.    

Specification by Example and Card-Conversation-Confirmation

By Alex Yakyma. 

Recently I happened to encounter a lack of specificity in a purportedly trivial situation. I was at a restaurant in Longmont, CO and wanted to start in my rather typical way, having tea with honey. Given that it was snowing, seems like a quite natural thing. I ordered "black tea with honey", the waitress asked "why black?", to which I answered that, in some countries, unless you say "black tea" or "black coffee", you will receive it with milk. Also, I noted that, this way, one could distinguish it from green tea, which is an entirely different thing. She smiled and, in a minute, I got my... you guessed it... glass of ice tea (actually with a lot of ice in it) and a little vessel with honey on the side.

 I was speechless, and she disappeared before I recovered. I felt like Eric Cartman with his funny-fuse, when he wasn't able to laugh any more :) Anyways, to make a long story short, I knew what I wanted, and we even had a chat about it. I thought we had even defined everything clearly, but turns out we didn't. Sound familiar?

I think XP folks impacted on the software development industry much more significantly than we may have otherwise thought. In fact, user stories aren't a Scrum invention; this was institutionalized in XP. In particular, Ron Jeffries, defined what we know as the "3C":  Card, Conversation and Confirmation - three important aspects of user stories (actually any stories, not just user ones). The "triad" actually means physical card, on which the story is written – a conversation between those who define and those who implement – to clarify detail and, finally, the acceptance criteria by which it is decided whether a job is done. This is a great conceptualization of what is necessary to keep in mind for teams. Nevertheless, the 3C itself is dependent on how effective a conversation is in terms of identifying detail; otherwise... you may get your ice tea with honey.

A useful technique to make the 3C work efficiently is Specification by Example - a collaborative approach to defining requirements based on concrete examples, instead of abstract statements. The "Second C" in this case gets recurrently supplied by examples, which become a natural way for requirements’ disambiguation for the team and stakeholders. What's important is that the "examples" are goal-driven and thus, the Conversation continues to be meaningful. So, had I said that, to warm up in such cold weather, I would need a cup of good Earl Grey tea with honey, then there likely would never have been such confusion in the first place. By the way, there indeed happens to be waiters and waitresses that have a good facilitation skill for thing like Specification by Example. But these are as rare as good facilitators among people in the software domain. Nevertheless, here's a set of simple recommendations that I give during ATDD coaching:

• Refine requirements collaboratively as a team, involve external stakeholders as much as possible; keep in mind that your co-dependent agile team is just another "stakeholder", so invite some representatives.
• Include the Specification Workshop (where Specification by Example actually comes into play) into every backlog grooming and sprint planning session.
• Actively use BVIRs (whiteboards, flipcharts, etc.).
• Use tables to represent the variations in data, flow conditions - virtually everything. Table format is the most powerful collaborative tool for refining requirements. Regardless of whether it happens later, it may or may not be represented in a table format in your ATDD tool.
• Do not allow any abstractions in the examples. Avoid those "X"s and "Y"s etc. at all cost. You will be surprised how much your team will learn about the seemingly familiar user story once they replace all Xs and Ys with real values.
• Stay timeboxed story-wise and move to the next story every 10-15 minutes. If not done with the story, no worries - you can return to it. Just don't let it go too deep and consume too much time.
• Capture all agreed upon examples - these are your acceptance tests. Capture all open questions (to stakeholders or otherwise) and resolve them first.
• Have Specification Workshops at every iteration - team self-discipline is key here. Scrum Masters can find this a real challenge for their facilitation, coaching and mentoring skills, which assist their team in adopting and sustaining Specification by Example as an effective cadence based activity.

 It is amazing how much a team can achieve when they know what they are building. User stories, as "containers" of value, in conjunction with concrete examples, represent an extremely powerful tool that makes a team extremely productive in delivering business value incrementally.