Legacy Mindsets in Test Automation

By Alex Yakyma. 

 In software development even these days test automation fails much more often than it succeeds. There are multiple reasons for this, but traditional thinking has a very strong impact in this regard – I’m referring to various legacy mindsets, originating from decades of the “divide and manage by function” pseudo-scientific set of beliefs that don’t withstand any practical validation, but unfortunately thousands of minds of otherwise smart people still adhere to such thinking. After one or two failed attempts, many organizations do not automate at all. Others do, but their process is not very effective even though automated test engineers manage to stay surprisingly busy. We will explore these false beliefs as well as working alternatives that evolved as a result of applying Lean thinking and Agile testing approaches inspired by Extreme Programming and its further evolution into areas such as BDD and ATDD. We will also be primarily concerned with principles that work at scale. 

Let’s consider these areas of concerns in more detail: 

Legacy Mindset #1: Automation should primarily be done by a separate automation team. 

This is one of the most severe issues with testing per se, creating endless list of problems, the most challenging of which are:

-          Developers never use these tests on a daily basis to prevent the propagation of un-validated code into the main code branch in the first place. This basically makes the feedback loop, which should be as short as possible, just the opposite, necessitating significant rework later when the tests are run. A lot of money gets wasted this way.

-          Developers and architects never bother with system testability and grow system design under completely different set of forces, making test automation an incredibly tough work.

-          Such tests never keep up with the latest requirements as developers (and possibly manual testers) move faster and are more flexible in responding to new requests and clarifications from the product owner and stakeholders.  

Agile team has to cover as many steps in the target value stream as possible. And even though something may be left outside their purview within iteration, this cannot be automated testing as it is indeed one of the key agile practices. If you cannot automate the tests for your own code, you are not a truly agile team. However, automation must be taken very seriously; many teams would be surprised at how much simpler automation actually is than they might have initially thought.

It is also important to note that the System Team can perform certain part of test automation within the program even in a very effective agile environment. However, if this is the only team creating automated tests and if the rest of the agile teams in the program do not automate, this just introduces equally poor waterfall-like behaviors with all the corresponding disadvantages, leading to a slowly moving, unreliable process. A good scenario entails just the opposite – when the system team actively collaborates with agile teams and aims to continuously give away more and more automation work to them while the system team themselves looks for the next systemic bottleneck for the program to cope with. This is part of the self-organization paradigm for the Agile Release Train as a team of teams capable of producing “the best architectures, requirements, and designs” at scale. 

Legacy Mindset #2: Automation is primarily done by specially trained automation engineers, using expensive automation tools. 

Companies waste hundreds of thousands of dollars on tools, and spend their time and effort hiring external people who can man them, essentially creating a hard dependency and largely ineffective process. These days, one can find very decent free tools with wide user communities, which are:

1.       Easier to use.

2.       Developed by agilists for agilists, so it’s no wonder it works for agile teams.

3.       Can be used by virtually anybody on the team.

Most unit testing frameworks are free for most platforms. Most BDD/ATDD tools are also free and work great for acceptance testing. Moreover, these tools are typically open source, allowing for customization whenever required. Using such tools helps to avoid multiple different bottlenecks that expensive commercial tools and employees with rare skills tend to introduce in the organization. 

Legacy Mindset #3: Testing via the User Interface. 

Do not test via the UI whenever it may be avoided. Always first look for alternatives and be ready to invest in those – it will tremendously pay off over time. Tests that operate via the UI are brittle and laborious both in terms of their creation and maintenance. Once again, a lot of money wasted. Even with such useful tools like Robot Framework or Cucumber (that leverage the re-use of “bindings”), their application to testing via the UI remains very fragile. Apart from very rare cases, following is the right thing:

Test via the programming API itself or simpler protocols such as XML or REST services, flat files, etc., but not UI. 

To enable this, the system under test should allow for testability. In very many cases, though, the system is not designed so. In this case it is time to revise key system design assumptions and to start gradually breaking hard dependencies in the overloaded UI and thus start moving towards the separation of concerns. System testability is one of the key nonfunctional requirements (just like performance or scalability or any other “-ility”) and should be considered very seriously as it ultimately enables everything else. 

Legacy Mindset #4: Manual testers cannot write automated tests.

In the post about ATDD and Exploratory Testing, I describe a scenario that enables manual test engineers to effectively create new automated tests from existing step definitions that have been created by others. While this process still requires little attention from developers (or automated test engineers), at some point, it allows manual testers to do a lot on their own without interrupting anybody else.  

Legacy Mindset #5: It is OK to write automated tests later. 

No it is not. “Building quality in” is one of the key Lean themes. It is too late and fairly pointless to write tests against the existing functionality. It requires too much effort to achieve an effect that is too marginal. 

Legacy Mindset #6: It is OK if only technical people are able to understand automated tests.

The disruptive success of new agile tools and ATDD techniques involving agile teams and their stakeholders in collaboratively defining system requirements in the form of executable examples (read: tests) is proving to be just the opposite. Tests can be written in plain English (or another natural language) and yet execute against the system and indicate what is passing and what is not. This type of tests becomes a kind of a cornerstone for conversation between business people, developers and testers. Nothing better sets the shared context. Tests in a natural language save significant effort and money by driving away ambiguities in requirements and their implementation. Gherkin-based tools (such as Cucumber, SpecFlow, JBehave, Lattuce, Behat, RSpec – for whole variety of platforms) that support business readable tests have already been widely adopted and are gaining more and more users as we speak. Other tools like Robot Framework or Fitnesse are also solving this problem from a slightly different angle.    

----------------------------

By taking an economic view and applying systems thinking, this allows organizations to critically look at their current practices and improve the quality and speed of the process. Automated testing is much more tightly connected with requirements definition, is primarily a collaborative practice used by agile teams, and relies heavily on paying continuous attention to proper design that would support testability.