In the software testing game it is all about the details. Testing is really a simple thing to describe; does the software behave as expected? The complexity comes in knowing the details of what to expect and then designing tests to verify the behavior is as expected. If I personally defined the requirements or designed the solution for an application then the details are in my head. I just need a simple list of things to test and away I go. I can find data, deal with complex navigation and work around dynamic flows and popup messages. I can visually recognize screens and read messages to get where I need to be and ignore everything that is not important. My human brain and eyes can recognize wrong behavior even when it was not expected. I can decide dynamically if it is a bug or expected.
Human brains are great at testing given knowledge of what the application should do. Take this test case for example.
Verify the system can create a new account for each valid region.
If I am very familiar with the system and I know the valid regions I can test this function reasonably well from this one simple sentence. Humans are Amazing.
But what if I am not a super expert at the system I am testing? I will need enough details to know how to logon to the application, how to navigate, what the valid regions are and how to tell if an account is added successfully. If someone writes this down for me, I can test it. For example the same test from above with more details.
Verify the “ACME” system version 2.4 on Server B47 can create a new account from the logon screen for each region shown in the attached list of valid regions. The Account is verified if the status on the confirmation screen is “Open”.
If I am a contract tester I will need even more details to be able to test effectively, a lot more. This next test case example is more typical of the level of detail I find in test management systems or written test plans of real customers.
- Open an administrator session on B47 (see screen shot)
- Select ACME version 2.4 from the environment list
- Logon to ACME with Tester1/pw47
- On the Home page click the New Account button
- Select a Region as specified in the attached list
- Fill in all required fields with data for a non-existing person/SSN (see screen shots)
- Enter your tester assigned email address for the confirmation code
- Click “Create Account”. Ignore any warning or popup unless you see a red X
- Retrieve the confirmation code from your email inbox and enter it into the Account Confirmation panel and click OK
- Verify that the screen shows the account information as entered and displays a status of “Open”
- Do it again for each Valid Region (see attached spreadsheet)
Just for fun imagine what it would take to instruct a 9 year old child to perform this kind of test case.
An average 9 year old could probably follow these instructions and do the testing. However, a test automation tool would have no chance at following these detailed human instructions. An Automation Tool requires many hundreds of detailed instructions to execute this kind of test. What an idiot! This is why I sometimes refer to Automation Tools as Subject Matter Idiots or SMI.
Most test organization have figured out the level of test case detail needed for their QA team whether SMEs or 9 year old keyboard pounders. However, all manual test projects rely heavily on the human brain to interpret instructions, fill in details as needed, dynamically react to application behavior as good, bad or unexpected and importantly, what to ignore. Even the most inexperienced tester can learn application behaviors and navigation very quickly and perhaps even a 9 year old. They have human intelligence.
Sooner or later that human intelligence realizes the potential benefits (faster, cheaper) and it becomes an imperative to automate testing. You should! However, it can be very difficult to make the transition from manual human testing to automated SMI testing. Much depends on the level of detail contained in your current test case and how much you are relying on the human brain.
No matter what level of detail you have for human testers (BA’s, SME’s or 9 year olds) there will be a large “detail gap” to fill for the SMI. How those gaps get filled in and who provides the details can make or break an automation project. I will discuss this in more detail an upcoming BLOG and discuss how Keyword Frameworks can be used to create detailed test cases at a magic level. For now let’s say the SME’s should do it.
Mapping of existing test cases to automated test scripts can be tricky. Most tests for humans try to do as much testing as possible in each test case. You will see notes or general instructions like “make sure logos and headers look correct on each screen” or “check date fields for localization” or “perform this test on all browsers and devices”. Tests for the SMI need to be much simpler and linear. For instance a common test description like “repeat for all valid regions” or “validate spelling on all screens as you test” will require many more tests case for the SMI than a SME.
As an automation engineer I know firsthand what has to happen to fill the detail gaps and translate test cases designed for humans into test that work for tools. I need to completely understand the original test descriptions even if I have to make things up and expand it to automation code. Let’s take a simple test case step “complete all required fields”. Pretty straight forward for a human. To automate this I need to create automation code that recognizes all the fields that are required but how? Color? A star next to the fields? Dark lines around the fields? What if the fields are dynamic based on the data like international address formats or gender specific fields? What about valid test data for each field? This seemingly simple test step can literally take hundreds of lines of automation code to accomplish and countless hours of design, testing and making up (filling in) the details. The less I get to make up the better. This is because I am a coder, not the SME and coders love to make up stuff. The SME’s should fill in the detail gaps as much as possible. Automation projects are much smoother, cheaper, faster and effective when the original test cases have extreme details. Automation projects are much rougher, costlier, slower and ineffective when the original test cases have few details.
Conclusions: One way to make Test Automation Easier
- Simple, extremely detailed test cases make the transition to Automated Testing easier (less hard).
- Write tests for 9 year old's, you will be half way home but only half way home.
- Reduce the detail gap from the top. Don’t let automation engineers fill in the detail gaps even though they want to.