In this article, we’ll talk about eliminating duplication, comparing images, explicit waits, and driving input from places other than the user interface.

Eliminate Duplication

Don’t Repeat Yourself, or DRY, is every bit as good advice for writing test automation as it is writing production code. The more places I have to make a change, the more likely I will miss something and have to hunt down multiple locations to modify. Also, if I had to hunt down those multiple locations at this time, it’s a good bet I’m going to have to do it again. This goes back to data‑driven testing and trying to keep specific items out of tests wherever possible. Hopefully, this goes without saying, but any time I find myself tempted to copy and paste something from one test into another, even if it saves me time, it means I’m setting myself up for tedious replacement and refactoring.

Better to make small snippets that you re-use, some of which eventually become functions. That means when a break happens the “fix” only needs to happen in one place.

Use Smart Image Comparison

It’s one thing to confirm that an image is on a page, and having a test to confirm the right image is there is certainly a test I might want to write. However, if there is an issue with the pixelation or the loading of that image, the higher the percentage of a match I select, the more likely I’ll get a false positive and fail a test. Fortunately, there are options with a variety of tools that, allow image compares with a percentage match, allowing “fuzzy matches” that are close enough – by dialing down the precision up or down. This can eliminate false failures, but too fuzzy a match might allow a real defect to slip in. Another approach is to use a tool like Appitools Eyes, which use AI to detect only differences in images that are perceptible to users.

Don’t Delay Explicitly

I’m certain that I could go through my tests right now and I could find several places where, due to the erratic response of resources, I’ve put in arbitrary delays. I know from experience that it takes about a minute from when I spin up an AWS environment until I can actually connect via ssh Entering in a delay of 60 seconds makes a lot of sense. However, there are times when the resource is available in less time than that, and sometimes, it’s not available even after 60 seconds. When it’s available sooner, I’m waiting longer than necessary. If it doesn’t respond after the sixty seconds, my test fails and I have to start again. Putting in an implicit wait or making a routine that polls for the availability of a service makes a lot more sense to me. That way, I only wait the required amount of time and then, if I decide that there is an unacceptable wait time, I can then consider a test to be a failure legitimately, not because an arbitrary timer completed. Additionally, if I get access to the resource in 20 seconds, so much the better. I’ve saved 40 seconds in that test step. Multiply by dozens or hundreds of tests and that’s a real time saver.

Change The Channel

I consider effective automation to help take out the steps that are tedious and that require a lot of repetition. In many cases, testing at the UI level makes sense, but there are times where there are better ways. If I’m looking to create test data, it makes much more sense to write scripts that let me create data in a database from a known good source. In the product that I currently work with, there are often varying needs for testing and rather than try to create that data from scratch each time or drive the UI to create that data, I can import accounts or other data structures that contain all of the pieces that I need and none of the pieces that I don’t. Additionally, API’s are my friend. I can send a few parameters via Postman or cURL and confirm that I am getting back the data I expect. With practice, entire suites of tests can be created that will exercise the feature functionality of my application without my having to drive a web browser at all.

All test suites will, at times, struggle with some or all of these issues, but if I take the time to consider these areas and implement them where possible, the odds of my test suite needing a lot of time and attention to debug or maintain will go down considerably.

Putting It All Together

Instead of this code:

wait 30
click_on(element_name)

Use an approach more like:

wait_for_element_maxtime(element_name, optional time)
click_on(element_name)

This function should end the waiting as soon as the element appears. Time should be a variable that you can change once that expands on all test — so if fifteen seconds is acceptable instead of ten, you only need to make the change once.

Expand that “don’t repeat yourself” (DRY) principle to the test automation code will make greening the tests easier. As you automate, consider how often the user interface will change and what parts of the screen might change, and by how much. Finally, look to import test data from an external source that is fresh for every build, instead of entering it through the user interface each time.

The results will be faster-running tests that have fewer false errors that are easier to fix.

What’s not to like?

You come into work one day, only to learn that an evil hacker has gotten into a web server and absconded with all the access credentials left behind in a plain text configuration file. Pandemonium breaks out. The bosses go berserk. The next thing you know, cables are being pulled from boxes in the data center, non-essential IP addresses and ports are closed on the firewalls and everybody wants to see the systems logs — right now!. Then, of course, security gets called in to figure out what happened and make sure it never happens again. It’s a tale too often told.

There’s a saying that goes like this: Insanity is doing the same thing over and over again and expecting different results. This absurdity has not been lost on those whose pride and passion are security. Bringing security in after the fact might be a nice way to do post-mortem forensics and establish policies moving forward, but reacting to disaster will only get you so far. Hence the evolution of DevOps into DevSecOps.

DevSecOps is, as the names implies, the integration of Development, Security, and Operations into a unified, self-directed, self-correcting team completely accountable for all aspects of creating and maintaining the software and digital infrastructure in its portfolios.

Those in the know have figured out that the best way to improve an enterprise’s approach to security is not only to give security experts a first-class seat at the table of software development before projects begin, but to create a culture in which security is a principled way of life.

Some might think that the only requirement for implementing effective DevSecOps is to make sure a security expert is present at daily scrums. It’s more than that, way more! In fact, DevSecOps.org has defined these way-of-life principles as follows:

Leaning in over always saying “No”

DevSecOps requires security interests to move past the typical knee-jerk “no”, to offering educated, informed input based on the situation at hand. Leaning in means being involved in development initiatives as an active, invested team member, not as an authority whose role is to do nothing more than bless the activities of others. DevSecOps means taking an interest in the work of others and being able to learn from them. It also means learning the dynamics of the team and the details of the work being done. It’s not simply being the expert who gets to call the shots about matters concerning security.

Data & security science over fear, uncertainty, and doubt

DevSecOps is about thinking in terms of the facts at hand and using the science available to create computing environments that are safe and reliable. It’s not about being driven by imaginary organization demons that threaten catastrophic disaster and bankrupting litigation. For example, acting on verifiable reports about a demonstrable hazard rather than reacting to a recent attention-getting telecast on a cable news network. A security practice that is based on facts and science is manageable. One based on fear, uncertainty, and doubt is beyond control.

Open contribution & collaboration over security-only requirements

Making software is an undertaking requiring the concerted efforts of many parties. It’s an orchestrated collaboration. And, just as a well functioning orchestra operates beyond the concerns of any one section in the ensemble — violin, oboe or trombone, for example — so too does the software development team. Hence, effective DevSecOps means that the security members of the development team need to have an operational awareness of all aspects of the development process– from design, programming, and testing to product release. DevSecOps means knowing the overall intention of the enterprise and how all the pieces, security included, fit together to make the entire operation work.

Consumable security services with APIs over mandated security controls & paperwork

DevSecOps understands the world of modern technology is fast changing, that no one product will ever satisfy a need forever. Today’s great security tool might be tomorrow’s ineffective hazard. Thus, DevSecOps promotes a services approach to security in which required functionality is represented as an abstract service backed by an ever-changing set of tools and security policies. Abstracting security into a set of services represented by way of APIs provides the flexibility needed to make sure that security practices stay current and effective.

Business-driven security scores over rubber stamp security

DevSecOps understands that appropriate security measures are a reflection of business needs that are continuously changing. A security policy that is written in stone will become outdated as the business changes over time. Instead of a binary, Yes/No approach to determining security compliance, DevSecOps promotes establishing compliance by using a passing grade method based on scoring. A score-based approach is one in which points are awarded according to metrics gathered from concrete security testing and subsequent analysis of the enterprise’s infrastructure. It’s the difference between “rubber stamping” that a penetration test has been conducted and awarding a score to the penetration testing based on breadth and depth of the penetration attempt. Attempting to penetrate all the access points of an enterprise will yield a higher score than attacking a single server.

How compliance is scored over time will change as the business evolves and the digital environment changes. Rubber stamping runs the risk of becoming obsolete. Business-driven security scoring provides the flexibility needed to create a secure computing environment that meets the current demands, both in terms of the business and the digital infrastructure.

Red & blue team exploit testing over relying on scans & theoretical vulnerabilities

A Red Team is an outside attacker. A Blue Team in an inside defender. DevSecOps promotes using the Red/Blue teams on an ongoing basis to emulate real word attacks against the enterprise and to defend against such attacks. While scanning for vulnerabilities and perpetrating attacks against theoretical vulnerabilities can be useful, such practices run the risk of becoming misguided and obsolete. DevSecOps asserts that Red Team/Blue Team attack and defense activity is a more real-world approach to security monitoring than prescribed scanning. Of course, this assumes that Red & Blue Team exploit is done continuously. The key here is the word, continuously.

24x7 proactive security monitoring over reacting after being informed of an incident

Football teams don’t plan defensive strategy after the touchdown has been made. Rather, they establish an overall approach to defense. The team gets drilled every day to hone defensive capabilities. A team will study the habits of other teams before game time to determine strategy. Then, they make adjustments as the game is being played. Yes, the team might alter defensive tactics on the field in response to another team’s touchdown, but momentary reaction is not the only approach to overall defense.

The same is true when it comes to DevSecOps. DevSecOps understands that security is a way of life and goes beyond one incident or response. Hence, DevSecOps advocates continuous 24/7 proactive security monitoring as a way of putting security into the very fabric of an enterprise’s daily operation.

Shared threat intelligence over keeping info to ourselves

DevSecOps understands that security is a team support. Therefore, sharing all information, all the time is critical to making each member of the software development team a contributing resource for a more secure computing environment. As security becomes a more pervasive practice among all members of the digital enterprise, the role of the DevSecOps engineer becomes one of the expert guide on the sidelines of activity rather than being a gatekeeper authority for work in progress. Therefore, the open, continuous sharing of information, particularly threat information, is critical for raising the level of security acumen of all contributors throughout the digital enterprise.

Compliance operations over clipboards & checklists

Compliance operations is the discipline by which an enterprise verifies that it adheres to the rules and best practices as defined for a given field of operation. Compliance operations is more than having a checklist on a clipboard and checking conditions as True or False as one would do during a car inspection — headlights work? Check!; brake lights work? Check!; horn works? Check!

Compliance operations requires not only knowing the rules in play, but also having an in-depth understanding of the reasoning behind the rules.

DevSecOps promotes compliance operations over clipboards and checklists. Putting compliance operations as a forefront of activity means having an ongoing awareness of the rules, regulations and best practices around corporate IT security. DevSecOps personnel apply what they know and understand to establish policies and procedures to ensure compliance with the most current rules and standards in force. Inherent in understanding is the ability to learn and adapt. Adaptability is essential to the practice of DevSecOps. Simply checking off items in a checklist on a clipboard does not scale nor does it provide the level of proactivity needed to operate securely on the modern digital landscape.

Putting it all together

It takes years of experience to develop the knowledge and intuitions required to be a first-class security professional. However, any security expert, whether from a traditional security discipline or from the modern DevSecOps school, cannot go it alone. In today’s digital infrastructure, security is everybody’s concern — from the maintenance worker noticing strangers dropping thumb drives with the label, “Use Me” in the IT breakroom, to the senior CISSP certified professional battling the newest menace threatening to bring the enterprise to its knees. DevSecOps is an important movement that will go a long way to foster the sensibilities needed to promote security best practices in the modern enterprise.

“Standards? We don’t need no stinkin’ standards!”

Not only are the individual things we try to base our standards on contentious, but the notion of having standards at all can be contentious, particularly on an agile team.

When someone says the phrase “coding standard,” usually the first thing that comes to our minds is formatting. This nestles us all snugly with very specific visions of braces and tabs dancing in our heads. Many of us feel cozier when things look a certain way; some us feel dread when we see things that offend our visual sense; and some have learned to not give a hoot — in fact, a recent co-worker suggested that we were all professional enough to be able to read code, regardless of the way it looked. Personally, I’ve seen enough languages and differently formatted code that I’ve learned how to wade through it, no matter how horribly it’s presented.

But of course, coding standards can cover so many more things than formatting:

• Naming: Are we using Hungarian notation? (Heck no, though you might be surprised at how prevalent it still is in 2018.) Upper or lower camel case? Snake case? How do we name tests and fixtures? Classes? Namespaces and packages? Parameter names? When are we using nouns vs. verbs? What are the proper domain names to use?
• Test organization: Where do the tests sit? What’s the granularity of the tests? Are tests allowed to produce console output? (No.) Are we using AAA?
• What language constructs are we preferring or avoiding? How do we distinguish among variables of different scope?
• Meta: Are we even writing unit tests? What programming languages are we using? Operating system? Editors? (Please, let’s not start the editor wars again — yet, if we are pairing …) Other tools? These items don’t necessarily impact the coding standard, but they might.

We could debate these standards elements, and so many more, for hours on end. Sometimes there’s not even a clear winner among competing choices: tabs vs. spaces, for example, divides the world into two mutually exclusive groups, and ne’er the twain shall meet. Members of each competing faction bolster their stance with solid, unassailable opinions. Some proselytizers wander the hills broadcasting their opinions, whether or not there’s anyone within earshot.

Be forewarned that advocates will sometimes fervently defend an opinion to the point of losing their sense of humor and rationality. I worked with a coach at a customer site who vociferously argued for using tabs. I blurted out, “Well, tabs are just wrong.” The coach seemed hurt and believed I was really an anti-tab zealot, even after I apologized for my sense of humor.

The Value of Standards

Standards exist for a few reasons. Adherence to standards can help prevent mistakes — this is why “safety braces” exist for single-statement code if blocks:

if (anOddCondition) {
    doSomeSingleThing();
}

The safety brace statement is a long-standing one. My violation of the standard offends developers far more often than not:

if (anOddCondition)
    doSomeSingleThing();

They’re worried I might accidentally code:

if (anOddCondition)
    doSomeSingleThing();
    otherThingApplicableOnlyToAnOddConditionBeingTrue();

I don’t make that mistake. Why not? Because I test-drive. My coding standards have changed. I no longer need the braces to protect me, because I have a better mechanism. Good teams learn how to evolve their standards, particularly when they exist to help prevent errors.

Simple standards have a greater value: They allow us to focus on more important things. Modern life is made less stressful because we have so many implicit standards. Book organization, time zones, electrical outlets, and automobile driving controls are all standardized, to an extent.

Travel to enough places, however, and you’ll find a different set of standards for each of these. When we encounter implementations that vary from the predominant standards we’re comfortable with, we’re taken aback. We waste time figuring out how to deal with the variance, and we might even do something dangerous as a result.

For example, most drivers today wouldn’t be able to get out of the rental car parking lot if given a car with a manual transmission. In the U.S., most new drivers learn on an automatic transmission. We quickly ingrain the location and operation of the brake and accelerator. This allows us to focus on the far more important task of paying attention to the road. Add a clutch and gear shifter to the mix, and we’re dangerously distracted while trying to figure out the new interface.

In programming, deviations from a standard can similarly slow us down, creating small bits of friction as we try to assimilate the logic represented by the code. It’s not atypical to encounter systems that embody a half-dozen different ideas about how to best present the code. Yes, we can grok the code, but we’ll spend a little more effort due to the friction of inconsistency.

The Cost of Standards

Deriving standards requires effort, of course, as does ensuring adherence to them. Discussions of standards invariably foster an even larger concern: Are they stifling creativity and innovation? This question hints at another cost of standards: We must revisit them often enough to determine what we need to update, lest we lose our ability to move forward or innovate.

As far as creativity is concerned, I imagine that it’s possible for coding standards to stifle it. Strong mandates about C++ programming styles might eliminate the possibility for an elegant (though potentially dense) solution using template metaprogramming. But I suspect these cases are extremely rare.

Code isn’t an art, though it can have artistic aspects. A core element of art is its ability to provoke different emotions and interpretations. Aesthetic beauty is another element of art; code can certainly exhibit this while adhering to standards. One might complain that the boundaries of what is readable kill some opportunities for pretty code. But I hope I never have to maintain code produced by e.e. cummings or submitted as an entry to the International Obfuscated C Code Contest. I just don’t have enough time to figure it all out.

A source file, in contrast, is a collaborative canvas. Teams producing code need everyone to be able to contribute to each canvas — which requires a common understanding of what its content (the code) accomplishes. We might be able to accept and admire aesthetic beauty on a canvas, but we can’t let the beauty get in the way of our ability to understand its meaning.

Agile Standards

I’ve been part of numerous debates about coding standards, and I’ve led a number of sessions to help derive them. Here are a few guidelines I’ve used in the past:

• Not everyone cares what the standard is. They don’t have to attend.
• Participants should come with a short list of things they really care about.
• Set aside an hour, no more, to pin down an initial standard. If no one cares about a specific standards element — for example, how to name constants (upper snake case, upper camel case, etc.) — go with the predominant standard in the codebase or community. Or don’t sweat it at all.
• If the initial standards document can’t be expressed on a single sheet of paper, it’s too much. Hint: Show it in a sample of code.
• The codebase becomes the embodiment of the standard. If we can’t toss the single sheet of paper after a few weeks, something’s not right.
• Every rule and standard is meant to be broken, but we should have a good reason. Breaking a standard might imply a conversation and subsequent transition to a new standard.

One way or another, review your standards and your team’s adherence to them at least every few months.

The Most Important Standard

After working with countless developers on countless codebases, I realized I wasn’t as concerned with the specific standards as much as the amount of care the developers put into their code. I have my standards preferences for code, and they’ve evolved over time, but I care most about crafting code that is consistent and accessible. I care because I don’t want other people to waste time deciphering my code.

Perhaps you’re a developer who dismisses coding standards because you don’t need them. No doubt that’s true for many experienced programmers. Think instead, however, that the primary point of standards is for other people:

• You should use descriptive identifiers because you care about other developers who need to understand the code’s intent
• You should consistently format your code because you care about your teammates who find it challenging to understand otherwise
• You should use spaces or tabs consistently (well, spaces… come on now!), because you care about Jeff, who gets frustrated when they’re mixed across the source (Yes, you can ignore white space on diffs; that’s only part of the challenge)

It costs little to adhere to a simple set of standards. You should also be gratified to know that you’re making life better for your fellow developer.

We’ve developed community standards around many programming languages. We’ve written countless books promoting language-specific standards. Some of these collections of standards contain hundreds of items — good luck remembering them all!

Ultimately, you need to remember only one standard: to care. With each line of code you write, think about how it will be received by the next person who must maintain it. Even if you’re not the caring type, don’t forget that the next maintainer is often yourself.

If you’re testing a web application, you naturally want to ensure that it works well with not just a single browser, but with all of the most popular browsers. This requires cross-browser testing.

This blog post will show you how to record your automated website browser tests; then automatically execute the recorded tests for compatibility on multiple browsers using the Ranorex Studio web automation engine.

In addition to executing web tests on Windows platforms using the Ranorex Studio web automation, you can also use the Selenium integration in Ranorex Studio to expand cross-browser testing to additional operating systems such as macOS and Linux.

For more information about how to use the Selenium integration in Ranorex Studio, please refer to:

• Understanding Web Testing Concepts in Ranorex
• Selenium WebDriver integration

Please note that when following the approach described below, you can at any point switch to the Selenium WebDriver integration and vice versa.

Sample Test Suite Project

To demonstrate how to perform a cross-browser test, I’ve chosen a small sample which performs a checkout process on an online store. The test case performing the web test has three parts:

1. A setup region containing the module OpenBrowser that starts the browser against which the test will be executed
2. CheckOutProcess, the actual recording module containing the test actions
3. A teardown region containing the module CloseBrowser that closes the browser after the test actions complete.

Cross-Browser Test

To perform these steps not only for IE but also for the other supported browsers, begin by configuring the browser in the OpenBrowser recording module as a variable.

To do this, open the OpenBrowser recording module and click the down arrow in the Browser column. Choose “As new Variable…” and add a new variable called BrowserName.

After making the browser a variable and binding this variable to a table containing the names of all supported browsers, you can use your test script for cross-browser testing to speed execution, rather than running your cross-browser.

There are two ways to set up parallel test execution:

• Use Ranorex Remote to automatically deploy and execute your tests on a remote machine running a Ranorex agent. During remote test execution, the Remote Pad keeps you updated on the test performance, while you can continue using your local machine. Once the test has been executed, you’ll receive a notification that the report is ready. Using the Remote Pad, you can execute multiple tests on multiple remote machines in parallel.
• Use the Ranorex Parallel Runner, a command-line tool that allows you to run a test across multiple capability sets on a Selenium Grid through a WebDriver endpoint in parallel.

A major goal of release 8.3 is to ensure that new users have a successful first experience with test automation and follow best practices for a robust and sustainable test automation project. To meet this goal, we’ve implemented the three major changes described below:

• The RocketStart new solution wizard
• Best practices in test automation – setting up and cleaning up your test environment
• The Recorder control center

The RocketStart new solution wizard

The RocketStart new solution wizard launches automatically the first time that you open Ranorex Studio, or when you choose “New test solution using wizard…” from the file menu or the start panel.

Begin by selecting the technology that you plan to test. The wizard will walk you through:

• Setting up your test automation solution for your needs
• Configuring your desktop, web, or mobile application for testing
• Selecting the desired recording behavior for error-free recording

  In selecting a recording behavior, you can choose from the following options:

  • Focus on single application: record only actions performed in the application under test, ignoring interaction with any other running applications
  • Record actions in specific applications: record interactions with several applications
  • No focus applied: record all interactions and consider all running processes when searching for UI elements.

  This approach is called whitelisting in Ranorex Studio. Focusing Ranorex Studio on only the processes relevant for your test has two advantages. First, it increases performance during test recording, test execution, and in Ranorex Spy. Second, it helps you create clean recordings since you can interact only with whitelisted processes. You can find the whitelisted process(es) in the Whitelist pad in Ranorex Studio:

After the wizard finishes, a new First Steps built-in tutorial automatically appears, as shown below. This tutorial demonstrates how to automate your first test.

Best practices in test automation – setting up and cleaning up your test environment

In Release 8.3, we have applied best practices by giving the test suite a more global view of the application under test. Instead of requiring you to choose the application under test each time that you start recording, we’ve moved this functionality to a separate “StartAUT” module. This module is automatically created by the RocketStart new solution wizard and appears in the setup region of the default test case. Additionally, the wizard automatically creates a CloseAUT module in the teardown region, to shut down the application under test.

As a result, you are no longer prompted to choose the application under test when you start a new recording session, as was the case prior to release 8.3:

Instead of selecting the AUT in the Ranorex Recorder, now you choose it in the RocketStart new solution wizard.

Setup regions are always executed before anything else in the direct parent item. It’s a best practice to populate the setup region with the modules and module groups needed to bring the AUT to the state required for the following modules to run, such as starting the AUT.

Teardown regions are always executed after everything else, or when an error occurs in the direct parent item. Here the best practice is to populate the teardown region with the modules and module groups needed to clean up the AUT after a test run, such as closing the AUT.

Following these best practices will help you set up robust and sustainable test automation projects. When creating a new test case, always keep in mind to include a clean startup and shut down of your application(s) under test, with all of the cleanup tasks necessary for an error-free next test run.

For an overview of a more complex test suite, refer to the desktop sample provided with Ranorex Studio.

As you can see in the screenshot above, there is a setup and a teardown region for the smart folder “AddNewEntry”. The setup region contains a module group which starts KeePass and logs in. The teardown region contains a module for saving the changes and a module for shutting down KeePass. The same goes for the test case “AddNewGroup”: a start and login module in the setup region, and a save and shut down module in the teardown region. In this test suite project, we also have a teardown section for the entire test suite. As you can see, whether or not the entire test succeeds, the modules in the teardown region will execute, thus cleaning up the previous test run.

When you are planning an end-to-end test spread over several different technologies, it is useful to separate the different technologies into individual projects. Simply create a module library project for each technology or app that you are testing and combine the execution of the generated modules in one test suite project.

Open the sample cross-platform solution for an example.

As you can see, there are several module library projects – one for desktop, one for web, one for iOS and one for Android. Additionally, there is a test suite project called “CrossPlatform” that combines the modules from the different projects together in one big test suite.

There are setup and teardown regions for all test cases in this test suite, which perform startup and cleanup actions for the different technologies using the modules from the technology-specific module library projects.

The Recorder Control Center

Last but not least, we have polished our Recorder control center to give you more control during the recording process. The recorder now displays a history of the last four actions, providing instant feedback during recording. You can delete recorded actions right in the recorder or add new ones using the recorder’s controls, making your recordings even more effective right from the start.

Restart Animation

To learn more, check out our user guide. You can also register for a free webinar on October 11, featuring a live demonstration of the new features in Ranorex Studio 8.3.

Proponents of test-driven development (TDD) can be overheard muttering their mantra to themselves and others: “Red, green, refactor.” The red-green-refactor pattern is a simplification of the TDD cycle, which requires that you demonstrate test failure prior to writing the code to make it pass.

The colorful part of the mantra derives from the tools used, which universally use a stoplight color scheme to indicate whether a test is passing or failing. Red indicates that you ran a test through a unit testing tool and at least one expectation described by the test was not met. Green indicates that the test passed: All expectations described by the test were met and no errors occurred during test execution.

In TDD, a test describes a new behavior through code that interacts with the underlying system. The test asserts that expected conditions hold true upon completion of that interaction.

Because the test is describing yet-to-be-implemented behavior, it might not even compile when first written. It might take the developer a few minutes and attempts to build enough code before the test can execute in a unit testing framework. These incremental steps are failures, but they’re not the red we’re looking for: A test must fully describe the new behavior before we consider the red part of the TDD cycle complete.

But what’s the point? If we know that a test describes nonexistent behavior, why take the time to run it through the testing tool? It might take only a couple of seconds to run the test and another couple to note the result, but we know that seconds are precious things that add up to real time.

We should probably be clear on why it’s important to first see red.

Heeding feedback

TDD is a simple feedback mechanism. The color at each step tells you whether you are ready to move on to the next step. A red (failing) test is feedback that says your test represents a proper description of a new behavior. The description contains at least one assertion that demonstrates whether the proper outcome has been achieved. A green (passing) test is feedback that says your system does all it’s been asked to do so far. You can choose to clean up its implementation through refactoring, to write another failing test to add more behavior, or to ship what you have.

One glaringly obvious conclusion you can make based on a red test is that things aren’t working yet! And they shouldn’t be — you’ve not written the code to make the test pass. (It’s possible, though, that the test is red for other reasons; we’ll touch on those later.)

Particularly as the behaviors in your system get more complex and the code gets more interesting, you might run your new test expecting it to fail, only to see it pass. Why? Sometimes the behavior you’re describing already exists in the system, but that fact isn’t known to you.

Seeing a test go green when you were expecting red can happen for a number of other reasons; we’ll talk about those as well. For now, the key thing to remember is that your reaction should always be the same:

A test that passes with no change to your classes is always cause for humility and pause.

“Oh! My test passed! That’s special,” you say (sounding like Dr. Seuss). It’s time to don your thinking cap. Why is it passing? Is the behavior already in the code? Or is something else amiss?

Catching dumb mistakes

I teach TDD from time to time in a classroom setting. I implore that pairs follow the TDD rules, but there are often students who don’t listen, or don’t figure they need that extra step. As students work on their exercises, I wander around the classroom to ensure they’re staying on track. In one class, I walked up behind a pair of programmers who proudly announced, “We’re writing our last test.” That was odd, as it was only about 20 minutes into an exercise that takes most students at least 45 minutes to complete.

A quick look at the code triggered my spidey sense. “Your tests are all passing? Let’s see this current test fail.”

“OK, but our tests have been passing all along.” Oh. Oops. I reminded them that it was important to stick to the cycle, and that absence of any failures meant that we really didn’t know what state things were in. Sure enough, their test run showed all tests passing, even though the new test should have generated a test failure. A bit of digging revealed a rookie coding mistake, and as a result, none of their new tests were included in the test suite that the tool was running.

The students fixed their problem, but pride turned to dismay when the next test run included their tests. Every single one was failing. “Looks like you have a bit of catching up to do,” I said in an upbeat manner as I moved to the next pair.

Here’s a list of some dumb reasons your tests might show green when you should be expecting at least one to be red:

• You forgot to save
• You forgot to compile
• The new test isn’t part of the current set of tests you’re running (hint: Try to run all the tests, all the time)
• You’re using a framework that requires you to explicitly add the test to a suite, but you forgot
• You didn’t properly mark the test as a test — for example, JUnit requires tests to be marked with an @Test annotation
• You’re running tests against a different version of the code (possibly due to path or classpath issues)
• You didn’t include an actual assertion in the test
• The test is disabled
• The test isn’t getting picked up for some obscure reason (Visual Studio’s Test Explorer continues to confound me this way occasionally)

Don’t feel bad; I’m pretty sure I’ve made every one of these mistakes at least once.

We make dumb mistakes all the time, but sometimes we make mistakes because things aren’t so clear. It’s possible, particularly if you are using test doubles, that the test is just not properly constructed. Maybe the assertion isn’t really expressing what you think it is, or maybe you are testing something that’s not really what you should be testing.

Because of all these opportunities (and many more) to make mistakes when constructing tests, you must ensure that when you finally get a test to pass, it’s for the right reason. Seeing the test fail first dramatically increases the odds that you’ve written a legitimate test and that the subsequent code you write properly implements the behavior described by the test.

One other hint: Keep track of the current test count, and make sure it goes up by one when you add a new test.

Violating the rules of TDD

Per Uncle Bob, you cannot “write any more production code than is sufficient” to pass your failing unit test. TDD newbies violate this rule wantonly. If a single statement would suffice to pass a test, they’ll introduce an if statement to guard against inappropriate input. If it’s possible that an exception might be thrown, they’ll sneak in a try/catch block to handle the exception. While these are good programming practices, the proper TDD approach is to first document these non-happy-path cases by writing a failing test.

Seasoned programmers often introduce constructs to support future generalization. While a simpler solution might exist to meet the needs of the current set of tests, these developers have a good sense of what’s coming, whether it’s in the near or distant future. Rather than go through a series of small test-driven increments, they choose to take a larger step with a prematurely complex solution.

Some problems exist with this sort of premature speculation:

• Premature complexity can increase development costs until the need for complexity arises
• We’re all wrong at times about what’s coming. If the need for the complexity never arises, we’re forever burdened with an unnecessarily challenging solution. Time to understand and maintain code increases. Also, sometimes complexity arises and it’s different than expected; it’s generally far more expensive to adapt an ill-fitting, overly complex design than a minimally simple design
• It kills TDD. Prematurely building support for not yet described cases means that it is no longer possible to stick to the red-green-refactor rhythm. Once I introduce the more sophisticated solution, any other tests that might be useful to write around these additional cases will pass immediately. It becomes very difficult to write a test that first fails, which in turn means that all such tests should be treated with suspicion

Even when not doing TDD — such as when writing a test after the fact against legacy code — it’s still important to ensure that you’ve seen the test fail at least once. With test-after, this usually means deliberately breaking or commenting out production code. Trust no test you’ve never seen fail.

TDD is a self-reinforcing rhythm. The more you follow its rule of writing no more code than needed, the easier it is to write subsequent tests that will first fail.

There’s not much to TDD: red, green, refactor. Hidden within this simple mantra, however, is a surprising amount of nuance around keeping things simple and safe. Always see red!

Attend one of my classes on test-driven development (TDD), and you won’t be programming alone. You’ll work in pairs or if the class is small enough, you’ll participate as part of a mob as you learn how to test-drive code.

The bulk of my past students worked exercises as part of a pair. Pairing shines in a classroom environment. The book “Training from the Back of the Room,” by Sharon L. Bowman, promotes having students learn from each other as a way to increase learning. Pairing gives students a chance to interact with each other, bounce ideas off each other, and lean on each other for help without the intimidation factor of having to ask me. With pairing, my students more quickly ramp up on TDD concepts.

Mob programming, or mobbing, takes this notion of collaboration to the extreme: Everyone in the room collaborates to solve the current challenge. I’ve used mobbing as a classroom technique for a couple of years now because, for companies that want a given team to learn together, mobbing provides a way to ensure everyone turns the same page at the same time.

Leveling Out

Most classrooms exhibit a bell curve distribution. The bulk of students are capable developers who work at a steady speed, in line with the other developers. A small number of students are very experienced and more effective. A comparable few are perhaps programming novices who struggle with the exercises.

Left to work alone, the capable developers in the class finish ahead of everyone else. They become bored and sometimes end up distracting other students, or even me, as a result. Sometimes I’ll throw these overachievers additional challenges for extra credit, but that too increases the amount of attention I must give them.

At the same time, the novices in the class lag behind everyone else. They often need special attention to fix mundane and unrelated problems. Their challenges distract me far too often in proportion to the rest of the class, and often for reasons unrelated to learning TDD.

If students pair with each other, I have them swap pairs from time to time. One person stays to work the current exercise, and the other gets up and finds a different person to work with. I sell the class on pair-swapping by suggesting that the rotation helps to disseminate information throughout the team, but I have students swap mostly to ensure that the pairs are reasonably balanced. Rather than a devastatingly off-track pair of underachievers, I’ll hook up an overachiever with an underachiever.

By the end of the class, the novices have usually picked up enough knowledge and even a bit of speed. Further, the experienced developers have gotten a little better about explaining to the novices. In a software development world where the average developer is left to their own devices in a cube, any sort of increase in “code socialization” can only help us as a team.

The use of pairing has always helped make for a more effective classroom environment, partly by leveling out the pace somewhat. No student left behind!

The Overhead of Pairing

Some raise the concern that pairing takes twice the time of solo development, but that’s only true if you think programming is merely about typing, or if you think code is written once and never read again. Having two people collaborate to solve a problem definitely generates some increase in quality—both from the notion of the solution’s design and the readability of the code to implement the solution.

Other costs of solo development, some intangible, continue to represent an increased ROI for pairing. I’ve written enough about these potential benefits of pairing; see “Pair Programming Benefits,” co-written with Tim Ottinger, for a long list.

However, there are certainly maintenance costs, or overhead, involved in pair programming that reduces its ROI. Switching pairs is perhaps the most obvious overhead: Ramping up on new code doesn’t come for free, and we’re asking you to switch pairs fairly often. Risks also exist when some unfortunate pairings occur—for example, when a novice pairs with a novice (“the blind leading the blind”), or when a dominating programmer steamrolls someone too timid to speak up. Clashes over development setup can hurt effectiveness, as can discomfort due to the physical environment.

Getting a team to pair effectively demands some level of attention, whether it be from a coach or from team members savvy enough to introspect well.

The Value of Minimizing WIP

Perhaps the largest cost to pairing, however, is that it—and solo development, too—are admissions that we are probably working on more than one thing at a time.

In a typical trying-to-be-agile shop, you’ll find considerably high amounts of WIP, which can mean either “work in progress” or “work in process.” If a team consists of five solo developers or five pairs, chances are good that five stories were started at the outset of the iteration. Chances are also good that a high percentage of these stories are at risk of getting completed by the end of the iteration. Many daily standup minutes and keystrokes in Jira are spent to help track progress toward a given in-progress story.

Given the nature of software development, interruptions and blockers are common enough to increase the WIP number even more. The resulting context-switching exacerbates the overall cost, which is further compounded by the cost of ramping up again on code that has likely changed since the last time you touched it.

You’ll find numerous articles on the benefits of minimizing WIP. To cut to the important part: “Work is not valuable until it reaches the hands of the customer.” Invariably, people who have promoted limiting WIP claim shorter cycle times, shorter lead times and fewer defects.

Speeding Us Up

One thing I’ve heard a number of times from teams who mob is “We go faster.” Every time, my mind races through lots of thoughts: Fascinating! Are they making that up? How is that possible? How would they really know? Prove it!

I’ve been trying to think about reasons mobbing might increase speed, occasionally iterating them. Any one of these reasons alone might be debatable. But the aggregate list shows enough weight to convince me that there might really be something there.

Here are the things about mobbing that incrementally can increase our speed:

• We’re not blowing 15 minutes every day listening to a glorified status report.
• We’re not planning the details of a future story.
• We’re not worrying about the mechanisms of how we work together (e.g., pairing).
• We’re minimizing context-switching costs.
• We’re not reworking the inadequate design of another pair or individual.
• We’re spending far less time making sense of our own code, because everyone was on the same page when we wrote it.
• We’re not subject to the bad code of a rogue or inept developer.
• We don’t have to call special meetings to engage members of our own team.
• We’re focusing on delivering value, not getting “my” work done.
• Through strong-style pairing and timed rotation, we’re ensuring that all our team members increase in individual effectiveness (and in a minimally intimidating fashion!)
• We learn how to describe code (and what needs to be done to it) in a more effective manner.
• We produce fewer defects—and the cost of a defect is probably far larger than most people think.
• We don’t spend time reading and reviewing pull requests, and we generally don’t do code reviews.
• We don’t blow hours trying to figure out an answer on our own when the person with the answer is already on our team and available.
  We adhere to the small set of standards we value, which includes things like doing TDD or producing a solid set of acceptance tests via behavior-driven development.
  We are more likely to refactor enough toward the simplest possible design, which helps reduce the cost of introducing new features.
• We minimize the angst over estimates and weak notions of commitment because the mob finishes the current feature, ships it and moves on to the next.
• We often address interpersonal and other team challenges immediately.
• We’re generally having more fun, which means we’re not surfing the net out of sheer boredom or anxiety about work.

Trustworthy scientific research won’t exist any time soon for the vast majority of these claims. Nor does it exist for the claims of pairing, or even solo developers. But the potential time savings are many enough to convince me to continue exploring mob programming as a way of developing software (in addition to continuing to pair and solo-program occasionally). As I do, I’m sure I’ll note additional factors that can speed us up. Please send me your discoveries too!

This article describes how to use the Ranorex Studio IDE and the Ranorex API for test automation in your behavior-driven development (BDD) process. BDD requires a cognitive shift: instead of thinking about testing functions, you are now thinking about behaviors. Following this approach enables seamless collaboration between business stakeholders (such as product management) and the QA or development teams. On the business side, the product management team can define feature specifications in natural language without writing a single line of code. On the technology side, the QA team and the development team add automation steps using recording modules and the repository.

Tool Stack

The diagram below shows a sample tool stack to create a BDD-based testing framework. Ranorex Studio provides the development environment, while the Ranorex test automation framework provides the automation API. To parse the feature files from natural language to written code, we are using SpecFlow as the BDD interpreter. Finally, NUnit is the unit test provider, which manages and executes the tests.

Process Overview

When following the BDD approach in an agile environment, the entire development process of a user story is divided into a business side and a technology side.

On the business side, we create user stories, which are taken and broken down into features stored in feature files. These features are broken down into scenarios. The features are written down in a natural language following the Gherkin syntax. A basic feature written in Gherkin might look something like this:

Feature: Serve coffee
 In order to earn money
 Customers should be able to
 buy coffee at all times

Scenario: Buy last coffee
 Given there are 1 coffees left in the machine
 And I have deposited 1 dollar
 When I press the coffee button
 Then I should be served a coffee

On the technology side, we need an interpreter to parse the feature files written in natural language (Gherkin syntax) and call test automation code. These Ranorex API calls or Ranorex recordings do the following:

• Automate the system under test
• Verify the expected output
• Returning a corresponding result

Prepare Ranorex Studio for BDD

The SpecFlow Add-in for Ranorex Studio

The SpecFlow add-in provides file templates for feature files, step definition files and event definition files. It also translates the features from Gherkin syntax to C# code. To install the SpecFlow add-in to Ranorex Studio, follow the instructions below:

Download or clone the SpecFlow repo and open the solution in Visual Studio.

If you have installed Ranorex in “C:Program Files (x86)Ranorex 8.0”, the correct references will be added automatically.

If not, open the project “TechTalk.SpecFlow.RanorexIntegration”, remove the following libs from the references:

• ICSharp.Core
• ICSharp.SharpDevelop
• ICSharp.SharpDevelop.Dom

and replace them with the dlls from <Ranorex installation folder>bin.

Next, build the RanorexIntegration add-in.

Extract the built “.sdaddin” file from the “IdeIntegration” folder of the SpecFlow solution to a new folder “TechTalk.SpecFlow.RanorexIntegration“.

Copy the extracted folder to <Ranorex Installation folder>AddinsMisc.

Make your Ranorex Solution BDD ready

Open the “Package management console” (View->Tools->Package management console) and run the following commands:

Install-Package SpecFlow.NUnit -Version 2.3.2
Install-Package NUnit.ConsoleRunner

Get started with BDD

Now that your Ranorex Solution is BDD-ready, you can begin defining features in SpecFlow and creating automated tests for them in Ranorex. You can see how to do this in the screencast “BDD and Ranorex,” or you can read the instructions below, whichever you prefer.

Define a feature from the business side

In SpecFlow, create a new feature file. To do so, right-click on the project, then choose Add > New Item > Category: SpecFlow > SpecFlow Feature.

Describe the behavior of your system using human-readable syntax in Gherkin. For Example:

Feature: Calculator
 In order to avoid silly mistakes
 As a math idiot
 I want to be told the square root of a number

Scenario: Square root of a Number
 Given I click 3 on calculator
 When I press sqrt
 Then the result should be 9 on the screen

Save the file. Ranorex Studio will create a <name>.feature.cs file containing the automatically-generated code that will execute the test scenarios using the configured test provider.

If the feature file is not created in Ranorex Studio, you have to set the “Custom tool” in the files properties to “SpecFlowFileGenerator” to make Ranorex Studio create the code file.

Create step and event definitions for the technology side

Create a new step definition file. To do so, right click on the project and choose Add > New Item > Category: SpecFlow > SpecFlow Step Definition.

Now, create methods for all “Given,” “When,” and “Then” actions described in the scenario description. Add “Given”, “When” and “Then” attributes with values matching the scenario steps defined in the feature file. To do so, follow the samples provided by the template from which the new step definition file has been generated.

     public class StepDefinition1
     {
         [Given("I have entered (.*) into the calculator")]
         public void GivenIHaveEnteredSomethingIntoTheCalculator(int number)
         {
             ScenarioContext.Current.Pending();
         }

         [When("I press add")]
         public void WhenIPressAdd()
         {
             ScenarioContext.Current.Pending();
         }

         [Then("the result should be (.*) on the screen")]
         public void ThenTheResultShouldBe(int result)
         {
             ScenarioContext.Current.Pending();
         }
     }

SpecFlow searches for the step definition text in the step definition files that matches the text in the scenario step.

Now you can implement the automated step, either by creating recording modules or by manually implementing the steps using the Ranorex automation API.

Your implementation might look something like this:

     public class sqrt
     {
         [Given("I have entered (.*) into the calculator")]
         public void GivenIHaveEnteredSomethingIntoTheCalculator(int number)
         {
             ClickNumButton.repo.numButton = number.ToString();
             ClickNumButton.Start();
         }

         [When("I press the square root button")]
         public void WhenIPressSqrt()
         {
             PressSqrt.Start();
         }

         [Then("the result should be (.*) on the screen")]
         public void ThenTheResultShouldBe(int result)
         {
             ValidateResult.Instance.result = result.ToString();
             ValidateResult.Start();
         }
     }

You can create additional steps to be executed before and after a scenario, such as performing setup and tear down tasks, including starting and closing the system under test.
These steps can be defined in an event definition file and might look something like this:

         [BeforeScenario]
         public void BeforeScenario()
         {
             StartSUT.Start();
         }

         [AfterScenario]
         public void AfterScenario()
         {
             CloseSUT.Start();
         }

Make the project ready for the unit test provider

After implementing the automation steps bound to the feature description, you can build your project as a library so that it can be passed to MSTest. To do so, open the project properties and set the output type of the project to “class library”. In addition, copy all Ranorex runtime DLLs to the output folder.

Run a BDD test using MSTest

Because this solution uses NUnit as the unit test provider, we use a NUnit Runner to execute our tests.
Your command line call should look something like this:

..\..\..\packages\NUnit.ConsoleRunner.3.9.0\tools\nunit3-console.exe <solution>.dll

Now the steps defined in the feature file will be executed and the results will be shown in the terminal. See the result of your test run for detailed information about the performed steps.

Conclusion

The preparation of Ranorex for BDD is complete. From this point forward, your business team can define user stories in a natural language following the Gherkin syntax, and your development and QA team can implement the automation steps in the step definition files.

As the need for coverage grows and calls for more overall tests, it makes sense to try to find ways to make tests that can address a variety of scenarios. It’s also a good idea to make sure that the need to update the tests or make radical changes is minimal. To that end, data-driven testing is a great option that fits into a variety of testing frameworks.

However, while data-driven testing is an excellent tool to have in our arsenal, it’s possible to have too much of a good thing. Over the past few months, I’ve been considering the way many organizations — mine included — work with data-driven testing, and I’ve come to the conclusion that perhaps we’re going about this the wrong way.

Defining data-driven testing

First off, let’s define what data-driven testing is. In a nutshell, it’s taking a test or series of tests and feeding it varying data values from a data source, such as a spreadsheet, text file or database query. From there, we can run the tests using those data values and perform comparisons to see if the data values help us see the values we’re expecting to see. From there, we could also use parameterization, where we create additional tests (or, more accurately, different examples of the same tests) by adding new lines of data.

The upside to this is that it is easy to add new tests by creating additional rows of data to process: one run for each row. By entering a new row of data that matches the variables we have chosen to work with, we can extend the tests as far as we want.

The downside is that this can make for lengthy test runs. Additionally, if the data entered is all the same format (i.e., correct, or what the application expects to see in those areas), then it’s not likely the tests are going to tell me anything interesting besides “OK, the system recognizes this type of data.”

With that in mind, what are some issues we may face, and how might we resolve them?

Creating wasteful tests

While data-driven testing can be automated to run a test multiple times with a variety of data, I often find that I create tests that are needlessly long and repetitive. As an example, I’ll share one of the most common interactions that might benefit from a data-driven test: user login. I can set up a test to log into a system, providing a variety of usernames and passwords. I could, potentially, log every user into the system to verify that everyone can log in, but that’s not really useful — or, should I say, not really useful beyond a few examples.

There are four examples that make sense to test a login process: the correct login, and the three examples of an incorrect login — wrong username, wrong password or a combination of both. Running this test for every user is wasteful and time-consuming, and it doesn’t provide additional information. If, however, the purpose of the test is to see if the system will crash when handling hundreds or thousands of logins in steady succession, then this could be a valuable test. It’s not one I’d want to run every single time I push a merge request or try to deploy a branch to a device, but it would be helpful at times.

Alternative method: testing the happy path and the errors

By setting up deliberate errors in my data (meaning the values I provide are a mix of valid and invalid input types), I can use a variety of data elements to examine the path and also check that my error-handling code is working. A quick win when testing software is to see what error-handling is in place and what triggers it. This lets me set up my tests to verify if those errors can actually be raised based on the data that is being provided. With one test containing multiple permutations with good and bad data, the net result is a lot of potential coverage and less need to write multiple test cases.

Relying on static data files

As I create data-driven tests, I start with a simple format such as a .csv file to keep my test data in a single place. The positives are that there is a known safe place for data and a specific format that is expected to be used and maintained. Adding new data is as simple as adding another row with data values. However, over time, if these data files get exceptionally large or there’s a new variable that we want to add to our tests, then it can be time-consuming to add a new column and populate those new values.

Alternative method: using a dedicated database

By contrast, with a database that can be queried, I can get the specific values in a similar format and feed those values to my script. Over time, as I add new variables, those values can likewise be queried from the database and used.

Not having a known or reliable data source

The example I just gave of using a database query to get the values to populate our tests is cleaner than maintaining data files individually, but it has its own downside. The database itself can be updated while tests are running, and such tests could change values in the database and cause issues of unreliable data.

Alternative method: creating a second database

A method to resolve this is having a cloned database that is never directly accessed by the application. Additionally, I could create a .csv file by setting up a query to retrieve the values important for my tests, and subsequently destroy the file after the tests complete. This way, my data source starts with a known state, and even if it’s modified during a test run, it can be easily regenerated with our known values. It’s also possible to have two databases, one being the specific test data values to be used and another to be the active database in use by the application. By querying the first database and getting the values needed, I can use them to populate our target database and examine those values to see if they are correct — or to raise errors if the values are not.

Conclusion

Data-driven testing can be useful to examine an application with a variety of data points, but there are challenges and pitfalls to the approach. Using the alternative methods outlined above can help make sure we have as good a chance to leverage the benefits of data-driven testing as possible.

You may be thinking, “Who is this guy with the audacity to write about his personal history? I’ve never heard of him.” Why should you, the reader, care?

We bring value to ourselves and others by introspecting about our experiences and mining the nuggets of wisdom they contain. An authentic story tends to resonate a bit more, and you might even remember the nugget it contains down the road.

With that clever rationalization for telling a bunch of tales from my past, I bring you a condensed history of my work experience: three and a half decades’ worth. I think you’ll be able to relate to how we all interact and work together on a day-to-day basis.

Early periods of isolation

In 1982, I was hired by the University of Maryland Central Administration (UMCA) as a “student programmer” for minimum wage — $3.25 per hour at the time. I found myself sitting in what was once a vending machine closet alongside another student programmer named Chris.

Too green to know how to take advantage of the proximity to someone else, I squandered my social time with Chris discussing things like music and movies. Chris worked on his software only, and I on mine. I don’t recall a single time across my three years at UMCA where we dug deep into each others’ code to learn or help out.

I moved to another tight-but-isolated work environment at the Federal Energy Regulatory Commission (FERC) in Washington, D.C., where I shared an office with two heavy smokers. (Yes, in 1985, you could light up all day.) Doug chain-smoked unfiltered cigarettes, and the elder Jay puffed on a pipe. My attempts at programming sessions were interrupted by the sound of Jay smacking his pants after ash had burned a hole into them.

The accounting system used by the FERC was convoluted, and the software naturally followed form. I was tasked with calculating interest due to the federal government for headwater benefits, because apparently most of the folks running hydropower projects weren’t paying up.

My task should have been straightforward. Unfortunately, no one was able to explain exactly what I was supposed to do — neither Jay, Doug, my manager, nor anyone else in the vicinity understood the complex accounting rules. I wallowed through the code and eventually got something running, at which point my consulting contract ended and it was time to move on. I have no idea whether my software changes were even correct, and I suspect no one else did, either.

It was bizarre to share a physically close environment yet experience almost zero collaboration. But I did get some value out of working in such an isolated fashion: I was learning how to be resourceful. I read manuals cover to cover and started deriving myriad problem-solving techniques.

My next two jobs involved being completely isolated: At both the American Institutes for Research and the Archdiocese of Baltimore, I was the sole programmer reporting to the manager of the computer department. I sat alone and coded by myself for a few years, again honing my survival skills.

The middle years: Cube farms and collaboration

In 1989 I was hired at Marriott in Bethesda, Md., where I was introduced to life in a cube maze. It was nice to be part of a larger team, and it was even better to be working full-time on a software application along with another programmer, Mark. We were both responsible for the development of an accounting system used in airports to support operations for merchandise and food and beverage.

For the first time, I was faced with the direct impact of the crud I coded. Mark and I learned from each other about how to write code that others could more easily understand. I started taking great pride in the quality of my code, often refactoring it for clarity. Mind you, this effort to clean up code that had already been proven to work was highly risky. But I felt the risk was worth it due to the importance of ensuring that Mark and future others could maintain my efforts.

My next cube farm was at MCI in Virginia, starting in 1991. I was now regularly working with a couple of other developers on any given project. We had business analysts and testers. We had a process! I learned object-oriented concepts from my BA (she was once a programmer), client/server architectures from my team lead and coding techniques from my fellow developers.

But the most impressive things I learned while at MCI weren’t technical. And unfortunately, I wasn’t smart enough at the time to heed the first lesson.

Two of the programmers in the group, women from India, had the audacity to sit together throughout the bulk of the day working together. The rest of us didn’t know what to make of it, and at least a couple of folks complained. I mistakenly pooh-poohed the idea myself, figuring that it suggested they hadn’t developed the resourcefulness I’d learned over the prior decade.

Around 1995, MCI shipped me to Atlanta for a few weeks to work on a special project. I was shunted into a dark room, in the center of which were a few tables arranged in an oval, with computers. I shook hands with a few other programmers and got to work.

We actively collaborated with each other throughout the day, initially because we were all tasked with learning the same framework targeted for the application. And once we began development, the heavy interaction didn’t abate; I saw firsthand how constant communication with your team dramatically sped things up. We weren’t wasting time figuring out things others had already mastered. We weren’t preplanning, planning or otherwise sitting in meetings figuring out how we were going to work together. And we weren’t stepping on each others’ toes in the shared codebase.

The increased collaboration in my middle career served me well. Working with others increased my knowledge, helped me understand how my code impacted others and opened my eyes to radically different ways of working.

Interlude: A step backward

Beginning in 1998, I spent two years at a Colorado Springs startup. I was back in a shared office working on something different from my office mate. I immediately felt my choice was poor — why did I accept working in an environment that actively shunned collaboration?

“Actively shunned” is not an exaggeration. When it came time for the engineers to select the office configuration for their high-ceilinged open space, they settled on floor-to-ceiling “cubes” with doors — essentially mini-offices with as much isolation as possible.

Another time, I had a skip-level meeting with a vice president named Fred, who asked what he could help with. I said that things were fine, though my manager was gone often on overnight trips, and I wasn’t sure how to get certain questions answered or roadblocks cleared in his absence. Rather than offer ideas about other channels, Fred instead immediately made me feel stupid for asking the question: “Why can’t you just wait until the next day?” Never mind.

Other events and decisions continued to convince me that the people at this startup felt we should already all know everything. I quickly began to actively seek something better. (Sure enough, the startup folded about a year after I left.)

Moment of enlightenment

Kent Beck, someone whose writings I’d read and enjoyed, was speaking at a Smalltalk conference I attended in 1999. I perused Beck’s sessions in the conference guide and noted one in particular: Extreme Programming. Huh? I had to hear what that was all about.

We’d done spiral development at MCI in the early 1990s and quickly abandoned it — the spiral model’s short iterative, incremental cycles were a great concept, but we couldn’t figure out how to make it really work. Any problems, such as big defects or missed requirements, threatened our ability to finish a cycle within the allotted time; what then?

As a result of my frustration with spiral, most of the talk on Extreme Programming (XP) was an “aha!” moment. Oh! TDD and acceptance testing are how to continually minimize defects. Oh! The planning game is how to negotiate work within a cycle.

The next year, I saw a trade publication classified ad soliciting a consultant, posted by a company named Object Mentor. (I only had a vague idea who its owner Bob Martin was, though I’d actually met and chatted with him at a prior conference.) Noting that the ad was a bit cheeky, I responded in a similar manner, scored an interview and, much to my excitement, was hired. My primary role was to teach customers about Extreme Programming and object-oriented design. Fun!

Pairing: Those who don’t do, teach

While consulting for Object Mentor customers, I found myself in the slightly awkward position of having to promote and teach pair programming. Pairing, where two developers sit side by side and jointly build software, was alien to me; I still thought of it as what those women at MCI did back in 1992. But the more I sat with other developers and helped them through things like test-driven development, the more I started recognizing the very real benefits that pairing can provide.

It wasn’t until after I’d left Object Mentor that I got to experience day-to-day pairing as part of a development team. I spent more than half a year in 2003 building software on a large (17-person!) programming team, all sitting around a large collection of tables. We paired long and hard for 10, sometimes 11 hours.

At the time I found it exhilarating, educational, tiring and sometimes frustrating. The pairing benefits I’d begun to recognize while at Object Mentor were again realized. I also began to pick up more nuggets about increasing pairing effectiveness (for one, 10 hours was very obviously at least a couple hours too many).

On my own as a consultant and trainer from there on out, I’ve continued to pair with customers. Every couple of years, I find a way to minimize travel and focus on just building software as part of a team. This traveling hiatus, in turn, provides me with more chops, which I turn into credibility for my eventual return to consulting and training.

Training: Pairing, randoris and mobs

Ever since creating and teaching most of Object Mentor’s test-driven development (TDD) training material, I’d insisted that students in my classrooms paired during most of their exercises. If I was supposed to be selling pairing, it made no sense to promote something else during training.

In a classroom environment, pairing helps level out everyone’s capabilities. Every class includes at least one stellar performer who completes their exercises well before everyone else. They then often begin to distract others or me, to the detriment of the rest of the class, or they check out entirely.

Just about every class also includes at least one complete novice who is barely able to program. Left to their own devices, they spin wheels. Once I can get around to help them, these novices often suck the majority of my time during exercises, which means other students get short shrift.

Pairing helps even out the rate of development. It also makes it easier for me to spot pairs in trouble. If I see a couple novices working together — the blind leading the blind — I split them up and ensure a seasoned developer works with each.

Some years back, I introduced the concept of a randori-style dojo in smaller classes. A pair of students would initiate a TDD exercise in front of the class, and another programmer would swap into the pair after a few minutes. The randori style keeps the class lively and provides a technique that students can use for entertaining sessions after the class is over.

A bit later, I discovered mob programming. Put simply, mob programming is the utmost in collaboration: The whole team works in the same room in front of a single large monitor. One programmer is at the keyboard listening to the rest of the room — the mob — and translating their directions into computer code. The programmer’s session is typically short, perhaps five or ten minutes, before another programmer takes the keyboard.

With mobbing, developers continually learn from each other and about each other. They dramatically increase their ability to converse about the code at high levels of abstraction. They minimize waste by working on one thing at a time and curtailing ceremony. For these and many other reasons, mob programming streamlines the development process.

Mob teams often tell me, “We go faster.” I believe them. I also find mobbing generally more enjoyable than either pairing or solo development, so it has become my preferred mode.

Although my style of working has gotten increasingly more collaborative over the years, solo programming, pairing, and mobbing each has its place in modern software development. Settling on one mode is likely a mistake. Consider instead that each mode is a tool that should be considered for any given job. It’s up to you how much collaboration is appropriate for your situation and your team.

The brave new world of continuous integration and continuous deployment is exciting: It promises, among other benefits, better and more consistent testing. Too often, though, the reality is far different. Despite our best intentions for better control over errors, our CI and CD practices introduce new sources for errors.

A little care with crucial details eliminates many of these problems. Your goal should be to have your continuous integration tool — for this article, I’ll talk about Jenkins, but it also could be Bamboo, TravisCI or another alternative — work for you rather than your having to work around it.

Here are five ideas to help ensure that outcome.

Test the Pipeline, Too

Left to themselves, developers are likely to think of “testing” narrowly, and the source code they write as the central asset of the organization. A testing department has the opportunity to communicate that the best results come when all the parts of a solution team up in a balanced way.

In this perspective, testing’s target extends beyond source code; it also includes other elements such as third-party libraries, the repositories used, documentation, and the automation of Jenkins and other tools. It is crucial, in fact, to validate all Jenkins configurations. That means not just verifying that they produce good applications or installations when the repository is healthy, but also that they respond appropriately to errors.

Consider this even more concretely: a typical test of a product might be something like: “Make sure that a login screen comes up within two seconds after the application is launched.”

To test a Jenkins configuration likely involves items such as: “If a source file in the repository includes a syntactic error, then a notice goes to $RECIPIENT within $CI_CYCLE minutes.”

Just as in other kinds of testing, it is important to test Jenkins both in its happy (successful) and unhappy (failure) branches.

Make sure that your tool is under control

Source code must have a home in an authoritative repository. While developers assume that to be true about what they write, that same repository is where your Jenkins configuration (all config.xml-s and *.txt-s, to start) also needs to live.

At the same time, the base Jenkins installation deserves to be documented. To have both base Jenkins and its site-specific customizations documented and controlled should give reproducible results.

Everyone who works in business continuity or disaster recovery knows, however, that backups only have value when recovery is regularly tested. This is another test that deserves periodic execution: ensuring the Jenkins installation can be recreated as though the local Jenkins server had been lost.

Jenkins creator Kohsuke Kawaguchi recognizes that Jenkins remains brittle. The value Jenkins provides is too great to rely on it without good backups and tests of recovery from those backups.

Minimize script complexity

A fully-automated, tested, backed-up Jenkins pipeline is an important milestone on the way to your continuous testing or continuous deployment goals. It is the base on which an organization begins to build real value. Once that base is in place, decide deliberately how much configuration to maintain through the web (TTW). Jenkins and similar products make it easy to edit and define simple pipelines in the Build tab.

In principle, those definitions can expand to handle more complex cases; the content of the Build tab can be arbitrarily long and involved. However, for most organizations, a better strategy is to restrict Build to something as simple as

/opt/local/$PRODUCT1/build.sh

or

powershell /special/build1.ps1 automatic

In this scheme, build.sh embeds all the construction logic necessary for $PRODUCT1; most crucially, build.sh can be tested, measured and versioned outside Jenkins. Reserve Jenkins for high-level automation of business operations, and keep the implementation complexity in scripts maintained independently of Jenkins.

Recognize the “jenkins” user

A conventional Jenkins installation executes processes with the jenkins user as owner. Depending on the details of your operating system and other services involved, this means system administrators might need to:

• Add the jenkins user to the docker group with usermod -aG docker jenkins
• Run triggers within Jenkins crontab
• Allow anonymous read access so that developers and testers can view job results

Security settings for Jenkins are a large and complicated subject, as is inevitable for an architecture that supports plugins and scripts; it can be nearly a full-time job to keep track of In-Process Script Approval for a highly customized installation. The first step to a comprehension of Jenkins security is awareness of the jenkins user.

Restart wisely

Jenkins’ restart behavior is subtly different in different releases. In general, Jenkins distinguishes “restart” from “safe restart”; the latter finishes running jobs before restarting.

One common way to restart safely is TTW at <URL: $BASE_URL/safeRestart > or the corresponding Jenkins command-line invocation:

java -jar jenkins-cli.jar -s $BASE_URL restart

There are also command-line variants of the safe restart that depend on the host operating system, such as the following for macOS:

sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist

Conclusion

Embarking on continuous integration and continuous deployment without also employing continuous testing will only introduce more sources for errors. A CI tool such as Jenkins can help you, as long as you are careful with the details.

Precisely because the tool is so powerful, it&#39;s essential to understand and test Jenkins pipelines with the same care as all the other software components that go into a finished product. By using these five techniques, you can get the most out of your CI tool.

Serverless computing is a game changer. Before serverless computing came along, the deployment unit for an application was a physical computer, a virtual machine or a container. In the serverless computing paradigm the function is the deployment unit. It’s a fundamentally different way of doing business. All computing resources, such as CPU, storage, network and memory have been virtualized away. All that remains is the function. (See Figure 1.)

Figure 1: The evolution of deployment units from physical computers to abstracted functions is core to Serverless Computing

The role of the developer in the serverless computing paradigm is to create sets of cloud-based functions that are accessed by way of an API. The API is realized as endpoints within an API Gateway scheme. (Figure 2.)

Figure 2: In a serverless computing environment, application behavior is defined by functions that are represented by endpoints in an API Gateway

Once the serverless functions are defined and bound to an API by way of an API Gateway, they are accessed by clients on the internet. The client can be a web page, a mobile app, an IoT device or another service on the web.

Also, a serverless function can be configured to respond to a system event, for example when a file is added to a cloud storage technology such as AWS S3 or Google Cloud Storage (see Figure 3), or when receiving a message from a message queue (see Figure 4).

Figure 3: A serverless function can be configured to respond to an event when a file is added to a cloud drive

Figure 4: A serverless function can be configured to subscribe to messages on a message queue

Serverless functions are compelling in that all the overhead of computing — memory, CPU, storage, and networking — is delegated to the cloud provider. The essential development work, creating the application logic, is left to the developer. And, as stated before, this logic is deployed as a set of cloud-based functions.

Most of the major cloud providers have serverless function products. AWS has Lambda. Google has Cloud Functions. Microsoft has Azure Functions. IBM has IBM Cloud Functions. Red Hat has OpenShift Cloud Functions. At the operational level, each works a bit differently in terms of implementation, but conceptually they are very similar.

Creating serverless functions cloud using a cloud console

As mentioned above, all the major cloud providers offer serverless functions. Serverless functions can be created manually using the console of a particular cloud provider. Figure 5 shows the online Amazon Web Services (AWS) console that is used to create a Lambda serverless function.

Figure 5: After you click the Test button (1) to exercise a serverless function in AWS Lambda, execution time (2) is reported in the test results

The AWS Lambda has a feature that allows you to run and test serverless functions directly in the console.

Figure 6, below, shows the console for creating Google Cloud Functions. This console is similar to the one in AWS Lambda in that you write code directly into and test with the console.

Figure 6: After you click the Testing button (1) to exercise a serverless function in Google Cloud Functions, execution time (2) is reported in the test results

Microsoft provides the capability to create a serverless function directly in the Azure portal. In addition, developers can create Azure Functions using Visual Studio or VS Code. Figure 7 shows an Azure Function being developed in VS Code using JavaScript under Node.js as the programming environment.

Figure 7: Running an Azure Function in debug mode (1) under VS Code using the Azure Functions extension reports function execution time. (2)

VS Code has an installable Azure extension that provides the capability to emulate the context and request that is passed to the serverless function. This means that Azure Functions can be debugged directly in VS Code.

A context object describes the run-time environment in which the serverless function is running. The request describes the HTTP request that is passed to the function. Context and request are not special to Azure Functions. All the serverless function technologies use context objects at a conceptual level.

Working with serverless functions at the command line

In addition to using a web-based console to create a serverless function, you can create one using the cloud provider’s SDK. As with consoles, all the major cloud providers publish an SDK that allows developers to create code in a local environment and then publish and test the serverless function in the cloud using the command line.

The following example from AWS shows you how to send a Lambda stored in a zip drive on a local system to the cloud using only the command line:

$ aws lambda create-function \
--region region \
--function-name helloworld \
--zip-file fileb://file-path/helloworld.zip \
--role role-arn \
--handler helloworld.handler \
--runtime nodejs6.10 \
--profile adminuser

Below shows the single command required to publish a Google Cloud Function from Node.js code stored in a local directory to Google Cloud using the Google Cloud SDK (For details of the example click here.):

$ gcloud functions deploy helloGET --runtime nodejs6 --trigger-http

The Azure CLI SDK makes it possible to publish Azure Functions from the command line too, with a process that is similar to the AWS and Google Cloud:

$ az functionapp create --name <app_name> --storage-account <storage_name> --resource-group myResourceGroup \
--plan myAppServicePlan --deployment-source-url https://github.com/Azure-Samples/functions-quickstart-linux

Command line creation of an Azure Function requires a bit of upfront configuration to make a deployment. You can read the details of the process at this link.

Being able to work at the command line is useful in terms of deployment efficiency. However, the real impact is that command line manipulation of serverless functions opens a door that allows the technology to be used in a Continuous Integration/Continuous Deployment CI/CD process. Controlling serverless function technology under CI/CD allows it to be used under automation at web scale.

Putting it all together

Serverless computing provides many benefits to the software development process. Eliminating the extraneous overhead that goes with the software development process will allow developers to focus on what they do best: create behavior. Isolating the deployment unit to the function means that more software can be written faster and tested at a finer grain. However, as with any new technology, there is a learning curve. Also, companies will need to adjust their source control, testing and release processes to accommodate the “functions only” approach to software development. But, after an initial adoption phase, the benefits that serverless computing provide will make it worth the investment of time and money required.

If you observe an average software development team, you’ll notice many things going on at once. You’ll see walls overloaded with story cards representing numerous pieces of work in progress (WIP). You’ll watch developers and testers move to and from meetings throughout the day, and occasionally turn their chairs to ask and answer questions. You’ll observe programmers slamming out a lot of code that accomplishes myriad bits of functionality. You’ll see people create involved automated tests to exercise and verify bits of behavior.

The hustle and bustle create the appearance that the team is getting things done. Apparently, we think multitasking is a good thing — and worse, we believe we’re effective at it.

Todd Herman, founder of the time-management program The 90 Day Year, tells us that “humans aren’t built to multitask.” He mentions the American Psychological Association’s contention that we can lose up to 20 percent of our time when we must switch between two tasks.

In software development, we are at risk of wasting even more time, because we lose more time as tasks increase in complexity. Gerald Weinberg’s oft-cited chart from his book “Quality Software Management: Systems Thinking” shows that you end up with maybe 25 percent of your time being spent effectively if you switch among five projects.

The core reason offered by Herman for the productivity loss created by context switching relates to the concept of flow state — the notion of “being in the zone.” Herman estimates it can take a dozen minutes to get back into a flow state once you’ve been distracted with something else.

The key, then, is to focus on only one thing at a time — OTAAT. Let’s look at some examples of where we can apply this method to regain some lost productivity.

Testing OTAAT

Even when we think we’re working on the same thing, we tend to conflate one concept with others. I often see automated tests that try to verify numerous things:

Bank account test

Create account A
Deposit $50 into A
Verify A has a balance of $50
Request withdrawal of $8 from A
Verify A has a balance of $42
Request withdrawal of $48 from A
Verify that the system presents an error message
Verify A has a balance of $42

It’s seemingly easier to create and run one long test like this. The problem is that when one element fails silently, it often takes longer to figure out why a subsequent verification fails, particularly as the test increases in complexity.

A longer test demonstrates and verifies numerous things. As the test becomes longer, its intent becomes less clear. You can attach only a vague summary description, such as “Bank account test,” to such a test.

OTAAT applies well to tests: Better we split one longer test into multiple tests, each of which verifies one useful behavior. We can name each test with a concise summary of what it verifies:

• Deposit increases account balance
• Withdrawal decreases account balance
• Excess withdrawal generates error

This way, we create more easily understood tests. When any of these one-purpose tests fails, our efforts to uncover the problem are less than with the multipurpose test.

Test-Driving OTAAT

Test-driven development’s cycle of test-code-refactor encourages developers to focus on one element of that short cycle at a time. Write a new test, watch it fail. Write code to make that test pass, ensure that the new test and all existing tests still pass. Clean up the code through refactoring, ensure that all tests still pass.

However, many developers find themselves tempted by bad code as they attempt to get a test passing: “This stuff is tough to understand; let me clean it up a bit.” They then find themselves in a bind — the tests are now failing, they’ve written a good amount of new code, and it’s become hard to separate the changed code from the new code. They waste gobs of time.

When stuck in a mire, the best solution often involves discarding both new and refactored code by reverting to the last known good state. From that clean starting point, the better approach is to refactor first, then write code, or vice versa. Small, discrete steps are almost always more effective.

Don’t refactor and add new code simultaneously. OTAAT!

Building OTAAT

The crowded story wall is a visible symptom of tackling too much work at once. We learned from lean manufacturing that having multiple tasks in process prolongs time to delivery and actually increases costs. In addition to the aforementioned cost of context switching that high amounts of WIP can encourage, high WIP levels create additional complexities in coordination, tie up resources, and allow problems to remain buried longer.

Perhaps the worst aspect of excessive WIP is that it distracts us from our core goal of delivering value to customers. In turn, we lose our ability to learn how to get good at continually doing just that.

I’ve often promoted pair programming, a collaboration model where two developers sit beside each other to actively build a software product. Developers produce higher quality software as a result because they actively review the code as they go.

For teams that pair, I also recommend frequently switching pair partners — at least once per day. A new person stepping into the mix for a not-yet-completed task provides a third-party review for software produced so far. Typically, by the time a pair marks a task as complete and integrates it into the source repository, it’s too late to remedy any quality problems. A third party can help prevent bad software.

Unfortunately, pair switching represents a context switch and thus engenders a bit of slowdown. We just have to bet that the cost of the context-switching slowdown is less than the cost of low-quality code produced by an isolated pair.

Lately, I’ve promoted mobbing over pair programming. Because everyone in the room works on the same thing at the same time, they don’t lose time to context switching. And in case it’s not obvious, we’re all working on one thing at a time. The effective WIP for a team that mobs is generally one item. We don’t experience the productivity loss of context switching. OTAAT for the win!

Hundreds of thousands of software applications are scheduled for migration from Python 2 to Python 3. The deadline is approaching fast: Support ends with 2019.

Abundant articles explain the importance of and techniques for this migration from a developer’s standpoint, including How to Port Python 2 Code to Python 3, Conservative Python 3 Porting Guide and official documentation for Porting Python 2 Code to Python 3, as well as several valuable video tutorials and other resources.

But how can testers help ensure a Python 3 migration is successful? Testing’s unique perspective has plenty to contribute.

Planning and preparation

First, recognize and promote the importance of planning and preparation. Programmers have a tendency to see programming as the solution to problems, and they often benefit from reminders to plan more strategically before the coding starts. A testing department can help developers clarify ahead of time such questions as:

• Is the product source migrated to Python 3 still expected to be able to run with Python 2.7 during a transition period? How long is that transition?
• Will the product source migrated to Python 3 be developed as a branch or on the main trunk of development?
• Once the migrated source goes into production, what will it take to revert to sources compatible with Python 2.7, if reversion is decided necessary?
• Do the testing sources need to migrate? How is their schedule coupled to that of the product sources?
• Can the deployment be targeted for a “quiet time” or “null sprint” when no functional changes are expected? Migration while trying to stay on top of other difficulties and priorities is asking for trouble.
• Will acceptance be on the same rhythm as other releases?

That last point deserves more explanation. Suppose the product to be migrated typically deploys a new release roughly weekly. Once deployments pass all automated tests in this hypothetical practice, they’re regarded as provisionally complete for another 24 hours, to give surprises time to surface. Although deployment schedules vary widely — from minutes to months — most organizations at least recognize these elements.

Should the move to Python 3 be on the same schedule as a “normal” deployment? Any rational answer depends on the comprehensiveness of tests. While a collection of unit tests and a few sanity checks at the integration level might be adequate for most releases, migration to Python 3 tends to bring out whole new categories of problems.

Anything to do with Unicode deserves careful inspection. Quite a few projects have thought their migration complete, then discovered problems only as foreign-language users began to exercise the application. Encoding dependencies pop up in places that don’t occur to the naive: CSV readers, pickles, cookie handlers, cachers and more. Defects or incompatibilities in these places usually have straightforward, backward-compatible resolutions, but someone needs to detect and identify them first. To do so for a release already in production is clumsy at best.

Does your team operate with automated tests? If it hasn’t made that step yet, can it install automated steps before the migration? If the product is a purely text-based library or service, maybe automation is straightforward and yields good coverage. On the other hand, if the product is a web, mobile or desktop application, only a specialized test automation tool can reduce the risk involved in a change as large as that from Python 2 to Python 3.

Ranorex, which underwrites this blog, offers a top-flight testing tool, Ranorex Studio; among other features, Ranorex Studio’s recorder generates customizable scripts in Python. Part of your analysis of the migration should be to define and plan adequate test automation well before deployment of the migrated source.

Performance surprises

Another significant risk of migration has to do with performance degradation. Python 3 is mostly faster than Python 2, and it uses less memory for many important situations — when the source is idiomatic. Problems can arise, though, with subtle semantic shifts.

Python 3 often provides iterators in place of corresponding lists from Python 2; range() is an example. Iterators are generally far thriftier with memory but demand more time when re-created repeatedly. Essentially, expect a Python 3 version to be quicker and take up less memory, but perhaps only after a few adjustments.

A few other hazards of migration are well-documented. Python 2 computes 5 / 3 as 1, while Python 3 gives a result of 1.6666666666666667. In Python 3, the way to arrive at 1 for this calculation is with 5 // 3.

Aligning expectations

All these coding pitfalls have relatively straightforward solutions. Each one demands careful scrutiny though, and errors can combine to upend schedules — and it won’t only be coders who are affected. More even than technical tools or programming heroism, it’s important for testing and development departments to align expectations ahead of time.

Python is on its way to becoming the world’s most popular programming language, so the move to Python 3 will affect many such departments. Explicit plans, well in advance of programming changes, give testers time to prepare and practice with all the materials and tools they need to verify a successful migration

We had recently hired a new software test engineer into our small group, and as part of the orientation, it fell to me to show them our automation suite. I walked through the examples I had prepared, showing what we did to create the tests, how they were integrated with our product, and the steps necessary to commit new tests and use them with our product. By the time I had finished the demonstration, I had hoped that they’d be impressed.

The reaction I received was a little different from what I expected: “Um, I don’t want to be rude … but have you considered just starting over?”

I was a little bit taken aback by this, but truth be told, it was something I had considered as well.

We had a test automation suite that was robust, provided broad and deep coverage, and took advantage of being integrated closely with our product. In fact, we ran our tests literally within the framework of our application and were able to edit our tests directly in our application. At the time it was designed, it was seen as elegant, was strongly supported by the development team, and had stood the test of time. However, to everything there is a season, and this conversation was the beginning of my realization that our test suite’s season had come and gone.

True, there was a lot of underlying code developed to help support our tests, but that also made it difficult to upgrade or update tests. Tests were written with Perl as the underlying language because once upon a time, our team had a lot of people who worked with Perl regularly. Over time, though, turnover and new hires caused a shift in that expertise. Where we once had a solid group of people able to create modules and maintain the code base, that number gradually shrank. The tight coupling we had with our own product and a variety of techniques we used to streamline our continuous integration approach was also showing its age.

As we weighed the options of adding a number of new features, we came to the conclusion that we could either work hard and aggressively modify what we had in place to focus on the new features, or we could, as our new hire suggested, think about “starting over.”

Ultimately, we decided that we would keep the existing framework in place for the older features but start with something new for the new feature development. This was pushed to the forefront of our efforts when our company was acquired and the new management team asked us to focus on streamlining technologies.

If a product or application survives in the market for long enough, it will go through several evolutions. Few products are exactly the same as when they were first introduced. Our product had been in the market for a decade by the time we were acquired by a larger organization. What they envisioned doing with our product was different from what we had originally planned for its primary use and in how it would integrate. Specifically, we were asked to consider how we could make a test automation infrastructure that would not just look at our specific product, but also consider how it might interact effectively with the broader range of products inside its ecosystem.

Starting over does not necessarily mean starting from ground zero

Legacy applications have a user base already in place, and that user base expects key functionality to work in a reliable manner. That means that, even if an application will be redesigned or features will be added, it’s not likely to have everything redone at the same time.

A good example is a web application that is enhanced to allow for responsive design. The look and feel of the application will change based on the dimensions of the browser or the reported user agent. Depending on the level of device support, this may provide two or more views and groups of elements to work with.

If a legacy application has existing test automation in place, odds are that that automation will still be useful for the existing functionality. But it will likely not be as useful for new functionality (if it can be used at all). Rather than focus on doing a full rewrite of the automation code, instead use the new functionality or feature set to create new tests, using new techniques or tools if necessary.

How far down the road can you see?

What may have been a promising platform at one time may fall out of favor down the road. Additionally, with companies merging or trying to streamline product offerings, a testing team may be asked to have their infrastructure fall in line with an existing framework. There can be a variety of reasons for this. Usually, it is because there is already a level of expertise in place, but just as important is the ability to train new people to use that technology.

When companies get acquired or merge, it is often advantageous to try to blend products to work as seamlessly together as possible. While it is possible to take products that exist on different platforms and with different operating systems and supporting tools and get them to work together, having to support multiple frameworks and languages can be more of a burden in the long run than standardizing on a few choices. That can be an advantage if the technologies chosen align with what the team is already using. However, it is just as likely that an engineering group will be asked to migrate to another platform to correspond with products already being used.

If a team has visibility into this and what the desired technology stack will ultimately be, it will be easier to train individuals on the team to be effective users, developers, and testers of those technologies. There still may be considerable need for learning and adaptation to get proficient with those technologies.

Separating the tools from the problems

One of the biggest challenges with any test automation strategy is understanding what problems actually need to be overcome. It is common to look at test automation as having a hammer and, thus, every problem to be addressed as though it were a nail. But there may be a variety of solutions that do not or will not fit into a standard test automation model.

It is common to consider the steps we take to run manual tests, look at what was necessary to work through those steps, and then automate them for future operations. Can those efforts be valuable? Certainly. Will they always be? Not necessarily, or at least not as valuable as we might think. Just because a test can be automated doesn’t necessarily mean that it should be, particularly when using a test tool means forcing it into that paradigm.

Consider the example of adding users to a system. If an application allows for adding individual users through a user interface, is it worth automating that process? To test a single user being added or a couple of different possible workflows, the answer is yes. Does it make sense to add hundreds of users with the same mechanism? If the goal is to stress test the front end over time with repetitive actions, then the answer could be maybe. However, if speed and volume are important, there are usually far better ways to do this type of action, such as with shell scripts or database queries to create bulk user transactions from existing data.

When in doubt, go for small and reusable

It is natural to think of tests or interactions with a system as a process of stringing together workflows. We rarely just perform atomic tasks with a system. Instead, we perform a variety of tasks so that we can accomplish a goal.

Likewise, it’s natural to consider test automation in the same way. We look to create tests that will allow us to get from point A to point Z. In some cases, having lengthy end-to-end tests is the right thing to do, but it comes at a cost. Maintaining such tests may prove to be difficult, if not impossible.

By breaking up individual tests into smaller components that can be run in a sequence, it is possible to modify workflows if needed. Additionally, if one step of a workflow changes, it is easier to make a single change to that one area than it is to try to go through and rework a long test with multiple steps. By using a modular approach, it makes tests and test components easier to maintain.

There are patterns in everything

While it may feel like a daunting task to put together new test automation, in truth there are a handful of basic interactions that are repeated time and again. If the focus is on a user interface, depending on the device there may be some variation, but most users will type text into a text box, select a drop-down menu to make a choice, click on a radio button or checkbox to select an option or set of options, and click buttons or links to perform operations or navigate to a new location. Those actions are not as numerous as we might initially think.

With a little forethought, it might be possible to make simple components that can be reused in multiple places. By looking for these patterns, we can determine which areas would be easier to make into small libraries or function calls. Rather than have a number of different statements to press a variety of buttons, it may be easier to simply have one method or function that presses a button, with the value of said button a variable that uses an element ID or class. Likewise, clicking on a link or selecting from a drop-down menu will probably be similar in most cases. Leveraging the patterns of interaction can help us make code that is easier to maintain, or at least not have us repeat the same process multiple times.

By taking the time to consider each of these aspects and how to work with them effectively, retooling test automation for a legacy system need not be a terrifying undertaking. Sometimes starting over may indeed be the best bet. Still, remember that what is new and shiny today may require the same overhaul sometime down the road, so get in the habit of starting over frequently. It may save headaches and heartaches down the line.

You might have already wondered; Ranorex does object-based recording, so a click action, for example, is always performed on a specific UI element, no matter how it looks or what screen coordinates it has. However, if you look closely on the action spot of a click action, you can still see that behind the word “Relative” a strange value like 51;13 is hidden. So, does this mean that Ranorex still records coordinate-based? And can I change this behavior? And what side-effects could this have? This small article should answer all those questions.

Coordinate Recording Mode

The setting that defines this behavior is the Coordinate Recording Mode that can be found in Settings -> Recorder Defaults. This setting has three possible values: Pixel, None, and Proportional. This setting defines how the exact action spot within a UI element is determined. The default is Pixel. This setting records the exact location of a click within the UI element. For example, if you click on a button in recording mode near its top left corner, you will get small coordinate value like 5;8, since the top left corner always has the coordinates 0;0. These coordinates use integer values and will also be used for playback. The setting None behaves differently: Here, it doesn’t matter where on the element you click in recording mode, Ranorex will use the action spot Center and in playback will always click on the middle part of the button. The setting Proportional uses coordinates in the form of .5;.6, thus using floating point numbers instead of integers. This means that if you click on a pixel in recording mode that is 30% of the button size from the top, then in playback Ranorex will also click a pixel that is 30% from the top in playback mode.

Advantages and disadvantages

All three of these settings each have a set of disadvantages and advantages. In fact, None might seem to be the best setting, but Pixel is the default value in Ranorex.

Pixel

This setting is the default in Ranorex. It recognizes the exact (=pixel-perfect) action spot in the UI element in recording mode and uses these coordinates for playback. This has one huge advantage: When running the test, Ranorex does exactly the same thing that you were doing when you recorded the test. At least as long as you stay on your same screen with the same screen resolution and the same screen scaling, that is. In modern applications, especially web applications, responsive design has become increasingly more important, and a button or a text field might change its size drastically, depending on the size of the application window, the screen, the device orientation, etc. For example, on the Amazon homepage, you might want to click into the search field, somewhere in the middle:

If we put our browser from maximized mode into a much smaller window size and try to run the test, this click suddenly doesn’t work anymore, because the coordinates that I have recorded (958;17 in my case) are suddenly outside of this element, since this text box has shrunk in size. This warning looks like this:

Suddenly, Pixel looks like a far from the optimal default setting. Especially in modern, web-based applications that use responsive design, this setting is far from ideal. So why is it that it still is the default setting in Ranorex? Consider this element in your application:

Unfortunately, the checkbox with its label is recognized as just one element. Especially in deprecated desktop technologies, this is very common. You also often see the small triangles or plus-symbols that should expand a section of a list or tree – those are also often not recognized as separate buttons. However, automating only works if you exactly click these buttons, not the text near it. This makes it necessary that a click is performed on the exact same spot within the control as it was recorded. Would the click just be performed on the Center location of that element, the checkbox in this example wouldn’t get ticked.

None

This setting works very differently, and it changes the behavior of the test run drastically compared to the recording. Every action spot is set to Center. This might sound as if it makes sense in many cases, especially when you have responsive design, but it has a few serious drawbacks. For example, consider you want to click on this headline on a web page:

When recording, you would click this text, and it would work. However, in the test run, the action spot would be Center, and therefore the click spot would not actually be on the text itself:

Although the value None might seem like a good setting on the first glance, you can run into these issues frequently. So, is there a better way to do it, if both Pixel and None have some big drawbacks?

Proportional

This setting tries to, similarly to Pixel, retain the action spot exactly where it was while recording, but in relation to the size of the element. This means that a click on a spot that is 30% from the left edge and 60% from the top edge will also be done 30% from the left and 60% from the top in the test run, no matter how big or small the element now is. This mitigates the problem that the Pixel setting had: if a button shrinks, the action spot is still inside of the button. And some issues that None has are also solved, but still, Proportional suffers from similar issues. For example, a click here…

…would translate to a click here in the test run, if the screen size becomes a bit smaller:

Thus, the setting Proportional suffers from very similar issues to None. This setting is represented by floating point numbers in Ranorex. For example, the Center location could also be written as 5;.5 in proportional notation.

The Best Setting

So, the question is: What is the best setting for me? The answer is pretty unsatisfying: It depends on your application. Pixel is the default because we think that if you record a test and then run it afterward without changing anything in between, it should run. None does not have this property, and while None looks logical up to a certain degree, Proportional seems to do very illogical things if you don’t know how the action spot is calculated. Therefore, we have decided to stick with Pixel as the default. This is often best for pure desktop applications and works fine if you don’t change the screen resolution. None and Proportional both have their uses in applications with responsive design (mostly in the web and on mobile devices), however, they can have strange side-effects. In all three cases, depending on your actual UI element, it might be necessary to change the action spot manually if it doesn’t do what you want. There is no cure-all type of setting. All of them have advantages and disadvantages. If you know what properties those settings have, you can easily pick the one that suits your project best.

When your Test Suite starts to grow and gets more complex, it’s a good idea to take a look at its overall performance and run duration. Identifying time-consuming bottlenecks in your tests has never been easier, as Ranorex Studio 9 comes with powerful analysis tools. Learn how Ranorex managed to tremendously cut down on their daily test run duration by using this new feature.

Fast feedback on new code changes marks one of the fundamentals of agile development processes. If you’re using continuous integration and run daily tests, you’ll want to make sure that results are available on a regular basis. Developers can then investigate where the Test Cases failed and react faster on bugs which might have occurred.

When your products grow more complex and developers add features to the mix, they also affect your Test Suite’s complexity. At a certain point the promise to deliver quick results might turn into a challenge. Sounds familiar to you? You might have reached a point where you should check the performance of your Test Suite.

To ensure constant high product quality, Ranorex runs an automated Test Suite to examine its own software builds every night. The massive Test Suite takes around 12 hours to complete. In order to cut down on this, it’s important to find out which parts are having an impact on the performance. In previous versions of Ranorex, one might have had to use the Report to find out about this, but they were limited in detail. Enter the new Performance Tracing feature in Ranorex Studio 9.

Performance Tracing in Ranorex Studio 9

Performance Tracing gives you all the insights you need to understand which parts of your Test Suite are having an impact at the runtime performance. You will receive detailed data reports on every action that occurred during the test run. You can enable Performance Tracing in the Test Suite settings by selecting a tracer. Ranorex Studio 9 comes with multiple tracers that let you specifically look into an area of interest.

Select a tracer you like to investigate in the Test Suite settings

The feature is disabled by default. You’ll notice that there are tracers for various areas of the test run. If you’re not sure yet in which area you would like to investigate, take the “All” tracer as a starting point. Once you have enabled a tracer and run your Test Suite, Ranorex will generate a CSV file containing a detailed overview of timings for each action. This is not limited to Recording actions, but also provides you insights in the overall workings of your test. This also works when you use Ranorex Remote to execute your test on an agent. Usually, you’ll find the file in the Documents folder on your SUT, but you can specify a different location in the Test Suite settings.

Having a first look on a Repository trace log. A search counter of 7 is a good indicator bearing some RanoreXPath optimization potential.

Working with the Data

When you open a CSV file you will find a log of all actions that Ranorex Studio executed during the test run. Depending on the tracer you’ve selected, the data generated might look different. If you choose the Repo tracer, for example, you’ll get an overview of the RanoreXPaths used and the number of attempts to find the said element.

You will also see the duration, in milliseconds, that it took to complete the action. Obviously, you’ll want to sort the list and have a look at the actions taking the most time. However, you can get much more information from the reports. You can see through the actions to spot delays you might have put during test development and forgot to remove or see unnecessary repetitions. Apart from that, you can use analysis tools like Microsoft Power BI to import the raw data and create dashboard-like reports on your test performance.

Use data analysis tools to get insights like never before

At Ranorex the performance analysis of their daily-run Test Suite revealed a lot of potential for optimizations. Thanks to the insights gained from the performance tracing reports more than 2 hours (almost 20%) of runtime could be conserved. Now it’s your turn to give it a try and optimize your Test Suites!

I remember quite vividly when I was confronted with one of the paradoxes of commercial computing. It was around 1983. The high school where I was teaching opened up a computer lab filled with Apple II computers running the command line-based Apple DOS operating system. I was hearing from a number of sources that computers were cool and I should learn how to work one, so I did.

At the time a friend was also getting into computers as a way of life. One day I was over at his place and he was showing off a program he liked. His computer of choice was an IBM PC running MS-DOS. I remember being impressed with the program and told him I might get a copy for myself. He told me I couldn’t. “Why?” I asked.

His response was, “It only runs on MS-DOS.”

My reaction was akin to a child learning that there’s no Santa Claus. At the time all I understood about operating systems was that a computer needed one to work. I really didn’t have any conception that there were operating systems out there other than the Apple DOS that ran on the machines in the computer lab.
Operating systems mattered a lot back then. In fact, they still do, despite decades of effort trying to unify the computing experience using cross-platform languages such as Java, JavaScript, and HTML.

Those of us who embraced virtualization and cloud computing hoped that these technologies would break the hold operating systems have on the computing experience. They really haven’t. While there have been efforts to make operating systems work in a conceptually similar manner, there are differences that need to be understood in order to have the full benefit of the cloud computing experience.

Remote access: A tale of two operating systems

Facilitating remote access is one such example. Linux supports SSH, which is the way most Linux admin access remote Linux servers at the command line. Windows, on the other hand, uses WinRM to facilitate remote access at the command line to Windows machines. This creates a problem when having to move between Linux and Windows targets using a cloud provisioning technology such as Ansible or a CI/CD product such as Jenkins.

If you want to work remotely across operating systems, you need to know how each OS facilitates remote access. Otherwise, you’re stuck in the water.

There are other differences as well. For example, Windows has a system-wide configuration storage and management technology called the Windows Registry, which is left over from the COM days. Microsoft has been trying to minimize the impact of the Registry through the .NET/CLR technology. But the Registry is still used by native Windows applications and .NET applications that use the COM interop, and vice versa.

A plethora of application installation methods

When it comes to installing applications, operating system matters. Linux has a variety of package management systems: yum, apt and dnf. The one you use is particular to the Linux distribution. Mac’s OSX, which is a flavor of Linux, uses homebrew, MacPorts, and pkg and dmg packaging. On the other hand, native Windows uses MSI Windows Installer, and there is an emerging package manager called Chocolatey. Also, for .NET apps, there is the NuGet platform, which facilitates downloading and installing external libraries.

You can’t run yum on a Windows machine. Conversely, you can’t run the MSI installer on Linux. This difference matters a lot when doing automated machine provisioning.

Linux, OSX and Windows will allow you to install a native app by doing nothing more than copying the executable file to disk. However, this method works only under certain conditions, mostly when there are no dependencies on external libraries. When the primary executable binary has a dependency on an external library, then the way the library is known and accessed varies by operating system and, to some degree, application framework.

For example, Windows might use the Registry to locate the dependency. Or it might rely on a library being present in a predefined location in the file system — for instance, by default .NET looks for dependencies in the bin subdirectory of an application should the given library not be present in the General Assembly Cache. (Of course, the base .NET binaries need to be installed beforehand in order for any .NET application to work.)

Linux will also follow file system convention in some cases. Java tries to make the dependency management the same across operating systems; sometimes it works, and sometimes it doesn’t — it depends on the Java Framework in play. For example, the Maven framework is cross-platform, but it has a dependency management system all its own.

Those pesky little command line differences

And then, of course, there are those pesky command line differences. For example, to copy a file in Linux, you use the command cp, while Windows has copy. To get network information in Linux, you use ifconfig. In Windows, it’s ipconfg. Linux has a filtering command, grep, which is bread and butter for Linux power users, but Windows does not support grep. These are but a few examples of many standard command differences.

In terms of the shell itself, Windows has the standard command prompt. To provide more power at the command line, Microsoft introduced a beefed-up shell called PowerShell, which has a learning curve all its own. PowerShell has some similarities to the standard DOS shell but has much more power. For example, PowerShell supports .NET system objects and C# programming at the command line. Linux goes another route: It has the standard shell, sh, as well as beefier shells such as bash, the korn shell and fish.

Two distinct approaches to doing security configuration

The way you do security for users and groups on a machine is different too. Handling users and groups is a lot easier at the command line in Linux, while Windows is a bit more complex.

The users and groups GUI hides a lot of the management complexity in Windows. Working through the GUI is the preferred method of interaction in Windows, even for Windows Server. However, this incurs a lot more overhead in the form of more disk, memory, and CPU required to support the GUI. Server editions of Linux ship command line only, so they are more lightweight. Yes, you can add a GUI to a Linux server, but that’s considered the sign of an amateur.

Putting it all together

Someone who is used to working with Linux in the cloud and then needs to move over to Windows — for example, to support working with Ranorex Studio — might experience a significant learning curve. The same is true for someone moving from Windows to Linux. Consequently, those working in the cloud will do well to have a basic familiarity working with each OS at the command line. Having such cross-platform understanding will save time and heartache if they ever want to work in an unfamiliar environment.

Not only that, but when you set up your environment, take notes. If it is in the cloud, you can script it up with code. The important thing is to make it reproducible, so the next hire, or replacement, won’t have to go through all that work themselves just to get started.

Consider a simple web application for a team transitioning to DevOps. The team wants to push new code to production at least every day, if not every few hours, and simply does not have enough time to explore it personally.

They want to use a tool to “drive” the application and make sure core functionality stays in place. To test the login process, the tool needs to find the username text field, type into it, find the password text field, type into it, and find the submit button and click it. Then it needs to find objects that appear on the next screen and make sure they are there, or not, depending on the type of test.

Then all of a sudden there is a user interface change and all the tests break. So, the team has to go in and change every test to use the slightly different connection string. That is a lot of rewriting.

The modern, under-the-hood hook to objects is often a CSS or XPATH string. For example, here’s the XPATH to a button labeled “Submit”:

//button[@value=”Submit”]

And here’s how to do it in CSS:

input[type=”button”,value=”Submit”]

Using these “magical strings” in tests means that when they change, testers need to do a global search and replace — except that the other submit buttons didn’t move, just the one on the login page. So, the search and replace becomes a manual process.

After making the fix, testers need to rerun the test and make sure they fixed the right things. That’s fine when you have five tests, or even when you have 500. But as Markus Clermont pointed out at Google’s Test Automation Conference over 10 years ago, once Gmail got to 5,000 tests, it was essentially unmaintainable.

The fix: Object repositories

An object repository is a collection of objects, their properties (exists, enabled, visible, innerText, tagName, class, etc.) and their tree traversal path locators, all in a structure that looks like a tree, just like the underlying code for a webpage. This allows controls with locators ranging from very specific to specific only at a component level, using wildcards like ? or * or other very general locators to reduce the possible impact from the changes to the page structure.

Visual testing extends this ability with screen capture and compare for objects. That way, if a test fails, there is a simple process to check that the test actually passed and now use the new image for future comparisons. This massively reduces the cost to update a test, making them maintainable.

Because the tree is modeled after the structure of the user interface (in the case of the web, the Document Object Model, or DOM), the object browser can access parts of the DOM that are perhaps not directly clickable, such as the page title or a linked JavaScript file.

Therefore, when a control transitions to a different location, a single click on the object opens the object repository and you can then adjust the locator right there, either by clicking Track to record a new one or by manually adjusting the XPath or CSS locator it was previously. Imagine we have a locator like:

“/dom[@domain='www.mydomain.com']//div[#'top-posts-2']/h4[@innertext='Top Posts & Pages']”

and that the location and inner text were both changed from being a reference to top posts to recent posts. This could become:

“/dom[@domain='www.mydomain.com’]//div[#'recent-posts-2']/h4[@innertext='Recent Posts & Pages']”

It all can be changed in line, without having to open every test to point it to a new object, thus saving you time as you automate the important things.

A good object repository will have a properties view that enables you to include, remove or update values for the properties to improve the locator in just a few keystrokes or mouse clicks. Notice in this example how the various nodes composing the locator path could also be updated at any point on the path for the needed web element:

An H4 element in the object repository with innerText equal to “Top Posts & Pages”

You may make the changes necessary, record or update the text in line, or change the properties collection that is included in the path, and then click Apply and save. Now you are ready to continue testing. Imagine the minutes you will save browsing through code on your own just to make a couple of small corrections for this one element.

The object repository goes a long way in helping automators focus on what matters. This solves most of the early issues you may encounter when creating test automation.

However, automators can still run into problems.

Issues for automators

Some components may not have strong, recognizable names — for example a radio button might be one of 10 for one radiogroup all called “radiobutton1,” “radiobutton2,” and so forth, and it may not be the only radio button group on the page, or even within the entire application under test. By using an object repository, you have the ability to change the name used in tests to something more meaningful. They may become “smbRadioBtn” and “‘entRadioBtn.” If the programmers do not provide names, an object repository can still find the object, and a tester can give that object a name.

The assumptions of what to use to match the object might involve properties that change frequently. That might make the locator fail, even though it’s still on the screen. The best-known practice is to tie the object to view properties that are unlikely to change and are likely to remain unique on the page.

Better locators will be able to find “button” with specific text, so that isn’t tied to where the button is on the page — unless that matters! Some tools observe these changes over time, along with the application, and can use machine learning to “guess” when an object has moved. We call these self-healing tools, and they have just begun to enter the market.

The object repository solves the stale and poorly named object locator problems, but it can introduce others. For example, when teams of teams are working on the same website, it is possible the teams step on each other, introducing changes in the same page that cause the tests the other team is working on to report a failure. These problems exist with or without object repositories, but you may need to have the object repository versioned right along with the code or placed in branches.

This extra layer of coordination does cause added complexity. Then again, giving names to objects and creating reuse creates simplicity. Think of the object repository, then, like a code library of sorts that holds how to connect to objects, making tests simple.

If your tests are becoming hard to maintain, you might look into tools that include an object repository. In my experience, if you don’t add a tool you’ll likely end up building your own anyway, but unless you’re careful, it will likely be awkward, be brittle and have half the functionality you’d like it to have. Save the time and effort and explore existing options for object repositories. You’ll probably be spared some headaches.

Login, search, create a profile are common things you have to do everywhere. eCommerce sites have “add to cart”, “view cart”, and checkout, while content-creation sites have tag, post, upload image, and edit a page. We can get enough understanding about these terms, easily enough, without even bothering to define the term pattern. So how do we test them?

Let’s start with Login.

Reusable Login Methods

The first thing you need to do in many application is login. It is a blocking condition. The way most testers solve this is by having two different tests. First, they test all the combinations of login, including the interesting failures, as one artifact. Then they have a simple, reusable login script, that appears at the top of every screen. This method combines four commands – type into username, type into password, click submit, wait for confirmation text that you are logged in, typically “hello, [full name as described in profile].” The screen might also verify some standard logged-in elements, or do an image compare. The simplest strategy is to cut and paste this into every test.

Then one day, something happens. The developers add a new field, “company id”, to login, in order to enhance security, like the below example. This is a real screen capture from a regional bank in the United States, which added the ‘commercial’ login flow, which has an additional text field.

All of a sudden, every single method fails. If the code is stored in plain text, a clever programmer can write a search-and-replace algorithm to fix it.

Then again, a clever programmer would have used a code snippet in the first place.

The Code Snippet

Most tools have a way of creating a snippet of reusable logic. The visual tools often create this as a sort of encapsulation. That is, create these four or five visual commands, then collapse them into a virtual command that can be cut and pasted around.

That has the same problem as general cut and paste.

What you want to do is create a method, or function, that can be stored in an external library. That library should be under version control, which we’ll discuss later. When the user interface changes to add a new field, the code only needs to change in one place, that code library. Make the change once and all the tests that failed suddenly pass. Problem solved.

In the case of login, we have some ‘pre-canned’ information. Username, password, and possibly, what to expect on the other side. If we change these to variables, then we can pass in a non-username, an invalid password, and expect an error message. This means our test software needs the capability to pass in variables.

We also want the ability to create variables on the fly. For example, we may want to create users in entirely different billing structures and make sure they cannot find each other with search. Or we may want to run multiple tests at the same time, and make sure those users have different names. My favorite way to do this is to user a date-time-stamp and stick it in a variable. For example “acount20180509130010010101” and “user@account20180509130010010101.com.” A code snippet can generate these. If you need to actually check those emails for received mail, consider using emails with a mailinator.com domain.

Once the pattern for snippets exists for login, you’ll find yourself making all kinds of snippets. Search is particularly easy – search for (term) expect (result text). Starting with a canned, known database of records, or at least one where you can calculate the result, search for terms and show how the search works. The result text could be “Showing 1-20 of 200”, along with the specific results – and these could be in order. Regular expressions are a great way to check order. For example:

/Test Results[\w\W]*Test Planning[\w\W]*Test Execution/

This code matches the three test terms in specific order, separated by any number (*) of word (\w) or non-word (\W). characters … which is anything. Here’s a similar code to match a div with the same text in Ranorex Studio:

div[@text~’Test Results[\w\W]*Test Planning[\w\W]*Test Execution’]

Snippets have different nicknames – in some tools they are called methods, functions, modules, or “user code”. Once you can add variables and regular expressions to your snippets, the opportunity to use them seems to pop up all over the place. The page object pattern is an attempt to define test automation itself in terms of reusable business functions (like login and search) then compose tests out of those functions. To the extent your tooling consists of code (as opposed to visual representations) it is a fantastic, if often misunderstood, place to start.

Use snippets enough and you see everything as a potential snippet.

Except sometimes you don’t want to go through the UI, and want to work at a lower level.

Command-Line Problems Vs Through the UI

Create a profile, in many ways, feels like login. Enter values into these fields, upload an image, confirm the results. If you leave a required field blank, see the correct error. All scriptable, all reusable.

Sometimes, though, you want to create a bunch of accounts. Perhaps you need three – two in the same account, to see each other when they search, and a third in a separate account.

In that case, calling the functions through the user interface will create delay. What we are doing is not really testing profile create. Instead, profile create is just a blocking condition for what we actually want to test. If I can, I prefer to offload the code from these to somewhere else — somewhere faster. We might store them in a test environment that we restore from disk. More likely, we’ll have a microservice or command-line function we can call to create profiles quickly.

Snippets are so good that we’ll be tempted to overuse them — especially for legacy applications where all the tester can see is the user interface. Don’t make that mistake.

This brings me to the final piece of code snippets – Version Control. If the code snippets (and tooling artifacts) are not versioned directly along with the production code, then the tests will be running against code that is different than the tests expect. The result will be false error reports, which requires the test-code to be “fixed.”

For today, it’s enough to say that all your testing code, including code snippets, should be safely stored under version control. Ideally in the same repository and branch as the production code.