Why should you test in production? Bluntly, the truth is you’re already doing it; the true question is whether to become deliberate, expert and reliable at it, or whether it continues to be an accident and a crisis.

You’re already testing in production in at least the sense that you have procedures for handling incidents and outages that affect customers directly. With a little more investment, that workflow can become an ongoing, manageable asset for your software development lifecycle (SDLC). Here’s how.

Definitions First

For current purposes, testing in production (TiP) has to do with web applications; the story for embedded software, installable software, and so on is outside the scope of this introduction. Also, TiP isn’t a trick that requires customers to find a mass of defects that the publisher should have found first. Most testing is still done as early in the SDLC as possible. Instead, TiP complements all other forms of testing for a few situations where the production environment is effectively irreplaceable.

There are several things TiP can give that can’t be matched testing in a staging environment:

• Delivery at extreme scale
• Practice at error recovery
• Data which respects customer privacy while representing customer diversity
• Good return on investment

Highlights of these four themes follow.

TiP for Scale

Some applications are simply big, and to replicate their facilities or loads would be prohibitively expensive. It might be impractical to maintain more than one global network topology, for instance.

In principle, most development organizations maintain a staging environment where a product can be tested thoroughly before final deployment. But in practice, essentially all staging environments are smaller than the production environment in at least one dimension. This means that, even when all tests pass in the staging environment, installation in production introduces new challenges: an order of magnitude more co-operating nodes, hundreds of consecutive hours of traffic to shuffle caches far more thoroughly, a longer tail of unusual data, a thread pool that’s a multiple of the size of the staging environment’s, entirely different query optimizations, and so on.

The staging environment becomes a model for production, then: a site to verify that the application behaves predictably under light stress. Full-blown execution only happens in production, though.

Business Continuity

One of the benefits of a focus on TiP is practice in useful disaster recovery of business continuity techniques. A good TiP team knows how to diagnose defects quickly, judge the best remedies for them, and, when the situation calls for it, revert back to a known good release reliably. TiP is an unmatched opportunity to verify an organization’s digital resiliency.

Real Data

Another aspect of TiP’s usefulness is the way it supplies test data. Conventional testing uses data that is similar to customer data, but of course not too similar, because that would violate the security of individual customers.

Production data, though, doesn’t just approximate the target; it is the target. Generating realistic data is a surprisingly expensive chore, and to be able to use real data correctly sometimes is a big advantage.

At the same time, rigorous controls need to be in place so that the data doesn’t leak out. For a programmer to disclose that “test-password” accesses a test database isn’t so bad; to share personal details about even one real customer, though, is a grave matter.

ROI

Engineering is always about choices — that is, finding the right balance between costs and benefits. TiP shouldn’t be seen as a last resort only for those situations when no other test is possible. Instead, think in terms of situations where TiP pays off.

It might be feasible, for instance, to create test data that adequately models customer data for a specific load test. If running such a test as TiP liberates an experienced analyst from data synthesis so they can focus on analyses that directly boost revenue, then TiP immediately becomes profitable.

Make the Most of TiP

The first and most important step toward TiP is the flexibility to consider it. Even with that in place, a great deal of technique remains. In fact, TiP has grown so big and beneficial that it’s a specialty unto itself. In particular, TiP techniques in the three phases of deployment, release and post-release are arguably as refined and complex as all of classical pre-production software testing. Comprehensive treatment of this range of methods is far beyond the scope of this introduction.

A few pervasive tips about TiP are worth recording, though:

• Work with feature flags and make them second nature in your designs and implementations
• Schedule slack resources so you can focus your first TiP efforts on the times and places your application is least loaded
• Instrument the application so you can see what customers see
• Create a variety of auxiliary test accounts that obey the same rules as govern other customers
• Enlist a security specialist to review your TiP for hazards and insights

By using these techniques, you can make TiP a manageable asset in your testing arsenal.

In practical terms, instead of programmers writing applications that testers then validate, the intent is that testing happens early and often. Developers continuously receive feedback on their artifacts — dashboards light up with problems after every check-in, or even after every new line is written.

If programmers are testing, does that mean testers are out of work? Far from it. The testing profession certainly adjusts as an organization shifts left, but it doesn’t vanish.

In broad terms, left-shifted testers act more strategically, attend to higher-value activities, and collaborate more deeply: The focus is on integration testing. Left-shifted testers have as much work to do as traditional testers; they just accomplish a lot more in the same amount of time, and they accomplish that work with fewer stressful “crunches” involving a bottle-necked delivery of a (nearly) working product.

These four tips can help you understand how testers are successful in a shift-left world.

1. Make Meetings More Important

Think for a moment of a traditional testing role: Programmers present what they think is a finished application, one ready for the marketplace, and then … wait on the testing crew to agree it’s good enough. This usually amounts to a series of meetings where testers are asked, “How long before you’re done?” “How quickly can you fix these problems?” “Is it supposed to do this? Is that what we want customers to see?” Front-line testers are busy in the back rooms, reverse-engineering requirements and pointing-and-clicking their way to specific outcomes.

Left-shifted testers also spend time in meetings, but at the beginning of projects. Testing professionals help define the charter of a new product by ensuring its requirements are testable. They help configure continuous integration (CI) that’s in place from the first day the source is written, and test-driven development (TDD) guidelines that ensure the application is close to passing its applicable requirements. Rather than being “monkeys” who bang at a keyboard until an error pops up, they systematically evaluate and apply tools to make tests reproducible, informative and transparent.

In a properly left-shifted team, there’s no need for extended, haphazard status meetings where lots of people tell (or cover up) what they’ve been doing. Instead, everyone can see pertinent statuses continuously, in real time. Meetings are still important; they’re just tightly focused on negotiating priorities, thinking through alternatives, and learning enough about each other’s perspectives to encourage collaboration and teamwork outside the meetings.

2. Embrace the Shift

A left shift challenges any individual tester at least a little. Such important changes always bring uncertainty and hazard.

The best way to address such negative qualities is to get ahead of them. Acquaint yourself with the whole team, insist that project plans make sense from the start, be ready to say no early when things don’t make sense, and practice new skills — scripting and XML transformation, for instance — that can keep you in command of what’s going on.

Testing becomes more strategic as it shifts left. It becomes bigger, more ambitious testing, including systematic environment or configuration testing, and even testing in production.

3. Get Help

Deeper collaboration is crucial for shifting left. To ask for help effectively is a fundamental team skill.

Your work as a traditional tester might have emphasized “staying in your lane” by following a recipe for tester activity. The shift left leaves that behind: Now you need to speak up when you run into problems, and to make the most of the abilities of the team as a whole.

A shift left is a particularly good time to work closely with documentation specialists. Like you, they’ll experience the novelty of being strategically involved early in a project. Insist on making the most of what they produce right away: Their documentation and your testing should complement each other from the beginning. When that’s not the case — a little clumsiness is inevitable in your new shifted-left territory — you’ll have the opportunity to find out early and correct any misalignment.

Your left-shifted work puts tooling in the spotlight. The shift also moves your time away from tedious, rote work and in the direction of learning and applying an expanding tool chest. Learn from others about how to make the best use of those tools.

4. Look for What Can Go Wrong

A crucial faculty you bring to the team is your ability to see what can go wrong. Many developers are conditioned to think about the “happy paths” of ideal operation. Your background is closer to the messy world of what end users actually do with applications, and how those applications run in true-life, not-always-ideal data centers.

Don’t assume the whole team sees what is obvious to you. Speak up. You’re an advocate for end users’ satisfaction and the quality of the application, and the team needs to hear your voice.

This Is an Opportunity for Improvement

Left-shifted testing certainly is a change. It deserves to be a good change. The correct attitude helps ensure the shift to the left improves not only the software and its business outcomes, but also your leadership and satisfaction as a tester.

Lines are too wide! Functions are too long! Everything is complex!

Working in a 21st-century software shop, you probably receive multiple reports that measure code by its stylistic quality, security, density and so on. What do you do with all those complaints?

One possibility is to ignore them. Just like “inbox zero” or recommendations about how much water to drink daily, plenty of our colleagues simply tune out these prescriptions with the thought that their finite attention is better spent on other matters.

You don’t have to make that choice, though; it is possible to bring your technical debt down to a low and sustainable level and keep it there. Here are a few guidelines to help get your automated metrics under control:

• Establish “green zones” of appropriate levels of technical debt
• Decide local coding standards with your team
• Establish daily and weekly routines for paying off debt
• Get help

Safe Levels of Debt

To clear all diagnostics — just like dealing with every piece of email you receive within a few minutes of first reading it — sounds like a good goal. Is it the best one, though? If five people suddenly write you in the last half-hour of the day, is it wise to stay late just to wipe your inbox clean before leaving?

Maybe not. At least consider the possibility that it’s OK to sustain technical debt at a small level that is still more than zero.

While I know of no rigorous evidence on what constitutes an acceptable level, practitioners and managers with whom I’ve spoken generally agree that for one developer or tester to have a backlog of more than a hundred items is a crisis. In contrast, a dozen tickets, issues, or tasks is probably appropriate for an experienced contributor. A newcomer to a domain does well to have no more than two or three open assignments.

Think of this another way: If your source code regularly reports thousands of warnings, or you have thousands of customer reports backlogged, it’s probably time to take extraordinary measures and reduce that accumulation quickly to a more manageable level.

Look for low-hanging fruit: Do different programmers on your team format function definitions in incompatible styles? Do you have someone on the team who regularly creates memory leaks? A team that addresses a new diagnostic system often finds it can knock out 80% or more of the initial volume of complaints by systematically addressing just two or three leading causes of debt.

When you first begin to work with a new diagnostic system — whether it has to do with test coverage, coding style, security hazards or something else — your first target should be to handle a few of the high-volume, “noisy” items. When you’re dealing with thousands of issues at a time, put a premium on consistency rather than careful prioritization: Pick any problem you understand, and solve it for good. Once you’ve slashed the backlog from the thousands to the hundreds or, ideally, a couple of dozen issues, you can return to more normal operations where you consider each item on its own merits.

That initial frenzy should only happen once. Reduce your count of issues to a sustainable level, and keep it there. “A sustainable level” varies a bit depending on individual psychology and team dynamics, but typical ranges are zero to five issues on the low side and twelve to twenty at peak. Anyone operating outside these ranges is probably under strain.

Local Standards

The first wave of results from running an industry-standard analyzer or application measurement tool is likely to be overwhelming; that’s the common reaction from the teams I’ve observed. Once you’ve followed the previous tip and shaved the diagnostics down to a manageable level, you can begin to consider individual items in a more nuanced way.

Maybe your analyzer expects a specific “header block” or “docstring” and complains when it doesn’t find one. Maybe your team has a reason to omit these particular source-code comments in specific circumstances. If so, it’s timely to reconsider the decision. Are you truly better off without a header?

If so — and there are plenty of reasons your team might choose to diverge slightly from common industry practice — make sure you configure your tools to reflect your standard. Don’t just expect the team to ignore a diagnostic that you’ve decided doesn’t apply; learn enough about the configuration of the tool to disable that particular diagnostic, preferably only in the situations where it shouldn’t appear. To leave the diagnostic turned on desensitizes programmers and makes it too easy to miss messages they should see. To turn off the tool entirely cuts down on the noise, but it also eliminates the signal of diagnostics that deserve more attention.

Consider this example: Your team writes (at least partially) in Python. Pylint expects all method definitions to have explicit docstrings, but your team typically doesn’t write docstrings for internal convenience methods.

You have a few possible choices:

• Don’t run Pylint, and don’t see the missing-docstring complaints. The problem with this approach is that team members miss out on everything else Pylint might find
• Run Pylint, but remember to ignore missing-docstring-s. This makes readers work too hard.
  Decorate individual instances of the pattern with “# pylint: disable=missing-docstring”. If there are only two or three instances of the pattern in all your source, this is probably a wise expedient
• Elaborate a common pylintrc with a specific syntax for the occasions where docstring-s might be absent. This is a good choice when you are dealing with scores of local methods
• Fill in minimal docstring-s, with the thought that, even though your team doesn’t like them, in this particular circumstance, it’s probably a healthy enough habit in general that its cost is less than the cost of making a special rule to handle it

In any case, make a conscious decision, and use the tools at hand to help enforce that decision.

Routines to Pay Off Debt

Make a habit. Pick an hour out of the day, or a half-day out of the week, to catch up on diagnostics from analyzers. Different people operate on different rhythms, so no one prescription fits all.

I’ve seen quite a few programmers and testers reach more success by figuring out a schedule that’s easy to follow. Soon they know more about their own capabilities: that they can consistently clear one routine diagnostic with half an hour of concentrated effort, for example, is a useful measurement.

Software Is a Team Sport

Frustration is a signal. If you find something blocking your progress — if you had good flow, for instance, in reducing your application’s technical debt from 6% to 2%, but you can’t seem to get past that — ask for help. Talk over what you see with someone you trust, and you’ll probably find that you understand the problem differently, and that the other person has a background, experience, and perspective that pays off in this situation.

Finally, a strong recommendation on what not to do: Don’t ignore software tools. When you first run one and see thousands of complaints about an application that you know has flocks of satisfied customers, it’s only natural to wonder what the point is. These tools are more sophisticated than might initially be apparent, though. Give them a chance.

Work through a sequence of diagnostics. Restyle the source as suggested in one or two examples, even when you don’t agree with the tool. At the very least, you’ll be in a better position to judge where you agree or disagree with the tool. You might find that you’re writing better source than you thought was possible. In many cases, you can even get help from “pretty printers” or “source formatters” to help improve your code composition automatically.

These opportunities can help you decide how to address your static source diagnostics and pay down some technical debt, so get with your team and make the most of them.

“It is generally recommended that you separate areas of concern by using one service per container. … It’s OK to have multiple processes, but to get the most benefit out of Docker, avoid one container being responsible for multiple aspects of your overall application.” Those words from the official Docker documentation site are typical of commercial practice with containers. By default, Docker runs one process per container.

But is there ever a good reason to change this prescription? When is it acceptable to run multiple processes in a container?

The short answers are “Yes” and “More often than people generally think.” To understand the long answers, consider first the essential context of containerization.

When to run containers

We host applications in containers to:

• Make underlying infrastructure more manageable
• Isolate services to make them more secure and reliable
• Create a more consistent operational environment

Containers aren’t the only way to achieve these effects, of course; good techniques exist for running directly within native operating systems, virtualized machines, or serverless hosts, for instance. Containers aren’t a magical technology, but thanks to a great deal of productive research and effort, they have become a widely used practice in the industry.

Different technologies have different profiles. Virtual machines and containers are similar in their standardization of operating environments and security concerns, but their practical application diverges a great deal. The best practice for containers, as the Docker documentation recommended, is to limit each container to a single service; with virtual machines, in contrast, conventional advice is to load them up with all the dependencies of a single application.

These are largely conventions, though: I maintain plenty of applications configured as distinct virtual machines operating together, and simultaneously I sometimes expand the borders of containers to run multiple distinct processes. Architectural rules such as “one application per container” are as much stylistic as functional.

True wisdom in container management, then, isn’t so much to follow “best practices” blindly as it is to know how to apply those best practices and when to adjust them.

Database choices

Consider this example. An organization maintains a complex equipment inventory in a database. All access to the database is through the application; it has no other entry points or uses. There is no particular need to scale the application horizontally or vertically.

A decade ago, a virtual machine might have naturally hosted such an application, although specific measurements should have been done to ensure that the database performed adequately in a virtualized filesystem. Nowadays, however, containerization might be more standard for that organization’s IT department. A conventional implementation would have separate containers for the database and its CRUD (create-read-update-delete) access. Separate containers can be tested and maintained in isolation.

Is that truly an advantage, though? Depending on the details of usage, perhaps not.

The CRUD application can’t be tested without at least a mock of the database. There are no scalability requirements. The description here makes it unlikely that the database will be run apart from the application, and while an efficient infrastructure department might need to re-host the application, there’s no compelling reason to split up the two component processes. In fact, realistic testing and monitoring might be easier with in-host communications and logging; there’s no need to introduce sophisticated, centralized scheduling and logging. Even Docker themselves has written about how to manage multiple processes within a single container.

Best practices exist for reasons, and they’re generally good reasons. At the same time, a sensitive, thoughtful DevOps team knows how to analyze specific requirements. A situation with low or peculiar scaling needs, that has little coupling to other applications, and is in a well-understood problem domain might be best designed as a monolith in a container, rather than a collection of microservices. Increase coupling to other systems, or decrease the tight coupling of the database and its CRUD, and design decisions shift.

In any case, it’s more important to analyze and make the most of resources, rather than just to apply received rules blindly.

What should be inside a container? Most likely the answer will be a single process. If you have good reasons for a different design, though, and sound plans to test and maintain that design, it’s OK to bend the single-process rule. In fact, your careful analysis may well yield you an even better result than automatic adherence to usual “best practices.”

Web element locators are used in UI test automation to identify unique elements within a webpage. It is important to have good locators to find the correct web elements, making sure that tests are faster, more accurate, and more maintainable. Locating web elements, however, isn’t that easy.

Anyone who does some kind of web UI test automation, with Selenium WebDriver for example, is likely to spend a big portion of their test development time trying to locate elements on a web page. Web elements could be anything that the user sees on the page, such as text fields, links, buttons, dropdowns, or any other HTML element.

Finding the right web elements is a vital requirement when creating an automation script, but it can be challenging. Frankly, it’s probably one of the biggest stumbling blocks in test automation.

Often, we end up working with incorrect or inconsistent web GUI elements, such as those lacking unique IDs or class names, or those which IDs or names vary from release to release, making it hard or even impossible to find the correct elements on the page. Failure to identify elements on a web page may result in less reliable tests.

What are web element locators?

Locators are basically the HTML attributes of a web element. They help identify unique web elements on a page and command testing frameworks, such as Selenium WebDriver, to perform the action on those elements. Locators are also known as selectors.

There are several types of web element locators in WebDriver, including:

• ID – Used to identify web element with an ‘ID’ attribute
• Name – Used to locate an element with a ‘name’ attribute
• Link Text – Used to identify the hyperlinks on a web page with the help of ‘<a>’ (anchor) tag
• CSS Selector – Used to locate complex elements on the page
• XPath – A language to query XML documents, consisting of a path expression along with some conditions

How do we usually find web elements on a page?

ID locators and Name locators are generally regarded as the most popular and the fastest way to identify web elements. To see how ID locators work, open Yahoo login page in Google Chrome, right-click an element on the web page, for example, a login field, and then select ‘Inspect.’

The ID locator looks for the ‘ID’ attribute as shown in the image.

This is a portion of code we’d use in Selenium to automate the action that would send values using the ID locator:

driver.get("https://login.yahoo.com/");
driver.findElement(By.id("login-username")).sendKeys("YOUR_USERNAME@yahoo.com"); //id locator for text box
WebElement searchIcon = driver.findElement(By.id("login-signin"));//id locator for next button
searchIcon.click();

Similarly, Name locators try to identify elements with the ‘name’ attribute.

This is how it looks when we want to automate it:

driver.get("https://www.google.com");
driver.FindElement(By.name("btnK")); searchButton.Last().Click();

IDs and names are a great choice for test automation as they make it incredibly simple to write locators with short queries and no extra anchors. This method, however, only works if developers write unique and consistent IDs or names for elements that don’t vary from release to release, which is not always the case.

For elements that lack unique identifiers, we must use more complicated locators, such as CSS selectors and XPath.

But, before we start explaining how to identify web elements using these two locator types, let’s check out Link Text locators.

Link Text Locators

We use this type of locator to identify the hyperlinks on a web page with the help of an anchor tag (<a>).
We’ll again use the Yahoo login page as the example. Right-click on the ‘Difficulty signing in?’ link and review the code.

Here’s how we automate it in Selenium WebDriver:

driver.get("https://login.yahoo.com/");
driver.findElement(By.linkText("Difficulty signing in?")).click();

Using these types of locators is ideal when testing navigation on the page, however, we need to know the exact names of the links.

CSS Selectors

A CSS selector is usually the best possible way to locate one or more elements in the web page, including those having no ID, class, or name. This way is faster and more readable than other locators. However, creating CSS selectors is not easy because they tend to be more complex and require a deeper understanding of CSS and JavaScript.

There are several types of CSS selectors. Each is used differently.

We have simple selectors that help us identify one or more elements based on element Class or ID. Refer to the following code example:

driver.findElement(By.cssSelector("#id"));
driver.findElement(By.cssSelector(".class"));

There are attribute selectors to find one or more elements based on their attributes and attribute values. Here’s the example in code where we use the Name attribute to locate an <input> element:

driver.findElement(By.cssSelector("input[name=username]"));

CSS selectors enable you to locate elements matching partial attribute values. This is very useful for testing applications where attribute values are dynamically assigned and change every time a page is requested.

We use the following CSS partial match syntax: ^= (starting with), $=(ending with), *= (containing)

Pseudo-classes are used to locate one or more elements that exist in a certain state, for example, elements being hovered over by the mouse pointer, or those disabled or checked.
For example:
:hover, :disabled, :checked, etc.

Pseudo-elements are used to style specified parts of an element, such as the first word of each paragraph, or to insert content before or after the content of an element.
For example:
::after (:after), ::before (:before), ::first-line (:first-line), etc.

Combinators are not selectors per se, but ways to combine two or more selectors to define specific selections. Sometimes one CSS selector may not be sufficient to locate an element, for example, one attribute, so we need to combine additional attributes for a precise match.

Here is an example of combinator use when we want to locate an element with two attributes:

driver.findElement(By.cssSelector("input[type='submit'][value='Login']"));

Similarly, multiple selectors are not separate selectors, but multiple selectors placed under the same CSS rule to apply a single set of declarations to all the elements selected by those selectors.

Here’s the example in which we use the element’s Id, class, and attribute value:

driver.findElement(By.cssSelector("#id .class input[type=text]"));

There’s a lot more to learn about CSS selectors as well as when and how to use them, such as universal selectors used to match elements of any type, sibling combinators, substring matching attribute selectors, structural pseudo-classes, etc.

Sounds complicated? You bet.

XPath locators

Using XPath locators is more efficient than using CSS selectors to find multiple elements on a web page. However, they’re even more complicated than CSS selectors. In fact, XPath locators are the most complex selectors to use.
XPath is a language to query XML documents and using XPath locators requires knowledge in XPath query language. So, it may be hard for anyone who is not familiar with this language to locate web elements using XPath.

Here’s how we use XPath locators when looking for web elements:

• In Google Chrome, navigate to google.com and inspect the ‘Search’ field.
• On inspecting the web element, we can see it has a ‘class’ attribute that we’ll use to construct an XPath locator which in turn will locate the ‘Search’ field.
• Next, click Elements tab and press Ctrl + F to open a search box in Chrome’s DevTools. Here, we’ll write the XPath like this:

//*[@class='a4bIc']

Or, in Selenium, this is how the code would look:

driver.findElement(By.xpath("//*[@class='a4bIc']"));

To locate the ‘Google Search’ button, we’d use this, a slightly different XPath:

//input[@name='btnK']

These are just basic XPath examples. But, as with CSS Selectors, using XPath locators is a broad subject that can’t be explained in a single article. Things such as methods, expressions, and functions used in XPath require knowledge, but also the time that many developers in test automation lack.

How to make it simple and faster?

The goal of testing is to make things as simple as possible so that we can focus on actual testing rather than coding or fixing the code. Finding and identifying web elements by manually writing CSS Selectors or XPath can be bothersome for many testers who want to speed things up.

Accelerating the testing process is actually possible with Ranorex Selocity, a free Chrome extension that auto-generates robust and reliable CSS, XPath and RanoreXPath selectors in the Chrome DevTools. You can use this extension with any IDE of your preference, but you’ll achieve the best results if you connect Ranorex Selocity to Ranorex Webtestit, the IDE for testing with Selenium and Protractor.

In the example below, we’ll demonstrate how to auto-generate locators with Selocity and send them directly to Ranorex Webtestit with just a few clicks.

First, open Ranorex Webtestit and create a new project or open an existing one.

Then, you need to install the Ranorex Selocity extension.

Open google.com in Google Chrome. We’ll try to identify the ‘Google Search’ button.

Click F12 to open Chrome DevTools on Windows/Linux, or Command + Option + I on MacOS. Find Ranorex Selocity in the Elements section (if you don’t see it, reload the page or expand the DevTools window.)

Connect Ranorex Selocity to Ranorex Webtestit with a click.

In Ranorex Webtestit, create a Page Object.

Better yet, you can create it directly from the Google Chrome DevTools page by clicking on ‘Create new Page Object’ button in the Selocity menu and then selecting the area of the page.

On the Google page, right-click on the element (button). You’ll see the list of selectors, including CSS, XPath, and RxPath.

Click on the ‘Send selector to Ranorex Webtestit’ button next to the selector’s name. We’ll send XPath.

The selector will be sent automatically to your Page Object in Webtestit. You can also view the screenshot of an element.

The imported element will have the default name (btnK) but you can rename it to something more coherent, such as searchButton or whatever else you prefer.

To initiate the action on the newly added element, simply drag and drop the element from the Elements tab into the Page Object code and choose one of the actions from the drop-down menu (in this case: Do > Click on element).

The action will be added to the code automatically. It will have a default name performNewAction which you can also change.

After you’ve added all necessary elements with the Ranorex Selocity extension, you can start writing your test which is also simple if you do it with Ranorex Webtestit.

Here’s a useful step by step guide on how to create a project and run your tests with Ranorex Webtestit.

If you use other testing frameworks, you can still utilize Ranorex Selocity to auto-generate selectors. Simply right-click the element and choose ‘Selector actions’ then ‘Copy css,’ ‘Copy xpath,’ or ‘Copy rxpath’ from the drop-down menu and paste them manually into your code.

It’s not as convenient as sending selectors directly to your code in Ranorex Webtestit, but it could still save you from writing all selectors manually.

Conclusion

As you can see, finding web elements doesn’t have to be a pain in the neck. The free Chrome extension, Ranorex Selocity, helps you handle complex CSS Selectors and XPath locators by generating them automatically with a single click. Moreover, if you use Ranorex Selocity with Ranorex Webtestit, these locators can be sent directly to your test code.

You can download a free, full-featured trial of Ranorex Webtestit and connect it to the free Ranorex Selocity extension and immediately discover all the advantages of painless testing.

A testing department’s true goal is to spark change, not just to write lists of errors. Clear thinking about what not to test helps liberate resources for more thorough testing in areas that make a difference. Here are five kinds of tests you should stop writing.

Tests of past requirements

The value of tests is that sometimes they fail. Tests that always pass provide little information.

Most of us have seen tests that fail occasionally, but only because requirements changed and notice of the update arrived after a test squawked. Tests like that invite everyone to go numb, and there’s no value in that.

Think of a concrete example: Early in a product’s history, a requirement for a visual display is expressed in terms of absolute position on a display. The first release passes all its tests. Soon, the absolute position changes a few pixels, to accommodate other screen elements. The test fails, is adjusted, passes, and the next release goes out. This cycle continues.

It continues to no one’s benefit, though. While some tests are rightly written in terms of physical location, this test has shown it’s not one of them. The test just costs tedious updates with each release and brings no value.

In such a case, it’s better to:

• Remove the test entirely
• Rewrite it to be more robust — perhaps as a test that has a widget on the screen, is visible, or has specific content
• Disable the test until product management establishes a different communication channel that allows testing to track true requirements with more precision

Continuing wasteful tests isn’t a virtue; it’s a kind of theft because it steals your attention away from more rewarding efforts.

Inconsequential tests

Suppose you notice a big ball of mud. Do you test it? Sometimes, but rarely. You shouldn’t test that legacy muddle if you know your organization is incapable of responding to your findings.

Don’t prioritize unit tests of existing code over unit tests of new composition: One of the great values of unit tests is their ability to make developers rethink implementations ahead of time. The “big ball” already exists, and its muddy architecture won’t change. Don’t pretend to write unit tests for code whose main value is integration or gluing; integration tests are a better match there. Don’t test the untestable: If A/B testing excites your organization in principle, but no decision-maker can articulate exactly what will happen with the results, the construction of the A’s and B’s is probably not worth your time.

This isn’t an invitation to nihilism. If you ever suspect your tests are headed toward irrelevance, don’t give up; just figure out what else needs to change. Put together a proposal that addresses the true need — the one whose correction will enable your tests to become meaningful.

Tests inferior to thoughtful inspection

Brief, idiomatic expressions are better regarded as configurations to be inspected than code for unit testing. Expert inspection gives more confidence for trivial getters, unbranched arithmetic and magic constants than onerous unit tests.

The big problem with inspection, of course, is that it doesn’t automate regression results. If you have mission-critical responsibilities, and exercising integration tests of already inspected segments doesn’t give all the confidence necessary, then, sure, generate tests that first + second actually sums two variables.

In many common situations, though, software inspection is more cost-effective than testing, or at least combines with testing efficiently.

Tests of throwaway code

To discuss “disposable programs” with a straight face is a challenge because software intended to survive a single weekend frequently ends up in use for a decade or more. However, if you truly are certain that a particular program will only support a unique report needed for an occasion never to be repeated, then it might make sense not to test that program. If in doubt, assume the program will be needed again, and that testing it does what it’s supposed to do is the right choice.

Tests you can delegate

Unusual situations sometimes arise when manual testing is particularly cheap. If passing a fraction of your testing responsibilities to a cost-effective alternative is possible, do it, and turn your attention back to the higher-value testing you do best.

Summary

Engineers judge trade-offs. Every test has not just benefits, but also costs. You’re not a janitor, sweeping up every crumb that programmers might have accidentally dropped on the floor; you’re an investor, looking to apply your high-value testing ability to the best possible portfolio of possible tests you might write. Treat your work like the precious commodity it is.

For more details on spotting particular technical situations where testing brings hazards, see guides by Anton Angelov, Mark Seemann and Adam Trilling.

You might have already wondered; Ranorex does object-based recording, so a click action, for example, is always performed on a specific UI element, no matter how it looks or what screen coordinates it has. However, if you look closely on the action spot of a click action, you can still see that behind the word “Relative” a strange value like 51;13 is hidden. So, does this mean that Ranorex still records coordinate-based? And can I change this behavior? And what side-effects could this have? This small article should answer all those questions.

Coordinate Recording Mode

The setting that defines this behavior is the Coordinate Recording Mode that can be found in Settings -> Recorder Defaults. This setting has three possible values: Pixel, None, and Proportional. This setting defines how the exact action spot within a UI element is determined. The default is Pixel. This setting records the exact location of a click within the UI element. For example, if you click on a button in recording mode near its top left corner, you will get small coordinate value like 5;8, since the top left corner always has the coordinates 0;0. These coordinates use integer values and will also be used for playback. The setting None behaves differently: Here, it doesn’t matter where on the element you click in recording mode, Ranorex will use the action spot Center and in playback will always click on the middle part of the button. The setting Proportional uses coordinates in the form of .5;.6, thus using floating point numbers instead of integers. This means that if you click on a pixel in recording mode that is 30% of the button size from the top, then in playback Ranorex will also click a pixel that is 30% from the top in playback mode.

Advantages and disadvantages

All three of these settings each have a set of disadvantages and advantages. In fact, None might seem to be the best setting, but Pixel is the default value in Ranorex.

Pixel

This setting is the default in Ranorex. It recognizes the exact (=pixel-perfect) action spot in the UI element in recording mode and uses these coordinates for playback. This has one huge advantage: When running the test, Ranorex does exactly the same thing that you were doing when you recorded the test. At least as long as you stay on your same screen with the same screen resolution and the same screen scaling, that is. In modern applications, especially web applications, responsive design has become increasingly more important, and a button or a text field might change its size drastically, depending on the size of the application window, the screen, the device orientation, etc. For example, on the Amazon homepage, you might want to click into the search field, somewhere in the middle:

If we put our browser from maximized mode into a much smaller window size and try to run the test, this click suddenly doesn’t work anymore, because the coordinates that I have recorded (958;17 in my case) are suddenly outside of this element, since this text box has shrunk in size. This warning looks like this:

Suddenly, Pixel looks like a far from the optimal default setting. Especially in modern, web-based applications that use responsive design, this setting is far from ideal. So why is it that it still is the default setting in Ranorex? Consider this element in your application:

Unfortunately, the checkbox with its label is recognized as just one element. Especially in deprecated desktop technologies, this is very common. You also often see the small triangles or plus-symbols that should expand a section of a list or tree – those are also often not recognized as separate buttons. However, automating only works if you exactly click these buttons, not the text near it. This makes it necessary that a click is performed on the exact same spot within the control as it was recorded. Would the click just be performed on the Center location of that element, the checkbox in this example wouldn’t get ticked.

None

This setting works very differently, and it changes the behavior of the test run drastically compared to the recording. Every action spot is set to Center. This might sound as if it makes sense in many cases, especially when you have responsive design, but it has a few serious drawbacks. For example, consider you want to click on this headline on a web page:

When recording, you would click this text, and it would work. However, in the test run, the action spot would be Center, and therefore the click spot would not actually be on the text itself:

Although the value None might seem like a good setting on the first glance, you can run into these issues frequently. So, is there a better way to do it, if both Pixel and None have some big drawbacks?

Proportional

This setting tries to, similarly to Pixel, retain the action spot exactly where it was while recording, but in relation to the size of the element. This means that a click on a spot that is 30% from the left edge and 60% from the top edge will also be done 30% from the left and 60% from the top in the test run, no matter how big or small the element now is. This mitigates the problem that the Pixel setting had: if a button shrinks, the action spot is still inside of the button. And some issues that None has are also solved, but still, Proportional suffers from similar issues. For example, a click here…

…would translate to a click here in the test run, if the screen size becomes a bit smaller:

Thus, the setting Proportional suffers from very similar issues to None. This setting is represented by floating point numbers in Ranorex. For example, the Center location could also be written as 5;.5 in proportional notation.

The Best Setting

So, the question is: What is the best setting for me? The answer is pretty unsatisfying: It depends on your application. Pixel is the default because we think that if you record a test and then run it afterward without changing anything in between, it should run. None does not have this property, and while None looks logical up to a certain degree, Proportional seems to do very illogical things if you don’t know how the action spot is calculated. Therefore, we have decided to stick with Pixel as the default. This is often best for pure desktop applications and works fine if you don’t change the screen resolution. None and Proportional both have their uses in applications with responsive design (mostly in the web and on mobile devices), however, they can have strange side-effects. In all three cases, depending on your actual UI element, it might be necessary to change the action spot manually if it doesn’t do what you want. There is no cure-all type of setting. All of them have advantages and disadvantages. If you know what properties those settings have, you can easily pick the one that suits your project best.

A testing department’s true goal is to spark change, not just to write lists of errors. Clear thinking about what not to test helps liberate resources for more thorough testing in areas that make a difference. Here are five kinds of tests you should reconsider.

Tests of past requirements

The value of tests is that sometimes they fail. Tests that always pass provide little information.
Most of us have seen tests that might fail from time to time, but only because requirements changed and we don’t get notice of the update arrived after a test squawked. Tests like that invite everyone to go numb, and there’s no value in that.

Here’s an example: Early in a product’s history, there is a requirement for a display field that is expressed as an absolute position. The first release passes all its tests. Soon, the position shifts a few pixels to accommodate other, newer, screen elements. The test fails, is adjusted to fit the new layout, passes, the release goes out. The cycle continues.

It continues to no one’s benefit, though. While tests can sometimes be rightly written for a physical location, this is not a hard and fast rule. When it does not work, the result is repeated, tedious updates each release and brings no true additional value.

In these cases, consider:

• Remove the test entirely. When it takes more effort to maintain it than the benefits it gives, get rid of it;
• Rewrite it to be more robust. For example, the test might have a widget on the screen, is visible, or has specific content. Or “more robust” may include using smart automation tools that use more reliable methods to identify UI elements, like Ranorex Studio.

Of course, you can always disable the test until product management finds a better communication channel to allow testing to track requirements with greater precision.

Continuing wasteful tests isn’t a virtue; it’s a kind of theft because it steals your attention away from more rewarding efforts.

Inconsequential Unit Tests

It is sometimes tempting to develop unit tests for existing code that do not have any existing unit tests. This might seem a good idea, however, without a thorough, deep analysis of the code before starting, it is easy to be pulled off well away from your primary purpose: developing unit tests for new code.

While it may be tempting to try and develop unit tests for the big ball of mud, don’t do it. Just don’t. Should you test it? No. Don’t try and create unit tests for existing, legacy, mud code without a corresponding effort to refactor.

One of the great values of unit tests is their ability to help developers consider implementations ahead of time. The “big ball” already exists, and its muddy architecture won’t change. Don’t pretend to write unit tests for code whose main value is not clearly understood.

This isn’t an invitation to nihilism. If you ever suspect your tests are headed toward irrelevance, don’t give up; figure out what else needs to change. Put together a proposal that addresses the true need — the one whose correction will enable your tests to become meaningful.

Test Vs. Inspection

Formal code inspections can be powerful tools to significantly reduce the odds of problems in a piece of software. By examining each command and each line, with great diligence, problems missed in simple unit tests can be uncovered.

Inspections can be helpful for more robust function or integration tests by highlighting areas that appear to be “correct” but may not behave as expected. Sometimes idiomatic expressions might be better candidates for inspection than some levels of unit or function testing.

In some situations, software inspection is more cost-effective than testing. Often, inspections used with various levels of testing make a potent combination to drive test quality.

Testing Throwaway Code

There is a school of thought that suggests “disposable programs” often survive long after their creators expect them to. There are times when code developed as a prototype can turn into the framework for a portion of the application. In these cases, the code often bears little resemblance to the original, “throwaway” code.

If a piece of code is intended for a very specific, short term purpose, it is possible it would not need to be tested, at least not rigorously. Given the nature of code itself, it is possible that the idea of “it is throwaway, don’t both testing it” can be dangerous. Code is still code and if people can allow for unusual behavior by imprecision in code itself, it will need to be tested.

Something simple, like a quick report, might not need the in-depth examination of code generating test data. If in doubt, presume you will need the program will again and that testing it will help you know if it does, or does not do what it is expected to.

Summary

Engineers judge trade-offs. Every test has not just benefits, but also costs. You’re not a janitor, sweeping up every crumb that programmers might have accidentally dropped on the floor. You’re an investor, looking to apply your high-value testing ability to build the best possible portfolio of tests you can write. Treat your work like the precious commodity it is.

For more details on spotting particular technical situations where testing brings hazards, see guides by Anton Angelov, Mark Seemann and Adam Trilling.

What does “testing artificial intelligence” even mean? AI is now often talked about in relation to how it can be used to help with testing, so how do we go about testing a technology employed in testing?

AI effectively means “advanced and difficult,” or even “whatever machines haven’t done yet.” AI is hard, and testing AI is hard. But a little care with basic concepts demonstrates that familiar testing skills apply, even on software’s bleeding edge.

The first and most strategic advance when testing AI is to take it a bite at a time: Identify, isolate and resolve individual difficulties in AI testing. What’s left is a little bit smaller and easier to tackle.

AI is part of a larger whole

AI applications inevitably have plenty of parts that are not AI; traditional testing applies to these just fine. Test the user interfaces, networking, storage, configuration, error-handling, arithmetic and so on as you would any other application, and the AI core that remains will be more manageable.

“AI” covers a number of specific technologies, including machine learning, natural language processing, perception and robotics. It’s generally important and valuable to identify and test different AI technologies individually.

A speech recognition application, for instance, probably combines a number of advanced technologies, such as digitization informed by neural networks, natural language processing, generative modelling, supervised training and more. Savvy testers will find ways to test the individual segments of a speech recognition system; some parts will involve testing big data at scale, variations on the histogram method are standard for analog-to-digital conversion, and standardization of test suites for natural language processing is in at least its third decade.

Don’t expect to be an expert in all of these technologies; your greatest contribution as a tester is to clarify the different components for which you’re responsible and construct realistic, useful test plans for each of them.

One common pattern in AI is that a process has different phases or aspects, which demand different kinds of testing. Often a training or learning phase results in a model; in use, the model is then applied to yield actionable results. Construction of the model demands one kind of testing; accurate application of the model is a different process, with different testing techniques.

Models tend to be “fuzzy”; a single set of training data can yield different models that only experts can judge and discriminate. Model validation often is hard to automate and involves considerable human effort. Model use, on the other hand, is more deterministic and easier to test in traditional ways. In all cases, it’s important to distinguish AI phases and craft tests appropriate to each one.

Yet another dimension of model testing focuses on transparency, intelligibility or interpretability. The literature is full of “deep learning” neural nets and other AI applications that produce good results but that no human understands. According to one popular article, “No one really knows how the most advanced algorithms do what they do.”

Whether that’s a problem or not, testing specialists can help in at least a couple of ways:

• Leverage experience with white-box and black-box testing to clarify which models are transparent and which can only be judged in terms of their outputs
• Teach implementers that interpretability is a virtue, like testability. Just as it’s often possible to rework naive conventional software to make it more testable, AI applications can often be instrumented and otherwise enhanced to make their models more transparent. Testable and transparent software is better and more reliable software, even if sometimes it requires a bit more initial effort

AI and UX

Another challenge of AI testing is that because AI is so advanced, users’ expectations about it are unrefined. Our collective expectations for how such legacy applications as a spreadsheet or web browser should behave have largely converged; not so, though, for an AI-augmented security surveillance tool. Consumers end up relating to AI applications in surprising, innovative and sometimes dysfunctional ways

Consider this example: A basic profile form might ask for first and last name. While even this basic interaction embeds plenty of cultural assumptions and pitfalls (people in some countries commonly possess only a single name, the meanings of first and last are reversed in certain societies, and so on), we at least can build on centuries of experience in government censuses and other contexts that range over the possible values for names. AI applications, in contrast, are so new that it’s not always clear what users will do with them, let alone when testing should pass or fail a particular result.

One consequence is that AI puts user experience testing at a premium. Whenever possible, AI testing should include a component of observing use by real users, because AI is new enough that all of us are learning what it can do.

Another way to say this is that we have good knowledge about the range space or possible inputs for conventional computing applications, like accounting or presentation or media playback software, and we can shape tests in terms of that knowledge. AI not only involves arcane technologies and meager testing backgrounds, but we don’t even have a firm grip on what users end up doing with the applications. What kinds of inputs might show up in the real world of users? Uncertainty about this question amplifies the difficulties of testing AI.

Put it all together

AI software is still software, and many of our typical testing techniques apply equally well to AI.

The hard parts of AI are hard to test, but we can usually isolate them for testing purposes, cutting the difficulties of testing down to size. Look for the different segments or phases of an AI application, and learn from the specialists for each technology involved. Work with your team not just on testability, but, in the case of AI components, interpretability.

And remember to emphasize observation of how real consumers relate to AI software; it’s likely to surprise you.

If testing still presents problems after you put all these tips in play, at least you’ll be in a position to report those problems cogently and constructively.

Test-driven development has filled up with esoteric techniques and beliefs. Here is a concrete example of a simplification and correction in testing.

This example is executable Python, but the ideas apply across most computing languages, and at least one of its generalizations is likely to apply in your situation:

check_result(handle):
“”” The context is that several long-running processes, 
indexed by user account, have been launched. The
current function determines whether a result is ready, 
and, if so, what it is for $USER.
If a result isn’t ready--if the long-running process 
is still running--just return None as a sentinel.
”””
outcome = done(handle)
if outcome:
return outcome[os.environ[“USER”]]
return None

While I doubt that this small function precisely quotes anything actually in use, it’s representative of many, many real-world definitions. Plenty of code in use that deserves testing differs from this example by only a single line or two.

Testability

One of the first reactions a tester should have to this tiny definition is that it’s hard to test. More precisely, when different developers, testers or continuous integration (CI) processes exercise a test of check_result, they’re likely to receive different results, because each one involves a different $USER.

A natural response would be to add another argument to check_result(). As is frequently the case, a more general or properly abstract definition turns out to be more testable. The signature of the function then becomes

check_result(handle, account):

At this point, the function looks more testable, but it’s less useful. The updated definition requires all clients of check_result() to compute and manage account for themselves. While testability sounds desirable, it certainly can’t be at the expense of a more complex implementation.

A better solution is possible. All check_result()’s current clients assume that check_result() manages $USER on their behalf. One possible response is to keep the original definition for check_result() but create test methods that save, alter and restore os.environ. Automatic testing of definitions that bind outside resources — for this implementation’s purposes, the user account is external — frequently resort to such invasiveness. It’s more feasible in a white-box situation, of course, when the implementation is visible to the testers. At the same time, to bind test definitions to implementation details makes the tests themselves more fragile.

An even better solution is possible, though. Consider:

check_result(handle, account=os.environ[“USER”]):
“”” The context is that several long-running processes, 
indexed by user account, have been launched. The
current function determines whether a result is ready, 
and, if so, what it is for $USER.”””
outcome = done(handle)
if outcome:
return outcome[account]return None

Reliance on a default argument allows all the clients of check_result() to continue to invoke it unchanged; for them, check_result() still takes responsibility for computation of account. At the same time, parametrization of check_result() makes testing easier; test definitions can pass in specific values for account that yield better-controlled results.

Some computing languages don’t directly support default arguments; C and Java are examples. From their perspective, this example looks suspiciously tricky. The larger principle — that a more general definition can sometimes by more testable — is so important, though, that it’s worth learning workarounds for these languages.

In Java, for instance, it’s natural to create the effect of default arguments through method overloading. In such a case, the main method would include all parameters, and a signature targeted for defaulting implements the default procedurally:

Result check_result(handle, account) {
…
}
Result check_result(handle) {
Map<String, String> env = System.getenv();
return check_result(handle, env.get(“USER”)); 
}

Whatever the implementation situation, a savvy tester looks for a way to cooperate with development to create an implementation that is both general enough to be testable and specific enough to be useful.

Ambiguity

We introduced a generalization of the check_result definition to make the original definition more testable. That testability pays off in uncovering a gap in the specification.

Recall the context of check_result: A long-running process handle indexes eventually completes, at which point it returns results for several different account instances. What if the process completes, though, but no result is available for account? What is supposed to happen then?

Our current version of check_result tosses a KeyError exception. Is that the right action? From the information available to us, we don’t know. Maybe an exception is needed; maybe the function should “silently” return a None, or even some third alternative.

This is the point at which an experienced tester reports back an ambiguity — that is, that a test has failed on the grounds that the requirements have a gap. Notice that parametrization of account makes it easier to identify that someone needs to answer, what if done() returns an answer with no entry for $USER?

Stylistic Hazard

That’s not all. The current version of check_result is so small that we should be able to understand it completely. That’s certainly a desirable property for an example. Even this little example, though, hides at least one more subtlety.

Focus on the test

if outcome:

From the materials available to us, we’ve already identified a gap. Without a formal specification of done(), we’re unsure whether check_result precisely matches the requirements it should meet.

It’s not just check_result’s behavior that’s in question, though; at least one more ambiguity about done() is lurking.

This one takes a bit more explanation. Some styles of Python return False as a sentinel, and a reader of the check_result above might well speculate that’s exactly the intent: If the computation hasn’t finished, then done() returns False. If the computation has finished, then done() returns a dictionary of results.

That’s a problem, though. Python style manuals prescribe that a correct expression for testing a value that might be False is indeed

if outcome:

The problem is that this test picks up other values as well. Suppose done() finishes but returns an empty dictionary. In Python, a simple

if outcome:

treats both False and the empty dictionary identically. Is that intended? Without a more detailed specification, we don’t know.

At least one improvement is likely. Whatever the decision about the empty dictionary, it’s slightly better style in Python to use None, rather than False, as a sentinel, and test it with

if outcome is None:

The only value that matches this test is None. Coding this way helps distinguish the sentinel from an empty dictionary.

This illustrates that sometimes — in fact, often! — a valuable result of a testing cycle is not “The software passed” or “Here are specific failures,” but “Gaps in the specifications result in these potential problems.”

An alert and knowledgeable tester generates higher-value results that identify not only errors, but also opportunities to improve the long-term resilience of the implementation. Complete specifications and good style help make software that is simultaneously more readable and more testable.

You’ve got your continuous testing (CT) humming: programmers check in their code, and within 10 minutes out pops a preliminary report confirming that all unit tests still pass. Lovely! And it’s a big benefit, too, because on the rare occasions that a problem does turn up, it’s solved with far less effort because it’s identified quickly, before it becomes obscured by all the distractions that accumulate over hours and days of focus on other issues.

That’s just the way CT is supposed to work. What your team might not realize, though, is how much wider CT can spread, well beyond the programming language the organization identifies with your output.

Do you build your software with Bash or PowerShell or Make? Those scripts deserve testing, too. Does SQL mediate your database access? Boost the productivity of your database specialists with automated validations. Are your CSS sources accumulating the technical debts of dead specifications and deprecated archaicisms? Bring in an automatic tool to help clean out that dirt.

Testing that more closely models customer-like interaction, is the usual focus of testing technology, and, of course, work at that level will always be primary. Automated confirmation of specific artifacts has a lot to offer on a smaller scale, though. Consider a few examples.

HTML

Even hardcore embedded programmers periodically emerge from their command-line sanctuaries to produce documentation, dashboards, and other HTML-based artifacts. For most of us, web applications or at least websites are part of daily work.

Is there any value to a focus on the HTML and its validity, though? If a web app passes all its tests, why check the syntax of the HTML that goes into it?

At least three answers broadly apply in commercial situations:

• HTML validation is generally a fast check. If someone accidentally garbles the punctuation of a tag — “<ui>” for “<ul>” during an edit, say — a validator can typically spot that in under a second, whereas a full test suite typically takes minutes or even hours to complete. The faster your CT finds errors, the quicker your coders are able to turn them around and bring your software back into a “green” state
• It’s widely believed that valid HTML raises search-engine rankings
• Valid HTML is more maintainable

That last point deserves a bit more explanation. It’s easy to come up with fragmentary HTML — source that lacks closing tags, for instance — that meets requirements, in the sense that all major browsers render it as expected. Slightly invalid HTML tends to be fragile, though. Adding another item near an invalidity can result in different browsers producing different results. When HTML is valid, though, and remains valid during maintenance, it’s far more likely that different browsers will continue to render it consistently.

HTML validation’s a good idea, right? In general, yes. Validation helps locate errors quicker and at less expense. I’ve certainly automated HTML validation for quite a few projects.

At the same time, be aware of some of the reasons not to validate your HTML:

• Your HTML is so small or static that it’s not worth the chore of automating its validation
• Your organization doesn’t value validity, and only treats findings as annoyances to ignore
• You have more pressing priorities for testing and choose to focus your attention on domains other than HTML

I most often rely on html5validator, which also knows how to handle CSS and SVG.

Shell

Shell (Bash, PowerShell, and so on) is another rich domain for automation. Even if you think you only have a few dozen lines of shell gluing together the other parts of your project, you should consider putting guards in place for their use:

• In-source configuration of “safe shell” with set -euf -o pipefail and similar
• Static analyzers such as shellcheck
• Study of Bash style guides

These tools are particularly good at spotting unassigned variables, whitespace errors, confused evaluation contexts and subtle spelling errors that can otherwise frustratingly elude capture.

Little pieces of Perl, PHP, VisualBasic, and other now-secondary languages also show up as “glue” in larger projects. Each of these has language-specific checkers that can help ensure the quality of your source.

JSON

JSON is often an alternative to XML. It’s usually simpler and easier to get right. It’s also more likely to be written by an inexperienced hand, and an automated checker quickly finds the out-of-place commas and quotes that plague weakly controlled JSON source.

Markdown

Markdown is roughly comparable to HTML: It’s a format that largely communicates textual content, and like HTML, its consumers are generally forgiving. Markdown renderers tend to show plausible interpretations even with content that isn’t well-formed Markdown.

As with HTML, also, well-styled Markdown is easier to maintain. Markdownlint helps keep Markdown sources properly syntactic.

Also common to both HTML and Markdown is that many projects rely on these languages but maintain templated sources. The individual pages of a website might be written in Jinja, for instance, which an application server then delivers as HTML. It’s often useful to automate separate tests, one to confirm that the template expressions are well-formed, then one to confirm that rendered content is also valid.

And more

As always, part of your opportunity as a tester is to adapt these general descriptions to your own situation. I’ve written special-purpose checkers to confirm stylistic details that the checkers above don’t cover. Also, many other languages — SQL, crontab, C#, and so on — have checkers available. My experience is that disciplined, automated application of these checkers always turns up errors that would have taken far longer to find without them.

The CT you have now is just a starting point. A little energy and creativity applied to your CT can make it considerably more valuable and help you significantly raise the quality of your software products and services.

This article describes how to use the Ranorex Studio IDE and the Ranorex API for test automation in your behavior-driven development (BDD) process. BDD requires a cognitive shift: instead of thinking about testing functions, you are now thinking about behaviors. Following this approach enables seamless collaboration between business stakeholders (such as product management) and the QA or development teams. On the business side, the product management team can define feature specifications in natural language without writing a single line of code. On the technology side, the QA team and the development team add automation steps using recording modules and the repository.

Tool Stack

The diagram below shows a sample tool stack to create a BDD-based testing framework. Ranorex Studio provides the development environment, while the Ranorex test automation framework provides the automation API. To parse the feature files from natural language to written code, we are using SpecFlow as the BDD interpreter. Finally, NUnit is the unit test provider, which manages and executes the tests.

Process Overview

When following the BDD approach in an agile environment, the entire development process of a user story is divided into a business side and a technology side.

On the business side, we create user stories, which are taken and broken down into features stored in feature files. These features are broken down into scenarios. The features are written down in a natural language following the Gherkin syntax. A basic feature written in Gherkin might look something like this:

Feature: Serve coffee
In order to earn money
Customers should be able to
buy coffee at all times

Scenario: Buy last coffee
Given there are 1 coffees left in the machine
And I have deposited 1 dollar
When I press the coffee button
Then I should be served a coffee

On the technology side, we need an interpreter to parse the feature files written in natural language (Gherkin syntax) and call test automation code. These Ranorex API calls or Ranorex recordings do the following:

• Automate the system under test
• Verify the expected output
• Returning a corresponding result

Prepare Ranorex Studio for BDD

The SpecFlow Add-in for Ranorex Studio

The SpecFlow add-in provides file templates for feature files, step definition files and event definition files. It also translates the features from Gherkin syntax to C# code. To install the SpecFlow add-in to Ranorex Studio, follow the instructions below:
Download or clone the SpecFlow repo and open the project “TechTalk.SpecFlow.RanorexIntegration.csproj”

You then need to ensure that the project is targeting the .NETFramework version 3.5. You will notice that some NuGet packages are missing, as indicated by the yellow warning symbol. Use the context menu and select Manage NuGet Packages, then click restore. Add the NuGet package for NUnit.Runners.2.7.1, it is important that you use version 2.7.1

The packages will be restored from your online sources, until you are left with only three warning symbols. The following packages will not have been restored and need to be removed, in this example using the context menu.

Then replace them using the .dll files found in <Ranorex installation folder>. Click on references and use the context menu to select add reference. You may need to click browse and navigate to the Ranorex installation folder.

Next, build the RanorexIntegration add-in.

A .sdaddin file will then be built in the IdeIntegrations folder. Use an extraction tool such as 7-Zip to extract the file and save it to a new folder named “TechTalk.SpecFlow.RanorexIntegration”.

Copy the extracted folder and paste in the <Ranorex Installation folder>Bin\Addins\Misc.

Make your Ranorex Solution BDD ready

In your Ranorex solution open the “Package management console” (View->Tools->Package management console) and run the following commands:

Install-Package SpecFlow.NUnit -Version 2.3.2

Install-Package NUnit.Runners.2.7.1

You will notice that the references are added to the projects view. To ensure that your BDD solution integrates seamlessly with SpecFlow and NUnit you will need to amend the AssemblyInfo.cs and app.config. Simply double click on each in the projects view.
For the AssemblyInfo.cs add:

[assembly: NUnit.Framework.Apartment(System.Threading.ApartmentState.STA)] 

as seen below:

For the app.config you will need to add <probing privatePath=”Runtime”/>
in ‘assemblyBinding’ as seen in line 17 of the below.

Get started with BDD

Now that your Ranorex Solution is BDD-ready, you can begin defining features in SpecFlow and creating automated tests for them in Ranorex.

Define a feature from the business side

In SpecFlow, create a new feature file. To do so, right-click on the project, then choose Add > New Item > Category: SpecFlow > SpecFlow Feature.

Create step and event definitions for the technology side

Create a new step definition file. To do so, right click on the project and choose Add > New Item > Category: SpecFlow > SpecFlow Step Definition.

Now, create methods for all “Given,” “When,” and “Then” actions described in the scenario description. Add “Given”, “When” and “Then” attributes with values matching the scenario steps defined in the feature file. To do so, follow the samples provided by the template from which the new step definition file has been generated.

Save the file. Ranorex Studio will create a <name>.feature.cs file containing the automatically-generated code that will execute the test scenarios using the configured test provider.

If the code/C# file is not created please refer to the troubleshooting section at the end of this blog. 

Create step and event definitions for the technology side

Create a new step definition file. To do so, right click on the project and choose Add > New Item > Category: SpecFlow > SpecFlow Step Definition.

Now, create methods for all “Given,” “When,” and “Then” actions described in the scenario description. Add “Given”, “When” and “Then” attributes with values matching the scenario steps defined in the feature file. To do so, follow the samples provided by the template from which the new step definition file has been generated.

SpecFlow searches for the step definition text in the step definition files that matches the text in the scenario step.

Now you can implement the automated step, either by creating recording modules or by manually implementing the steps using the Ranorex automation API.

Your implementation might look something like this:

You can create additional steps to be executed before and after a scenario, such as performing setup and tear down tasks, including starting and closing the system under test.
These steps can be defined in an event definition file and might look something like this:

If your assembly cannot be found please see the troubleshooting section at the end of this article for further details. 

Run a BDD test using NUnit

Because this solution uses NUnit as the unit test provider, we use a NUnit Runner to execute our tests.
Your command line call should look something like this: (Use root folder)

..\..\..\ .\packages\NUnit.Runners.2.7.1\tools\nunit-console-x86.exe <solution>.dll

Conclusion

The preparation of Ranorex for BDD is complete. From this point forward, your business team can define user stories in a natural language following the Gherkin syntax, and your development and QA team can implement the automation steps in the step definition files.

Troubleshooting

The code/csharp file is not generated

If the code file is not created in Ranorex Studio, you have to set the “Custom tool” in the files properties to “SpecFlowFileGenerator” to make Ranorex Studio create the code file.

Assembly not found

If your assembly was not found you will need to build your project as a library. To do so, open the project properties and set the output type of the project to “class library”. In addition, copy all Ranorex runtime DLLs to the output folder.

Many teams measure velocity. However, cycle time often provides a more accurate measure and better data. I’ll discuss when you might use each.

Kathy, a tester, knew there was something wrong with the way this new-to-her team worked. Her previous team had used velocity to estimate their stories for the iteration. This new team also tried to use velocity, but they never quite estimated correctly. This new team used the same techniques. But, they almost always fell short of what they thought they could finish.

In her first retrospective with this team, Kathy asked this question, “Is it me, or do we seem to have trouble using velocity to estimate what we can do in an iteration?”

One of the developers, Dan, grinned. He said, “Oh, it’s all you. No, seriously, we never seem to get it right. Have you seen velocity work?”

“I have,” Kathy said. “On my previous team we used velocity and we were spot on with our estimates.”

“Do you know what’s wrong?” Dan asked.

“I have a suspicion,” Kathy said. “But I need to gather some more data first. I don’t think we’re all wrong. I think our context is different than my previous team. Is it okay if I gather more data?”

Dan frowned. “You can. But why don’t you tell us what you’re thinking and we can help gather that data?”

“Okay,” Kathy said. “But I could be wrong. As long as you know that.”

Dan and the rest of the team nodded.

“I think we have too many wait states,” Kathy said. “You guys develop larger chunks than I’m accustomed to. I’m not saying that’s wrong. But, then you have to wait until you’re done developing before you do code review.”

Dan nodded. “True,” he said.

“After you guys are all done with your code, I get to test it. And, because you merged several pieces of functionality, my little tests aren’t enough. I have to write more tests. I don’t always finish before the end of the iteration.”

“Yeah, I noticed that,” Dan said. “But, it does mean you have work at the start of the next iteration.”

“True,” Kathy said. “And, we can’t take credit for partially completed work. We account for all of that story’s points in the next iteration. So, instead of our velocity being a somewhat standard 32, we go up to 39 or 40. Or, we go down to 25 or even 20. We’re just not consistent.”

Sam, another developer asked, “What’s really the point of consistency? So we get 32 or 20 or 40 points? Does it really matter?”

“It only really matters when an external date is on the line and we can’t predict when we’ll be done,” she said. “Yeah, it matters to us, too. But when a product manager or a salesperson wants to tell an important customer, and we don’t have good enough data, we look like fools. Worse, the people make dates up if we can’t be consistent.”

Dan nodded. “We’ve been there.”

“That’s only half the story, though,” Kathy said. “What really counts is we don’t know why we have delays and where those delays are. Some of the development delays turn into testing bottlenecks. You know how you’re waiting for me to finish the testing?”

Most of the team nodded their heads.

“Yeah, I become a bottleneck because the work I get is too large for me to finish faster.”

“We’re going as fast as we can,” Sam said.

“I agree,” Kathy said. “That’s why I need to gather data. I can’t tell if it’s the size of the work, or the handoffs between the developers, or the final handoff to me. It might not be any of that. It might be that Prudence, our PO, isn’t around enough to guide us. I can’t tell. I need to measure the cycle time.”

Dan said, “That’s a great idea. Let’s all do that, not just you. Fist of five, everyone?”

Every team member held up either four or five fingers, signaling their agreement.

“Excellent,” Dan said. “We’ll measure cycle time and then decide what to do in our next retro.”

Velocity Is Not Just Story Complexity

Many agile teams think of story points as a way to see the story’s complexity. However, story points don’t tell the entire story. Story points don’t take the team delays into account.

Cycle time is the measure of how long it takes a team to complete an entire story, and the cycle time includes the delays. It doesn’t matter how complex the stories are. Teams can measure their cycle time and use an average cycle time to predict what they can complete in an iteration, or when the team can finish certain features.

It all depends on how long the average cycle time is for the team.

Kathy’s team had these questions about their cycle time:

• Are our stories larger than we might like?
• Do we have more handoffs and cycling back to previous states?
• How long does work wait for us to finish?

Without data, Kathy and the team can’t know if any or all of these ideas are true. They don’t even know if the problem could be elsewhere. That’s why they need to gather data.

Map Your Team’s Value Stream

A value stream map shows you a team’s various work and wait states. Someone (or people working together) take an item off the Ready column and start to work on that item. They work on it until they can do no more. Do they need to wait for someone? If so, the work goes into a wait state until the next person can take the work. That person then works on the item and when they’re done, the item waits for the next person. That repeats until the team has finished the work item.

Add all the times, for all the work states and the wait states. That’s the team’s cycle time for that item. If a team swarms or mobs on the work, all the time is work time. There is no wait time.

However, in this case, Kathy knows that the team uses several work states: in development, in code review, and in test. That means there is a wait state between all the work states. And, if someone finds an issue in code review, the work might cycle forwards and backward until the team finishes the work.

It doesn’t matter if your team is collocated or distributed. When teams don’t work on one item at a time, they incur wait states. Once you see your value stream, you can calculate the cycle time.

In the next iteration, Kathy calculated these cycle time measurements, with the help of her team:

At the next retrospective, the team wasn’t done with Story 3. However, even with this limited data, they could see some of their problems:

• Their stories were too large. Any time a story stayed in Code Review for 2 days, that story was too large.
• They had to wait for Code Review.
• They had varying work time and wait time, but they appeared to be able to do not more than 2-3 stories per iteration, because of their cycle time.

They knew they had to change something. The created several possible experiments:

• Always pair so the developers had built-in code review.
• Create smaller stories.
• Check smaller chunks into the code, so they could review more often.

They decided to start with checking smaller chunks into the code first. That would allow them to review more often.

Measure the Experiment

The team still had to complete Story 3. Then they could see if their smaller chunks worked. That worked better for reducing the time waiting for code review, but Kathy still needed the entire chunk so she could test and know if the item met the acceptance criteria.

Over the next three iterations, the team tried pairing more often. That also helped. Then they tried swarming and that reduced their wait time dramatically.

After two more iterations, they asked the product owner to work with them to create much smaller stories. They wanted to create one-day stories.

They’re still working on how to create one-day stories. And, even without stories that small, they have their average cycle time down to 2.5 days. That means that for a 10-day iteration, on average, the team can complete six or seven stories.

That’s good enough estimation.

Decide When to Use Velocity or Cycle Time

I no longer use velocity to estimate or predict. I only use cycle time or count stories. The larger your stories, the less predictable your cycle time is. I find that measuring cycle time becomes a forcing function for the team to create and finish smaller stories.

If you’re not sure what to do, visualize your team’s work states and wait states. Measure the time the team spends in each of those states. Then, you can create options for experiments or changes.

Velocity might be useful for your team. However, cycle time will offer your team more value over the long run. You can see and measure how much time your team spends in the work states and the wait states. And, you can see alternatives to create better flow through your team.

Maybe I’m reading the wrong articles. Maybe I am listening to the wrong podcasts. Maybe I am finding the wrong sources and references.

Why do I think I’m checking all the wrong sources and “experts”? Because loads of them will say things that sound great. The problem is, when it comes to the actual doing of those things the great sounding stuff suddenly gets vague.

Testing is supposed to “provide value.” Software Testers are supposed to be the ones who “provide value.” Except, somehow, these same experts usually fail to say how they are to do this.

Traditional Scripted Testing

I work hard to understand testing. I also work hard to understand concepts like quality. I work hard to understand how testing is done well, and how the quality of a product can be improved.

When I look for ideas on how I can become better at these I often come across writers, speakers, pundits, and “experts” who advocate ideas I have heard over many years. These ideas might be valid in some situations. They are not all universal. They do not really tell us anything about adding value, except by inference.

What do I mean? The list often looks something like this:

• Software testers find bugs
• Software testers verify conformance to requirements
• Software testers validate functions
• Software testers improve quality

There are different versions of these ideas; they may be expressed in different ways. Some organizations embrace one or more of these ideas. They define and direct testing based on their understanding of what these ideas mean. They insist that testing be done a specific way. They insist that practices in place that have “always been followed” must not be disturbed. That these practices ensure maximum effectiveness and the best possible results.

In some ways, I get it. There are loads of reasons to not change how things are done. People are comfortable with how the organization has worked in the past. Change is uncomfortable and can sometimes be messy.

I understand why people might like having detailed scripts that lay out precise steps to be followed. I also understand why the “expected results” are so important to many organizations. I get the comfort of standardized approaches. I really do.

For several years I was quite comfortable with them. I worked in shops where these were the norm. Detailed scripts direct people’s efforts. They cut down on distractions. They drive the focus of work to be done. But except in narrow instances — for example, those working under strict contractual requirements  — I have come to firmly oppose using them.

Problems with Scripted Testing

Proponents of using detailed test scripts often suggest they can help train new people in how the software functions. Another argument is to make sure everyone understands what is expected to happen. Both arguments focus on the “expected results” format for test cases and test steps.

My concern with that is, if the “expected results” state one thing, people executing the steps usually find themselves focusing only on that. This is not a criticism of the people doing the work. It is a criticism of the belief that very limited focus will determine success or failure of the work.

We may very well get the precise “expected result” to appear in the screen or the column in the database. Did anything else happen? Were there any error messages in application or system logs resulting from this test? Were there any odd behaviors not noted because the “expected result” happened?

One obvious solution is to have people to look at broader aspects than the documented “expected results.” Except, this sets up a conundrum around what is and is not part of what should be looked for. Many of us would assert that the tester should track down apparently odd behavior out of professional responsibility and due diligence. Testers should take the time to investigate what is going on. I absolutely agree.

If the team is to reduce variation and execute the scripts as written, is that likely to happen? If tests have explicit time frames for execution, how likely is it a tester will do anything to impact those targets? What is the reasonable outcome if something odd is seen but the “expected result” is met? People tend to ignore these errors if they feel constrained from investigating them.

Of course, management can insist that testers be “professional” and investigate off-script issues. Will testers follow those instructions if their performance measurements drop? If part of their performance review and their pay/bonus is tied to those measures, can we really expect them to branch out from documented steps?

Teams relying on “click and get reports” tools for functional or UI testing are set up for a similar problem. Without careful investigation of the results in tool and application logs, problems not accounted for in the expected results will be missed. That means the error must be anticipated for the automation code to look for it.

A New Path

I tried these techniques for several years and have seen their shortcomings firsthand. I’ve explored the consequences of these ideas. Early in my career I tried to make them work. Believe the problem was I was not doing things “right” I doubled down and worked harder. Then I began looking for ideas that might work instead.

It has become a decades-long search that has led me down many paths and investigated many possibilities. I don’t believe that the ideas described above work as people say they do. To be honest, I’ve found few approaches that work every time in every situation. This led me to another idea. This one I have been pursuing for several years, and so far, I find it generally holds up. It is not an academic testing definition. However, it is a reasonable working definition:

Software Testing is a systematic evaluation of the behavior of a piece of software, based on some model.

Don’t look for bugs exclusively. Don’t look for proof of requirements, exclusively. Instead, look at the software’s behavior.

If we have a good understanding of the intent of the software, we can develop useful models around that understanding. We can consider logical flows people using the software may use to meet their needs.  Noting what the software does, we can compare that behavior against the expectations of our customers.

These observations can serve as starting points for conversations with product owners and managers. They can incorporate documented requirements along with the product owners’ expectations and expertise.  This means the project team can choose the path they wish to examine next based on its significance and the likelihood of providing information of interest to the stakeholders.

Instead of a rote checklist, testers working with product owners, the development team and other stakeholders can compare their understanding of the software and ask the crucial question of “Will this software meet our needs?” Comparing system behavior with the documented requirements means that testers can initiate and participate in discussions around the accuracy of the requirements (do they match the expectations) and the way those requirements are communicated. Thus, testers help reduce the chance of misunderstanding. This helps the Business and Requirements Analysts do a better job writing requirements and position the organization for conversations around how to make requirements and the software better.

By changing what we are looking for from specific items in a checklist to looking at overall behavior with specific touch points, we change testing from an activity that merely checks a box, to something very different.

Value

If you, like me, have been told what “good testing” or the “role of testing” is from someone who has not done software testing or developed actual working software in 20 years, if ever, I have an invitation:

• Join me in rejecting grand pronouncements with no real meaning, like “add value.”
• Walk away from limiting definitions like “find bugs” and “verify requirements.”

Instead, make the conversation about what we can do as testers:

• We can identify behavior.
• We find limits and characteristics of the system.
• We can compare these findings with the expectations and documented requirements and understanding.
• We can bring awareness of the reality of the software to the development team. We can bring awareness of the state of the software to leadership and organizational management.
• We can make clear ideas and information that is not clear now. We can shine light into areas that are murky. We can bring them truth and fact.

That is what software testers do.

That is the value in software testers.

Testers find defects and raise awareness about quality. What happens after the bugs are found can be any tester’s guess, though. Bugs may get delayed, postponed, go unnoticed or linger on due to lack of information.

Testers need to champion the cause of their bugs in order to avoid unneeded delays in fixing defects that are important. At the same time, testers should maintain a distance to make it an impersonal and impartial experience. Testers need to master the art of bug advocacy.

Why is advocacy important?

Advocacy is basically pleading the case for a bug to be fixed. The testers who find the bugs are the ones who need to advocate for their bugs. It is important that they take a stand and voice their opinions.

Some bugs may not be deemed important from a business perspective, as they seem too small. But in reality, they may be blocking an important feature for a particular user group. On the other hand, some bugs may seem more critical than they truly are, and while fixing them may be important, it may not be the highest of priority.

Whatever the case, testers must aim to present the facts and data in such a way that decision-makers are able to make well-informed resolutions about the issue.

Communication is key

Advocating for anything is not a one-way street. It takes discussion, debate and reaching a consensus on key points to make a collective decision. This is where testers’ communication skill plays a key role. Testers need to have good communication, both verbal and written.

When writing bug reports, testers need to focus on providing every possible piece of information that can help the team in understanding and triaging the issue. Missing, incorrect or ambiguous details make the bug look less relevant and important and may lead to the issue being ignored or deferred. To stress the importance of the issue, testers should mark the correct bug severity and use a good, descriptive title.

Beyond this, if you observe that the bug you reported is being postponed or not looked at, you must be able to initiate a discussion or raise a point during a triage meeting. Good verbal communication skills include being able to present your argument, discuss the repercussions and reach a conclusion with the team in a professional manner.

How and when to advocate

Since more and more teams are following agile development methods, communication is mostly verbal, and triages are quick discussions of bugs found in a day or week within a sprint. A short, live demonstration of the issue is the best way to present the bug if the agile team is not using detailed bug reports.

Testers can also collaborate and pair with developers for short test sessions and showcase, resolve and retest the issue together in the least possible amount of time. But to do this, testers must understand that in order to work on resolving issues, developers need to switch context from the new features’ development. Testers should find the appropriate time to discuss an issue so that it makes sense for everyone.

It’s not personal!

Bug advocacy is not about finger-pointing or blame games. Advocating for an issue does not mean testers criticize developers’ work or take the opportunity to blame people for their errors. It is simply putting forth the assessment of the bug, its impact and the consequence of not fixing it.

Once that is done, testers need to learn to disassociate themselves from the issue on a personal level. If the decision-makers choose to still defer or not fix the issue, it is not personal!

Testers need to remember that it is not up to them to decide whether a bug gets fixed. The business makes the decision based on multiple factors, like the risk it poses, time constraints and budget. Testers can keep track of postponed or deferred issues and raise flags when the list grows beyond an expected number in order to escalate more bug fixes.

The important things to focus on are bringing forth the facts, presenting them in an understandable and relevant fashion, and then leaving the decision to the business.

In recent years, the technological needs of consumers have exponentially increased, forcing companies to build applications that are more scalable, secure, multi-threaded and faster to respond than ever before. To ensure all these attributes are present in consumer applications, APIs were created, providing developers with the ability to tap into external programs instead of creating a new one from scratch.

APIs, or application programming interfaces, provide a level of abstraction by exposing only the data needed to communicate with another system. For example, Google’s Maps API is used by many organizations to provide Google Maps functionality, rather than companies building their own map applications. The same holds true for Mint, the finance application that connects to different banks via APIs to fetch required customer data.

The growth of new software development techniques such as microservices has served as a catalyst in the API movement by removing dependencies and establishing easier data exchange between different software components.

More often than not, organizations overlook testing APIs because they opt to focus on other aspects of the system, such as the UI, databases, application code, security and performance. API testing is also bypassed because it is time-consuming and requires certain specialized skill sets, tools and approaches.

But like the ugly duckling from Hans Christian Andersen’s timeless tale, API testing shouldn’t be overlooked. Organizations often fail to realize the critical importance of APIs and the impact of their failure in production.

Here are some different aspects of testing APIs and what to specifically look for when performing this kind of testing.

Develop an API Testing Strategy

Before testing APIs, the following questions should be discussed by teams to get a better understanding of what needs to be tested:

• Is the API public-facing or internal?
• Who is the target audience of the API?
• Is it necessary to integrate with components outside the system or application?
• What are the endpoints and value types?
• Where is the data coming from? Can the data be accessed freely, or is a particular key or authentication required?
• How can the expected result be verified? How will it be determined if the API has performed as intended under the specified circumstances or scenarios?
• Is the API logic simple, or is it more sophisticated, with dependency-based decisioning?
• Under what conditions is a site, application or function most likely to fail?
• The answers to these questions will help you to plan the API testing effort more effectively.

Cover 3 Aspects of API Testing

There are three major aspects to cover when performing API testing.

Connectivity

To test API connectivity, make a call to the API using its URL. If a 200 response is returned, the API is connected. If no response is returned, or a connection failure error is returned, then connectivity failed, meaning the request was not received by the server.

Responses

All API calls return different status codes indicating what happened during the interaction. This serves as a reference to see whether the APIs are working as intended.

These are the main different status codes to pay attention to while doing API testing:

• 400 BAD REQUEST: Generic error that is returned when no other 4xx status code is appropriate, such as for domain validation errors, missing data or improper API requests
• 401 UNAUTHORIZED: Missing or invalid authentication token
• 403 FORBIDDEN: The user is not authorized to perform the operation
• 404 NOT FOUND: The requested resource is not found—it may not exist, or there may have been a 401 or 403 error and, for security reasons, the service masks it as a 404 error
• 409 CONFLICT: Resource conflicts, such as duplicate entries are found, or an attempt to delete root objects when cascade-delete is not supported
• 500 INTERNAL SERVER ERROR: General catch-all error when the server side throws an exception and the consumer cannot identify the exact error from their end

Performance

An API’s performance can be its most valuable feature. Each time an API request is made, the response must be returned in a matter of milliseconds.

Depending on the application, thousands if not millions of API requests can be made at a time, as in the case of Facebook, Google and Twitter. Therefore, APIs should be able to handle large amounts of load without failing. If the API does not have fast response times and is unable to handle large numbers of requests, it is essentially worthless.

Choose Your API Testing Tools

There are various approaches and tools to test APIs. Some can be tested using standalone tools specifically meant for API testing, and there are others that enable performing API testing along with other kinds of testing using additional code. 

We are living in a world of APIs. Every second there are thousands of API calls happening in the back end without us even noticing it. Performing API testing is a critical part of the overall testing process, and it is about time we start paying attention to it.

The Satires by Juvenal has a line in it which often gets pulled into things that may (or may not) have little to do with the original intent: “Quis custodiet ipsos custodes?”

Generally translated as “who watches the watchmen” or “who guards the guardians.” Never mind that a closer translation would run “who guards the guards themselves,” but there you have it. (Four years of Latin has some benefits.)

In the Satires, this was in a section where a fellow was uncertain of women and their reliability. It did not matter if it was in his family, or women in general. He was lamenting that anyone who would watch them (either to control or for protection) could be corrupted. They would need to be watched themselves.

Who tests the testing?

Software testing can be complex. I’ve found that things that are complex get glossed over by people who are not deeply engaged in the activity. This can be managers, developers, other analysts and other testers.

We often rely on some vague notions about test plans and test scripts to demonstrate the testing is being done well. We might also rely on “automation” to demonstrate that the testing is good.

How do we really know the testing is good? How confident are we in the effectiveness of our software test efforts? Are they doing what we think and hope they are?

Testing

Even when done well, testing as an independent activity is problematic. People working in an environment that expects or requires formal test plans for each project spend loads of time preparing documentation based on their understanding of the system function.

If they are moving onto a system area that has not had anyone testing it before, they might find themselves in a situation with no documentation around testing at all. They also might have little meaningful or accurate system notes about how pieces work and interact with each other and how the system acts with external systems.

Combine these needs with a common attitude of “everything you need to know is in the documentation.” Then mix in the expectation that testers develop tests strictly from the specifications, or the “story cards” if they are in some form of “Agile.”

It is impossible to develop meaningful tests in these conditions, particularly if there are time constraints around them, for example, the “testing must be done in a two-week sprint” because the shop is “Agile.”

It is inconceivable that any meaningful testing work can be done. It will be more rote “check the box” work and not insightful or informative to the organization.

Is it any wonder that under situations like this, any attempt at thoughtful deliberate testing is set aside in favor of a test automation tool? Writing code to handle automation might give people the experience needed to say they have “worked in” automation.

Test Automation

Test automation can contain some of these same serious challenges. If information is limited to documented requirements, system documentation and limitations on conversation with system or application experts, the automated tests will be like the tests created for “manual” testing described above. They will be narrow focus and shallow in depth.

If the intent of test automation is to speed testing and make testing efficient by executing the lower-level, mundane tests, are the narrow-focused tests going to do what needs to be done? Will the portions of the system that need testing regularly actually get tested?

Test automation comes with the same bonuses of any other piece of software. Does the code do what we expect it to do? Does it do what we need it to do?

How carefully is the code for the automation tested? This code is prone to the same problems (bugs) as any other code. How carefully will this code be tested? How certain are you it will be working as needed?

As with any code, over time, some functions will become redundant as new functionality is added or behavior changes. Will the tests that exist for those functions be maintained appropriately?

In my experience that is not always the case. This leads to failed tests which get ignored or written off as “that always fails.” The result is people don’t pay as close attention to the test results as they normally would. Legitimate failures get missed.

Agile

Agile practices introduce their own issues and problems for testing.
The point of Scrum, which many people mean when they say “Agile”, is to deliver a working piece of functionality at the end of each sprint. Where it gets interesting is when the teams take the approach that the work to be done is development alone – that testing can happen later. The “later” is in a different sprint.

Which means, the idea of delivering a working piece of software is rather lost in translation.

Many organizations will respond with “we do automation” to test software and test it quickly. Great! Is it done in the same sprint as the development work? Are you wrapping up “stories” and handing them off tested and ready to go, confident in their correct behavior?

The times I have been told “We do TDD, then automate those tests” are too many to count. I get it. TDD has “test” in the name. Still, that is a development and design technique to speed development. It is, effectively, a set of unit tests with the expectation that if the unit test “works” then the goal is met.

Automating that is fine. At least, it is fine if someone tests the incremental changes between sprints.

Does the work of the last two or three sprints, the “incremental improvements”, all work together? Are there any issues found that would not be found in a unit test? Have you exercised the potential environmental variables?

How about error conditions? Have those been exercised? If not, do you really have a product you could safely deliver to customers?

Depending on how often the organization releases software this may not be a big deal. It seems to me, however, that if a team is not doing these things, then the broader organization has some challenges beyond testing. It also has challenges around what “Agile” means.

In companies where I’ve seen this quasi-Agile-ish approach, they have also had challenges with how to deliver working software.

Testing Testers

And so, to the rub of the problem. The center of the conundrum.

How do we know that testers are doing things well? If we reward them for stopping significant problems released into production, the best we can do is reward them for writing bug reports. This can be painfully counterproductive as bug reports are often seen as criticism of the code writers in the development team (of which testers are a part.)

What if the people doing the testing are engaged with the people who wrote the piece of code they are testing? What if they can send a message to them over a chat tool, or turn around in their chair, or look to their left or right and ask a question?

What does it mean to be a “tester”?

When testing is seen as a step removed from making the product, testing will always be a “cost.” When testing is integrated in making the product, it is simply “making the product.”

The most effective testers I know are not the ones who raise the most interesting bugs or write the most detailed bug reports. They are the ones working with their teams, side by side as equals, making sure the bug reports don’t get written in the first place.

I’ve come to believe that this then, is the only meaningful “test” of testing. It is the measure of software such that the delivered product has no notable, identifiable problems found by customers of the product.

This is also the only meaningful measure of software.

The testers must test themselves.

Everyone is a tester.

Using the 12-factor methodology is the right way to write applications in 2020. These dozen factors are guidelines for ideal app development that take into account maximum possible agility, onboarding new developers, and how an app will scale over time.

The first of those 12 factors prescribes one codebase tracked in revision control. Let’s look at our codebases more closely and consider frequently misunderstood or overlooked examples of what “in revision control” means for development teams.

Think of it this way: If a meteorite landed square on your office and obliterated all your hardware, how long would it take you to restart normal development activities? For some of the projects I manage, that time to recovery would be around three minutes: pull a convenient desktop off the shelf, clone a copy of the repository and launch a standard installation.

Other projects would probably never recover. Even though their repositories are formally comprehensive, the QA department keeps its own copies of tests on a server outside central control. Or unit tests depend on environment variables whose configuration no human entirely remembers. Or the user experience team hosts its visual designs on a fileshare whose backup I can’t access. And so on.

Test that it’s all in there

Make a task out of this. Bring in an experienced developer or tester who doesn’t know your project, and have them go as far as possible going off just your README.

Does it take four pages of project-specific instructions before they can run the first project-specific unit tests? That probably represents a lot of opportunities to script more of your installation. Does your project install easily, but programmers new to it require a couple of hours of training in the IDE and tool configuration before they accomplish anything useful?

Prep default configurations and make them part of the standard installation. Script the first-time installation for a beginning programmer, and make sure that is in the repository. Include explicit details about how to take the first steps in a README, which of course is itself under version control.

Do you have utilities that you use occasionally, perhaps to help generate favicons or run business metrics for the project? Make sure they’re in the repository. Do you need to verify backups periodically, or scan include libraries for security vulnerabilities? Script those scheduled processes, and put the script under version control.

Your repository automatically generates a history or log of commits, plus their annotations. Consider writing an additional narrative of prominent milestones — not for every business-oriented story or card or enhancement, but for a couple of the biggest technical advances from each month, with hyperlinks to pertinent commits or tickets. Maintain that narrative under version control, and help your team be able to find details about how, why, and when your project changed its front-end framework or swapped its back-end load balancer or migrated to an industry-standard library for a specific algorithm.

“Red books,” or crib sheets that help operators diagnose problems in production, also deserve to be under version control. It is advantageous both to coordinate their management with the artifacts that produce application behavior and to document who committed which updates when.

Does your red book tell operators what to do when a particular process is anomalously slow? Are its example screenshots from the version from three years ago, and the threshold for “slow” outdated by fourteen months? Those details are easier to get right when coordinated with the “canonical truth” of the central repository.

Too many organizations say they have DevOps but just proliferate silos under new names. A consolidated, well-maintained central source repository is one symptom of healthy cooperation among different roles.

Just as all artifacts deserve testing, they also all deserve to be under version control.

There are some exceptions

A few exceptions to this commandment are important to understand. Binaries generally don’t belong under source control, especially distributed source control like Git. Secrets are another important exception. Do not check passwords or customer data into source repositories, even if the repositories themselves are maintained as trade secrets.

More subtly, some things don’t belong under source control because they already have other and better homes. Your team probably has a dedicated task manager or problem ticketer. Don’t duplicate the information there; simply refer to it. Don’t write up an emergency incident in the source comments of a particularly crucial algorithm; instead, provide URLs in source comments — something along with the order of:

/* See <URL: http://$SOME_URL > for an example of why it’s
 important to limit MAX_HOSTS to 16. */

Check your code repository before you need it

Programming and testing teams usually point confidently to a particular repository with the thought that it holds their sources. That naive belief is different, though, from 100% certainty that everything is in its proper place.

Just as IT regularly tests backups, it’s equally imperative that your team verify that your repository truly does all for you that you expect. Trace the steps and examples above, and fill any gaps before an emergency arrives.

The world has drastically evolved in the way teams work together. Employees used to have to go to a physical office where they had desktop computers set up for them. All the work was performed from the company location.

Then, teams were provided laptops, which was a welcome change — but they were still mandated to come into the office to perform work. In the last few years, the times have changed, and more teams work from remote locations and are still able to figure out ways to collaborate with one another.

We are living in the era of remote, self-organized teams, with individuals who can complete tasks at their own pace and still schedule a time to work collaboratively to deliver high-quality products. Yet even though these teams are self-organizing, they still require management, and managing remote teams requires different skills from managing collocated teams. Here are five ways to effectively manage distributed teams.

1. Schedule meetings to check-in

Organizations with distributed teams often have people in different locations and even different time zones. It becomes important to periodically communicate with all of them to ensure they are progressing as expected. There are two effective ways to accomplish this.

One is by having regular weekly or biweekly one-on-one meetings with team members to ensure they are getting the needed attention and their ideas and views are heard. This also has been known to increase employee retention and job satisfaction.

Another way is to let teams meet periodically to discuss each other’s work and get clarity on the goals, tasks, and objectives for the upcoming period. This could be done through daily standup meetings, retrospective meetings, planning meetings, or any other meeting where all the individuals in the team are given an opportunity to share their opinions and status on different tasks.

2. Invest in collaboration tools

Collaboration is the key to success for any agile team, but this is especially true when it involves remote employees. To foster collaboration, invest in collaboration tools that include chat and video-conferencing functionalities. Research suggests 93% of our daily communication is nonverbal, and it significantly affects behavior in the workplace. Being able to see people’s body language and facial expressions through video can help to proactively prevent gaps in communication.

3. Respect time zones, schedules and holidays

When working with teams in different regions, we need to be mindful of the way we communicate and set expectations. For example, if a company has teams in the United States and China, we need to realize that China is about eight hours ahead of the US. So, when scheduling meetings or expecting replies, ensure it is at a mutually agreeable time.

If a compromise in time is not possible and some meetings have to be scheduled early or late in the day for one of the teams, alternate the meeting times so that the same team isn’t always having to meet outside regular work hours.

When working within strict deadlines, make sure to mention the time zone along with the time you expect something to be delivered. For example, if a status report is expected from a remote team working on the West Coast of the US at 11:00 a.m. and you are located on the East Coast, make sure to mention that it’s the Pacific time zone to prevent ambiguity in expectations. This sounds simple, but it is often a major cause of confusion.

Finally, pay attention to domestic and national holidays for teams working in different regions, as it may affect timelines in planning and completing tasks.

4. Encourage social events

Although there are cutting-edge collaboration tools available for teams, nothing can replace face-to-face communication. Encourage having on-site social events where every team member is expected to attend. This gives an opportunity for people to get to know each other on a personal level and foster more connections between teams.

5. Appreciate good work

People often value words of encouragement and appreciation more than monetary benefits. To build better relationships with employees and coworkers, make your appreciation known via emails or thank-you notes, and let their peers know when someone does a great job. This appreciation has a positive impact on the whole team as well as the individual.

When asked about what things are important to track when testing, I find most people give a pretty detailed list of items. They’ll mention system or machine states, DB or other data system availability, structure or readiness. They might point out things like successful builds, available devices and tools, server and network readiness. They could point out mobile devices and emulators devices used by their customers to access their app.

Even when test teams can “control” their environment, there are some things they cannot control. Depending on the nature of the software and organization, teams may be able to control certain features on their test devices, laptops, desktops, mobile devices, emulators and the like. The challenge I sometimes see is if they can control all the features on all their devices?

Again, that will vary from organization to organization. Sometimes, from team to team.

Who is in control?

There is the question of the software under test. Development teams are making fixes and deploying them to some environment. Are the testers aware of when this happens? The question of controlling when the new builds get deployed to test environments where testing and what kind of notification occurs is extremely important. This is particularly true when the test environment is shared across multiple teams.

If there is more than one project running in a single environment, make sure the question of changes and possible impacts to each team are made fully public. There needs to be a clear notion of how system and environmental variables are changed. It might be that any testers from any of the teams can make these changes. It may also be some admin not known to you.

What about switches that might be enabled at the database level, or in partitions of the test environment? If you can set them to emulate a particular customer setting, for example, how well can you control them to keep someone else from changing them for their own testing?

Look for program switches that impact functionality. It is possible these get changed in the test environment without everyone being aware of the change. Being aware of what the settings do, how they impact your software and other elements of the broader application is crucial to understanding how the software works.

Make sure you know when they are changed and who controls them in your test environment.

It is important for teams to verify what other teams may be working in the same environment. Teams need to communicate as much as possible over when code is being checked into a shared environment to keep from stepping on each other’s work.

What might testers control

Every project is different. Every team and software organization are different.

When asked, most testers would say they have complete control of their test environment. People will tend to believe that until something comes along to shake their beliefs. For example, asking questions around specific functions, system and environment settings and who can change them.

Are there other things they cannot control themselves?

There may be server settings which impact the stacks in the test environments. This can cause no end of consternation if not well managed. Then there are other questions around server stacks, for example:

• Can one stack’s variables be set independent of the other stacks?
• Can testers control which stack they are running against?
• Does the software under test depend on specific stack level settings configured set a certain way?
• Which values impact how the software works and who can set them?
• Is there a current list of valid environment parameter settings, like stack variables, that can be used?
• What happens if new ones are added?

These are questions I typically did not consider until I ran full-speed into them. Until then, I was certain I “controlled” the test environment I was working in. Then I found I knew less than I thought I did.

I now have a very different understanding of what “control” of environment and environmental variables looks like.

If you are working in a “cloud” or “server-less” environment, perhaps exercising a SaaS model product, these are concepts you might be familiar with. If you are not, it might be a good idea to find out if there are any present and how they can impact the software and your testing.

If you begin to work in such environments, these are questions to definitely think and ask about.

And what you can’t control?

These are variables we might be able to control. Completely outside of what any test scenario may call for, these combinations of settings and conditions can influence how the software functions.

We have a reasonable chance of controlling them if we know they are there, and how they impact the system.

What has not been considered yet is the level of access testers have to workstation/laptop/desktop settings and server settings. Some questions you might want to consider include:

• Do testers have administrative rights on their machines?
• Does this include mobile devices that might be needed for the application in all their forms?
• When testing POS and payment system applications, does the tester have administrative rights to configure the test devices?

In many shops I have seen, but by no means all, most testers do. Sometimes, not all. There might be one administrator for certain types of devices beyond computer workstations, tablets and smartphones.

Then there are the server and database control variables. Do testers have the access needed to configure those to match specific conditions and scenarios?

If you have customers, internal or external, do you have the means to configure systems to match how they are using it? What if they are more than one OS upgrade back from “current.” Can you test against all mobile device versions that are supported?

If we can’t control these things, which if we think about it might be kind of obvious, what else might we have a problem with “controlling”?

What about time?

There is one thing we can’t control, any of us: time. Things will happen whether we are aware of them or not. They may be happening all the time and we don’t notice.

For example, do you know what the “automatic” updates on workstations and servers do before they run? Do you have any control over disabling them? When do you find out about them?

One thing you might want to think about is after a day or two executing complex test scenarios, the workstation or server might get an “automatic” update. Can this impact our tests? We like to think not. Still, if it does, what happens if system settings are changed by default in the upgrade?

And when out of time?

What happens to the quality of testing as the deadline for the project or release approaches? The might team launch into overdrive and do excellent work. Of course, the team might put in massive hours and rush through a minimum number of steps to “check the box” that a feature has been tested.

Unless you have a time turner it is very likely that you are not able to do any form of time travel. This limits the amount of “last minute testing” you and the team can do. The problem, of course, is with rushed work, the odds of problems sneaking past you tends to increase.

If the tester is concerned about what is left to do, how much (or little) time there is to do it, and is not focusing attention on what is happening right now, it is likely that small, but potentially important problems will not be noticed.

The view of the testers tends to get pulled to other areas and concerns that the software at hand. Is there a way to avoid this? There may be, but that involves much more than “test faster” or “use more automation” or “deliver the code sooner.”

Even when we think we can control something, it appears we still cannot control when testing can start or finish.

Can you manipulate time?

There are other things that can happen that are beyond our control or expectation.

I remember one instance where a group of colleagues were heading out for after-work drinks on a Friday. I had been doing some complex configuration changes on the test system and decided to simply leave them in place until Monday. That would save me massive time reconfiguring the system.

When I came in that Monday, the system was down. There was a memory leak in the application I was testing. By limiting the amount of available memory on the virtual server it was running on and allowing the application to simply idle all weekend, the leak revealed itself by bringing down the server.

While I had not planned on doing that “test” it found an important problem I likely would not have found. It might be wise to consider other things related to time. For example::

• What is the result if you enter data too slowly or too quickly into tests?
• What happens if you enter data into random fields on the screen then walk away for 20 minutes or longer?
• What if you recreated my “test” above and left it overnight?

Consider what might impact the software and what happens all the time? What if you open a new browser window the next morning, without going back to the original window? What happens if you start multiple sessions and let them all simply idle?

Do you know what might happen if the host/server does something unusual while these multiple sessions are running? What is the result if they all end abruptly, say in rebooting the workstation?

What about one more weird what-if?

Are the workstations set to detect time zones? What happens if you go back an hour, for example, move from Eastern US Time to Central US Time? What if you move up an hour and go the other direction? Can you force the system clock on the workstation to change?

The ideas, questions and concerns raised are only a starting point to think about. Each organization will have a different set of challenges. What else can you not control that you have not thought of trying to control?