Three Tips to Improve Your Requirements-Based Testing (RBT)

By Moty Aharonovitz

Our primary job, as QA and testing professionals, is to find defects in software builds. Fortunately, however, most of us have moved beyond exposing and tracking bugs to the more critical role of ensuring that the software our company is delivering meets customer expectations before it is released. To do this, many organizations have recently embraced requirements-based testing (RBT).

Teams can be more effective using RBT because it enables them to target the root causes of quality issues by being tightly coupled with application requirements and integrated through various activities throughout the software lifecycle. As a result, it helps organizations deliver measurably high, systematic and cost-effective test coverage, and ensures that what our teams test is really what matters to customers.

The challenge as we roll out RBT to our organizations is to ensure we are coupling its processes with existing QA and testing best practices. To that end, I offer three practical suggestions to improve your RBT practices:

The Quality Crisis

The Standish Group’s Chaos Report and other studies describe what has been the industry reality for decades: the majority of software projects fail to achieve schedule and budget goals. Poor software quality is a primary factor behind many failures, and often results in massive rework of application scope, design and code. Such rework extends release cycles and consumes significant additional budget. To reduce software failures, it is imperative that we better understand the quality initiatives behind the products being developed for today’s global economy.

Industry experience and studies show the two primary reasons behind poor application quality are: 1.) defects in requirement specifications and 2.) problematic system test coverage.

Defects in Requirements Specifications

Some of us are still not surprised when we hear about end user complaints and lack of adoption for applications that went through a rigorous QA and testing process that found few bugs. The reason we are not surprised about the uproar is simple: the requirements were wrong from the very beginning.

Requirements of complex software applications are often negotiated through two concurrent dialogues, which continuously evolve throughout the project lifecycle. These dialogues are focused around two questions: “what do we need to build?” and “what can we build?” The quality of these dialogues often determines the ultimate quality of the constructed application.

A study by James Martin¹ shows that the root causes of 56 percent of all defects identified in software projects are introduced in the requirements phase. Further, the study states that approximately 50 percent of requirements defects are the result of poorly written, ambiguous, unclear and incorrect requirements. The other 50 percent of requirements defects can be attributed to incompleteness of specification (i.e. requirements that were simply omitted.)

Diagram 1: Distribution of Defects in Software Projects

We conclude the two primary issues with requirements quality are that requirements and specifications are incomplete (due to lack of input from users and other key stakeholders), and that requirements are specified poorly (by domain experts who lack the skills to define consistent, accurate and testable requirements, and whose tools do not allow for efficient validation and/or review of the requirements with the users).

Problematic Test Coverage

When quality is handled as an afterthought, quality verification efforts typically begin only once code has been completed. At this point, the testing team is under the gun to certify the application as quickly as possible, and such certification is often perceived as a bottleneck that prevents deployment into production. In this environment, it is not only difficult to ensure that the requirements are correct, but also to properly plan tests, to ensure correctness, completeness and solid coverage of requirements and to gain visibility into the various quality aspects of the tested application. It is no wonder that frequently this proves to be a costly and frustrating exercise to all involved stakeholders.

On top of that, achieving satisfactory test coverage is challenging. The complexity of modern applications—in particular distributed application—has made it more difficult to cover all of the possible scenarios, due to the sheer number of alternative paths through the application. Any attempt to systematically (or automatically) consider all possible combinations results in a set of test cases that is simply to large to manage and takes very long time to execute, which can mean further delays in certification.

In addition, change management has become unwieldy in many organizations with application requirements changing hourly or daily. However without the ability to properly manage these changes, traceability between requirements and test cases is often lost, making it even harder to ensure continuous test coverage.

RBT Best Practices

As you know, RBT is an approach that is implemented through well-defined processes, which target the root causes of the quality problems mentioned previously. Even if your organization is not using the practice today, most of us are familiar with the key processes within RBT, highlighted in this diagram.

Diagram 2: RBT Process Flow

In addition to existing testing processes, organizations should adopt the following best practices to be successful with RBT:

1. Test Early and Frequently

As soon as requirements, design and code artifacts are ready, review them against business objectives, requirements, use cases and test cases. Test early and frequently because defects detected in earlier development phases are cheaper to fix and result in significantly fewer surprises when the code is actually delivered. Make testing a parallel activity to the development process to make all stakeholders aware of quality objectives. By doing this, testing will be seen less as a bottleneck activity—performed after code is delivered under extreme time pressures—and more as a key contributor in the overall quality process.

Since software project success can be directly correlated to a solid understanding and articulation of requirements, organizations leveraging RBT place a great deal of importance on the quality of requirements. For example, best practices within the RBT Requirements Quality process include the:

Validation of requirements against business objectives: This optimizes project scope by ensuring that each requirement satisfies at least one business objective. If there is no match between the requirements and business objectives, refinement is necessary.

Review of requirements by stakeholders: Feedback of users and domain experts should be used to refine the requirements before additional work is done.

Development of use-cases: Map requirements against a task-oriented or interaction-oriented view of the system. If one or more use-cases cannot be addressed by the requirements, then the requirements are not complete.

Application of language analysis techniques: Detect problematic phrases in the requirements and fix them to guarantee the consistency and clarity of the requirements. A problematic phrase is language that is ambiguous, unclear or inconsistent. Such phrases can be interpreted in multiple ways by different people, which can lead to confusion and errors later. Ambiguous terms also result in requirements that are not testable.

2. Test with Your Head, Not With Your Gut

Most organizations value experienced testers, and count on you to “catch” what others might overlook. While a single “go-to quality expert” may be great for an individual contributor’s career, it can also put unnecessary pressure on that person and the other members of the team because the organization is relying on instinct, rather than collective heads to achieve quality goals.

Today, it is uncommon that test case design is performed in a systematic and rigorous manner. Rather, many tests are designed based on gut feel and intuition, which leads to unpredictable quality. Best practices within RBT require organizations to support methodical and systematic test case design to ensure predictable test coverage. A rigorous Logical Test Case Design process facilitates test case design that does not rely on the skill or experience of specific testers. Rather, it helps organizations inject repeatability and method to the test planning process in order to make test coverage more predictable. Organizations using this method can also apply various optimization techniques to produce the minimum number of test cases required for sufficient test coverage, making the testing cycle faster and more manageable.

In most organizations, textual requirements are very important given that almost all business analysts and domain experts use natural language to express requirements. However, using textual requirements can make it more difficult to achieve and validate high test coverage. As a result, testers relying exclusively on natural language can only use their intuition to claim that the set of test cases that they designed covers 100 percent of the functionality captured in the requirements. In other words, with non-structured textual requirements these organizations have no formal way to show complete test coverage, and therefore often “corner” or even major scenarios remain untested until defects occur in production.

To systematically achieve high test coverage, organizations must aim to create more formal and structured representations of requirements. Once this is done, it is possible to define “logical” (skeletal) test cases and ensure optimal coverage of requirements, as these logical test cases are eventually evolved into the actual set of tests that are ran against the system.

Multiple techniques can be used to provide structure and formality to natural language requirements. The purpose of these techniques is to reveal cause-effect relationships embedded within requirements, that is to express requirements as a set of conditions (causes) and resulting actions (effects). Cause-effect charting is one of these techniques. Another way to achieve similar goals is to express requirements as flow charts, since they naturally depict precedence dependency between actions as well as conditional branching of activities.

Once expressed in the structured manner describe above, it is much easier to derive the equivalent test cases for the requirements. A set of logical test cases can be defined (manually or automatically), which is exactly equivalent to the functionality captured in the requirements. However, this set of test cases may include many redundant cases (i.e. overlapping with other test cases). To optimize the number of test cases but still provide full coverage, techniques such as decision (truth) tables can be applied if cause-effects charts were used to structure the requirements. If flow charts were used for that purpose, then generation of optimal set of test cases means finding all unique paths on the flow chart, for which there are known techniques.

Once test cases have been designed, there is a probability that some errors still exist in them. To detect errors in test cases and fix them, organizations using RBT best practices will incorporate reviews of the test cases by the authors of the requirements and by end-users and domain experts. This practice provides key stakeholders with an opportunity to review the requirements and the test cases in their structured form, and catch any defects that were introduced in the transition from natural language to formal form.

In addition, organizations taking a lifecycle quality approach go the extra mile to integrate the quality process into the development process, using the structured, validated, high-quality test cases generated during the previous phases.

For example, they review design against the logical test cases—as they are simply a different representation of the requirements—to ensure the design is robust enough to satisfy the requirements. If the design cannot meet the requirements, then either the requirements are problematic and additional refinement is necessary, or the design requires rework.

To avoid costly rework, test cases should also be reviewed by the developers who wrote the code to ensure they understand what will be tested and to provide them with a structured view of requirements.

Finally, individual code modules should be reviewed against the structured requirements that trace to them to ensure that the delivered code conforms to the requirements. It is much easier to match the algorithmic expressions captured in code to the formal view of requirements than to compare it to their non-structured view.

3. Test with Measurement and Improvement in Mind

It is often said that what cannot be measured does not exist. Using RBT best practices, organizations can manage and improve quality processes through measurement. Throughout the RBT process, multiple measures can be used to quantify the status of deliverables and activities. This helps managers and process experts oversee quality initiatives across the IT application portfolio.

The Role of Traceability in RBT

Traceability also plays a critical role for organizations using RBT since maintaining traceability information between requirements and test-cases and tests is crucial. This information is required for monitoring progress and coverage, as well as properly managing the impact of changes in requirements. Without it, it is more difficult to determine which test cases or tests should be changed if a specific requirement changes.

While capturing and maintaining traceability data is very important, many software development organizations find that it is extremely difficult to get right. The primary reason is because most tools in the market require the individual practitioners in the software delivery chain to manually create and manage traceability information. This is an overhead that many are not willing to accept. To deal with this challenge, strongly consider tools that can automatically manage links between work products.

Taking the first step to Lifecycle Quality Management

By deploying solid RBT and QA/testing best practices, organizations will find they are better equipped to maximize the business value of their software through quality initiatives that span the complete software lifecycle.

Teams embracing RBT best practices are in essence taking the first step to a more comprehensive quality approach, known as Lifecycle Quality Management (LQM). By moving beyond just a focus on code to addressing quality into each phase of the lifecycle, LQM helps software delivery organizations ensure higher standards of quality and service, while systematically reducing costs associated with rework and maintenance.

Further, LQM accelerates the path to Software Delivery Optimization (SDO), which helps transform software development into a managed business process to increase control, predictability, visibility and efficiency over the entire software delivery process.

About the author
Moty Aharonovitz, senior director of Product Strategy at Borland Software, has more than 15 years experience in software development. At Borland, he is leveraging his experience and helping the company solidify its Software Delivery Optimization vision and supporting the Borland Lifecycle Quality Management Solution. He can be reached at moty.aharonovitz@borland.com.

¹James Martin, “An Information Systems Manifesto”
²Martin & Leffinwell
³Standish Group: Chaos Report
⁴Standish Group: Chaos Report
⁵Jacobs

• Company
  • Our Vision
  • Leadership Team
  • News
  • Events
  • Awards
  • Investor Relations
  • Careers
  • Newsletter
    • Archive
  • Contact Us