Trust, But Verify, That Systems Will Do the Job

May 1, 2004 12:00 PM, By Don Sturgis

Technology is always changing. Its rapid pace continuously prompts the security industry to evolve — often leaving little or no time to test new security products. Yet with the extent and complexity of today's software-based and networked systems, testing installations is more important than ever before.

The role of testing

End-users should answer the question of what they need when it comes to a specific product. The technology may be full of different features, but only the functions that will be helpful to the user on a regular basis are truly important. In this case, the relevant questions become:

• Does the system provide the specific features that are needed?

• Are they set up correctly?

• Do they operate correctly and as expected?

The majority of users, and even some security system vendors, do not know the full role of testing and do not realize its importance in security system projects. This article — first in a series covering security system testing — will inform end-users how to get what they want and how to ensure that systems operate the way the vendors say they should. This series will explain the role of testing, show its many significant benefits, and outline a baseline of best practices for physical security system testing — thus empowering both vendors and customers to achieve better security system projects.

System testing

System testing is a primary ingredient of on-schedule and in-budget security system projects. The larger the project, the more critical the testing role is — but testing is important in any project.

Electronic security systems are an integration of multiple systems from one or more manufacturers. Each of the systems — alarm monitoring, access control, CCTV, etc. — is an integration of various components. Critical to the systems are the network connections over which the various parts of the systems communicate, and system installation and design are also important. Unfortunately, fixed deadlines often lead to installations being attempted before full testing has been completed, and sometimes before the full design itself has been completed.

Testing is also important because the exact combination of components and systems, set up and used in the way a user needs, may not have been installed before by the vendor providing the system.

Scenario-based testing

To ensure that a system will support an organization's Emergency Response Plans and Security Response Plans and Procedures, scenarios are developed that describe the ways in which the system would be used under normal operations, as well as various emergency and security conditions. These scenarios are incorporated into the test plans.

For this series of articles we define testing as “a process to ascertain that a product or system performs as expected.” So how does a user know what is expected? Product brochures, data sheets and specification sheets are readily available from each vendor. This literature is usually part of the proposal supplied during the bidding process. A vendor's proposal should also provide a description of how the various components or subsystems are connected to each other and how they operate as an integrated system.

There are several approaches to establishing among bidders what a user expects from his/her system. They include:

• a functional requirements document;

• an architect's and engineer's specification (“A&E spec”) provided by a manufacturer (naturally written around that manufacturer's product);

• a hardware specification or equipment list that tells what manufacturers' products to use; and

• a walk-through of the facility with the bidding vendors to indicate where card readers, monitor switches, motion detectors and CCTV cameras should be installed.

While some small systems can be described using the walk-through approach, most systems also need a written functional requirements document, which describes what the user expects from the system by way of performance. The vendor can use that information to develop a complete specification and/or an equipment list along with descriptive design information that explains how the key features of the system will function compared to the user's requirements.

There can be only one basis for testing: the written information that the vendor and his customer provide to the project.

The more detailed information the user provides, the better chance the system will perform as needed.

When examining the total of the design information provided, consider this question: If the system satisfies every documented requirement, will the system be completely acceptable?

Do other considerations apply and which are not included in the written information? If so, they should be committed to writing and included in the system design documentation. The design is complete if the user can confidently answer “yes” to the previous question.

Baseline shortcut — Test someone else's system

One way to lighten the design documentation requirement is to establish a baseline for a system, identify the most important aspects of that baseline, and then describe needed features that are different from or in addition to the baseline. This can be achieved by finding a system that performs exactly or closely to what is needed, and then “testing” that system against expectations by having the system demonstrated by its owner. Then it's possible to tell a vendor, “I want my system to be just like that one except for the following things …” The test plan would include the most important aspects of the baseline, and the needed things that are different from the baseline system.

Testing and documentation

To perform testing requires documentation. The vendor's team needs documentation to know what to install and how to set it up. It should be no surprise to learn that the systems with the most problems (i.e. customer dissatisfaction) are generally the ones with the weakest test plans and the poorest documentation.

One of the values of test planning is that it requires in-depth consideration of what the system is expected to do — but full value is only realized if that exercise is performed before a system is installed. This is where many projects go wrong: The test planning is performed at the end of the project instead of early on.

A good test plan will require a complete set of “as built” drawings when the final acceptance test is performed, since they will be used during the testing process. Being involved in the testing process as a user will also provide a more complete understanding of the operational characteristics of the system and of how to make changes and additions as the facility expands.

What is involved in test planning?

Testing for end-user systems is different from product development testing that manufacturers perform. User testing is acceptance testing. At various stages of the project, tests are performed to ensure that the project is ready to move forward to the next phase, until the final acceptance test is complete, at which time the system installation is considered complete (with the exception of some punch-list work items that may remain).

Test planning specifies what will be tested, when it will be tested, what type of tests will be used, the exact test steps to be taken (test cases or test scripts) and who will perform the tests. Test planning documentation includes:

• project background information;

• project and performance assumptions;

• test milestones;

• test terminology;

• list of tests to be performed;

• reference information (requirements, specifications, product literature, etc.);

• resources (equipment, time, participants and observers);

• schedule and timeline;

• test setup requirements;

• test conditions;

• test start, stop and restart criteria;

• test scripts or test cases (with pass/fail criteria); and

• test tracking and reporting.

Each test script should reference the specific requirements it satisfies. It is common to satisfy more than one requirement in a single test script. Each test requires one or more customer personnel to witness the test and sign off on successful test execution. Someone on the customer team should be designated as the test manager, who assigns and schedules customer personnel for test participation, tracks and reports on the test results, and who collects the completed and signed test forms to compile with the test reports into a test records book at the completion of testing.

Test Strategies

There are two basic acceptance test strategies:

• vendor-directed

• user-directed (also called customer-directed)

A vendor-directed test is performed by the vendor, with some level of participation by the customer. A user-directed test is performed entirely by customer personnel, who follow detailed test scripts that specify what steps to perform and what the outcome should be for each step. A user-directed test may be preceded by pre-test training, done to familiarize the customer with the system sufficiently to enable performance of the test. The two approaches can be combined — where early tests are vendor-directed and later tests are user-directed. The smaller and simpler the system is, the easier it is to perform user-directed tests — but that should not be the only criteria. Even complex systems can be successfully tested by users who are generally familiar with the type of system being tested.

One benefit of user-directed testing is that the testing process can provide a bridge for users between how they expect the system to operate and how it actually operates with regard to the most important features. It's much better to have that happen early, in a controlled environment, than on the job during a security incident or emergency.

Specific recommendations for test strategies will be given in future articles in this series that deal with each type of acceptance test.

Acceptance test phases

Three phases of testing are recommended for a large physical security system: Factory Acceptance Test, Site Acceptance Test and Operational Acceptance Test.

Factory acceptance test

A Factory Acceptance Test (FAT), is often no longer conducted at the “factory,” but at the system integrator's or customer's facility. For this test, all of the major system components are assembled in one room or area of a building even though they will be dispersed over many sites when the system is finally installed. The purpose of this test is to prove to the customer that the major features of the systems function properly when connected over a simulated network like the one to be installed. It will prove correct redundant server takeover operation and demonstrate correct backup and restore operations, if these functions are part of the system.

Vendors often try to omit the Factory Acceptance Test. They consider it costly and time-consuming and, since they feel they know their product well, they also often consider it unnecessary. However, vendors who have actually participated in a properly planned and executed FAT understand the many benefits for them as well as the customer, and will be eager to include it in the project planning.

The FAT offers the user the only opportunity to witness the major operational issues of the system at a single location before any equipment is installed at customer facilities. As systems become more and more complex, the FAT becomes increasingly important. Any significant interconnection problems and integration problems should be discovered during the FAT and resolved.

A factory test may be performed in more than one part. For example, a factory acceptance test on new network technology may be performed separately (and in a different location) from a test of the security system components.

Site acceptance test

The Site Acceptance Test (SAT) is also called a Field Acceptance Test, because it is performed in the “field” (i.e., at the customer's facility). There is one Site Acceptance Test for each physical location at which system components are being installed. It is not uncommon for site acceptance tests to be performed on an ad-hoc basis, without prior planning. This approach often leads to end-user problems and dissatisfaction that can be troublesome to rectify. Proper test setup and user preparation and participation are important factors that, when done correctly, lead to smooth turnover of the system and a high satisfaction rating from the users.

Operational acceptance test

The Operational Acceptance Test (OAT), which is sometimes called a Field Reliability Test, is usually conducted as a 30-day test, the purpose of which is to ensure that the system can operate reliably for an extended period of time. Sometimes test exercises are included in the test if normal operations are not expected to sufficiently exercise the system.

A factory test may be performed in two or more parts. A site acceptance test is required for each site or facility connected to the system. An operational acceptance test spans 30 days or more, and can involve periodic inspections and reports. Thus, in spite of their being named “tests,” it is more accurate to think of each of them as a test phase.

Ongoing system operational testing

Ongoing System Operational Testing (OSOT), also called Ongoing Maintenance Testing, is performed periodically throughout the life of the security system. This periodic testing should incorporate testing the people who operate the system, in support of security response objectives. The frequency and extent of Ongoing System Operational Testing vary significantly from one facility to another, and are highly dependent upon the application of the system. For example, airports test their duress alarm systems once or twice daily, often as part of the work shift changeover.

Types of Tests

Any of the test phases described above will incorporate many different types of testing, such as:

• Functionality Testing
• Security Scenario Testing
• Performance Testing
• Stress Testing
• Load And Capacity Testing
• Fault Tolerance Testing
• End-to-End System Testing

All of this may sound like a lot of work, and in fact, it is. But the payoff is in reduced project costs, shorter project execution, and proof to the user that he is getting what he pays for. Properly planned and executed testing makes a big contribution to the peace of mind of the project managers on both the vendor side and the customer side. It is also a best practice element of security project risk management.

FOR THE RECORD

About the author

Don Sturgis, CPP, is a senior security consultant for Ray Bernard Consulting Services (RBCS), a firm that provides high-security consulting services for public and private facilities. This article is excerpted material from the upcoming book The Handbook of Physical Security System Testing by Ray Bernard and Don Sturgis, scheduled for publication by Auerbach Publications in the spring of 2005. For more information about Don Sturgis and RBCS go to www.go-rbcs.com or call 949-831-6788.

COMING NEXT MONTH: How Factory Acceptance Testing ensures that all the aspects of systems will actually work correctly when the components are connected.

Want to use this article? Click here for options!
© 2008 Penton Media Inc.

This month in Access Control

• Opening Up About Door Closers
• An Enterprise Approach
• The Framework For Open Systems
• On A Higher Plane
• More from April's issue