Analyzing cost and performance of physical security systems at Sandia National Labs

Sep 1, 1997 12:00 PM, LARRY ANDERSON

Cost and Performance Analysis is a prototype integration of existing PC-based computer tools that provides useful information for decision-makers on the cost and effectiveness of security systems.

Evaluating cost and performance presents a challenge for any security system manager, especially in a time of shrinking budgets, evolving security missions and changing threats. A project at Sandia National Laboratories, Albuquerque, N.M., addresses both cost and performance issues-and their complex relationships-using a computer. CPA-Cost and Performance Analysis-is a prototype integration of existing PC-based analysis tools. CPA is being developed at Sandia for use by the U.S. Department of Energy (DOE).

In the mid-1970s, DOE designated Sandia as the Lead Laboratory for Physical Security Research and Development to protect nuclear weapons and their components from theft or sabotage. As the Lead Laboratory, Sandia has developed expertise in security systems modeling, analysis, engineering, integration and implementation. Sandia also operates test facilities to help maintain a knowledge base in security technologies, equipment and components.

CPA incorporates two existing software tools. One is ACEIT (Automated Cost Estimating Integrated Tools) developed by Tecolote Research Inc. for the U.S. Air Force. ACEIT is widely used throughout the Department of Defense (DOD). The other is ASSESS (Analytic System and Software for Evaluating Safeguards and Security) developed jointly by Lawrence Livermore National Laboratories and Sandia National Laboratories for the DOE. ACEIT supports costs analysis over the full life-cycle of a system; ASSESS supports performance analysis. CPA organizes the cost and performance data generated by ACEIT and ASSESS into Excel workbooks. These workbooks make the data more accessible to analysts and organize the results for decision makers.

Access Control & Security Systems Integration interviewed Mary Jane Hicks, Ph.D., a senior member of the technical staff in the Security Systems Analysis and Development Department at Sandia, about CPA, a tool under development.

Q Why was this system needed?

A DOE wanted a quantitative system for evaluating security technology alternatives. If some new technology offered a lower false alarm rate, they wanted a way of understanding-quantitatively-what that new technology would cost and how it would affect system performance. So, Sandia was tasked by the DOE to develop a physical protection benchmark in terms of both cost and performance. The cost and performance data needed to be organized into a "table-top" tool that would show where costs are high and performance is low. The challenge was to organize the data to support informed decisions.

We've used a pyramid (see illustration at left) to show where CPA fits in the process of going from data to decisions. Two blocks of data form the base of the pyramid; cost data and performance data. Costs are illustrated by the left block of the pyramid base. The right block of the base indicates performance data from analysis tools used within the DOE community. Data collection was not the problem.

What was needed was a way of organizing, a way of structuring, cost and performance data that would both support analysis of alternatives and summarize results in a format useful to both the analyst and the decision-maker. To me, supporting analysis of alternatives means using a consistent approach. A well-designed architecture implemented in computer software can provide that consistency. We found several excellent examples of analysis of alternatives; but these were all for specific sets of choices. We were not able to find a general approach for analyzing cost and performance physical security systems. That's what CPA is designed to do: to organize/structure volumes of data in a way that analysts now have information from the data. Information, not data, is what managers need to make decisions.

Q Is the data in your pyramid real-world data?

A Yes, the data is real-world data. For costs we use Activity-Based Costing (ABC). ABC is an approach to cost estimation that's being taught in management schools and being used by industry. With ABC you follow the allocation of resources and labor (both direct and indirect) to the goods produced or services provided. Say you want to estimate how much it costs to have one guard posted at one access control point. You know the hours that a guard is posted. From your general ledger accounting system you can estimate the average loaded hourly rate of a guard. You can now calculate the cost of that activity-the posted guard. Or you may want greater detail. You may want to know how much it costs to have that posted guard check credentials. Now you need to know how long, on average, it takes the guard to check a credential; and how many credentials, on average, the guard checks per hour, per shift, per day-whatever interval of time is of interest to you. Now the activity that the activity of checking credentials.

OK. Now let's talk about performance data. We collect test data on components and subsystems, we develop performance models based on the test data, and then we validate the models with more test data. So when we use ASSESS for a specific security system, we are estimating performance of that system based on test data. So whether we are talking about "raw" data or performance models, it's all based on "real-world" data.

Q Can you tell us a little more about how you model system performance?

A We use an Adversary Sequence Diagram (ASD) to visualize a physical protection system. An ASD compresses the site plan of a physical protection system into a form that our computer program ASSESS can accept.

We think of security as concentric layers of protection with increasing security at each layer. We call this "protection in depth." As you can see in the illustration (facing page), the areas of increased protection are represented by the long bars. The target is represented by the long bar at the bottom of the illustration. The path elements, represented by the small boxes between the bars provide the layers of security.

Path elements function as either barriers (such as fences) or access control points (such as gates and portals). Each path element may have multiple safeguards. And each safeguard will have threat-specific effectiveness. For example, a credentials check and biometric device, such as a hand geometry unit, will have little effect on a adversary attempting to drive a truck right through a gate, but these safeguards should deny access to an adversary with false or stolen credentials.

In our models, safeguards provide detection or delay. Our models assume that an adversary must be either denied access to the target or must be contained within the perimeter of the physical protection system. Denial or containment require timely response. However, in order to have timely response, the adversary must first be detected. After detection, the adversary must then be delayed long enough for a force to respond in time to either deny access or to prevent escape. The concepts of denial and containment differ from the forensics, or police, approach of capture away from the scene of the crime based on evidence found at the scene.

Q How does the CPA architecture work?

A The whole purpose of the architecture is to enable analysts to do three things: to quickly identify performance and cost issues; to systematically identify how to address those issues; and to communicate their findings clearly to decision-makers.

It's easier to talk about the architecture if we refer to the diagram (on page 63). We start with (1) ASSESS, our performance analysis tool. We use (2) EXTRACT to read system definition and performance results from ASSESS. Results from ASSESS are post-processed in (3) PERFORM. System definitions from ASSESS are used to launch (4) CATSS-Cost Analysis Tool for Security Systems. Through CATSS and PERFORM, metrics of cost and performance for each path element are aligned (9 and 6). We use the structure already defined by the ASD to align costs with performance.

Although the cost analysis tool ACEIT does not appear in this illustration, it is the computational engine behind CATSS. All the cost and economic analysis capabilities of ACEIT are available through CATSS.

Q How do you measure performance?

A At the system level, we talk about risk. Risk is a function of three things: the probability of attack; the probability that the security system fails given that it is attacked; and the consequences of an attack. When we talk about adversaries or threats, we talk about threats with different skill sets and tactics. Threats can range from a group of school boys to highly trained terrorists. Threats may use force, stealth or deceit to get through a security system. We call this set of threat capabilities and tactics the threat spectrum. So, at the system level, we measure risk across the threat spectrum (5 in illustration). Each bar shows the risk for a specific threat. This is the information the decision-maker needs. These are the data that show what the risk issues are-but not how to address them.

It's the analyst's job is to identify how to address risk issues. The analyst first looks at where threats are getting through the security system (according to the ASSESS model). Then the analyst looks at how the threats are getting through the system.

ASSESS defines a critical path as a series of path elements most easily defeated by a specific threat. CPA assists the analyst by providing a bar chart (6) that shows how often each path element is in the critical path across the threat spectrum. This bar chart quickly shows the analyst where the threats are getting though the system and thus where better security is needed. Finally, the analyst examines tabular data (7) to identify how the threats are defeating the elements in the critical path and what type of additional security is required to reduce risk-detection or delay.

Q What about the cost side?

A We've already talked about Activity-Based Costing, so let's talk about how we put it all together.

The cornerstone of CATSS, the cost analysis tool, is a Summary Costs Spreadsheet (8). System elements are listed down the first column. They are grouped by path elements, infrastructure, and assessment. We've already talked about path elements. Infrastructure is all those facilities and activities (excluding assessments) associated with security that cannot be directly assigned to a path element. Some examples of infrastructure are: badge offices where badges and other access control credentials are issued and associated records are maintained; security system designers and management; and equipment and facilities provided for the guard force and for the security maintenance personnel. Assessments are externally imposed evaluations of the security system, much like FDIC audits of banks.

Cost types are listed across the first row of the spreadsheet. When we consider the cost of a system or subsystem we consider the cost over the full life-cycle; that is, the cost to install, operate, maintain and retire the system. Installation and retirement costs are usually one-time, or non-recurring costs. But operations and maintenance costs are recurring costs, paid each year. In fact, you can expect those costs to grow with time as the system ages and the labor force matures.

The cells of the spreadsheet are populated using the principles of Activity Based Costing. The cost data are now organized so that the costs of path elements can be aligned with our metrics of path-element performance (9). These data show how costs are distributed across the system and how path element costs relate to path element performance. These are the data that the analyst would want to work with.

Finally, system or subsystem costs can be represented graphically (10). Pie charts show relative costs; line charts show cumulative costs over time. These are the costs data, along with the system risk data, that a decision-maker should see.

Q Some businesses are having the same problem making the con-nection between benefit analysis and cost analysis-filling the gap your system fills. How applicable would your system be in a business environment?

A On the cost analysis side, we've applied Activity-Based Costing, an approach already used in industry. So we're learning from business in that arena. On the performance analysis side, we use a performance-based approach as opposed to a compliance, or check-list-based approach to physical security. A compliance approach to security asks if you have fences, if there are sensors on the fence, if you have access control, etc. A performance-based approach asks how effectively do the sensors at the fence detect intruders; is there reliable notification of sensor alarms to some central alarm station; is the response to alarms timely; and how effective is the access control system at preventing unauthorized access attempted by deceit or by force, etc.

Security is like insurance; no one likes to pay for it, but everyone wants to know where it is when something happens. Just as businesses do not want to be over-insured, they also do not want to be over-protected or over-analyzed. Businesses with high-value items to protect or high consequence of loss may want to examine the rigor of their approach to physical security. In fact, all businesses should take the time to identify potential targets (those things required to do business) and the consequences of loss of those targets (the time and cost that would be required to recover from a loss). If your computer goes down, you get it fixed or buy a new one. If all your business records were on the computer and you didn't have a backup, you now have a high-consequence loss!

Q What kinds of cost questions might this system answer?

A Let me answer that by going back to our initial task, which was to develop a physical protection benchmark in terms of both cost and performance as a basis for evaluating technology alternatives. Because this tool would be used to evaluate technology alternatives, we needed to design in a lot of flexibility. But let's start with the most straightforward: What am I paying and how well is it doing the job? How do the costs of my various path elements relate to the performance of those path elements? Am I paying a lot for a path element that performs poorly? I might be a manager with a proposal on my desk for a new intrusion detection sensor, but the analysis shows that I don't need better intrusion detection, I need better access control. But if I go back and check the costs of maintaining my current intrusion detection system, I might find that I have an aging system with escalating maintenance costs. I can use this tool to evaluate alternatives to my aging intrusion det e my need for better access control.

Cost and configuration of the direct and indirect labor that support security are a user-defined input to CATSS. Managers can use this tool to identify the distribution of labor to support security activities at path elements to evaluate the relative costs of alternative labor configurations.

About CPA

CPA-Cost and Performance Analysis-was defined, designed and prototyped by Sandia National Laboratories and Tecolote Research Inc. (under contract to Sandia) to support automated definition of a physical protection benchmark. The illustration on page 63 shows the CPA architecture. It uses ASSESS (Analytic System and Software for Evaluating Safeguards and Security), an existing Department of Energy (DOE) performance analysis tool for physical security systems; and ACEIT (Automated Cost Estimating Integrated Tools), an existing Department of Defense (DoD) life-cycle cost analysis tool. CPA integrates these tools and offers results in familiar Excel workbooks. PERFORM is the Excel application that accepts performance results from ASSESS through EXTRACT. CATSS (Cost Analysis Tool for Security Systems) is in an Excel application that accepts system definitions from ASSESS through EXTRACT. ACEIT is the computational engine behind CATSS. CATSS and PERFORM organize cost and performance data and offer a predefined set of tabular and graphical data analysis; and because they are Excel applications, the user also has access to the full graphical and functional capabilities offered by Excel.

Want to use this article? Click here for options!
© 2009 Penton Media Inc.

This month in Access Control

• Targeting The Customer
• Electronic Pedigrees
• One Hero Among Many
• Who? What? When? Where? Why?
• More from September's issue