Technology Evaluation process of the EU PRoTECT project

Within the EU PRoTECT project, a large part of the activities involved setting up an evaluation process and evaluating potential solutions for mitigating the earlier identified vulnerabilities for public spaces that are soft targets for terrorist attacks. This evaluation process involved setting up a Request for Information process, deciding on an evaluation method (in this case a Multi-Criteria Analysis), deciding on evaluation criteria and performing evaluations on the results of the RfI with a team of experts and demonstrating potentially fruitful solutions for the public spaces that were focused on in the PRoTECT project. This evaluation process is an example and one way of setting up such an evaluation process and has been developed into a Technology Evaluation Framework.

In the context of PRoTECT, the evaluation is organised by a municipality, whereby the degree in which solutions meet certain criteria, set by the municipality, are determined. In the context of PRoTECT, there are two types of evaluation: an evaluation of information given by providers and an evaluation of the demonstration of a solution. A third type of evaluation, based on executing operational scenarios in tabletop sessions, was defined in PRoTECT but not executed.

Request for Information

A Request for Information (RfI) is a process whereby solution information is solicited from providers based on a list of solution requirements. In the context of PRoTECT, an RfI announcement was made public by a municipality, requesting providers to offer information on a solution which will mitigate a PSOI security vulnerability (a result of the EU Vulnerability Assessment).

Multi-criteria analysis

An MCA is a method to compare optional solutions based on selected criteria. This method was used for PRoTECT as each city had different needs and different public spaces. However, some general requirements were considered for limiting the solutions to a specific type of technology, the operational circumstances conserving the solutions and conditions for the responding solution providers. This step needs to result in setting up the requirements needed to decide on criteria and to set up the RfI.

The following general solution requirements could be considered:

• Required functionality (e.g. must produce a certain output)
• Required interfacing, interoperability with other solutions, or standards
• Required type of user
• Required level of organisational and operational impact
• Required level of maintenance
• Required compliancy to (national) ethical and legal codes
• Etc.

The following general solution provider-related requirements could be considered:

• Minimum provider experience, service organisation, etc
• Minimum security clearance, confidentiality requirements, need for a non-disclosure agreement, etc
• Latest delivery date
• Requirements concerning carrying out a demonstration and costs involved
• Required maximum solution costs (e.g. total cost of ownership)

Evaluation criteria

The team of experts will decide on the criteria for assessing the suitability of a solution in addressing the vulnerability and describe on the basis of which solution information the degree to which the criteria are met will be determined. The selection of criteria depends on the characteristics of the PSOI and of the vulnerability and the requirements for solution participation. The assessment will be based on solution information offered by the providers. The criteria will be communicated to the providers during the solution information gathering process and used in the solution evaluation process. It is fruitful to set up a brainstorm with a team of experts and discuss the following aspects:

Quality:

• Check that the most important criteria are included
• The criteria should not be redundant (even partial overlap can influence the evaluation score)
• Scores assigned to a solution under one criterion may not be affected by the scores assigned under another criterion
• The criteria must be suitable for evaluating all solutions (i.e. not specific to only some of the solutions)
• The criteria must have the same meaning for each of the solutions
• The criteria must be clear and if possible measurable
• The criteria must be suitable for distinguishing between good and bad solutions

Applicability:

• The criteria must be relevant to the vulnerability scenario determined in Step 1
• The criteria must be in line with the objectives noted in Step 2
• The criteria must be in line with any minimum requirements set Step 2

General list of criteria

The following non-exhaustive list of criteria potentially applicable for the security solutions foreseen in the context of the EU PRoTECT project was used:

• Total cost of acquisition (TCA)
• Total cost of ownership (TCO)
• Physical (e.g. the team might consider it important that the presence of solution in the PSOI is not too obvious):
  • Volume
  • Weight
  • Dimensions (e.g. height, length, etc.)
  • Robustness (vandal resistance)
  • Visual prominence
• Compliance (e.g. the team might consider it important that the solution is compliant with privacy legislation):
  • Privacy (e.g. EU GDPR legislation)
  • Security (e.g. antisabotage)
  • Safety (e.g. specific national legislation)
  • Ethics (e.g. municipality policy)
• Performance (e.g. the team might consider it important that the solution functions reliably):
  • Reliability (e.g. a high detection rate, small chance of malfunction, redundant)
  • Accuracy (e.g. able to count people accurately)
  • Timeliness (e.g. processing time)
  • Capacity (e.g. processing throughput)
• Environmental (e.g. the team might consider it important that implementing the solution will have no negative impact on the environment):
  • Protection (e.g. against rain, dust, wind, etc.)
  • Emission (e.g. CO₂)
• Operability (e.g. the team might consider it important that the solution is highly user friendly):
  • Usability (e.g. can be operated easily)
  • Staffing (e.g. number of operators required)
  • Deployment (e.g. time necessary to deploy the solution)
• Interoperability (e.g. the team might consider it important that the solution can deliver information which can be used by another system protecting the PSOI):
  • Information (e.g. required by another system or party)
  • Interfacing (e.g. a specific IT-communications protocol)
• Maintainability (e.g. the team might consider it important that the solution can easily be repaired if damaged):
  • Availability (e.g. mean time to repair)
  • Support (e.g. 24-hour service)
• Training (e.g. the team might consider it important that the operators can be trained quickly):
  • Education (e.g. level, duration, course availability, location, etc)
  • Experience (e.g. duration)
• Maturity (e.g. the team might consider it important that the solution is to some degree field proven):
  • Technology Readiness Level (e.g. high TRL)
  • Availability (e.g. development time, time to production)
  • Roadmap (e.g. expected short-term improvements)
• Other (e.g. the team might decide to also consider some general aspects of the solutions):
  • Miscellaneous advantages
  • Miscellaneous disadvantages

The abovementioned list is not complete and is primarily intended as an aid to identify criteria.

Examples of general evaluation criteria for the purpose of PRoTECT

Within the RfI process and based on the general list of criteria, the EU PRoTECT project evaluated solutions based on the following criteria.

	Please Select One Point Rating						Notes	Weighting factor	Score
CRITERIA GROUP- Technology							Total 80%
Accuracy	0	1	2	3	4	5		10%
Compliancy	0	1	2	3	4	5		10%
Reliability	0	1	2	3	4	5		10%
Security	0	1	2	3	4	5		10%
Ease of use	0	1	2	3	4	5		10%
Maturity	0	1	2	3	4	5		10%
Portability	0	1	2	3	4	5		10%
Maintainability	0	1	2	3	4	5		10%
CRITERIA GROUP- Additional information							Total 20%
Additional benefits	0	1	2	3	4	5		10%
Sufficiently detailed and accurate information	0	1	2	3	4	5		10%
Total Solution Score

Evaluating solutions

Paper exercise with team of experts

To evaluate the results of the RfI and thus, identifying potential relevant solutions for protecting public spaces against terrorist attacks, an initial paper exercise was organized. This paper exercise, is a type of exercise amongst stakeholders (seated around a table) to determine the solutions' performance based on the information given by the solution providers.

Demonstrations on-site

A demonstration is an event, organized by a municipality, and usually involving a solution's provider, whereby specific functions of the solution are proven in a relevant operational environment, using scenarios. In the case of PRoTECT, each municipality's PSOI was used and multiple solution providers demonstrated their solutions on-site.

Next steps

This EU project has set out to provide municipalities and their stakeholders the knowledge and skills to protect their public spaces. The actual implementation of measures lies beyond the scope of this project, but is in essence one of the goals to mitigate vulnerabilities and protect public spaces.

Relevant documents

The PRoTECT Technology Evaluation Framework can be downloaded here.