How to Conduct Safety Performance Assessments in Aviation SMS With Part 5

Written by Tyler Britton | Dec 11, 2019 10:47:00 AM

Purpose of Safety Performance Assessment in Safety Assurance Pillar

If you cannot measure safety performance in your aviation safety management systems (SMS), then:

How do you know whether your operations are as safe as can be reasonably expected?
How can you set safety goals and objectives when you don't have actionable intelligence to assist in fact-based decision-making processes?
How do you know your aviation SMS is driving operational performance improvements?

Safety performance assessments to comply with FAA's Part 5 SMS requirements play a critical role in bridging the gap between:

higher-level risk management processes; and
day-to-day aviation SMS activities.

More practically speaking, this component of safety assurance helps bridge the gap between:

everyday hazard mitigation efforts; and
large scale decision making and accountability for the overall direction of the safety management system.

Obtaining Assurance/Evidence Your Aviation SMS Is Functioning

By the time you get to safety performance assessment activities in your SMS implementation, you have been:

Gathering safety data (safety reports, audit findings, and other risk management inputs);
Processing safety events through your risk management "machine;"
Analyzing and categorizing safety data as safety events are processed;
Monitoring safety performance (either formally using defined processes or informally by keeping an eye on important safety metrics);
Measuring and communicating safety performance to regulators and senior management; and
Setting future goals and objectives with the aim of demonstrating continuous improvement.

As the safety team compiles and communicates the results of the safety analysis, senior management's role involves making decisions about what to do with that data, based on:

Organizational goals and objectives (not necessarily safety goals);
Regulatory compliance;
Operational vs safety performance;
Effectiveness of risk controls to ensure alignment with organizational goals;
Changes in operational environment; and
Newly identified hazards.

All of these are part of the decision-making process, which arises from the analysis of safety data.

Safety Performance Assessment Is Where You Make Safety Decisions

Making safety decisions involves asking the following questions:

Is the safety performance of SMS implementation adequate?
Is the safety performance meeting defined safety goals and objectives?
Is the safety performance consistent with regulatory and contractual compliance expectations?
Does safety performance align with the expectations of existing risk controls and documentation?

Answers to these questions will largely dictate what happens next. There are two general types of outcomes from the safety performance assessment:

overall satisfaction or
overall dissatisfaction.

If answers to these questions indicate dissatisfaction:

“…the certificate holder must use the safety risk management process described in [the safety risk management process]” as described in 14 CFR Part 5.73(b); and
The safety concern will also move on to the Continuous Improvement SA element, where safety risk controls will be developed to correct substandard performance

If the answers indicate satisfaction:

It indicates that all aspects of safety performance for the safety element in question are adequate; and
The element in question will cycle back through the SA process to be monitored, measured, and more data acquired and analyzed to ensure ongoing performance.

It’s very important to be clear about what your outcomes are, as outcomes dictate which decision you make.

Outcomes of Safety Performance Assessment

After reviewing the questions listed in the previous section, there are four distinct outcomes, each with a distinct decision that should be made:

Performance is acceptable and meets all-important expectations, whereby performance will continue to be monitored;
Performance is unacceptable, and the problem seems to lay in compliance/safety policy and/or risk controls, whereby the SA’s Continuous Improvement element will be triggered to develop corrective actions;
Compliance and safety policy are acceptable, but performance remains unacceptable, whereby the SRM process should be triggered to review how the system addresses the safety concern; and
New or uncontrolled hazards are discovered, whereby the SRM process will be triggered to document the hazard and/or needed risk control and assess how the system addresses it.

These outcomes should be documented, whether it’s on a spreadsheet, paper, or in aviation safety software.

How to Conduct Safety Performance Assessment

Part 5 indicates that conducting safety performance assessments involves using acquired data and analysis to assess:

Compliance with your safety risk controls – 5.73(a)(1);
SMS performance – 5.73(a)(2) – though this is a very vague requirement;
Effectiveness of safety risk controls, including the identification of any ineffective controls – 5.73(a)(3);
Changes in the operational environment that may induce new hazards – 5.73(a)(4); and
Any newly detected hazards – 5.73(a)(5);

Point numbers 2 and 4 above need some clarification, as they are quite vague.

What does it mean to evaluate the performance of your SMS? The important implication here is the performance of your SMS in terms of your goals and objectives. A significantly better way that Part 5.73(1)(2) can be phrased is to indicate that you should assess whether or not performance meets the expectations of your goals and objectives.

Secondly, “changes to the operational environment that may induce hazards” seems to be a long-winded way of describing hazard mechanisms. Hazardous mechanisms are one of the following:

Hazardous sources (i.e., a mountain, machine, chemical, etc.);
Initiating mechanisms (Human factors, safety culture, environmental changes, etc.); and
Targets of potentially dangerous condition, such as an aircraft, person(s), etc.

Undocumented changes to any of these areas should be considered potentially unsatisfactory for safety performance. Evidence of these undocumented changes indicates that your organization needs to either:

Implement processes to manage change projects;
Acquire/develop data management strategies to easily manage change;
Review and/or revise your list of triggering conditions that prompt formal "change management;" or
Enforce defined processes to document change related to "Management of Change" projects;

Final Thought: Indications of Safety Performance Assessment Compliance

Now, on to the main point of this article that you probably care about most – how do you know if your safety performance assessment is compliant with Part 5.73 requirements?

Here are some good indications that you will perform well on audits that deal with the SA Safety Performance Assessment process:

Safety goals and objectives have been established for which to evaluate SMS performance;
Key safety performance indicators have been defined that align with defined safety goals and objectives;
When reviewing safety issues, you can clearly identify outcomes;
Risk controls are regularly reviewed and verified for effectiveness;
There is a strong documentation trail that indicates assessment outcomes, and how you proceeded further (i.e., a note or built-in process indicating the next steps taken);
Specific evaluations exist for compliance, risk control analysis, identified hazards, and goals and objectives considerations; and
Which data was used to perform these assessments?

The keyword here is documentation. When you conduct safety performance activities, SHOW YOUR WORK!

Safety Performance Assessment Dependent on Data Management Strategy

Safety performance assessments are reliant upon the mountains of SMS documentation stored by the aviation SMS' data management tools. This data includes inputs from:

Safety reporting system;
Reactive risk management and investigation procedures;
Hazard identification and risk assessment processes;
Safety performance indicators;
Safety training and communication;
Continuous improvement analysis;
Safety audits and subsequent findings management;
Management of change projects; and
Emergency response planning.

If you are a very small company with fewer than 100 employees, you may not have enough data to be statistically relevant. Furthermore, smaller aviation service providers are not expected to have advanced SMS data management capabilities; however, this does not mean they are excluded from participating in safety performance assessments.

How your aviation SMS stores and retrieves safety data will determine your organization's ability to demonstrate continuous improvement and participate in predictive risk management activities. From over a dozen years of SMS database management, I believe that companies with more than 40 employees should be using an aviation SMS database to store and retrieve safety information.

Aviation SMS Database Facilitates Safety Performance Assessments

Several low-cost, commercially available SMS databases are definitely worth the expense when one considers what they can offer your organization. I recommend acquiring a complete SMS database solution to ensure all your required SMS documentation requirements are stored in one centralized location.

Besides offering your organization the means to acquire and store safety data, an SMS database provides aviation service providers with industry-tested workflows that satisfy the risk management requirements of

Civil aviation authorities;
Standards setting bodies (IATA, IS-BAO, Flight Safety Foundation, etc.); and
Clients that require SMS implementation from their vendors (oil companies, hospitals, government agencies).

Not only does an SMS database facilitate safety performance assessments, but it also becomes a real-time saver whenever auditors come to review your SMS implementation. I have heard of operators reducing their auditing time from several days to less than four hours when the operators use the SMS Pro database. Furthermore, the SMS Pro database has all the required elements that SMS auditors are looking for.

Are you using an SMS database? If you have more than 40 employees, this should become a top priority for your SMS implementation. Do you have fewer than 40 employees, but experience high employee turnover? This is another indicator that a smaller organization should adopt a low-cost, commercially available SMS database.

Below are some short demo videos that will show how your organization can benefit from an SMS database. Safety performance assessments using an SMS database will certainly become more valuable to your company than if you were using the alternatives: paper and spreadsheets.

Last updated in July 2024.

View full post