Why Do We Measure Aviation Safety Performance?

Aviation safety managers are tasked with a Safety Assurance (SA) element labeled "Continuous Improvement." Not many aviation safety management systems (SMS) training courses go into much detail about "Continuous Improvement" which leaves many safety managers confusedly scratching their heads.

Safety Assurance is the third ICAO component or pillar of an ICAO-compliant SMS framework. Annex 19 - Safety Management and the four components are described in detail in ICAO's Document 9859.

Related Articles on Aviation SMS Four Pillars

• What Are the 4 Pillars of SMS?
• History of Aviation SMS and Four Pillars - with Free Tools
• Which of the Four Pillars of SMS Carries the Most Weight?

The effectiveness of any implemented SMS cannot be determined whenever it cannot be measured. Aviation SMS' safety measurement process must be able to:

• Define;
• Develop; and
• Implement processes to determine and measure key safety performance indicators.

In every business, two limits must be considered when measuring SMS performance:

• Financial resources to devote to safety management; and
• Production capabilities that can realistically support operations, shareholder profits, and SMS budgets.

Safety managers tasked with measuring SMS performance will also be required to consider:

• Organizational goals, including the level of commitment to the SMS; and
• Available resources.

Safety managers who neglect to consider these two important factors will not be able to realistically develop SMS performance monitoring processes that complement organizational goals using budgeted resources.

Safety Performance Monitoring and Measurement Element

"Safety assurance activities focus on assessing the health of the organization, with an emphasis on safety. Specific goals for improvements in all areas should be set for all senior operational managers..."

ICAO Annex 6, 3.3.5; Appendix 7, 3.1

Safety measurement is an essential component of every SMS. How else can one set SMART goals and objectives? For review, SMART goals are:

• Specific;
• Measurable;
• Attainable;
• Realistic; and
• Timely.

If we are not measuring performance, there will never be any concrete proof that our objectives were met. Abstract or ambiguous goals should be reworked until they meet the SMART requirements. SMS regulatory auditors start to look for SMS activity that is measuring and monitoring the SMS starting from Phase 2 and Phase 3 of SMS implementation.

Knowing which phase you are in your SMS implementation is important for managing expectations. Safety managers may have just inherited an SMS implementation. Others are under pressure to address SMS implementation deficiencies. In both cases, safety managers need to take an objective assessment of the SMS. If you don't know where you are, then you are less certain of which strategies to incorporate to achieve your safety goals and objectives.

To determine which phase you are in, three tactics come to mind:

• Redo the gap analysis process;
• Review an SMS implementation plan; or
• Take an SMS assessment.

Related Articles on Aviation SMS Goals and Objectives

• What Are Safety Objectives in Aviation SMS - with Examples
• How to Create Safety Objectives in Aviation SMS with Examples
• How to Shape Your Corporate Vision with Aviation Safety Goals and Objectives

Understanding What to Measure with Gap Analysis

A gap analysis is the process of assessing the state of an aviation SMS using an industry-accepted model. These models are commonly structured around the four pillars, meaning that the most common section heading you'll find in a gap analysis checklist is named after ICAO's four SMS components or pillars:

• Safety Policy;
• Safety Risk Management (SRM);
• Safety Assurance (SA); and
• Safety Promotion.

I have to warn you. There are really good gap analysis models and substandard models. The best advice I can give you is to stay away from the IS-BAO gap analysis checklist. It shocked me as a safety professional that it was even considered to have value.

We refused for many years to add the model to SMS Pro's gap analysis tool because I believed the content was too elementary to be useful for anyone. We relented about five years ago when a client insisted he needed it to evaluate their SMS against the prescribed model of the same standards body that was about to audit their operations.

The only benefit I see from the IS-BAO gap analysis model is that it was designed to "not scare" anybody, or overwhelm operators with a large laundry list of required activities and processes.

If you look at the IS-BAO gap analysis model, you may even laugh. In the early days of SMS implementation, there was considerable resistance in the general aviation (GA) community toward SMS. IS-BAO attempted to seize an opportunity to provide SMS guidance to biz-jet operators who needed proof of an SMS to fly to foreign countries, namely France and the Caribbean. This was the beginning. Now biz-jet operators are expected to have a demonstrable SMS in many more parts of the world.

When IS-BAO provided their gap analysis model to their followers, they were right. It didn't scare anyone away. One look at the IS-BAO checklist and a GA operator may reasonably think: "Wow, this doesn't seem too bad. I can do this."

Maybe this was IS-BAO's intention to increase the voluntary adoption of SMS. I don't know the full story. If anyone else does, please leave a comment or send me an email. However bad the IS-BAO gap analysis model is, IS-BAO really made up for it on their SMS implementation plan. It is first class!

I may be too hard on the IS-BAO gap analysis model because I'm a firm believer that the gap analysis is a great tool besides using it to define the gaps in the SMS. A gap analysis model is an excellent educational resource to train management and new safety managers on what is required in a fully implemented SMS.

By reviewing a gap analysis model, safety managers can conceptualize which SMS documentation requirements and processes will be required to develop a fully ICAO-compliant SMS. Furthermore, you also learn, at a high level, which aviation SMS activities support measuring and monitoring, such as:

• safety reporting metrics,
• SMS training effectiveness,
• responsiveness to accomplish risk management activities, etc.

If you are going to select one Gap Analysis model, I'd choose the ICAO gap analysis checklists. We have other gap analysis models that you can find from the links above.

Related Articles on Aviation SMS Gap Analysis

• What Is a Gap Analysis in Aviation SMS?
• SMS First Steps - Gap Analysis
• 4 Best Aviation SMS Gap Analysis Strategies for SMS Implementations

Align Measurement Strategy with Accountable Executive Expectations

Safety managers inheriting a functioning SMS implementation are in a great position to recharge an aviation SMS. Resistant safety cultures retard sincere SMS implementation, but a "change of face" with a new safety manager can revitalize or redirect a failing SMS. New "energy" can improve safety cultures, especially when the safety manager gets off on the right foot with both the accountable executive and employees.

The safety manager's expectations must align with the accountable executive before spending too much time and energy measuring SMS performance. When the accountable executive only wants a "paper SMS" to check the box, there is no need for robust, detailed SMS data management strategies. In this case, you only want to do the bare minimum, which can be accomplished with spreadsheets.

For aviation service providers who cannot operate with a "paper SMS," you will need SMS data management tools to:

• collect data;
• store data;
• set goals and objectives;
• measure collected data to support goals and objectives; and
• present the data in usable charts to facilitate decision-making.

Aggregated safety reports, dashboard charts, and trending charts help demonstrate dutiful safety performance monitoring and measurement compliance.

Related Aviation SMS Performance Monitoring Articles

• How to Monitor Aviation SMS Performance - Safety Chart
• How to Monitor Aviation SMS' Safety Culture Performance
• How to Set and Monitor Key Performance Indicators (KPIs) in Existing SMS

SMS Implementation Plan Helps Determine Measurement Activities

Both the gap analysis and the SMS implementation plan serve as "insight" into what SMS data management strategy you will need to demonstrate compliance for measuring safety performance. If you only need a paper SMS, then spreadsheets will work. Otherwise, save yourself time, energy, and money by acquiring an SMS database as soon as possible. Spreadsheets are simply the wrong technology for a sustainable process to measure and monitor SMS performance.

If your existing SMS database does not allow you to easily monitor and track required SMS documentation, then I advise that you find one that addresses all SMS requirements in a single platform. Do this as quickly as possible. Just rip off the band-aid. I've seen hundreds of SMS implementations and scores of failed SMS implementations. There are common threads about why aviation SMS fails.

Lack of top management support is the number one reason for SMS to fail. If you cannot get top management support for tools to measure and monitor SMS performance, you need to remind the accountable executive that:

Every accountable executive is responsible for properly implementing the SMS and making sure the SMS performs as designed in all areas of the organization. In order to determine whether the SMS is properly implemented and functioning, the accountable executive must regularly review organizational safety performance.

In order to monitor SMS performance, you need to measure SMS activity. Furthermore, you will need to set goals and objectives and be able to easily track your defined safety goals and objectives. When this process is not easily repeatable, it will not be performed for a prolonged period. An aviation SMS is not a short-term project, but one that will stretch out for many years.

Considering the breadth and scope of the SMS implementation, an SMS database is required in most cases. Otherwise, there is too much risk of things slipping through the cracks when safety managers leave the company. Furthermore, without tools to adequately measure SMS activity, it becomes impossible to effectively monitor SMS performance and direct actions necessary to address substandard safety performance.

The second reason SMS fails is the lack of data management tools to satisfy the mountains of SMS documentation. The SMS implementation plan helps safety managers understand which data point may be available to measure and monitor SMS performance. Within each SMS implementation plan, look for hints from keywords as to which data may be available to support measures such as:

• reactive risk management (safety reports);
• proactive and predictive risk management (safety reports and historical data);
• training and communication;
• safety assurance (audits and submitted safety reports)
• safety performance indicators (SPIs); and
• safety goals and objectives.

Larger operators may already have a quality management system that collects considerable amounts of data. Quality management systems (QMS) and SMS are very similar in that they share many of the same elements, especially in the Safety Policy and Safety Promotion components. Many tools in the QMS can be re-purposed to collect and store SMS data.

SMS implementation plans are checklists developed either by:

• ICAO;
• Regional civil aviation authorities (CAA); or
• Recognized standards bodies, such as IS-BAO.

When selecting an "SMS implementation checklist," consider who will be auditing your SMS. If you are serving two masters, such as the FAA and IATA, I always recommend using the ICAO SMS implementation plan. All CAAs and standards bodies model SMS requirements after ICAO. After all, ICAO is the driving force behind the SMS initiatives.

Related Aviation SMS Implementation Articles

• Why Should We Implement Aviation SMS?
• How to Implement SMS in Small Organizations
• SMS First Steps - How to Create an SMS Implementation Plan

SMS Assessments Indicate True SMS Phase

In order to reach a goal, you need to know two key elements:

1. Where am I currently at in the SMS implementation (honest assessment); and
2. Where do I need to go?

The first question is the easiest to answer. By taking an honest assessment of the aviation SMS, you can develop a strategy to get to the end game. What is the end game? We can see a glimpse of what the end game is by reviewing the gap analysis and SMS implementation checklists. What will the end game look like for your organization? This will depend on several factors, including:

• Accountable executive's priorities;
• Top management support;
• Available budget to manage the SMS;
• Proficiency of regulatory SMS oversight;
• Considerations of additional standards bodies' programs to satisfy, such as IOSA, ISAGO, BARS, IS-BAH.
• Timeline.

In project management, one of the first things we learn is that three elements affect the successful completion of a project:

• Time (how soon does it need to be completed?);
• Budget (more budget, you can complete the project faster); and
• Features (fewer features means less budget and time to implement).

Understanding these basic elements of project management will help safety managers plan more effectively for both new SMS implementations, and for planning the revitalization of the SMS. Each of these three elements is related to the others. For example, if you only need a paper SMS, you will not need significant amounts of time or budget.

Understanding where your SMS implementation sits today is critical for planning future moves. In addition, when you know the maturity level of the SMS implementation, you can manage the expectations of:

• upper level managers; and
• auditors.

Furthermore, understanding where you need to go will affect the three project management elements we just discussed, namely time, budget, and features.

SMS assessments are useful for identifying where you are in the aviation SMS implementation process. While the gap analysis checklists and SMS implementation checklists are useful for guiding safety managers on "where to go," the SMS assessment advises safety managers "Where are we now?"

Below are four useful assessments to determine your SMS maturity level. If you take these assessments, be honest with yourself. Only you are seeing these assessment results and these assessments are for your guidance only.

• Phase 1 SMS Assessment
• Phase 2 SMS Assessment
• Phase 3 SMS Assessment
• Phase 4 SMS Assessment

Related Aviation SMS Assessment Articles

• The Real Way to Assess the Effectiveness of Your Aviation SMS
• Is Your Aviation SMS Implementation a Farce? - with Self-Assessments
• Performing Vendor Safety Assessments for Aviation SMS

Setting Goals and Objectives to Measure SMS Performance

During the first year or two of the SMS implementation, you may not have developed adequate data acquisition and data management strategies to manage all the SMS documentation requirements. You may still be in spreadsheets or diapers, if you want to take an analogy of an SMS maturity being equal to a baby, toddler, juvenile, and adult stages.

Other companies are using point solutions that are disconnected from other SMS data sources, such as a stand-alone

• safety reporting system;
• risk management system; or
• auditing system.

Goals and objectives are very important to understand especially when you are considering strategies to monitor SMS performance. When you have goals and objectives to start with, finding the necessary data elements to support the measurement and tracking of objectives becomes much simpler.

If you are using spreadsheets, you will know which spreadsheets to sift through to support measuring progress on your safety objectives. Likewise, when safety teams use point solutions, they will have to visit each "system" to aggregate and prepare reports for management to determine whether safety objectives are met.

If you don't have any goals set, consider upper management's organizational goals before tackling safety goals and objectives. This will save the overzealous safety manager from becoming disillusioned when upper management's business strategy does not align with safety goals and objectives.

Safety goals and objectives need to have data points behind them in order to facilitate timely and accurate measurement. Otherwise, there is little objective evidence to support SMS performance monitoring activities. Examples of goals in an SMS include:

• Identify and eliminate hazardous conditions within our aviation-related processes and operations;
• Perform hazard and risk assessment for all proposed new equipment acquisitions, facilities, operations, and procedures;
• Provide relevant SMS training/ education to all personnel;
• Minimize accidents/incidents that are attributable to organizational factors;
• Prevent damage and injury to property and people resulting from our operations;
• Improve the effectiveness of the safety management system through a yearly safety audit that reviews all aspects of the SMS.

By analyzing each of these goals, visualize where in your organization you can find data to support tracking these goals. Will you be using:

• safety reporting system?
• auditing system?
• spreadsheets with training documentation?

Every company is a bit different. When aviation service providers use SMS databases to manage SMS documentation requirements, all SMS data resides in one location. This is a best practice and facilitates SMS data measurement and monitoring. Reports are automatically generated, thus freeing up management to focus on data analysis instead of data collection, cleaning, and preparation.

Related Aviation SMS Database Articles

• What Is an Aviation Safety Database
• 5 Most Important Things to Know before Buying Aviation SMS Database
• Pros and Cons of In-House SMS Database and Off-the-Shelf-Solutions (COTS)

Key Performance Indicators Measure and Monitor SMS Performance

Up to now, we have identified our data sources. We've also set our goals and objectives for which to measure and demonstrate continuous improvement of the SMS. If you have been following along and reviewing the SMS implementation plan, you may have noticed "safety performance indicators" popping up in Phase 4 of the SMS implementation.

Safety performance indicators are KPIs (Key performance indicators). They are basically the same animal, but SPIs are a subset of KPIs. KPIs commonly focus on organizational performance tracking, while SPIs focus on safety. If you have an integrated safety and quality management system, then you will probably be using the KPI term almost exclusively to prevent confusion among management personnel who are accustomed to hearing about KPIs.

SPI = KPI

KPIs support your safety goals and objectives. Do you remember when we set our goals and objectives in the previous step? We identified the data sources needed to support the measuring process to determine whether we are accomplishing our safety objectives.

During your risk management activities, you will be working in several disparate systems, such as:

• safety reporting;
• SMS training;
• Safety communications;
• auditing;
• safety risk analysis; and
• hazard register.

There are many ways to measure data and generate reports. KPIs are the data points that help you either directly or indirectly monitor safety objectives. If you are interested in measuring safety performance in SMS, this is where you want to be: key performance indicators.

So a workflow to measure safety performance in aviation can be:

• Understand SMS requirements (very important);
• Set or review safety goals and objectives;
• Determine a plan to achieve safety goals and objectives;
• Set KPIs that allow you to monitor progress toward achieving safety objectives;
• Continue to collect data;
• Monitor KPIs to confirm whether the organization is on track;
• Repeat.

Related Aviation SMS KPI Articles

• 3 Best Tips Selecting Aviation SMS Key Performance Indicators - Free KPI Resources
• 6 Sources for Aviation Key Performance Indicators
• How to Automatically Monitor KPIs in Aviation SMS

Leading Indicators Are the Most Crucial Key Performance Indicators

The most important key performance indicators (KPIs) are leading indicators, also known as predictive indicators. Predictive indicators are not always accurate but exist as the most realistic "future vision" for safety planning and risk mitigation strategies.

Leading factors are measurable factors of safety performance that change before underlying safety-related events begin to manifest themselves. Leading indicators tend to follow particular trends. Since these safety-related statistics precede a string of accidents, incidents, and irregularities, they are used to forecast changes in operating budgets, including

• Safety promotion activities;
• Training; and
• Risk mitigation efforts.

One should logically assume that the accountable executive, upper-level management, and safety managers participate in establishing:

• Key safety performance indicators; and
• Goals and objectives to reach desired levels of safety performance.

Related Articles on Aviation SMS Leading Indicators

• What Are Leading Indicators in Aviation SMS
• How to Use Leading Indicators in Aviation SMS
• 8 Aviation Leading Indicators That Successful SMS Measure

Resources to Help Measure Aviation Safety Performance

There are many resources and software tools to help airlines and airports monitor safety performance. Below are some educational articles regarding key performance indicators, also known as safety performance indicators.

• Choosing Key Performance Indicators - Airlines and Airports
• 7 Quick Tips for a Key Performance Indicator (KPI) Strategy
• Compare Aviation SMS Key Performance Indicators - KPI Trends
• Tips for Using Aviation Key Performance Indicator (KPI) Statistics
• 3 Tips Setting Goals for Aviation Key Performance Indicators
• 4 Steps to Automate Key Performance Indicator KPI Monitoring

Final Thoughts on Measuring Aviation Safety Performance

If your company has not yet defined any KPIs, you undoubtedly don't have a mature SMS program. When you set KPIs, please follow the best practices outlined above in the offered resources.

The last words: if you don't measure it, you don't know where you are going and you don't know where you have been.

If you need aviation SMS database tools to help you measure and monitor SMS performance, we can help. These short demo videos illustrate how an SMS database can save your safety and quality assurance team considerable effort.

Live SMS Pro Demo

After watching the demo videos, you may have questions. Does it look like we are a good fit for you? We've been working with aviation service providers around the world since 2007. SMS auditors love SMS Pro because it is a complete SMS data management suite with all SMS data in one easy-to-access, secure system.

Last updated November 2023.