How to Use Key Performance Indicator Statistics?

Aviation safety management systems (SMS) revolve around the four ICAO pillars:

• Safety Policy;
• Risk Management;
• Safety Assurance; and
• Safety Promotion.

KPI statistics can be effectively used in each of these four pillars to monitor SMS performance.

In this post, we will explore some common and perhaps overlooked ways to make the most effective use of KPIs in your organization.

Related Articles on Aviation SMS Four Pillars

• What Are the 4 Pillars of SMS?
• Who Started Four Pillars of Safety Management in Aviation?
• 4 Pillars | What Is Safety Promotion Component (the "Overlooked" Pillar)

Aviation SMS KPIs are used by safety teams and the accountable executive to determine:

• SMS is properly implemented;
• SMS can be demonstrated as "functional" in all areas of the organization;
• Safety performance monitoring is actually happening (big point);
• Key important SMS data points are being measured;
• Employees are participating in the SMS;
• SMS has been accepted (safety culture measurement and monitoring);
• Employees are properly trained in a timely manner;
• Whether safety promotion efforts affect safety reporting culture;
• How effective regulatory agencies view your SMS; and
• Whether risk mitigation strategies are working as designed.

We could probably fill this page with ideas as to how KPIs benefit aviation SMS implementations. Your imagination is the limiting factor here. The point is that KPIs can be applied across all four pillars and not focus only on the "sexy" parts of the SMS, which are namely:

• Safety Risk Management; and
• Safety Assurance.

Effectively Use KPI Statistics With ICAO Four Pillars

Each of the four ICAO four pillars or components possesses opportunities to effectively use KPI data to monitor and correct SMS performance.

Aviation safety managers spend considerable time collecting data that is required to demonstrate a fully functioning, performant SMS. This data comes from a wide variety of sources, including:

• Hazard register;
• Safety reports (number and severity);
• Proactive activities vs reactive activities;
• Risk management activities (i.e., the responsiveness of the safety team);
• Management of change activities;
• Internal and external audits;
• SMS training records;
• Safety committee meeting minutes;
• Safety communication metrics (how often sent to how many?)
• Metadata from monitoring risk controls;
• Accident and serious incident investigations; and
• Safety goals and objectives.

You can see from this quick list above the data points you are probably capturing today. You may have more, or you may have less depending on whether you:

• Are using spreadsheets to track SMS data;
• Are using an SMS database (and where it came from);
• Have a mature SMS or a relatively new SMS; or
• You have a paper SMS (check the box).

If you do have data, it only makes sense that safety managers can now leverage this carefully collected and classified data to maximize the potential use in their SMS. If you are just getting started, or you don't have the capability to capture and store SMS data, don't be too concerned yet. Here are some resources to help you get started on your SMS implementation. Just be aware that this article applies to you and will give you a taste of what to expect in the future as SMS regulatory auditors come to visit.

Related Articles on Key Performance Indicators (KPIs) in Aviation SMS

• What Is a Key Performance Indicator (KPI) in Aviation SMS?
• How to Set and Monitor Key Performance Indicators (KPIs) in Existing SMS
• How to Automate Key Performance Indicator KPI Monitoring

Some uses for using key performance indicator data include:

Using KPIs With Safety Policy

Safety Goals & Objectives

Setting goals and objectives is a requirement in every aviation SMS. KPI statistics are necessary to understand where your company has been for the past year(s). KPIs should be able to track the performance of your safety goals and objectives.

Carefully selected key performance indicators are essential to be able to monitor goals and objectives. You have to be able to easily track these KPIs as safety reports and audit data become available. You may have other safety goals and objectives that rely on data coming from safety surveys or interviews to help measure and monitor safety culture.

A centralized SMS database helps monitor safety goals and objectives more effectively and efficiently than a group of "point solutions." Point solutions are software products that do one thing, such as:

• Safety Reporting System;
• Training Management System;
• Survey tools; and
• Auditing System.

Point solutions require more work to monitor safety goals and objectives, but they are better than spreadsheets.

Data gathered for goals and objectives can originate from a multitude of different systems, even systems outside the control of the safety department. This point is driven home so the safety team thinks outside the box when they are looking for ways to take measurements or gather data to measure a particular objective.

The process of setting safety goals and objectives can be challenging for safety managers that have poor safety cultures or they don't know what sort of goals and objectives are reasonable in a new SMS implementation. More than once I have heard honest feedback from safety managers reporting that they use our default safety goals and objectives for their first year. It is a no-brainer if you have some inspiration. There is no rule against using predefined objectives that come from a template.

For those who are paying attention, I'll post these default goals here. They have helped many other safety managers and I'm sure this quick list will help you. But there is another reason I'm posting this following list. It is for you to be inspired as to what sort of data points will be required when you want to monitor your safety goals and objectives.

To help with the following data analysis, examples of safety goals are listed below. We'll analyze data sources by their respective numbers:

1. Identify and eliminate hazardous conditions within our aviation-related processes and operations;
2. Perform hazard and risk assessment for all proposed new equipment acquisitions, facilities, operations, and procedures;
3. Promulgate an ongoing systematic hazard and risk assessment plan;
4. Provide relevant SMS training/ education to all personnel;
5. Provide a safe, healthy work environment for all personnel;
6. Minimize accidents/incidents that are attributable to organizational factors;
7. Prevent damage and injury to property and people resulting from our operations;
8. Improve the effectiveness of the safety management system through yearly safety audit that reviews all aspects of the SMS

Related Articles on Safety Goals and Objectives in Aviation SMS

• What Are Safety Objectives in Aviation SMS - With Examples
• How to Create Safety Objectives in Aviation SMS With Examples
• How to Shape Your Corporate Vision With Aviation Safety Goals and Objectives

KPI Data Sources for Safety Goals and Objectives

Based on the above goals, you can see you will need multiple data sources. I'll list these data sources with their associated number from the above list:

1. Hazard Register;
2. Management of Change data; and Hazard Register;
3. Hazard Register; Safety Reporting System; and Risk Management System (or Investigation data);
4. SMS Training System; (or spreadsheet for smaller, less sophisticated SMS);
5. Safety Reporting System;
6. Safety Reporting System;
7. Safety Reporting System; Risk Management System (or Investigation data);
8. Auditing System; (or spreadsheet for smaller, less sophisticated SMS);

We can see a few major data stores that we must account for when we set our goals and objectives. This is a very important point I'm making. Here is a scenario:

Imagine you are in your annual or semi-annual safety review meeting. You are directing the safety goals and objectives discussion, and a Department Head asks for a particular safety goal or objective.
The second thing you should say after: "That is a great idea," is "Where will the data come from?"

Each objective that is tied directly to a goal will require at least one data point to monitor. These data points are the KPIs to verify the accomplishment of each safety goal and objective.

Related Articles on Setting Aviation KPIs

• Choosing Key Performance Indicators in Aviation SMS
• 3 Best Tips Selecting Aviation SMS Key Performance Indicators
• 3 Goal-Setting Tips Using Aviation Key Performance Indicators - With Free KPI Resources

Risk Management

Hazard Register

A hazard register offers employees and managers a glimpse into the most actively monitored hazards affecting the organization at that given time. KPI statistics should be used to determine which hazards belong at the top of the Hazard Register, and those hazards that pose minimal risk.

Not every hazard should factor into your KPI selection. Only choose those that are significant and meaningful to the success of your airline or airport operations.

A challenge of new and immature SMS implementations is that the safety manager doesn't have a hazard register. Furthermore, there is no sustainable way to monitor hazards effectively unless you are

• using an integrated SMS database or
• copying and pasting from your safety reporting system into your risk management system.

The first approach is preferred, as copying and pasting reported safety issues from one system to another is a sloppy hack. Yes, if you are using "point solutions," then you may not have much choice. But the truth is that these point solutions are not long-term options. It becomes more difficult to monitor as your company grows and the safety culture matures.

If you are managing your hazard register and safety reporting system in a spreadsheet, you are in even worse shape to track KPIs unless you are very small, such as fewer than 50 employees at the most.

KPIs from the hazard register are the number one data source auditors look for.

Trend Analysis and Trend Identification

Trending aviation SMS data can be surprisingly daunting for safety managers who lack adequate tools. Safety managers of both small and large operations tell us that they cannot manage the data manually.

Manually managing SMS data for tracking and analyzing trends is nearly impossible when you are simply the "part-time" safety manager. Part-time safety managers are the "norm" at most smaller organizations. Carefully selected KPIs allow safety managers to focus their trend analysis efforts on a significantly smaller subset of data.

Data collected for trend analysis comes normally from a database. You can do it manually with spreadsheets, but this system is:

• time consuming;
• prone to inaccurate data collection; and
• painful.

There are very few safety managers that I know that like collating data and generating trend charts. The easiest way is to acquire a low-cost, commercially available SMS database that does it all for you. With the SMS database, any manager with the proper security access can easily generate trending charts and perform their own analysis.

Trending charts are a safety manager's best friend and there is MORE! Trending charts can be the safety manager's best tool to "win friends and influence operational department heads!" The SMS database can capture considerably more than safety reports. The SMS database can easily capture:

• quality and customer complaint data,
• lost and found data;
• environmental data; and
• security data.

There is no sense in spending extra budget on other systems when the SMS can capture and monitor KPI data related to other operational areas. This is how your SMS database and SMS trending charts can help safety managers acquire "safety champions." When safety teams show operational department heads how their SMS data can help them make better business decisions, then you are on your way to reducing resistance to the SMS in that department.

Trending data tells a visual story that is hard to beat.

Related SMS Trend Analysis Articles

• Aviation Safety Managers' Best Friend - Trending Charts
• How to Use Trending Charts in Aviation SMS
• Six Steps How to Perform Trend Analysis in Aviation SMS

Safety Assurance

Performance Monitoring

Aviation managers at all levels need to stay abreast of the major issues affecting the performance of the aviation SMS. These managers typically acquire this information from canned reports, dashboards, and automated trend-monitoring tools.

Dashboards could become easily overwhelmed with too much information. KPI statistics, therefore, are the perfect data points to incorporate into any well-developed dashboard.

Along the same line, automated trend monitoring tools should focus on meaningful and potentially costly elements affecting operations. When we consider the safety goals and objectives, we realize that many data sources are needed to present a full picture to management in order for them to be responsive to substandard safety performance.

What types of reports do we need to present to the accountable executive so that his responsibilities are fulfilled? Is it just safety report metrics? Or audit performance? Before we become too hasty and answer this question, what are the accountable executive's responsibilities to the SMS?

For every aviation SMS, accountable executives must ensure the SMS is properly implemented and performing across all areas of the organization, not just flight ops, or ground handling. Operations that could logically affect:

• Flight operations;
• Maintenance activities;
• Manufacturing and design;
• Flight training; or
• Navigation and marshaling of aircraft.

We can see from both this list and the requirement to ensure the SMS is properly implemented AND performing that we need to be able to monitor all SMS documentation requirements to ensure the SMS is performing as designed. Furthermore, the accountable executive is required to review organizational safety performance on a regular basis. Whenever substandard safety performance is identified, such as through reviews and audits, the accountable executive is expected to lead the team that fixes these problems.

Related Articles on Aviation SMS Performance Monitoring

• My Safety Score - What's Yours? Aviation SMS Performance Monitoring
• How to Be Compliant With ICAO Safety Performance Monitoring and Measurement
• 4 Pillars | How to Conduct Safety Performance Monitoring and Measurement

Monitoring SMS Performance Using Dashboards

Safety managers are at a distinct advantage when creating SMS performance monitoring dashboards. They don't usually understand the data requirements to create a particular chart, or they may not know how to access the underlying data to populate dashboard charts.

The easiest way is to acquire an SMS database that already has SMS performance monitoring dashboard charts. If you are building your own charts, you may be interested in knowing what sorts of charts to put into your dashboards. If you are an SMS database competitor, you are also interested in knowing what sorts of dashboard charts are being delivered to aviation service providers.

This list below (SMS KPI Performance Monitoring Charts) is meant to inspire you to:

• either create your own SMS performance monitoring charts; or
• ensure your existing database has similar dashboard charts that are available in real time.

List of SMS KPI Performance Monitoring Charts

Based on your organizational needs at any given time, these charts allow the accountable executive and safety teams to continuously monitor the health of the SMS. These charts can be tied to KPIs, but since KPIs are always evolving, a chart that is valuable one year may lose its value the succeeding year as organizational safety goals and objectives evolve.

1. Number of reported safety issues compared to previous month/year
2. Overdue safety tasks by days overdue, such as (< 7 days; 1-2 weeks; 2-3 weeks; etc.);
3. Average number of days since hazard identification training before reporting first safety issue;
4. Types of reported safety issues (maintenance, flight ops, general occupational safety, etc.);
5. Number of reported safety issues based on employee role or position;
6. Number of issues by location (base, region, country, etc.);
7. Percentage of reported safety issues by management level;
8. Number of "in progress" issues by department or division;
9. Top 5 (or 10) hazards associated with reported safety issues;
10. Top 5 root causes from investigations;
11. Top 5 contributing factors from investigations;
12. Percentage of reported issues by risk for the monitored time period;
13. Number of expiring/expired employee training and qualifications;
14. Top 5 (or 10) procedures associated with reported safety issues;
15. Number of "in progress" reported safety issues associated with identified hazards grouped by severity;
16. Number of overdue safety tasks (corrective and preventive actions and reported safety issues) grouped by severity;
17. Average number of days to close reported safety issues by severity (acceptable, acceptable with mitigation, intolerable);
18. KPI exceedances (advanced KPI monitoring required);
19. Number of users not trained in SMS;
20. On-time safety performance last 30 days
21. Number of reported safety issues and status for past report period;
22. SMS implementation progress by element or pillar;
23. Percentage of employees accessing safety database within last period;
24. Top safety reporters (for safety promotion);
25. Latest newsletters

This list above is a sample of charts that safety teams can build for the accountable executive and the safety manager to continuously monitor safety performance. Depending on your role in the aviation SMS, you may require different charts. The best solution is to have a user-configurable performance monitoring dashboard. This will allow each manager to focus on charts that are most relevant to the manager's duties and responsibilities.

Predictive Analysis

KPI statistics hold a significant role in the predictive analysis functions of an aviation SMS. To be valuable in predicting the likelihood of certain types of events occurring, you must have meaningful and well-classified KPI data.

Predictive analysis is based on predicting future types of events based on historical data. You will be collecting and analyzing data to detect trends. Above we discussed trend analysis. We won't repeat it here, as there are additional resources listed above in the "Trend Analysis and Trend Identification" section.

Related Articles on Aviation SMS Performance Monitoring

• 10 Great Reports to Monitor Safety Performance in SMS - With Examples
• How to Monitor Aviation SMS' Safety Culture Performance
• How to Monitor Aviation SMS Performance

Safety Promotion

Finally, KPI statistics are meaningless if they are not used to communicate risk to organizational stakeholders.

For safety promotion activities, consider using KPI statistics:

• In newsletters;
• At safety meetings to communicate progress;
• In safety articles; and
• Sprinkle them into your safety surveys to educate.

Final Word on Effectively Using KPI Statistics to Monitor SMS

As you have seen, one can use KPI statistics in every component of an ICAO-compliant SMS. We see a tragedy every day when safety managers spend so much time each day collecting and classifying data, but they fail to capitalize on the benefits.

Hopefully, this post will prove useful for you to make more effective use of your KPIs. KPIs help managers focus on important elements of the SMS. As you have seen, most KPIs are best tracked using an SMS database. To comply with the intent of aviation SMS, one needs a database to spot trends unless you are an incredibly small company. A small company does not have enough safety data to easily spot a trend in time.

How big should a company be before it should invest in an SMS database? If you have more than 50 employees, you need an SMS database.

Related Aviation SMS Database Articles

• What Is an Aviation Safety Database
• How to Choose the Best Aviation Safety Database Software
• How to Manage Aviation Safety Programs Without Complex SMS Databases

For smaller companies, you may get by with spreadsheets, but why suffer the pain? There are low-cost, commercially available SMS databases that take the pain away for even these smaller companies.

When you consider that the accountable executive has to monitor the entire SMS to ensure it is performing properly, then an SMS database is the best way to mitigate risk to the accountable executive.

For a few hundred dollars per month, these small companies can get a safety reporting system with all the risk management benefits, including SMS performance monitoring dashboards.

Why continue to struggle with point solutions and spreadsheets? The technology for a complete SMS data management solution has been available for over ten years. The technology is mature. Why wait?

We can help.

If you need help to monitor SMS performance, check out these videos. We have been developing SMS databases since 2007 and serve all industry segments.

Live Demo of SMS Performance Monitoring Database Solution

Ready for a live demo of SMS Pro?

Last updated November 2024.