Curiosity and upper management may frequently demand that safety managers create reports from safety data that compare their operations with their competitors or with a national, regional, or global "average."
Displaying hazard reporting data over time increases management's understanding of the real performance of airline and airports' safety processes.
Safety managers at all but the smallest aviation service providers are required to monitor the performance of their aviation SMS using data accumulated from the SMS documentation processes and detect trends.
Tracking trends using KPIs are naturally extended to the airline or airport's established goals and objectives that monitor the continuous improvement of the SMS.
This post describes an approach to compare your airline or airport's KPIs with those from your civil aviation authority, a national average, or against your competitors. These steps will include:
Key performance indicators are commonly discussed in the news, civil aviation regulatory reports, or aviation SMS trade journals. The media may not explicitly call them KPIs, but may use language similar to: "From 2010 to 2015, there were 2.5 bird strikes per 10,000 operations."
In this case, "Bird Strike" is the KPI to focus on.
In the scenario above, there were 2.5 events in 10,000 operations; therefore the rate of interest is 2.5 per 10,000. In most cases, your airline or airport will never see 10,000 operations. Your organization may have several hundred or tens of thousands of operations, but probably never the exact number under consideration.
The safety manager's challenge is to compare 2.5 events in 10,000 operations with the number of events at his organization. For the sake of this discussion, let's call the "10,000" our "normalization rate."
In order to compare "apples to apples," safety managers will need to determine how many similar events their airline or airport had within the established time period. Aviation safety management databases should easily provide this KPI data, in this case, the number of bird strikes from 2010 to 2015. Safety managers will also need to know how many operations their organization performed each year from 2010 to 2015.
Trying to accumulate this data from spreadsheets will be too burdensome. You may make this comparison, but it will take considerable time and effort to aggregate the data. An SMS database is a proper technology if you are routinely working with statistical data across multiple disparate data sets.
What remains is for the safety manager to adjust the values from these different scales to come to some comparison. In short, he must normalize his company data with the 2.5 bird strikes in 10,000 operations.
Normalizing rates multiply your KPI classification frequency by your normalization rate divided by an annual number of operations (departure/landings or hours).
What remains is to chart your data. This can be performed in MS Excel or your aviation SMS software may already provide a way to monitor KPI trends. The better aviation SMS software solutions have this ability.
Your KPI data may be skewed. Having a high number of occurrences in a given time period or a lack of data may cause your aviation KPI trends to appear suspicious. In this case, you should understand:
Standard Deviation -- a measure of how to spread out numbers is - the square root of the Variance.
Variance -- the average of the squared differences from the Mean.
I'll leave it to you to look up your college statistics professor.
To accurately compare your KPIs against industry norms, you must:
Remember, "garbage in is garbage out". When your manager asks you to compare your airline or airport's performance with others, remember this easy formula described above. You may wish to bookmark this post or even share it with others.
Staying on top of classifying reported safety issues is a BIG DEAL! When safety managers neglect to classify reported safety issues, they are wasting an opportunity to accurately identify trends and effectively monitor the health of the aviation SMS.
Reported safety issues may be classified in multiple ways. In a spreadsheet, there are some severe limitations, but an SMS database will normally allow you to classify reported safety issues in countless ways.
A quick rule of thumb is that if your organization has more than 50 employees, you need an aviation SMS database. If you are smaller and are under EASA jurisdiction, an SMS database is required to store reported safety data. Spreadsheets are no longer permitted.
When you are normalizing data to monitor KPI trends, you will need proper tools to have a sustainable SMS performance monitoring process. Otherwise, the process will not be sustainable and management cannot be assured of the SMS requirement of safety performance monitoring and measurement are being met.
If you lack the tools to adequately monitor the performance of your SMS, we can help.
Last updated August 2024.