Monitoring safety performance in aviation safety management systems (SMS) is undoubtedly one of the most important tasks you will undertake in your SMS. Regulatory requirements turn to monitoring the performance of the SMS from a "nice-to-have program" for reducing risk and avoiding accidents to a "must have and must demonstrate proof" to comply with SMS regulatory requirements.
Safety performance monitoring is integral for maintaining Safety Assurance and the integrity of your SMS program. This is "pie in the sky," telling you the purpose and benefit of monitoring SMS performance, but the real reason safety managers and accountable executives must pay attention is because it is required. You may get away with "satisficing" SMS auditors for a few years, but by year four or five, the chickens will come home to roost.
The decisions you make regarding how you monitor safety performance can have drastic effects on safety performance, for better or worse. Do we need to mention SMS audits anymore?
The most important questions regarding your safety performance monitoring process are:
To answer these questions, you first need to define:
When it comes to core performance, performance monitoring is most concerned with:
You should have reports ready at hand so safety teams and the accountable executive can quickly assess different elements of SMS performance. Having real-time reports is a best practice and allows more responsive decision-making to:
Regarding monitoring the SMS, we have discussed:
What we haven't discussed is how and when or how often.
Monitoring a system requires charts, graphs, and tabular data. Data is required for safety teams to collate and format into user-friendly reports that add value to managers' decision-making processes. Where is your data stored?
Each of these styles of managing SMS data has its strengths and weaknesses. In very young, immature aviation SMS implementations, spreadsheets are the most common technology employed to store SMS data. As the SMS matures, point solutions may be cobbled together from other systems that exist within the organization, or purchased from software providers. These point solutions usually address a single pain point in the SMS, such as:
Aviation service providers that have more than 100 employees may already have some of these point solutions in place before beginning their SMS implementations. These companies become "wed" to these systems and are loathe to switch systems despite their shortcomings. Their reluctance to switch is related to the "sunk cost fallacy." Aviation service providers will continue using these disjointed, imperfect systems because they have historically spent considerable time, money, and effort on these systems. To abandon these systems is not seen as a strategic win to adapt to changing environmental pressures but as a management failure for selecting these short-sighted programs in the first place.
Examples of cobbled-together solutions are called "hacks" in the IT world. A solution may be created for one thing but re-purposed for another. SharePoint and email ticketing support systems are often converted into SMS database tools to manage:
Safety promotion may be handled by Survey Monkey and an electronic message board, such as can be put together by SharePoint. Spreadsheets are used considerably in this "SMS data management maturation phase" mainly for tracking SMS training requirements and the "Hazard Register."
After years four to six of SMS implementation, medium-sized aviation service providers will realize that spreadsheets and point solutions are not sustainable solutions. It may take two or three years of the same repeat audit findings before these companies realize that their spreadsheet and hybrid point solutions introduce more risk than they are worth. Furthermore, accountable executives possess limited ability to effectively provide oversight to monitor SMS performance with all these cobbled-together point solutions.
It is painful to listen to frustrated accountable executives when they feel powerless to monitor the performance of their SMS. Accountable executives' frustration stems from their lack of ability to control decision-making because the safety team may only show the accountable executive what the safety team believes the accountable executive needs to see. Or accountable managers may fear their safety managers are not doing their job and they have no way to prove it.
For the accountable executive to ensure the SMS is performing properly, the accountable executive needs access to timely, unbiased reports. Obviously, spreadsheets are not going to satisfy the accountable executive's need to monitor the SMS, except for the smallest companies. Point solutions are also unwieldy because there are many required elements to track in SMS documentation. These point solutions are only addressing a particular requirement (or set of requirements).
An executive monitoring dashboard is what will scratch the accountable executive's itch for monitoring SMS performance. What you call your monitoring dashboard really doesn't matter. Some call them:
Dashboards are effective in displaying monitoring data from multiple, disparate data sets. Dashboards in spreadsheets are a hack and not worthy of anyone's time except in a company with fewer than 40 employees. Medium to larger companies require dashboards to pull data from all their point solutions into a single dashboard.
Integrating point solutions into independent performance monitoring dashboards can be expensive and risky whenever any of the point solutions change. Creating dashboard charts and graphs may be difficult for IT staff who cannot easily access the data layer behind the point solutions.
To overcome the hodge-podge, cobbled-together point solutions, SMS database solutions started appearing in 2007 and 2008. These SMS databases were designed to manage all SMS requirements using a single, centralized data management strategy. Having all data in a secured repository also allows better data management capabilities, especially when it comes to presenting performance monitoring dashboards.
The best-of-class SMS databases may have scores of various dashboard charts for managers to select and save their views according to their needs. In these cases, each manager may have their own personalized dashboard that is being fed data in real-time, as soon as it is entered into the SMS database.
SMS databases are the final solution for operators who need to:
We have seen the limitations spreadsheets have for demonstrating SMS performance monitoring. Point solutions are a bit better, but still not the best way to go for long-term monitoring satisfaction. If you have more than 100 employees and are starting to implement an SMS, here is a word of advice:
You will save yourself a lot of grief and frustration. Yes, I admit I'm biased because I develop SMS database software, but I'm first a data management expert and I understand SMS requirements. This is free advice and will save you money and maybe your credibility as a safety manager.
Don't go to your account executive and tell him you are going to implement an SMS that will save him tons of money by using spreadsheets. This is both a disservice and shameful if you do because now you know better than to manage SMS data using spreadsheets. Ask your account executive how he plans on monitoring the performance of his SMS. He may ask you for suggestions. Don't tell him spreadsheets. Yeah, it will save him some money today, but later it will cost him more.
When the accountable executive asks you for recommendations, you should give him options to monitor the SMS:
In most cases, there will be no additional costs for developing dashboard monitoring charts with an SMS database because most low-cost, commercially developed SMS databases have very flexible and powerful performance monitoring dashboards. Some of these solutions are very complete and will replace many of your point solutions that were discussed earlier. Therefore, it is important when you present the accountable executive with options that you subtract costs associated with any point solutions that will be replaced by the SMS database.
Did I mention that your recommendation should be an SMS database from the start of your SMS implementation?
We have answered the who, what, where, and how of SMS performance monitoring, but we have not discussed when should monitoring be performed. Part of this answer relies upon selected data management technologies for your SMS documentation requirements. Spreadsheets are not real-time and are traditionally not shared across a collaborative risk management environment. When you will monitor SMS performance charts when your company is resigned to using spreadsheets?
Based on second-hand accounts from safety managers upgrading to SMS databases, these safety managers claim to only create executive-level SMS monitoring charts and graphs once per year. In one case, a helicopter operator with 300 employees at the time, this safety manager spent three weeks each year collating data and creating management reports.
Therefore, if you are using spreadsheets, I'm expecting you will be monitoring the SMS less frequently than with hybrid point solutions and SMS databases. A good assumption will be semi-annually at best but expect to do it at least once per year. Is this enough? Maybe not. At the bare minimum, I believe SMS performance monitoring activities should occur at least quarterly, and more often if you have access to real-time performance monitoring dashboards.
Safety managers may monitor SMS performance on a daily basis as they manage reported safety issues. The quality department will monitor SMS performance whenever their team performs audits/inspections on a recurring basis. Trend charts may be reviewed quarterly or at random intervals by safety committees.
Trend charts should be reviewed at least once per year regardless of your data management capabilities. Per the FAA,
The objective of Safety Management Systems (SMS) is to proactively manage safety, identify potential hazards, determine risk, and implement measures that mitigate the risk. The FAA envisions operators being able to use all of the components of SMS to enhance a carrier’s ability to identify safety issues and spot trends (emphasis added) before they result in a near-miss, incident, or accident. For this reason, the FAA is requiring carriers to develop and implement an SMS.
Source: Federal Register / Vol. 80, No. 5 / Thursday, January 8, 2015 / Rules and Regulations
Trends are a big deal in aviation SMS. Spotting trends and mitigating substandard performance when, and as soon as possible after trends are detected is a lofty ambition of SMS. But we don't care about the lofty ideals, do we? Maybe I just want to check the box with my "paper SMS."
How do I collect data, make sense of the data, and present trend data whenever a manager requests it from me? The answer is simple. Get an SMS database that does it for you. In the best world, managers will have their own accounts so they can investigate trends at their convenience.
Up until now, we've laid the groundwork for SMS performance monitoring. We know who, what, where, when, and how.
Let's now look at ten common charts that can be used to demonstrate SMS performance monitoring. Your SMS database may already have these, or you can use these reports as inspirations for your performance monitoring dashboards.
If you are a competitor with an SMS database, you may as well copy these reports, because other competitors have already been here before you. We recognize that being an industry leader, others are sure to copy what works and what makes the most sense.
As safety issues are reported and managed, you will classify each one with a relevant primary hazard. In a safety concern, a hazard:
Over time, you will have a well-developed hazard risk register, which is simply a term that describes a report for:
You need to keep tabs on your top hazards to assess what your organization's most pressing safety concerns and needs are.
This is a special report that many organizations are not using, but most definitely should be. This report lists:
In other words, this report shows you hazards that have been identified in your operational environment but have no documented risk controls assigned to. This might be because you simply have not yet documented these risk controls, or it may be because you have not implemented risk controls.
Either way, having uncontrolled hazards is just asking for an audit finding or repeat safety issue occurrence.
Key Performance Indicators (KPIs) are the metrics that correspond directly to your safety objectives. As such, KPI reports demonstrate safety performance that your organization has flagged as “most important.”
KPI reports can be any simple or complex reports that are dedicated exclusively to your KPIs.
High-risk issues are the most time-sensitive safety concerns in an aviation SMS. High-risk issues are identified after an initial risk assessment is performed on a reported safety concern.
Each organization's risk management processes should have default time periods that are set to close high-risk issues, such as within 3 days after the issue is reported.
The on-time issue management reports will show you the percentage of reported safety concerns that were closed on time, per policy. The best version of this report will allow you to compare the current time period for on-time closures vs. the previous time period.
Much like your top hazards report, an equally valuable report is to see a report for your top root causes. This report would simply show you a list of your top root cause classifications, and the number of times each root cause was classified.
This report will allow you to assess the most pressing organizational concerns in your company, and where to spend energy developing risk controls.
As your SMS matures and improves its documentation, control measures, and safety oversight, you will experience a decline in the number of high-risk issues that are reported.
Why? Several reasons:
Having a report for several high-risk issues over time provides you with a fantastic high-level picture of continuous improvement in your organization. This particular report easily demonstrates safety reporting culture over time as you can see the number of reports by month, based on severity.
Much like the on-time reported safety issue closure for "high-risk safety issues" report, an on-time corrective action and preventive action (CPA) report shows you a percentage of all CPAs that were closed on time.
This measurement assures several things:
You should compare the current time frame of this report with the previous time frame.
Reported safety issues by department is a simple report showing the number of reported safety issues per department. Though simple, it allows you to ensure that each department is participating in the aviation SMS as expected. This report quickly allows a decision-maker to evaluate the safety reporting culture of the department.
For example, if you have a department with significantly fewer reported safety issues than other departments, then you may have a problem with non-participation or management silos. When these trends are detected, a safety culture survey is recommended to help identify potential areas to focus on safety promotion efforts.
Being able to generate reports for monitored risk controls with both positive and negative associations is the best way to monitor the effectiveness of safety control measures. What this report shows, is:
When you have controls that have many negative associations, you know it’s time to update them. The recommended action is to start a "Management of Change" process to analyze the control and to generate a list of alternatives. In short, you are back to the safety risk management side of the SMS (the design side).
One final report that is extremely valuable is simply a report that lists all safety policies, and the last time each policy was reviewed. There is a requirement that each aviation SMS has processes in place to ensure policies are regularly reviewed and updated as required.
Having such a report ensures that a portion of critical SMS documentation is up to date – this will significantly reduce your chance of having an audit finding for non-compliance related to SMS documentation.
Want to see a real SMS program that includes all these reports, among many others? See these demo videos.
If you need help monitoring the performance of your SMS, we can help. Our SMS database has been specifically designed for the aviation industry to address ICAO SMS requirements. Guess what? We also satisfy all the other regulatory and standards bodies' SMS requirements as well.
If you want the best and if you want to reduce your risk, check out SMS Pro.
Last updated May 2024.