Most Important Questions to Monitor Safety Performance in SMS

Monitoring safety performance in aviation safety management systems (SMS) is undoubtedly one of the most important tasks you will undertake in your SMS. Regulatory requirements turn to monitoring the performance of the SMS from a "nice-to-have program" for reducing risk and avoiding accidents to a "must have and must demonstrate proof" to comply with SMS regulatory requirements.

Safety performance monitoring is integral for maintaining Safety Assurance and the integrity of your SMS program. This is "pie in the sky," telling you the purpose and benefit of monitoring SMS performance, but the real reason safety managers and accountable executives must pay attention is because it is required. You may get away with "satisficing" SMS auditors for a few years, but by year four or five, the chickens will come home to roost.

Related Articles on Monitoring Aviation SMS Performance

• How to Be Compliant With ICAO Safety Performance Monitoring and Measurement
• 5 Useful Safety Performance Monitoring Tools in Aviation SMS
• 4 Pillars | How to Conduct Safety Performance Monitoring and Measurement

The decisions you make regarding how you monitor safety performance can have drastic effects on safety performance, for better or worse. Do we need to mention SMS audits anymore?

The most important questions regarding your safety performance monitoring process are:

• Who is performing the safety performance monitoring?
• What tools do you use to monitor safety performance?
• What types of performance are the most important?
• How do you monitor safety performance?
• Who accesses SMS performance monitoring reports?

To answer these questions, you first need to define:

• What are you trying to measure?

When it comes to core performance, performance monitoring is most concerned with:

• Hazards;
• Risk controls;
• Safety objectives; and
• SMS bureaucratic oversight.

You should have reports ready at hand so safety teams and the accountable executive can quickly assess different elements of SMS performance. Having real-time reports is a best practice and allows more responsive decision-making to:

• take advantage of opportunities;
• direct actions necessary to address substandard safety performance;
• fulfill regulatory requirements of regularly reviewing organizational safety performance;
• satisfy auditors requests to demonstrate safety performance monitoring and measurement; and
• satisfy SMS requirements for continuous improvement of the SMS.

Related Articles on Continuous Improvement of Aviation SMS

• What Continuous Improvement Compliance Looks Like in Real SMS Program
• FAA Part 5 Compliance | Safety Assurance Continuous Improvement
• 4 Pillars | The Truth About SMS Continuous Improvement and KPIs – With Free KPI Resources

What Is SMS Performance Monitoring?

Regarding monitoring the SMS, we have discussed:

• What is being monitored;
• Who will be monitoring;
• Why you will be monitoring hazards, risk controls, etc.

What we haven't discussed is how and when or how often.

Monitoring a system requires charts, graphs, and tabular data. Data is required for safety teams to collate and format into user-friendly reports that add value to managers' decision-making processes. Where is your data stored?

• In spreadsheets?
• In multiple systems scattered around the company?
• In a centralized database?

Each of these styles of managing SMS data has its strengths and weaknesses. In very young, immature aviation SMS implementations, spreadsheets are the most common technology employed to store SMS data. As the SMS matures, point solutions may be cobbled together from other systems that exist within the organization, or purchased from software providers. These point solutions usually address a single pain point in the SMS, such as:

• Safety reporting system;
• Auditing system;
• Training management systems; and
• Electronic communication platforms for emails and company messages.

Aviation service providers that have more than 100 employees may already have some of these point solutions in place before beginning their SMS implementations. These companies become "wed" to these systems and are loathe to switch systems despite their shortcomings. Their reluctance to switch is related to the "sunk cost fallacy." Aviation service providers will continue using these disjointed, imperfect systems because they have historically spent considerable time, money, and effort on these systems. To abandon these systems is not seen as a strategic win to adapt to changing environmental pressures but as a management failure for selecting these short-sighted programs in the first place.

Examples of cobbled-together solutions are called "hacks" in the IT world. A solution may be created for one thing but re-purposed for another. SharePoint and email ticketing support systems are often converted into SMS database tools to manage:

• reported safety issues;
• safety documentation;
• safety communications.

Safety promotion may be handled by Survey Monkey and an electronic message board, such as can be put together by SharePoint. Spreadsheets are used considerably in this "SMS data management maturation phase" mainly for tracking SMS training requirements and the "Hazard Register."

Related Articles on Using Spreadsheets in Aviation SMS

• See How Spreadsheet Not EASA Compliant Aviation Hazard Reporting Database
• 5 Things Spreadsheets Can’t Do for Your SMS
• Spreadsheets vs Software for Aviation Safety Management

SMS Performance Monitoring Culture Matures

After years four to six of SMS implementation, medium-sized aviation service providers will realize that spreadsheets and point solutions are not sustainable solutions. It may take two or three years of the same repeat audit findings before these companies realize that their spreadsheet and hybrid point solutions introduce more risk than they are worth. Furthermore, accountable executives possess limited ability to effectively provide oversight to monitor SMS performance with all these cobbled-together point solutions.

It is painful to listen to frustrated accountable executives when they feel powerless to monitor the performance of their SMS. Accountable executives' frustration stems from their lack of ability to control decision-making because the safety team may only show the accountable executive what the safety team believes the accountable executive needs to see. Or accountable managers may fear their safety managers are not doing their job and they have no way to prove it.

For the accountable executive to ensure the SMS is performing properly, the accountable executive needs access to timely, unbiased reports. Obviously, spreadsheets are not going to satisfy the accountable executive's need to monitor the SMS, except for the smallest companies. Point solutions are also unwieldy because there are many required elements to track in SMS documentation. These point solutions are only addressing a particular requirement (or set of requirements).

An executive monitoring dashboard is what will scratch the accountable executive's itch for monitoring SMS performance. What you call your monitoring dashboard really doesn't matter. Some call them:

• simply "Dashboard";
• SMS Dashboard;
• Risk Management Dashboard;
• Safety Dashboard, etc.

Dashboards are effective in displaying monitoring data from multiple, disparate data sets. Dashboards in spreadsheets are a hack and not worthy of anyone's time except in a company with fewer than 40 employees. Medium to larger companies require dashboards to pull data from all their point solutions into a single dashboard.

Integrating point solutions into independent performance monitoring dashboards can be expensive and risky whenever any of the point solutions change. Creating dashboard charts and graphs may be difficult for IT staff who cannot easily access the data layer behind the point solutions.

To overcome the hodge-podge, cobbled-together point solutions, SMS database solutions started appearing in 2007 and 2008. These SMS databases were designed to manage all SMS requirements using a single, centralized data management strategy. Having all data in a secured repository also allows better data management capabilities, especially when it comes to presenting performance monitoring dashboards.

The best-of-class SMS databases may have scores of various dashboard charts for managers to select and save their views according to their needs. In these cases, each manager may have their own personalized dashboard that is being fed data in real-time, as soon as it is entered into the SMS database.

SMS databases are the final solution for operators who need to:

• demonstrate SMS compliance;
• reduce risk from cobbled together hacked systems; and
• reduce costs by supporting many duplicate systems.

Related Aviation SMS Database Articles

• What Is an Aviation Safety Database
• How to Choose the Best Aviation Safety Database Software
• How to Manage Aviation Safety Programs Without Complex SMS Databases

How to Monitor SMS Performance

We have seen the limitations spreadsheets have for demonstrating SMS performance monitoring. Point solutions are a bit better, but still not the best way to go for long-term monitoring satisfaction. If you have more than 100 employees and are starting to implement an SMS, here is a word of advice:

Start with an SMS database from the very beginning!

You will save yourself a lot of grief and frustration. Yes, I admit I'm biased because I develop SMS database software, but I'm first a data management expert and I understand SMS requirements. This is free advice and will save you money and maybe your credibility as a safety manager.

Don't go to your account executive and tell him you are going to implement an SMS that will save him tons of money by using spreadsheets. This is both a disservice and shameful if you do because now you know better than to manage SMS data using spreadsheets. Ask your account executive how he plans on monitoring the performance of his SMS. He may ask you for suggestions. Don't tell him spreadsheets. Yeah, it will save him some money today, but later it will cost him more.

When the accountable executive asks you for recommendations, you should give him options to monitor the SMS:

• Spreadsheets (with the cost of developing reports);
• Dashboard fed by data collected from point solutions (and costs); and
• SMS database (include costs, and cost-saving from ditching point solutions).

In most cases, there will be no additional costs for developing dashboard monitoring charts with an SMS database because most low-cost, commercially developed SMS databases have very flexible and powerful performance monitoring dashboards. Some of these solutions are very complete and will replace many of your point solutions that were discussed earlier. Therefore, it is important when you present the accountable executive with options that you subtract costs associated with any point solutions that will be replaced by the SMS database.

Did I mention that your recommendation should be an SMS database from the start of your SMS implementation?

When to Monitor Aviation SMS Performance?

We have answered the who, what, where, and how of SMS performance monitoring, but we have not discussed when should monitoring be performed. Part of this answer relies upon selected data management technologies for your SMS documentation requirements. Spreadsheets are not real-time and are traditionally not shared across a collaborative risk management environment. When you will monitor SMS performance charts when your company is resigned to using spreadsheets?

Based on second-hand accounts from safety managers upgrading to SMS databases, these safety managers claim to only create executive-level SMS monitoring charts and graphs once per year. In one case, a helicopter operator with 300 employees at the time, this safety manager spent three weeks each year collating data and creating management reports.

Therefore, if you are using spreadsheets, I'm expecting you will be monitoring the SMS less frequently than with hybrid point solutions and SMS databases. A good assumption will be semi-annually at best but expect to do it at least once per year. Is this enough? Maybe not. At the bare minimum, I believe SMS performance monitoring activities should occur at least quarterly, and more often if you have access to real-time performance monitoring dashboards.

Safety managers may monitor SMS performance on a daily basis as they manage reported safety issues. The quality department will monitor SMS performance whenever their team performs audits/inspections on a recurring basis. Trend charts may be reviewed quarterly or at random intervals by safety committees.

Trend charts should be reviewed at least once per year regardless of your data management capabilities. Per the FAA,

The objective of Safety Management Systems (SMS) is to proactively manage safety, identify potential hazards, determine risk, and implement measures that mitigate the risk. The FAA envisions operators being able to use all of the components of SMS to enhance a carrier’s ability to identify safety issues and spot trends (emphasis added) before they result in a near-miss, incident, or accident. For this reason, the FAA is requiring carriers to develop and implement an SMS.

Source: Federal Register / Vol. 80, No. 5 / Thursday, January 8, 2015 / Rules and Regulations

Trends are a big deal in aviation SMS. Spotting trends and mitigating substandard performance when, and as soon as possible after trends are detected is a lofty ambition of SMS. But we don't care about the lofty ideals, do we? Maybe I just want to check the box with my "paper SMS."

How do I collect data, make sense of the data, and present trend data whenever a manager requests it from me? The answer is simple. Get an SMS database that does it for you. In the best world, managers will have their own accounts so they can investigate trends at their convenience.

Related SMS Trend Analysis Articles

• Aviation Safety Managers' Best Friend - Trending Charts
• How to Use Trending Charts in Aviation SMS
• Six Steps How to Perform Trend Analysis in Aviation SMS

Actually Monitoring Performance of the Aviation SMS

Up until now, we've laid the groundwork for SMS performance monitoring. We know who, what, where, when, and how.

Let's now look at ten common charts that can be used to demonstrate SMS performance monitoring. Your SMS database may already have these, or you can use these reports as inspirations for your performance monitoring dashboards.

If you are a competitor with an SMS database, you may as well copy these reports, because other competitors have already been here before you. We recognize that being an industry leader, others are sure to copy what works and what makes the most sense.

1 – Top Hazards in Company

As safety issues are reported and managed, you will classify each one with a relevant primary hazard. In a safety concern, a hazard:

• Is the point at which safety control is lost – i.e. when operations are no longer within an Acceptable Level of Safety; and
• Requires immediate action to bring operations back into ALoS, or the hazard occurrence very well may lead to a risk occurrence.

Over time, you will have a well-developed hazard risk register, which is simply a term that describes a report for:

• All hazards that have been classified;
• The number of times each hazard was classified in issues; and
• All risks and times each risk has been classified.

You need to keep tabs on your top hazards to assess what your organization's most pressing safety concerns and needs are.

2 – Hazards Without Assigned Control Measures

This is a special report that many organizations are not using, but most definitely should be. This report lists:

• All hazards in your company that have been classified in reported safety issues; but
• Have no documented risk controls.

In other words, this report shows you hazards that have been identified in your operational environment but have no documented risk controls assigned to. This might be because you simply have not yet documented these risk controls, or it may be because you have not implemented risk controls.

Either way, having uncontrolled hazards is just asking for an audit finding or repeat safety issue occurrence.

3 – Any KPI Reports

Key Performance Indicators (KPIs) are the metrics that correspond directly to your safety objectives. As such, KPI reports demonstrate safety performance that your organization has flagged as “most important.”

KPI reports can be any simple or complex reports that are dedicated exclusively to your KPIs.

Related Articles on Key Performance Indicators (KPIs) in Aviation SMS

• What Is a Key Performance Indicator (KPI) in Aviation SMS?
• How to Set and Monitor Key Performance Indicators (KPIs) in Existing SMS
• How to Automate Key Performance Indicator KPI Monitoring

4 – On Time Safety Management for High-Risk Reported Safety Issues

High-risk issues are the most time-sensitive safety concerns in an aviation SMS. High-risk issues are identified after an initial risk assessment is performed on a reported safety concern.

Each organization's risk management processes should have default time periods that are set to close high-risk issues, such as within 3 days after the issue is reported.

The on-time issue management reports will show you the percentage of reported safety concerns that were closed on time, per policy. The best version of this report will allow you to compare the current time period for on-time closures vs. the previous time period.

5 – Most Common Root Cause Classifications

Much like your top hazards report, an equally valuable report is to see a report for your top root causes. This report would simply show you a list of your top root cause classifications, and the number of times each root cause was classified.

This report will allow you to assess the most pressing organizational concerns in your company, and where to spend energy developing risk controls.

6 – Number of High-Risk Safety Issues Over Time

As your SMS matures and improves its documentation, control measures, and safety oversight, you will experience a decline in the number of high-risk issues that are reported.

Why? Several reasons:

• Better risk controls mean hazard mechanisms are cut off before they can develop into hazard occurrence;
• Improved safety culture and safety awareness allow hazards to be identified early in the hazard/risk occurrence lifecycle; and
• Strong controls will effectively mitigate the severity of hazard occurrence.

Having a report for several high-risk issues over time provides you with a fantastic high-level picture of continuous improvement in your organization. This particular report easily demonstrates safety reporting culture over time as you can see the number of reports by month, based on severity.

Related Articles on Aviation Safety Reporting Culture

• How to Develop Healthy Safety Reporting Cultures in Aviation SMS
• How Aviation Safety Managers Can Improve Safety Reporting Cultures
• 5 Simple Tips to Improve Aviation Safety Reporting Cultures

7 – On-Time Corrective and Preventive Action Performance Percentages

Much like the on-time reported safety issue closure for "high-risk safety issues" report, an on-time corrective action and preventive action (CPA) report shows you a percentage of all CPAs that were closed on time.

This measurement assures several things:

• Safety concerns are being corrected quickly;
• Employees who are assigned CPAs are complying with the safety program; and
• Department heads or other subject matter experts who are assigning CPAs are taking continuous improvement sincerely.

You should compare the current time frame of this report with the previous time frame.

8 – Reported Safety Issues by Department

Reported safety issues by department is a simple report showing the number of reported safety issues per department. Though simple, it allows you to ensure that each department is participating in the aviation SMS as expected. This report quickly allows a decision-maker to evaluate the safety reporting culture of the department.

For example, if you have a department with significantly fewer reported safety issues than other departments, then you may have a problem with non-participation or management silos. When these trends are detected, a safety culture survey is recommended to help identify potential areas to focus on safety promotion efforts.

9 – Positive/Negative Associations With Risk Controls

Being able to generate reports for monitored risk controls with both positive and negative associations is the best way to monitor the effectiveness of safety control measures. What this report shows, is:

• A list of all safety risk control measures;
• The number of times each control measure was marked as actively mitigating safety issues; and
• The number of times each control measure was marked as not mitigating a safety issue as it should.

When you have controls that have many negative associations, you know it’s time to update them. The recommended action is to start a "Management of Change" process to analyze the control and to generate a list of alternatives. In short, you are back to the safety risk management side of the SMS (the design side).

10 – Safety Policy Items Review Date

One final report that is extremely valuable is simply a report that lists all safety policies, and the last time each policy was reviewed. There is a requirement that each aviation SMS has processes in place to ensure policies are regularly reviewed and updated as required.

Having such a report ensures that a portion of critical SMS documentation is up to date – this will significantly reduce your chance of having an audit finding for non-compliance related to SMS documentation.

Want to see a real SMS program that includes all these reports, among many others? See these demo videos.

If you need help monitoring the performance of your SMS, we can help. Our SMS database has been specifically designed for the aviation industry to address ICAO SMS requirements. Guess what? We also satisfy all the other regulatory and standards bodies' SMS requirements as well.

If you want the best and if you want to reduce your risk, check out SMS Pro.

Last updated May 2024.