Aviation safety managers are tasked with a Safety Assurance (SA) element labeled "Continuous Improvement." Not many aviation safety management systems (SMS) training courses go into much detail about "Continuous Improvement" which leaves many safety managers confusedly scratching their heads.
Safety Assurance is the third ICAO component or pillar of an ICAO-compliant SMS framework. Annex 19 - Safety Management and the four components are described in detail in ICAO's Document 9859.
The effectiveness of any implemented SMS cannot be determined whenever it cannot be measured. Aviation SMS' safety measurement process must be able to:
In every business, two limits must be considered when measuring SMS performance:
Safety managers tasked with measuring SMS performance will also be required to consider:
Safety managers who neglect to consider these two important factors will not be able to realistically develop SMS performance monitoring processes that complement organizational goals using budgeted resources.
"Safety assurance activities focus on assessing the health of the organization, with an emphasis on safety. Specific goals for improvements in all areas should be set for all senior operational managers..."
ICAO Annex 6, 3.3.5; Appendix 7, 3.1
Safety measurement is an essential component of every SMS. How else can one set SMART goals and objectives? For review, SMART goals are:
If we are not measuring performance, there will never be any concrete proof that our objectives were met. Abstract or ambiguous goals should be reworked until they meet the SMART requirements. SMS regulatory auditors start to look for SMS activity that is measuring and monitoring the SMS starting from Phase 2 and Phase 3 of SMS implementation.
Knowing which phase you are in your SMS implementation is important for managing expectations. Safety managers may have just inherited an SMS implementation. Others are under pressure to address SMS implementation deficiencies. In both cases, safety managers need to take an objective assessment of the SMS. If you don't know where you are, then you are less certain of which strategies to incorporate to achieve your safety goals and objectives.
To determine which phase you are in, three tactics come to mind:
A gap analysis is the process of assessing the state of an aviation SMS using an industry-accepted model. These models are commonly structured around the four pillars, meaning that the most common section heading you'll find in a gap analysis checklist is named after ICAO's four SMS components or pillars:
I have to warn you. There are really good gap analysis models and substandard models. The best advice I can give you is to stay away from the IS-BAO gap analysis checklist. It shocked me as a safety professional that it was even considered to have value.
We refused for many years to add the model to SMS Pro's gap analysis tool because I believed the content was too elementary to be useful for anyone. We relented about five years ago when a client insisted he needed it to evaluate their SMS against the prescribed model of the same standards body that was about to audit their operations.
The only benefit I see from the IS-BAO gap analysis model is that it was designed to "not scare" anybody, or overwhelm operators with a large laundry list of required activities and processes.
If you look at the IS-BAO gap analysis model, you may even laugh. In the early days of SMS implementation, there was considerable resistance in the general aviation (GA) community toward SMS. IS-BAO attempted to seize an opportunity to provide SMS guidance to biz-jet operators who needed proof of an SMS to fly to foreign countries, namely France and the Caribbean. This was the beginning. Now biz-jet operators are expected to have a demonstrable SMS in many more parts of the world.
When IS-BAO provided their gap analysis model to their followers, they were right. It didn't scare anyone away. One look at the IS-BAO checklist and a GA operator may reasonably think: "Wow, this doesn't seem too bad. I can do this."
Maybe this was IS-BAO's intention to increase the voluntary adoption of SMS. I don't know the full story. If anyone else does, please leave a comment or send me an email. However bad the IS-BAO gap analysis model is, IS-BAO really made up for it on their SMS implementation plan. It is first class!
I may be too hard on the IS-BAO gap analysis model because I'm a firm believer that the gap analysis is a great tool besides using it to define the gaps in the SMS. A gap analysis model is an excellent educational resource to train management and new safety managers on what is required in a fully implemented SMS.
By reviewing a gap analysis model, safety managers can conceptualize which SMS documentation requirements and processes will be required to develop a fully ICAO-compliant SMS. Furthermore, you also learn, at a high level, which aviation SMS activities support measuring and monitoring, such as:
If you are going to select one Gap Analysis model, I'd choose the ICAO gap analysis checklists. We have other gap analysis models that you can find from the links above.
Safety managers inheriting a functioning SMS implementation are in a great position to recharge an aviation SMS. Resistant safety cultures retard sincere SMS implementation, but a "change of face" with a new safety manager can revitalize or redirect a failing SMS. New "energy" can improve safety cultures, especially when the safety manager gets off on the right foot with both the accountable executive and employees.
The safety manager's expectations must align with the accountable executive before spending too much time and energy measuring SMS performance. When the accountable executive only wants a "paper SMS" to check the box, there is no need for robust, detailed SMS data management strategies. In this case, you only want to do the bare minimum, which can be accomplished with spreadsheets.
For aviation service providers who cannot operate with a "paper SMS," you will need SMS data management tools to:
Aggregated safety reports, dashboard charts, and trending charts help demonstrate dutiful safety performance monitoring and measurement compliance.
Both the gap analysis and the SMS implementation plan serve as "insight" into what SMS data management strategy you will need to demonstrate compliance for measuring safety performance. If you only need a paper SMS, then spreadsheets will work. Otherwise, save yourself time, energy, and money by acquiring an SMS database as soon as possible. Spreadsheets are simply the wrong technology for a sustainable process to measure and monitor SMS performance.
If your existing SMS database does not allow you to easily monitor and track required SMS documentation, then I advise that you find one that addresses all SMS requirements in a single platform. Do this as quickly as possible. Just rip off the band-aid. I've seen hundreds of SMS implementations and scores of failed SMS implementations. There are common threads about why aviation SMS fails.
Lack of top management support is the number one reason for SMS to fail. If you cannot get top management support for tools to measure and monitor SMS performance, you need to remind the accountable executive that:
Every accountable executive is responsible for properly implementing the SMS and making sure the SMS performs as designed in all areas of the organization. In order to determine whether the SMS is properly implemented and functioning, the accountable executive must regularly review organizational safety performance.
In order to monitor SMS performance, you need to measure SMS activity. Furthermore, you will need to set goals and objectives and be able to easily track your defined safety goals and objectives. When this process is not easily repeatable, it will not be performed for a prolonged period. An aviation SMS is not a short-term project, but one that will stretch out for many years.
Considering the breadth and scope of the SMS implementation, an SMS database is required in most cases. Otherwise, there is too much risk of things slipping through the cracks when safety managers leave the company. Furthermore, without tools to adequately measure SMS activity, it becomes impossible to effectively monitor SMS performance and direct actions necessary to address substandard safety performance.
The second reason SMS fails is the lack of data management tools to satisfy the mountains of SMS documentation. The SMS implementation plan helps safety managers understand which data point may be available to measure and monitor SMS performance. Within each SMS implementation plan, look for hints from keywords as to which data may be available to support measures such as:
Larger operators may already have a quality management system that collects considerable amounts of data. Quality management systems (QMS) and SMS are very similar in that they share many of the same elements, especially in the Safety Policy and Safety Promotion components. Many tools in the QMS can be re-purposed to collect and store SMS data.
SMS implementation plans are checklists developed either by:
When selecting an "SMS implementation checklist," consider who will be auditing your SMS. If you are serving two masters, such as the FAA and IATA, I always recommend using the ICAO SMS implementation plan. All CAAs and standards bodies model SMS requirements after ICAO. After all, ICAO is the driving force behind the SMS initiatives.
In order to reach a goal, you need to know two key elements:
The first question is the easiest to answer. By taking an honest assessment of the aviation SMS, you can develop a strategy to get to the end game. What is the end game? We can see a glimpse of what the end game is by reviewing the gap analysis and SMS implementation checklists. What will the end game look like for your organization? This will depend on several factors, including:
In project management, one of the first things we learn is that three elements affect the successful completion of a project:
Understanding these basic elements of project management will help safety managers plan more effectively for both new SMS implementations, and for planning the revitalization of the SMS. Each of these three elements is related to the others. For example, if you only need a paper SMS, you will not need significant amounts of time or budget.
Understanding where your SMS implementation sits today is critical for planning future moves. In addition, when you know the maturity level of the SMS implementation, you can manage the expectations of:
Furthermore, understanding where you need to go will affect the three project management elements we just discussed, namely time, budget, and features.
SMS assessments are useful for identifying where you are in the aviation SMS implementation process. While the gap analysis checklists and SMS implementation checklists are useful for guiding safety managers on "where to go," the SMS assessment advises safety managers "Where are we now?"
Below are four useful assessments to determine your SMS maturity level. If you take these assessments, be honest with yourself. Only you are seeing these assessment results and these assessments are for your guidance only.
During the first year or two of the SMS implementation, you may not have developed adequate data acquisition and data management strategies to manage all the SMS documentation requirements. You may still be in spreadsheets or diapers if you want to take an analogy of an SMS maturity being equal to a baby, toddler, juvenile, and adult stages.
Other companies are using point solutions that are disconnected from other SMS data sources, such as a stand-alone
Goals and objectives are very important to understand especially when you are considering strategies to monitor SMS performance. When you have goals and objectives to start with, finding the necessary data elements to support the measurement and tracking of objectives becomes much simpler.
If you are using spreadsheets, you will know which spreadsheets to sift through to support measuring progress on your safety objectives. Likewise, when safety teams use point solutions, they will have to visit each "system" to aggregate and prepare reports for management to determine whether safety objectives are met.
If you don't have any goals set, consider upper management's organizational goals before tackling safety goals and objectives. This will save the overzealous safety manager from becoming disillusioned when upper management's business strategy does not align with safety goals and objectives.
Safety goals and objectives need to have data points behind them in order to facilitate timely and accurate measurement. Otherwise, there is little objective evidence to support SMS performance monitoring activities. Examples of goals in an SMS include:
By analyzing each of these goals, visualize where in your organization you can find data to support tracking these goals. Will you be using:
Every company is a bit different. When aviation service providers use SMS databases to manage SMS documentation requirements, all SMS data resides in one location. This is a best practice and facilitates SMS data measurement and monitoring. Reports are automatically generated, thus freeing up management to focus on data analysis instead of data collection, cleaning, and preparation.
Up to now, we have identified our data sources. We've also set our goals and objectives for which to measure and demonstrate continuous improvement of the SMS. If you have been following along and reviewing the SMS implementation plan, you may have noticed "safety performance indicators" popping up in Phase 4 of the SMS implementation.
Safety performance indicators are KPIs (Key performance indicators). They are basically the same animal, but SPIs are a subset of KPIs. KPIs commonly focus on organizational performance tracking, while SPIs focus on safety. If you have an integrated safety and quality management system, then you will probably be using the KPI term almost exclusively to prevent confusion among management personnel who are accustomed to hearing about KPIs.
KPIs support your safety goals and objectives. Do you remember when we set our goals and objectives in the previous step? We identified the data sources needed to support the measuring process to determine whether we are accomplishing our safety objectives.
During your risk management activities, you will be working in several disparate systems, such as:
There are many ways to measure data and generate reports. KPIs are the data points that help you either directly or indirectly monitor safety objectives. If you are interested in measuring safety performance in SMS, this is where you want to be: key performance indicators.
So a workflow to measure safety performance in aviation can be:
The most important key performance indicators (KPIs) are leading indicators, also known as predictive indicators. Predictive indicators are not always accurate but exist as the most realistic "future vision" for safety planning and risk mitigation strategies.
Leading factors are measurable factors of safety performance that change before underlying safety-related events begin to manifest themselves. Leading indicators tend to follow particular trends. Since these safety-related statistics precede a string of accidents, incidents, and irregularities, they are used to forecast changes in operating budgets, including
One should logically assume that the accountable executive, upper-level management, and safety managers participate in establishing:
There are many resources and software tools to help airlines and airports monitor safety performance. Below are some educational articles regarding key performance indicators, also known as safety performance indicators.
If your company has not yet defined any KPIs, you undoubtedly don't have a mature SMS program. When you set KPIs, please follow the best practices outlined above in the offered resources.
The last words: if you don't measure it, you don't know where you are going and you don't know where you have been.
If you need aviation SMS database tools to help you measure and monitor SMS performance, we can help. These short demo videos illustrate how an SMS database can save your safety and quality assurance team considerable effort.
After watching the demo videos, you may have questions. Does it look like we are a good fit for you? We've been working with aviation service providers around the world since 2007. SMS auditors love SMS Pro because it is a complete SMS data management suite with all SMS data in one easy-to-access, secure system.
Last updated October 2024.