Many airlines that conduct LOSAs internally have been confronted with a problem that has significantly affected the validity of their observations.
The problem has to do with observer calibration.
Observer calibration (which may be referred to more formally as inter-rater reliability) is the process of ensuring that each LOSA observer is coding threats, errors, and UASs (Undesired Aircraft States) in the same way. For instance, if three out of eight observers are recording an inoperative lavatory as a UAS then there would be a serious calibration issue between those observers. That will undoubtedly cause a lot of extra work in the data cleaning and verification process. And in some cases, there may be a lot of data that unfortunately wind up being completely unusable.
To preclude this from happening, LOSA observers need to be thoroughly calibrated and reliability testing needs to be done before any actual observations are made (this can be done using paper scenarios in the classroom).
Observer calibration will help to ensure that each observer is seeing, processing, and recording the same thing (i.e., on the same sheet of paper)—to the extent possible. Although it would be great to have perfect reliability among all observers, we know for a fact there will always be some variation. This is normal. However, if there is significant variability then there is a good chance that your LOSA results will not be very truthful and those results will be met with much management skepticism.
Learn how LOSA can benefit your company. Contact us today for a free consultation or LOSA training.
Last updated February 2024