Identifying and classifying the human errors behind rail incidents and accidents is the basis for targeting safety countermeasures, yet the two techniques compared by Baysari, Caponecchia, McIntosh and Wilson (2009) - HFACS and TRACEr-rail - each captured only part of the causal picture (organisational and latent conditions versus driver task context, cognitive error type, and error detection and recovery) and both achieved only moderate inter-rater reliability when applied to Australian rail investigation reports. The source's own recommendation - that the tools be modified, or a new tool developed, for complete and consistent error classification - defines the problem it left open: develop, and validate on real investigation data, a rail-specific human error identification framework that is simultaneously complete (covering driver task context, cognitive error type, error detection and recovery, and organisational conditions at the resolution the two compared techniques reached only in their respective strong domains) and consistent (unambiguous, mutually exclusive categories - including a workable violation/error boundary - that independent analysts apply with inter-rater reliability above the moderate levels the two compared tools achieved). No published framework jointly meets both requirements. The problem remains open.

Background

Human error identification (HEI) techniques are structured taxonomies and procedures for classifying the errors and contributing conditions behind accidents and incidents, so that recurring patterns can be counted and targeted with countermeasures. Two such techniques dominate retrospective safety analysis. The Human Factors Analysis and Classification System (HFACS), developed for aviation accident investigation, is a four-level framework that organises causal factors into unsafe acts, preconditions for unsafe acts, unsafe supervision, and organisational influences; it is strong on latent, organisational conditions but coarse on the cognitive detail of the front-line error. The Technique for the Retrospective and Predictive Analysis of Cognitive Errors (TRACEr), developed for air traffic control, classifies an error through interlocking taxonomies - task error, cognitive domain, internal error mode, psychological error mechanism, and performance shaping factors (the conditions that influence the likelihood of error) - together with taxonomies for error detection and error recovery; a simplified retrospective rail version, TRACEr-rail (Retrospective TRACEr Lite), was later developed for the UK Rail Safety and Standards Board. A recurring scientific requirement for any such taxonomy is inter-rater reliability: independent analysts classifying the same incident should agree, conventionally measured with statistics such as Cohen's kappa, Krippendorff's alpha, or the index of concordance (the proportion of agreeing code pairs among all code pairs); without adequate agreement, counts of error types depend on who does the coding and cannot support valid conclusions about prevalence or countermeasure effectiveness. A further classic distinction separates errors (unintended actions or judgments) from violations (intentional departures from rules or procedures), a boundary that is hard to draw after the fact because intentionality is rarely documented; signal passed at danger (SPAD) events, in which a train passes a stop signal without authorisation, are a prototypical rail outcome of both. Baysari, Caponecchia, McIntosh and Wilson (2009) applied both HFACS and TRACEr-rail to 19 publicly available Australian rail safety investigation reports of train-driver accidents and incidents. HFACS found skill-based slips of attention to be the most common unsafe acts; TRACEr-rail found most train driving errors to be violations and most train stopping errors to be perceptual. Both tools pointed to decreased driver alertness and incorrect driver expectations as the largest underlying factors. Crucially, each tool neglected different parts of the picture - HFACS produced extensive organisational and latent-condition entries but little on the driving task and its aftermath, while TRACEr-rail gave detailed task, detection and recovery information but missed many organisational factors - and both possessed only moderate inter-rater reliability. The authors recommended that the existing tools be modified, or a new tool developed, for complete and consistent error classification in rail. Their companion work framed the same gap from each side: an HFACS application to 40 Australian reports concluded that practical, reliable rail use requires rail-specific category descriptions, formal rater training and taxonomy refinement, and an application of Retrospective TRACEr Lite to Australian and UK reports showed the tool could classify every sufficiently described driver error while revealing unclear and non-mutually-exclusive category descriptions. What no existing technique offered was a single rail-specific framework that is at once complete across cognitive, task, detection/recovery and organisational levels, and consistently applicable by independent analysts.

Problem Statement

Develop and validate a rail-specific human error identification capability for the retrospective classification of errors in rail incidents and accidents that is simultaneously: (1) Complete: within one coherent framework it captures the train driver's task context, the cognitive error type (including the error/violation distinction), whether and how the error was detected and recovered, and the wider performance-shaping and organisational conditions behind the incident - the complementary strengths that HFACS and TRACEr-rail each delivered only partially. Completeness requires categories that discriminate within each of these dimensions at a resolution no coarser than that which the two compared techniques reach in their respective strong domains (HFACS's four causal levels for organisational and latent conditions; TRACEr's internal error mode, error detection and error recovery taxonomies for the cognitive detail of the front-line error), so that acknowledging a dimension without categories that distinguish within it does not satisfy this clause. (2) Consistent: its categories are defined unambiguously and are mutually exclusive, including a workable boundary between violations and errors, such that independent analysts classifying the same investigation material achieve inter-rater reliability demonstrably above the moderate agreement that HFACS and TRACEr-rail attained in the source comparison. The source comparison reported its agreement as raw percentage agreement - the index of concordance (the proportion of agreeing code pairs among all code pairs, as defined above) - at only moderate levels: pairwise HFACS agreement of approximately 30-86% (average super-category agreement 79%) and TRACEr initial-extraction agreement of approximately 50% (detection/recovery agreement approximately 17%). The new framework's index of concordance must exceed the source's reported agreement in each dimension for which the source comparison reports a figure - organisational and latent conditions above the 79% HFACS super-category agreement, cognitive error type above the approximately 50% TRACEr initial-extraction agreement, and detection and recovery above the approximately 17% TRACEr agreement - and, in the overall index of concordance across the full coding, above the highest source figure (79% of coded pairs); the train-driver task-context dimension, for which the source reports no figure, is reflected only in that overall figure. The reported agreement must be measured on the same index of concordance as the source comparison, so that the values are comparable. Demonstrating consistency requires application to real rail incident or accident investigation data with a reported inter-rater reliability study, because consistency in the sense of the source is a property of how the technique performs when independent analysts apply it to investigation material. The investigation material is the corpus of 19 publicly available Australian rail safety investigation reports of train-driver accidents and incidents analysed in the source comparison. The reported agreement must come from genuinely separated coding passes that are independent of the framework's development, with all disagreements reported; agreement measured on a single self-consistent pass is not inter-rater agreement.

References

  1. Shorrock, S.T., Kirwan, B. (2002). Development and application of a human error identification tool for air traffic control. Applied Ergonomics 33(4), 319-336. DOI: 10.1016/S0003-6870(02)00010-8. https://doi.org/10.1016/S0003-6870(02)00010-8
  2. Shappell, S., Detwiler, C., Holcomb, K., Hackworth, C., Boquet, A., Wiegmann, D.A. (2007). Human error and commercial aviation accidents: An analysis using the Human Factors Analysis and Classification System (HFACS). Human Factors 49(2), 227-242. DOI: 10.1518/001872007X312469. https://doi.org/10.1518/001872007X312469
  3. Baysari, M.T., McIntosh, A.S., Wilson, J.R. (2008). Understanding the human factors contribution to railway accidents and incidents in Australia. Accident Analysis & Prevention 40(5), 1750-1757. DOI: 10.1016/j.aap.2008.06.013. https://doi.org/10.1016/j.aap.2008.06.013
  4. Baysari, M.T., McIntosh, A.S., Wilson, J.R. (2008). Analysis of Australian and UK rail incidents and accidents using Retrospective TRACEr Lite for train drivers. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 52(21), 1761-1765. DOI: 10.1177/154193120805202115. https://doi.org/10.1177/154193120805202115
  5. Baysari, M.T., Caponecchia, C., McIntosh, A.S., Wilson, J.R. (2009). Classification of errors contributing to rail incidents and accidents: A comparison of two human error identification techniques. Safety Science 47(7), 948-957. DOI: 10.1016/j.ssci.2008.09.012. https://doi.org/10.1016/j.ssci.2008.09.012
  6. Kim, D.S., Baek, D.H., Yoon, W.C. (2010). Development and evaluation of a computer-aided system for analyzing human error in railway operations. Reliability Engineering & System Safety 95(2), 87-98. DOI: 10.1016/j.ress.2009.08.005. https://doi.org/10.1016/j.ress.2009.08.005
  7. Baysari, M.T., Caponecchia, C., McIntosh, A.S. (2011). A reliability and usability study of TRACEr-RAV: The technique for the retrospective analysis of cognitive errors - For rail, Australian version. Applied Ergonomics 42(6), 852-859. DOI: 10.1016/j.apergo.2011.01.009. https://doi.org/10.1016/j.apergo.2011.01.009
  8. Read, G.J.M., Lenne, M.G., Moss, S.A. (2012). Associations between task, training and social environmental factors and error types involved in rail incidents and accidents. Accident Analysis & Prevention 48, 416-422. DOI: 10.1016/j.aap.2012.02.014. https://doi.org/10.1016/j.aap.2012.02.014
  9. Olsen, N.S. (2013). Reliability studies of incident coding systems in high hazard industries: A narrative review of study methodology. Applied Ergonomics 44(2), 175-184. DOI: 10.1016/j.apergo.2012.06.009. https://doi.org/10.1016/j.apergo.2012.06.009
  10. Ergai, A., Cohen, T., Sharp, J., Wiegmann, D., Gramopadhye, A., Shappell, S. (2016). Assessment of the Human Factors Analysis and Classification System (HFACS): Intra-rater and inter-rater reliability. Safety Science 82, 393-398. DOI: 10.1016/j.ssci.2015.09.028. https://doi.org/10.1016/j.ssci.2015.09.028
  11. Madigan, R., Golightly, D., Madders, R. (2016). Application of Human Factors Analysis and Classification System (HFACS) to UK rail safety of the line incidents. Accident Analysis & Prevention 97, 122-131. DOI: 10.1016/j.aap.2016.08.023. https://doi.org/10.1016/j.aap.2016.08.023
  12. Zhan, Q., Zheng, W., Zhao, B. (2017). A hybrid human and organizational analysis method for railway accidents based on HFACS-Railway Accidents (HFACS-RAs). Safety Science 91, 232-250. DOI: 10.1016/j.ssci.2016.08.017. https://doi.org/10.1016/j.ssci.2016.08.017
  13. Zhou, J.-L., Lei, Y. (2018). Paths between latent and active errors: Analysis of 407 railway accidents/incidents' causes in China. Safety Science 110, 47-58. DOI: 10.1016/j.ssci.2017.12.027. https://doi.org/10.1016/j.ssci.2017.12.027
  14. Hulme, A., Stanton, N.A., Walker, G.H., Waterson, P., Salmon, P.M. (2019). What do applications of systems thinking accident analysis methods tell us about accident causation? A systematic review of applications between 1990 and 2018. Safety Science 117, 164-183. DOI: 10.1016/j.ssci.2019.04.016. https://doi.org/10.1016/j.ssci.2019.04.016
  15. Duncan, D., Klockner, K. (2020). Development of the contributing factors Framework interview guide (CFF-IG): All aboard the rail safety truth train. Safety Science 129, 104853. DOI: 10.1016/j.ssci.2020.104853. https://doi.org/10.1016/j.ssci.2020.104853
  16. Ciani, L., Guidi, G., Patrizi, G. (2022). Human reliability in railway engineering: Literature review and bibliometric analysis of the last two decades. Safety Science 151, 105755. DOI: 10.1016/j.ssci.2022.105755. https://doi.org/10.1016/j.ssci.2022.105755
  17. Zhou, J.-L., Tu, R.-F., Xiao, H. (2022). Large-scale group decision-making to facilitate inter-rater reliability of human-factors analysis for the railway system. Reliability Engineering & System Safety 228, 108806. DOI: 10.1016/j.ress.2022.108806. https://doi.org/10.1016/j.ress.2022.108806
  18. Ahmadi Rad, M., Lefsrud, L.M., Hendry, M.T. (2023). Application of systems thinking accident analysis methods: A review for railways. Safety Science 160, 106066. DOI: 10.1016/j.ssci.2023.106066. https://doi.org/10.1016/j.ssci.2023.106066
  19. Wang, Y., Guo, M., Ye, L., Shi, J., Song, K. (2026). Human factors configurations of driver maladaptation in urban rail accidents: An fsQCA-based analysis. Journal of Transportation Safety & Security 18(7), 966-990. DOI: 10.1080/19439962.2026.2620374. https://doi.org/10.1080/19439962.2026.2620374