
We Say Life is our Priority Until it Comes to
Data Collection
By Brian Brush
In a traditional sports season, the evaluation of a team’s performance comes down to wins and losses. For 2020, sports seasons have been anything but traditional, and standard performance metrics have been challenged. With delays in conference starts and or game cancellations, teams are finishing the same season with varying numbers of total games. This has a profound effect on number of losses as an evaluation tool. A team with one loss in a six-game season is not the same as a team with one loss in a 12-game season. A team with two losses in a 12-game season is not the same as a team with one loss in a six-game season.
When the number of losses does not provide the best picture of a season, discussion moves to the quality of a wins. Team X lost to team Y, but they beat team Z by XX. This metric is argued to be more subjective. The counter to the subjectivity argument is the depth of a quality assessment. The win quality can be evaluated by looking at the record of the other team, the margin of victory, the location of the game and other factors beyond the booth announcer’s view of a team being hot or not.
Beyond wins and losses, another tool that surfaces with regard to record is the strength of schedule. Strength of schedule is a gauge of how much true competition one team sees. Teams that acknowledge this as an important metric will purposefully seek out top level competition in their non-conference or preseason schedule to improve strength of their schedule, essentially their season resume. In 2020, due to shortened seasons, the added exposure potential of travel and other factors, the opportunity for out of conference or intentional elevated competition has been severely impacted. For this reason, the way strength of schedule has been viewed in the past as an elective component and measurement to the level of risk to gain a team is willing to take into their season has been removed.
Given this backdrop, and familiar parallel, the American fire service needs to use this opportunity to proactively challenge the way we view our performance metrics.
Fire Service Losses
Formal fire service reporting in the United States has been in place for almost 50 years. The establishment of a national fire reporting system was born in 1973 out of America Burning. The group of fire professionals and congressional members assembled to take the fire problem head on found the greatest barrier was the lack of understanding with regard to the problem. “The commission found an appalling gap in data and information that effectively separated us from sure knowledge of various aspects of the fire problem” (National Commission on Fire Prevention and Control, 1973, p. 9). To address this problem, the federal commission issued a recommendation that a “National fire data system be established to provide a continuing review and analysis of the entire fire problem” (National Commission on Fire Prevention and Control, 1973, p. 9). Within a few years the National Fire Incident Reporting System (NFIRS) was fully operational.
Looking back to the beginning, a time where the American fire service was unsure of the fire problem, it is understandable that the focus of all efforts would be concentrated on the problems (loss). This concentration established the first fire service inputs and metrics that as quantitative and negative. From day one through 2020, what the fire service measures and reports are, number of fires, property lost or damaged, the number of injuries and the number of fatalities. An unintended result of this loss centric methodology is a myopic data set with an analysis that is limited to loss relationships. To clarify this with an example, demonstration of a reduction in civilian fire deaths is only that; it cannot conclusively show more lives are actively being saved.
The 2020 sports season analogy, or this article is understandably not enough to drive a paradigm shift away from a loss-based performance standard but a call to action within our most trusted data and standard setting organizations should be. For more than forty years, the central data source for evaluation of the nation’s fire problem has been the National Fire Incident Reporting System (NFIRS). The United States Fire Administration (USFA) and the National Fire Protection Association (NFPA) uses this data to prioritize and inform prevention interventions. This practice is reinforced as the frequency reduction data continues to demonstrate the success of prevention.
The 2018 NFPA report on home structure fires evaluated American fire service data from 1977 to 2017. The report shows home fires and home fire deaths have been reduced by about 50% since 1980. Over this same time period, the death rate per 1,000 home fires has remained fairly consistent. In recent years the death rate per 1000 fires has been slightly higher than it was in 1980 (Ahrens, Home Structure Fires, 2019). The report concludes that the reduction in home fire deaths over the past 40 years has been due to the reduction in the frequency of fires and not related to any reduction in severity of the event (Ahrens, Home Structure Fires, 2019). In short, the American fire service is now seeing that 40 years ago we were a 12 and 2 team and today we are 6 and 1. We are losing less, but we have not improved our record.
What is distilled from this divergence is that prevention success is purely pre-event. Even with the greatest preventative efforts available at any point in the history of human intervention, fires still occur. It is possible, that given the level of modern data analysis and sophistication we need to consider that the true fire problem is uninformed operations?
The most recent and most comprehensive evaluation of fire service data collected from the United States Fire Administration was released in 2019 and it directly speaks to this challenge. “The lack of data, especially for these residential fatal fires masks the true picture of the fire problem.”. How is it possible that this statement did not spark the same reaction to lack of understanding that was witnessed with America Burning. From over 40 years of data collection, the most advanced nation in the developed world has directly recognized a failure to reduce the severity of residential fires and identified the presence of a blind spot in understanding why.
Fire Service Wins
To maintain consistency, it shall be repeated that the traditional measurement and reports of performance for the fire service are explicitly defined as the number of fires, amount property lost or damaged, the number of injuries and the number of fatalities. The clarity and history of this practice provides the solid known of loss and how many losses are experienced every year. What I believe is embedded in the statement from the USFA that “The lack of data, especially for these residential fatal fires masks the true picture of the fire problem.”, is the lack of clarity and history of wins. As presented earlier, demonstration of a reduction in civilian fire deaths is only that; it cannot conclusively show more lives are actively being saved.
To date, the American fire service has not officially collected the number of or means of which civilians are rescued at fires. The result of this information gap is a service unable to quantify if or how, presence, actions, or operations result in saving lives. For the fire service to deliberately improve outcomes and not just reduce loss, the mission (saving lives) must match the metrics (lives saved). Until then, the operational impact of the American fire service will remain unknown.
To further demonstrate the unintended consequence and widespread impact of language selection when defining mission look no further than the USFA page for civilian home fire fatalities. This page is arguably the most referenced in “real time” data source available to the American fire service. The energy put into maintaining this resource at this level is very deliberate and the explanation is very concise.
“Home fire fatalities in the news: Information on home fire fatalities is compiled through a daily Internet search (Monday-Friday) of U.S. news media reports. The primary purpose of this information is to help raise awareness about the danger of fire and the frequency of home fire deaths.” USFA Civilian Home Fire Fatalities in the News
Drawing from this brief paragraph, the two areas of focus for the USFA is “raise awareness of danger” and “frequency of home fire deaths”. I would offer these translations, “raising awareness of danger” intentionally highlights loss. Attention given to “frequency of home fire deaths” demonstrates the entrenched vision that our interventions are limited to making a difference in frequency and not the severity of the with regard to fire risk in this country. As a test of theory, I elected to flip language and vision that is used by the USFA while utilizing the same data collection tool. For a 15-day period from December 1st through December 15th, 2020; using Google Alerts, social media and fire department press releases through a daily internet search a collection of U.S. reports of home fire civilian rescues by fire departments was established. These reports were confirmed and categorized to help raise awareness about how often fire department operations are responsible for the potential difference in the outcome of civilians trapped by fire.
For a 15-day sample, mission (saving lives), matched our metrics (lives saved) and we created a record of our wins. At the time of capture, the record for the operational American fire service was 127 wins and 122 losses. The depth of vetting in confirmation of these reports confirms that a rescue was associated with victims leaving the scene either under their own power or by the determination of EMS transport that the patient had a chance. Over the course of the 15-day period and going forward, some of these rescues transition into fatalities. Fire department operational interventions cannot influence the conditions prior to arrival in the structure or in the patient. The impact of moving victims from a toxic, hostile atmosphere to one of treatment and care in fresh air is an intervention that can be measured, and this should be both highest priority intervention and data collection point for the discipline of operations.
It is important that the American fire service recognizes performance cannot be adequately measured with a single lens. The unrealistic expectation that any occurrence of fire is reflected as a failure of our efforts has had a profound negative impact on our service as a whole. We are not “quitting” when we face the fact that fires will occur despite our best efforts, this is acceptance of reality and the history of man. Realistic vision creates realistic comprehension and higher quality evaluation and strategy. The pre-event record belongs to prevention, when the fire does start, this is where we measure the performance of operations. We must continue to support the wins of prevention through the mission of reducing the number of fires and going forward we need to be more direct in defining and reviewing the wins of operations through the mission of improving outcomes for fire victims.
Strength of Schedule
According to the 2019 Fire Loss in the United States report from NFPA, there were 361,500 home fires which resulted in 2,870 civilian deaths for the calendar year of 2019. The combination of these two numbers creates the civilian fire deaths per 1000 fires ratio which has served as the unofficial record of the fire service for decades. From the same 2019 NFPA report; “For overall home fires, the death rate per 1,000 fires was 15 percent higher in 2019 than in 1980, while the comparable civilian injury rate was 34 percent higher than in 1980.”. It is here that the issue of negative input only reporting is highlighted as the analysis is limited to loss relationships.
Utilizing the same ratio 40 years apart, NFPA data can show we are losing more today than we were 40 years ago and the loss only record provides no further context. We are not afforded the opportunity a general manager or a head coach would get to immediately question if the cause of our regression was due to poorer performance or more challenging competition. As mentioned before this is the deeper look provided by determining the quality of wins. Given the improved staffing, equipping, training, standardization and communication provided today one could argue that the “modern” fire has increased the level of competition. Given the reduction of fires, the increased safety measures with regard to live fire training, broader scope of responsibility and demands of today’s fire service, one could argue that we aren’t as good at fighting fire as we used to be. Both positions would just be arguments because there is very little additional data to triangulate with. For discussions sake, let’s say rescues had been recorded for the same 40-year period and it demonstrated today’s fire service was removing more victims from fires than ever before. If the fatality rate continued to rise, it is much easier to direct discussion on the toxicity of the fire environment or the preexisting conditions of the civilian population and less on the performance of operations. Until we reach this point the operational impact of the American fire service will remain unknown and vulnerable to question.
The ratio of civilian fire fatalities per 1000 home fires has been one of the long-standing benchmarks metrics of the fire service. The importance of this metric is clearly demonstrated with the level of concern the 2019 NFPA report gives to the deterioration of it. It is my opinion that the level of attention without question this metric receives has thrown shade over the impact of fireground operations and perpetuated a critical flaw which has led us to the point that our fire service data analysis professionals report “The lack of data, especially for these residential fatal fires masks the true picture of the fire problem.” USFA 2019. It is understood that most of the discussion here is a long-term paradigm shift so as this piece comes to a close, I feel it is important to present a small and simple solution with a big result.
With regard to strength of schedule in the sports world, this metric is the gauge of how much true competition one team sees. According to NFPA, in 2019 there were 361,500 home fires which resulted in 2,870 civilian deaths, this number is most often reduced the ratio of civilian fires per 1000 home fires. The critical flaw with this presentation is it subconsciously pushes the assumption that civilian lives are at risk at every home fire and it skews our success. Given 361,500 home fires and 2,870 civilian fire fatalities, one could potentially present this same data in a statement that less than 1% of all home fires in the U.S are fatal. This is the reason the data set is moved up to “per 1000” fires, because “per 100” would not be a whole number and that optic is all it takes to diminish perceived risk.
The purpose of the strength of schedule metric is to evaluate how much true competition one team sees. Saving civilian lives on the scene of a fire is the championship game for fire service operations. A working structure fire in an abandoned house that burned two days before is a day in the batting cage, yet in our current system this is logged as a fire which did not have a civilian fatality and it goes in the book as a win. Fire service reporting is done after the fire, whether there was a fatality or a rescue at the scene, at the time of reporting it is known if the structure was occupied on arrival of the fire department. With one question answered, one box checked our traditional reporting system would gain a massive amount of context. Currently we can only report that 2,870 civilians died out of 361,500 home fires because we have no idea how many of those structures actually had people in them. One simple check box; was the building occupied upon arrival, two simple choices, yes or no. The question is not, was the building “vacant” or “abandoned”, it is not “did you know”, it is a very clear post incident response to answer, was anyone inside upon arrival.
With just one year of this one question, we could better evaluate performance and communicate a more accurate and stronger message. With just one year, our fire service data analyst could finally focus in on evaluating the performance record of the fire department when faced with true competition of parties trapped. This one question would reduce the 361,500 fire immediately to a much smaller population where the USFA and NFPA could go deeper into what factors were the difference in outcome between and associated fatality associated or not. Our current language is that the civilian death rate per 1000 home fires is X.X. (less than 1%) The desired communication would be, the civilian death rate for fires in occupied dwellings is…………. if we only knew.
2021 Civilian Fireground Rescue Research Project
With the support of Oklahoma State University, through a graduate research practicum, reports of fireground civilian rescues from news outlets, press releases and social networks will be actively collected, categorized and confirmed for a period of 3 months. Organizations responsible for rescues will be directly contacted, informed of the research project, provided a rescue reporting procedure template, and directed to complete a Firefighter Rescue Survey (FFRS). The FFRS is an online, qualitative research tool for collection and classification of data from the first-hand reports of those directly involved in fireground civilian rescues.
The purpose of this graduate research project is to demonstrate the scope and value of fireground civilian rescue reporting using qualitative survey methods. The results will support an improved understanding of fireground operational influence on outcomes. A clearer vision of the nation’s fire problem for the future includes the knowledge of both the parameters of our problem (loss) and the dimensions of our success (saves).
A final report will present with a three-month period of parallel collections from the U.S. Fire Administration. To report fireground civilian rescues, for more information about the research project, or to continue the discussion contact me at Brush@okstate.edu visit www.firefighterrescuesurvey.com or take in the article Search and Rescue by the Numbers By Nick Ledin. Using the collection of 1,500 fireground civilian rescue surveys, Nick provides detailed look at how qualitative data collection can inform fire ground operations. https://www.fireengineering.com/2020/02/01/484614/search-and-rescue-by-the-numbers/.
BRIAN BRUSH, a 20-year veteran of the fire service, is a Training Chief at Midwest City (OK) Fire Department. He has a bachelor’s degree in fire and emergency services administration, a graduate of the EFO program and a Chief Training Officer designation from the CPSE. He instructs on a national level and writes for Fire Engineering.