By Jane Moore, Jennifer Mieth & Cynthia Oulette
Fire and burn injuries are the third leading cause of unintentional injury and death among children between one and 10 years old. Fire and life safety education (FLSE) is an important strategy in reducing unsafe behavior and environments, according to both the Centers for Disease Control and Prevention (CDC) and the NFPA. Fire service professionals understand this and fire departments throughout the U.S. provide an array of public education programs–nearly 90% of which focus on children in schools, as reported in a 2008 survey by Johns Hopkins Center for Injury Research and Policy. Nevertheless, we lack reliable, consistent evidence that children learn and retain fire safety teaching. Fire services rarely use structured programs with built-in evaluations; Johns Hopkins also reported that while slightly more than half of all fire departments assessed their education efforts, fewer than one in 10 of those used formal measures.
Incident reports are one source of data to assess learning. In the 19 years since the Massachusetts Department of Fire Services (MA-DFS) launched statewide support for fire and life safety education through the state (the Student Awareness of Fire Education [S.A.F.E.] Program), child fire fatalities have decreased from an average of 18.5 to an average of 5.5 per year–a drop of nearly 70%. Children’s behavior also demonstrates learning: 288 children have received ‘young heroes’ awards for using what they learned in S.A.F.E. programs to save lives. These results are telling, but MA-DFS sought a system to produce reliable, consistent data about the efficacy of the teaching itself. To achieve this, MA-DFS has implemented ongoing, structured assessments of the S.A.F.E. Program. The complete system–with components for third, sixth and 10th grades–is available on the MA-DFS S.A.F.E. website: www.mass.gov/eopss/agencies/dfs/dfs2/osfm/pubed/s-a-f-e/public-education-evaluation.html.
Staying S.A.F.E.
S.A.F.E. is a coordinated statewide effort providing grants, resources, training and technical assistance to local fire services in partnership with schools and communities. S.A.F.E. grant recipients send a lead educator to the Massachusetts Firefighting Academy’s five-day training to learn about child development, how children learn, lesson planning, public speaking and program evaluation. The Massachusetts Fire and Safety Education Curriculum Planning Guidebook supports firefighter-educators in designing their programs. This approach accommodates the wide variations among the 351 cities and towns in Massachusetts, allowing each grantee to design a program responding to their community’s changing needs. More than 200 communities receive S.A.F.E. grants, providing FLSE to more than 300,000 children in classrooms and community events statewide. More than 60% of these children were in pre-K through fourth grade, providing early learning and consistent reinforcement.
Evaluation is required, and to support firefighter-educators, MA-DFS created a system to assess students’ understanding of critical fire and life safety concepts, based on standard classroom tests for third-, sixth- and 10th-grade students.
Design and Implementation
To be certain that the tests cover critical concepts, and are appropriate for the target age groups, MA-DFS sought input from fire and life safety professionals and elementary grade educators; studied fire and burn incident and unintentional injury reports; and reviewed existing curricula and research literature. Questions and test layout were reviewed and rated by a panel of experts in FLSE, elementary and secondary level education and literacy. This process was followed by pilot testing with students in the classroom. Pilot testing proved critical in ensuring students’ comprehension of illustrations, stories and questions, and in determining how much time students needed to complete the tests. The MA-DFS Public Education Unit, in collaboration with the Massachusetts Public Fire and Life Safety Education Task Force, monitored each stage.
A common barrier to effective ongoing evaluation is a lack of time and staff to deal with the burden of collection, analysis and reporting of data. The Massachusetts program minimizes this barrier by providing a user-friendly database into which firefighter-educators record test answers. Embedded formulas automatically calculate the number of tests entered and percentage of correct answers by key concept, e.g., escape plans. A user guide describes how to administer the test, use the database, use results to report outcomes and plan teaching.
Positive Outcomes
The third-grade system was implemented in the spring of 2010, when 16 communities voluntarily used the system with more than 3,000 students. By the end of fiscal year 2012, 79 communities were using the third-grade system, reporting data on more than 12,000 third graders. Reports since 2010 have shown consistent improvements. In 2010 for example, only slightly more than half of students tested could identify the correct Stop-Drop-Cover-and-Roll posture, with hands covering the face. By 2012, more than 70% of students identified the correct response. One firefighter-educator expressed a common response to such findings: “When DFS first rolled out the third-grade evaluation tool, I thought [they] were crazy. Once I used it and saw the results, I was stunned, and I am changing my lesson plans.”
Firefighter-educators have used the data to define their own goals. One found that nearly one-third of students did not know that a family meeting place should be outside of the structure, and only 57% could correctly identify ‘two ways out of every room’ as an important part of an escape plan. Over the next year the firefighter-educator focused on improving escape plan understanding. In 2011, 88% students knew a family meeting place should be outside the building, and nearly 80% knew ‘two ways out’ is an important part of an escape plan.
The sixth-grade system, implemented late in 2012, has not yet produced data on the scale of the third-grade system. The 10th -grade system will be implemented in 2014. But the widespread use of the third-grade system suggests that firefighter-educators now have tools they can and will use to assess their teaching.
In Sum
The system used in Massachusetts for assessing fire and life safety education provides a structured, standardized method for monitoring program effectiveness and measuring and tracking children’s understanding of safety practices. Critical to this effort has been the broad, system-wide support of the MA-DFS. In addition to improving teaching and safety outcomes, the system produces data supporting forceful arguments to continue funding of the S.A.F.E. program, as it confirms that kids are truly learning what we are teaching. All components of the system are available without charge on the MA-DFS website: www.mass.gov/eopss/agencies/dfs/dfs2/osfm/pubed/s-a-f-e/public-education-evaluation.html.
The Massachusetts Public Fire and Life Safety Education Task Force assisted in concept development, test and system revisions, and pilot test site recruitment. Students and teachers in the following Massachusetts communities assisted in testing the evaluation tools: Boston, Everett, Holliston, Holyoke, Hudson, Stow, Upton, Westport, Weymouth and Worcester. Jean Egan, Ed.D., and Janet Ryan, M.Ed,. provided expertise in test design and literacy assessments. Stan Jaskiel provided illustrations for all three tests.
