The Survey | Results | Looking Ahead | Conclusion | References
Evaluating the Evaluator: Volunteer Firefighter Opinion of the Public Protection Classification Program
—James P. Festa
Fire departments provide crucial services to the community, specifically fire protection and response to emergencies. The ability of a fire department to provide these services is evaluated by the Insurance Services Office (ISO), which then issues a rating on a scale of 1 (exemplary) to 10 (no fire protection provided). The ISO uses its Fire Suppression Rating Schedule (FSRS) to determine this rating, taking quantitative account of such factors as department personnel, equipment, and water supply. This rating from 1 to 10 is known as the fire district's Public Protection Classification (PPC). The stated purpose of the FSRS is to review the available public fire suppression facilities, and to develop a Public Protection Classification for fire insurance rating purposes" (1). However, this program also serves fire departments by helping them evaluate their ability to provide fire protection. It is primarily this program that evaluates the quality of fire protection in a community, and this evaluation can affect fire district budget, spending, and insurance rates.
Those who provide a service are perhaps the best judges of its methods of evaluation. A 2001 survey of fire chiefs conducted nationwide indicated that 97% of fire chiefs felt that the ISO's Public Protection Classification was important in helping communities save lives and property (2). However, while fire chiefs command the fire scene, it is the firefighters who physically perform the vital functions of a fire department. Therefore, firefighter evaluation of the PPC's role in helping the community is also important in determining its importance and effectiveness. In order to make a sound evaluation of the standards by which they are measured, firefighters must first possess knowledge of those standards.
This study sought to accomplish two objectives: first, to evaluate how familiar the surveyed firefighters were with the PPC program; and second, to obtain an evaluation by knowledgeable firefighters of the PPC program's effectiveness for helping fire departments protect their communities. Top of page>>
The Survey
In 2002, ninety-one firefighters from seven different fire departments in Saratoga County, New York, were asked to complete a self-administered survey evaluating the ISO's PPC program. I coordinated meeting dates with each respective fire department and was present at the designated meetings in order to distribute the surveys and then to collect them. All of these surveys were completed by the firefighters present during single sessions for each department.
The eight fire departments were selected on the basis of population density and geographical location within the county. County fire officials, for the purpose of administration, divide Saratoga County into four geographical quadrants. Two departments were selected from each quadrant in order to get a geographical sample spread. Of these eight departments, two were selected from each population density quartile countywide. The intended sample spread thus included two fire departments from each geographical quadrant and from each population density quartile. However, one fire department did not participate in the survey and a replacement was not found due to time and resource constraints; hence, the number of departments was reduced to seven. The absent fire department is located in a community that is in the third population density quartile. The seven fire departments provided 17, 9, 18, 8, 13, 14, and 12 firefighters, respectively, producing a sample of 91 firefighters.
In Part I, which consisted of nine close-ended questions, firefighters were asked to evaluate their knowledge of the ISO's PPC program. In Part II, consisting of ten close-ended questions plus one open-ended question, firefighters who indicated knowledge of the PPC program were asked about the program's accuracy in evaluating fire departments and its effectiveness In helping them perform their job. Results were compiled into one data set. The raw responses were tallied for each answer, and the total number of firefighters choosing each specific answer was then converted to a percentage of the total number of firefighters in the survey. Because the survey utilized a selective sample based on geography and limited by available resources, a margin of error is not calculated. Non-respondents were included in the results as non-respondents and not as part of any other category unless otherwise noted. Top of page>>
Results
This survey yielded results that are significant to the fire service and to the insurance and public administration communities. These firefighters were largely unfamiliar with the PPC program and its rating system, and only about a third (34%) of those surveyed knew the PPC assigned to their district by the ISO. However, most (74%) would care to know more. The firefighters surveyed indicated that it should be the responsibility of both the ISO (88%) and their fire department (82%) to educate them about the PPC system. Firefighters responded that the most helpful ways to learn about the PPC program were (in descending rank order) presentations given by the ISO at local fire departments, the distribution of educational/explanatory materials by the ISO to fire departments, classes given by the fire department itself, classes given by the county training authority, and classes given by the state. While half of the responses indicated that all of the five ways would be helpful, a significant proportion indicated that department classes (15%) and state classes (15%) would not be helpful at all. The final question asked in Part I was about familiarity. Only one fifth (18%) of firefighters responded that they were somewhat familiar or very familiar with the PPC program. Almost two thirds responded that they were either not very familiar (37%) or not familiar at all (26%).
Because Part II was focused on informed opinions, only those who considered themselves to be very familiar or somewhat familiar were asked to complete it. This reduced the sample size from 91 to 18. Only a few Part II respondents (16%) indicated that the PPC program in general was very accurate; while 61% said it was only somewhat accurate, and 6% said it was not accurate at all. However, when asked about their own department's assigned number, one third responded that it was very accurate, and 44% that it was somewhat accurate; no respondents said that it was not accurate at all. Forty-four percent of respondents indicated that the PPC program was very (33%) or extremely (11%) important in justifying changes to their district's fire protection. The effect such changes would have on their PPC number was felt to be very (44%) or extremely (17%) important.
As for the effect the PPC program has had on their district's fire protection, 39% said that fire protection had increased and 44% said there was no change. No respondents said that fire protection had decreased. Responses about the degree to which the PPC helps in providing fire protection were mixed: 17% said very effective, 56% said somewhat effective, and 22% said not effective at all. Thirty-nine percent of respondents did not know what kind of effect the PPC has had on insurance rates in their district, while 44% saw no change, six percent an increase, and 11% a decrease in rates. Eleven percent indicated that the PPC does not help their department evaluate itself against a national standard, while 44% said it is somewhat useful, 17% found it very useful, and 6% responded that it was extremely useful for that purpose.
The PPC program was somewhat understandable for 56% of respondents, but very understandable for only 17%; no respondents felt that it was extremely understandable. When asked to comment on the strengths and weaknesses of the PPC program, only a third of the subjects wrote comments in the space provided. These comments included "not explained clearly by ISO rep.," "probably outdated," "accuracy is not the point, relevance is," "a lot of energy for little change," and "encourages fire department to look at its fire protection effectiveness." When asked if any weaknesses in the system affect the program's accuracy, six percent responded that weaknesses had an extreme effect on accuracy, 11% saw a significant effect, and 22% a slight effect. Six percent said there was no effect on accuracy, and no subjects felt that the PPC program had no weaknesses at all. Top of page>>
Looking Ahead
Three primary aspects of these survey results could be further developed in future research, specifically the sample selected, the questions asked, and the investigative scope. Although this purposive study included 91 respondents for part I, this sample size could be increased. Participants could also be chosen by random sampling rather than by selective sampling, as was done due to time and resource constraints. Future research should also include more than 18 knowledgeable respondents for evaluative opinion questions such as those in Part II. More open-ended questions might be included so that firefighters could comment on broader aspects of the PPC program. A third limitation is the scope of the research: it only focused on the issue from the perspective of the firefighter. To gather a more encompassing view of the PPC program's use, relevance, and effectiveness, future research could investigate the program through the perspectives of the town resident, the public administrator, the insurance assessor, or the ISO. Top of page>>
Conclusion
According to the volunteer firefighters surveyed in Saratoga County, NY, the Public Protection Classification program is only somewhat effective in helping fire departments provide fire protection, and it is only somewhat accurate, in general. Still, the program is one of the only objective measures of a fire department's quality, and while some raise questions about the current relevance of the PPC program, insurance companies, as well as current and prospective district residents, evaluate departments by the PPC rating (3). Therefore, it is essential that fire department personnel become more familiar with the program; only twenty percent of the volunteer firefighters surveyed considered themselves somewhat or very familiar with it.
Most surveyed firefighters believed that both their own departments and the ISO have a responsibility to educate them on the ISO and its PPC program, which suggests that more could be done by the ISO and by local departments to address the lack of familiarity with the program. As for what could be done specifically, firefighters overwhelmingly responded that classes and information materials given by the ISO would be most effective in helping them learn more about the PPC program. Perhaps if these educational methods were utilized, more firefighters would become informed about the PPC program and be better able to collectively focus on ways to improve the quality of fire protection in their communities. The ISO could then be more effective at helping fire departments provide the fire protection that is so vital to the communities they serve. Top of page>>
I wish to thank Dr. Cliff Wirth for his guidance during all stages of this project, as well as the firefighters of Saratoga County, NY, for making this project possible. Read about the author>>
References
1. Insurance Services Office, Fire Suppression Rating Schedule, Edition 08-98, 2001, 1.
2. Opinion Research Corporation International, January 12-19, 2001, study# 33343 [press release online] http://www.iso.com/press_releases/2001/02_21_01.html
3. Interview, Chief Sean Dooley, Quaker Springs Volunteer Fire Dept., Stillwater, NY, June 2002. While Chief Dooley acknowledged that residents had called him to inquire about the PPC number for the fire district, he noted that it was a rare occurrence during his tenure as Chief.
