Fire Assessment: Verification Calibration and Validation

1. Reference Fire History
   • Wildland Fire Library
   • Near Real-Time Remote Sensing Products
2. Key Spatial Analysis Factors
   • Landscape (LCP) Characteristics
   • Crown Fire Potential
   • Spotting Spread Potential
   • Burn Period
   • Burning Thresholds and Burn Day Frequency
   • Wind Speed and Direction
   • Weather Observations and Forecasts
   • Dead Fuel Moistures
   • Live Fuel Moistures
3. Evaluating and Adjusting Spatial Assessments
   • Verification of Analysis Models
   • Analysis Adjustment and Calibration
   • Validating Spatial Analyses

Reference Fire History

Wildland Fire Library 
Evaluate reference fire history and other local knowledge for insight into current fire situation.

 

Near Real-Time Remote Sensing Products

• MODIS, VIIRS, GOES, and Firehawk IgPoint Fire Detections are available on the Enterprise Geospatial Portal (EGP), in WFDSS Situation and Analysis Maps, and from Google Earth tools.
• National Infrared Operations (NIROPS) Interpreted Infrared Products may be produced to support incident specific needs.
• Web-Cams and FLIR resources may be assigned to specific incidents, or in some cases, available publicly from airports and other locations.

Return to Top

Key Spatial Analysis Factors

Consider what analysis settings and inputs can be used to represent real factors influencing current and expected fire behavior.

Landscape (LCP) Characteristics

The fuel model and canopy selections provide important influences on all of the factors that follow. In some areas, default LANDFIRE data need evaluation primarily for recent changes (burn scars, other significant disturbances) and barrier depiction. In other cases, significant editing of fuel models and/or canopy characteristics may be needed to reflect seasonal changes and errors in classification.

Crown Fire Potential

If the landscape (and the analysis) present potential crown fire spread events, growth estimates need to reflect significantly different spread rates. As currently configured, the models present three different approaches for estimating crown fire growth and behavior:

• In the spatial analysis tools, there is a choice between two different implementations of the Van Wagner transition model used to select surface, passive crown fire, or active crown fire for fire behavior estimation (Comparison Table). The Finney method, produces less active crown fire and models passive crown fire with the spotting model only. The Scott & Reinhardt method, produces more active crown fire and models an intermediate spread rate when passive crown fire is anticipated.
• There are surface fuel models (generally in the shrub category) that have been touted and used to represent crown fire behavior.

Spotting Spread Potential

Estimating spread potential across barriers and from other random movements (e.g. rollout) generally requires a stochastic model that factors in frequency potential, or probability. The spotting models implemented in spatial analyses provide a frequency modeling of the firebrand production and movement as well as the likelihood for a new start.

Burn Period

The NWCG Fire Glossary defines the burn period as that part of each 24-hour period when fires spread most rapidly; typically from 1000 hours to sundown.

In most cases, the burn period refers to the period when fire is actively spreading at the head of the fire. If the six categories of visual fire behavior are considered, the 24-hour day includes all of them. Field Observers should be careful to report/describe their estimate of burn period accurately and purposefully and relate it to the type of fire observed:

• Smoldering fire behavior continues around the clock for most active fires. It does not represent any part of an active burn period if reported at the head.
• Creeping fire behavior may continue through the night, but is generally transitional between smoldering and running fire behavior. Generally, it produces little overall fire spread and is not considered part of the burn period if observed at the head of the fire.
• Running and Torching/Spotting fire behavior describes what is encountered during the burn period on most days when fire spread and overall growth is low to moderate. However, it may represent transitional fire behavior when more significant Torching/Spotting or Crowning fire behavior occurs during peak hours.
• On days when Active Crowning fire behavior is predominate, the burn period used for analysis should generally reflect only the period of significant spread.

Burn period can vary from day to day for a variety of reasons:

• Solar Radiation heats fuels as well as warming the air and lowering relative humidity. These influences lower fuel moisture, creating conditions favorable for active burning. Affected by the sun angle based on the time of year and latitude. Cloud cover and canopy shading can further reduce solar radiation.
• Fuel bed Characteristics can influence burn period as well. Moisture content of light fuels, such as grasses, respond more quickly to changes in temperature and humidity.
• Diurnal Fuel Moisture Trends are affected by the quality of night time humidity recovery and inversions. Slope/aspect and recent precipitation all affect the length of the burning period for a given situation.
• Drought can influence the length of the burn period through the heat produced in the burning of heavy fuels.
• Direction of Spread can be an important factor as well. Backing spread can start later and end earlier in the day for a given situation.

In the validation of your estimate, there are tools and criteria that can help identify when the burn period starts and ends.

• Fireline Observations are probably the first and most important source of information for determining the burn period. Try and get answers to specific questions as you pursue a reasonable estimate. When and where did fire begin to move and when did it slow down on previous days? Was there significant spread during the night? What were observed spread rates and when?

Sometimes these reports are incomplete and need to be correlated to other information as suggested below. FSPro seeks burn period information for different types of days. These factors suggest that fireline observations should be reinforced with these other information sources where possible.

• Sunrise-Sunset Tables (time of year and latitude) from BehavePlus and solar radiation sensors can show periodicity and suggest timing of beginning and end of active spread.
• Diurnal Wind, Weather and Fuel Moisture Trends can similarly show a periodicity that can suggest timing of active spread. Graphs displaying these trends are readily available at MesoWest.
• Fire Progression Maps suggest the overall daily spread around the fire, and with knowledge of weather conditions, fuels, slope and spread direction, can be compared to modeled growth. A new resource called the Wildland Fire Library provides a variety of historical references including fire progression maps.

WFDSS Help suggests that “The default burn period in NTFB is 24 hours; however, modeling a fire overnight is generally not advised. NTFB, like FARSITE, has a tendency to over-predict overnight fire spread. For this reason, most analysts shorten the duration that the modeled fire is allowed to burn each day.”

Each fire growth projection, whether using non-spatial tools (BehavePlus) or spatial tools (WFDSS analyses NTFB and FSPro) specify a duration as the number of hours or minutes to obtain a resulting fire size and/or perimeter. Characterizing the duration as the number of hours or minutes in a day (burn period) for a projection allows the user to model growth for multiple days.

Burning Thresholds and Burn Day Frequency

Most of the area burned across any management unit, geographic area, or ecological region occur on relatively few days over the life of the fires that are included in them. Burn Day evaluation can help identify which days have conditions receptive to significant fire spread and can serve as indicators of important transitions in fire behavior.

The concept recognizes that there are thresholds in environmental conditions below which significant spread is substantially limited, even when fire spread models estimate that spread will continue. A simple classification of days into [Burn Days] and ‘non [Burn Days]’ can greatly improve estimates of area burned. (Podur and Wotton, 2011). There can be important within and between season variation in this pattern.

• WFDSS FSPro analysis defines a threshold ERC for each analysis, below which no fire spread is modeled for the entire day. Among its outputs, a distribution of burn days spread across the ERC classes is provided. It is possible to define daily environmental conditions favorable to significant growth and use those to estimate the frequency with which they occur to adjust the default 60^th percentile ERC as that threshold (Ziel, 2015).

• WFDSS NTFB analysis requires input of a burn period for each of the included days. Specifying same start and end hour for a given day essentially defines it as a non-burn day. The same daily conditions mentioned for FSPro can be used as criteria to exclude specific days or adjust burn period in NTFB analyses.

Wind Speed and Direction

With a sparse network of RAWS stations that provide hourly reports, it may be difficult to find a weather station that represents both wind speeds and directions appropriately for the analysis. Review observations and forecasts carefully for both speeds and direction. Try analyses with alternatives being considered.

Weather Station climatology in FSPro analysis is generally restricted to RAWS installations. For that reason, many analysts choose a combination of 10-minute average and gust wind speeds in those climatological distributions. This results in approximately a 50% increase in wind speeds. ASOS/AWOS climatologies often reflect this higher wind speed. View wind roses for ASOS/AWOS stations. This difference is often found in NDFD forecast wind speeds as well. Consider a similar adjustment to 10-min RAWS winds when conducting calibration analyses in STFB and NTFB.

Weather Observations and Forecasts

In most cases, forecasts have difficulty outperforming climatology beyond 48-72 hours. Most important are confidence in wind and precipitation elements. Read the area forecast discussion for insight into the confidence horizon.

Extend STFB and NTFB analyses only to the limits of very high confidence, usually 1-3 days. FSPro Analyses combine forecast and climatology. Extend use of forecast inputs out beyond 3 days only if confidence for the extended forecast is unusually high. Critically examine the wind speed and direction in the forecast.

Dead Fuel Moistures

Fine dead fuel loading and moisture generally determine day-to-day variability in flammability in specific locations. Generally, fine fuel moistures are set by weather inputs. However, they may not reflect sufficient range in an analysis period to produce the observed variability. Evaluate landscape distribution in 1 hour and 10 hour fuel moistures using the Basic Outputs from Basic and STFB analyses. Consider the range of variability in these fuel moisture categories applied in the ERC table of FSPro analyses.

Live Fuel Moistures

Among all fuel moisture categories, live fuel moisture levels can have a dramatic influence on results in fuel models that include them. Among the original 13 fuel models, the most dramatic effect is in the shrub models FB04 and FB07. With the inclusion of the 40 Scott and Burgan fuel models, herbaceous (grass) and shrub live fuel loads have been included separately. With these, the most dramatic effects have been shifted to fuel models that include herbaceous fuel loads (Grass, Grass/Shrub, some Shrub and some Timber Understory).

In these fuel models with live fuels loads, herbaceous fuel moisture trends establish curing levels and transfer of herbaceous fuel loads to dead fine fuel loads.

• Where curing is a critical cue to increasing fire potential, live fuel moisture levels can be set to simulate current conditions. New NFDRS tools for estimating live fuel moisture can be useful in setting current levels.
• Where critical thresholds for active fire behavior can be reached in green landscapes with growing vegetation, NFDRS estimates generally produce high live fuel moistures. This usually overstates its influence as a heat sink and can confound modeling efforts. In these environments, default estimates are often adjusted downward in modeling efforts. Consider adjustments for only fuel models that represent fuel beds that are burning actively.

Return to Top

Evaluating and Adjusting Spatial Assessments

"You have to study, figure out what [the fire is] seeing and represent what is the fire seeing and how is it responding to the various factors in reality. And then can you replicate that for the right reasons with the model? Can you get pretty close to it? If you're not getting the right answer for the right reasons, I don't care how pretty your output is..." Mark Finney, Research Forester

Verification of Analysis Models

Verification is a demonstration that the modeling formalism is correct.

Though much of the verification work was completed in the development of a model, end users have a responsibility to evaluate large scale factors to ensure that the range of variability used produces the range in results that generally fit observed outcomes. Learning regional preferences and default analysis inputs are an important first step.

Consider the static factors applied to analyses:

• Is crown fire an issue for this fire? What methods and spotting inputs are preferred in the region?
• Are there standard landscape edits identified for the area?
• How are slowly changing seasonal and drought factors typically represented there?

Analysis Adjustment and Calibration

Calibration is the estimation and adjustment of the model parameters and constants to improve the agreement between model output and a data set.

Persistence prediction estimates applied from observations in the recent past and comparison of inputs with forecast elements are the most common form of fire behavior calibration. They can be applied as spread event size, fireline intensity, and crown fire behavior, or as specific adjustments to input elements (e.g. local wind speed and direction).

Otherwise, careful evaluation of important input factors should allow the model to reflect the variation in conditions truly influencing the fire.

Otherwise, careful evaluation of important input factors should allow the model to reflect the variation in conditions truly influencing the fire.

Calibration Input Factors to Consider (categories Bold,; edit locations in normal text):

Steps to consider:

• Consider including last known day within the forecast analysis for quick, embedded validation.
• Edits to Large Scale Constants first.
  • Fuel Model & Canopy Layers can require local edits due to disturbance, classification errors, and possibly seasonal trends. These fuel characteristics can have a large impact on outputs. It is CRITICAL to account for things that will check or stop fire spread, so ensure that recent historical fires are reflected in the LCP. Also, make sure that rock and water are adequately mapped. However, other fuel layer adjustments should be considered only when the changes you contemplate are appropriate and will result in significant changes in fire growth. Otherwise, edits may mask necessary variability.
  • Barriers are critical factors in anticipating fire growth.
  • Crown Fire Method and Spotting Frequency are frequently settings applied consistently with experience in particular landscapes. As these become standard, they become verified.
• Edits to seasonal factors as well as medium and large scale Day-to-Day factors.
  • Burn Period Length and Burning Threshold edits should be consistently applied according to seasonally changing day length, in view of drought assessments, and according to daily variability when limiting burn period.
  • In most situations, Herbaceous and Woody Fuel Moisture trends should reflect reasonable ranges for season, drought and curing levels. Where active fire is anticipated in green growing landscapes, it may be best to identify fuel beds that are most prone to active burning and edit live fuel moisture specifically for those fuel models.
• Edits to diurnal weather-related inputs (temp, rh, wind speed/dir, cloud cover) should be limited to standard kinds of adjustments to preserve relevance of observed calibrations to forecast projections.
  • Wind Speed and Direction is most important factor. Ensure that observed portion of weather stream reflects observed fire spread and that winds observations are appropriate for situation. Precipitation estimate may not represent fireline amounts. Consider adjustments carefully.
• Calibration may require a couple iterations.
  • Do not over-calibrate to over fit the result.

Validating Spatial Analyses

Validation is a demonstration that a model, within its domain of applicability, possesses a satisfactory range of accuracy consistent with the intended application of the model.

Validation is used to evaluate forecast projections from applied calibration adjustments. Here are some methods and some considerations:

To determine if analysis prediction should be accepted/completed:

1. In any validation effort, evaluate weather forecast first to frame consideration of analysis prediction results.
   • Compare winds, cloud cover, and precipitation in the analysis forecast to other forecasts you are using.
   • Check forecast portion of ERC Streams in FSPro. Are the number of days appropriate? Are the wind forecasts reasonable?
   • Is the analysis duration appropriate with the given forecast confidence for STFB and NTFB?
2. Evaluate ignition locations/sizes and barrier locations for appropriateness. They may be too large or insufficient.
3. Compare analysis inputs and adjustments from the analysis (using the table above) to known factors influencing fire growth and behavior.
   • Review notes in each tab for the analysis to determine what adjustments were made and why.
4. Critique the results. Are they surprising? Are they useful? How will you brief them? Judgment and experience are important.
   • FSPro: The Big Red Cloud and the Small Rare Event.
   • NTFB/FARSITE: Review Other Characteristics (flame length, spread rate, crown fire activity) for reasonableness.
   • STFB/FLAMMAP MTT: Look at Basic Outputs.
5. Document your conclusions in the analysis notes.

When analyses need updating or have expired:

1. Compare weather forecast or climatology in the analysis to observed weather to determine how that influenced analysis accuracy.
2. Compare subsequent observed perimeters with predicted perimeters and flow paths for the concluding date of the analysis period. In FSPro, there may be benefit to comparing both the concluding observed perimeter and the observed perimeter at the date where the forecast stream concludes.
   • Do predicted perimeters and flow paths for STFB and NTFB represent actual spread directions and produce a reasonable representation of the size of observed spread event?
   • Does NTFB predictions reflect day-to-day variability effectively?
   • Did the probability contours in FSPro provide useful insight for contemplated strategies and priorities for planned actions?
   • Does the analysis effectively represent fire behavior observed over the period (surface, spotting, and crown fire behavior)?
3. Critique levels and adjustments to inputs factors that relate most closely to real influences and variations seen over the analysis period.
   • Check Ignition and barrier files. Are they responsible for significant differences?
   • Evaluate burn periods, burning thresholds and burn day frequencies to develop a sense of day-to-day and diurnal variability in the analysis and on the ground.
   • Are the seasonal factors effectively representing observed departures from normal levels?
4. Update Analysis notes with the evaluation.