Challenge to Develop and Demonstrate the Best New User-Oriented Forecast Verification Metric
The public, industry, emergency managers and other decision makers can use weather, climate and impact forecasts more effectively in their decision making when the quality of the forecasts is measured in terms that are meaningful to them.
Yet few metrics exist to measure forecast quality in user-relevant terms. A good example, developed by the UK Met Office in consultation with the aviation industry, is the Flight Time Error which measures the difference between the actual flight time taken by an aircraft and the time estimated to "fly" the plane along the same path through the predicted wind field. Errors in arrival time are meaningful to airlines, who can calculate the costs associated with touching down early or late.
To encourage the development of user-oriented verification approaches, the World Meteorological Organization's Joint Working Group on Forecast Verification Research (JWGFVR) is conducting a challenge to develop and demonstrate new user-oriented forecast verification metrics. The contest is in support of the WWRP projects on High Impact Weather (HIWeather), Subseasonal to Seasonal Prediction (S2S), and Polar Prediction (PPP).
The challenge will consider all applications of meteorological and hydrological forecasts that are relevant to user sectors such as agriculture, energy, emergency management, transport, etc. The metrics can be quantitative scores or diagnostics (e.g., diagrams), but they must be new to be considered for the prize.
Criteria for judging the best new metric include:
- user relevance
- simplicity and ease of computing
- resistance to hedging
Metrics that have a clear statistical foundation and can be applied to a broader set of problems are especially encouraged. The judging panel will comprise members from the JWGFVR and WWRP Societal and Economic Research and Applications (SERA) working groups and representatives from user sectors.
Entries may be submitted by individuals or teams. Multiple entries are permitted. The winning entry will be awarded with an invited keynote talk and one paid attendance at the 7th International Verification Methods Workshop in 2017. All participants will be invited to submit abstracts to this workshop.
The timeline is:
- Challenge begins – 10 September 2015
- Deadline for completed entries – 31 October 2016
- Announcement of winner – January 2017
We hope you will participate in the challenge to develop and demonstrate new user-oriented forecast verification metrics, by developing a new metric or by reviving and applying old user-oriented metrics.
If you have any questions about the challenge please contact the Verification Challenge Organizing Committee at firstname.lastname@example.org.
An entry form is provided below:
Entry Form Deadline: 31 October 2016
Flyer to download
Verification Challenge Organizing Committee
Beth Ebert, chair, BOM
Barbara Brown, NCAR
Martin Göber, DWD
Thomas Haiden, ECMWF
Nanette Lomarda, WMO
Marion Mittermaier, Met Office
Pertti Nurmi, FMI