Identifying Users, Diagnosing Understandability Challenges, and Developing Prototype Solutions for NOAA Climate Prediction Center’s Seasonal Climate Temperature and Precipitation Outlooks
Environmental Decision Support Science
Melissa A. Kenney
Michael D. Gerst
NWS NCEP CPC OSTI Modeling Branch
2017 ANNUAL REPORTS
The National Oceanic and Atmospheric Administration (NOAA) National Weather Service Climate Prediction Center (CPC) provides future-oriented scientific information on various climate outcomes in the range from one week out to one year. This information can be viewed as a leading indicator, as it provides predictive, albeit uncertain probabilistic information tracked over time that is intended to be useful for decision-making. Structuring the delivery of this information is challenging for many reasons, foremost being that visualizing uncertainty for geospatial data has been an open area of research for many years (MacEachern, 1992). However, careful attention to delivery of uncertain information is of high importance because incorporating uncertainty into decisions can be challenging, even when uncertainty is small. Larger uncertainties compound this problem because users often conflate larger uncertainty with less utility of the information to decision making (Montibeller and von Winterfeldt, 2015). In addition, CPC publishes predictions for temperature, precipitation, and drought as well as a synopsis of potential hazards. Each type of prediction engenders its own uncertainty characteristics and feeds into decision-making differently. Consequently, web-based information delivery must take into account these aspects so that users can find information efficiently and understand its meaning.
Over the two-year period, research activities that will inform the potential adjustments or revision of CPC’s temperature and precipitation outlook visuals shall be conducted by the research group at the Cooperative Institute for Climate Studies (CICS) in order to determine how to better communicate risk and uncertainty associated with the forecasts. The work will be divided into two components: (i) assessment of individual visualization of CPC forecasts and (ii) overall product design.
Phase 1: Initial Scoping of User Communities
The first phase proposed to assess existing CPC visualizations is to understand CPC’s current and aspirational goals of temperature and precipitation outlook images. The Environmental Decision Support Science Lab (indicators.umd.edu), led by Kenney and Gerst, identified the objectives and intended user audiences through semi-structured interviews with eight NOAA experts identified through CPC. In close collaboration with CPC, we developed an interview protocol that focused on participants’ perception of how the products are designed and how well they work / their utility. For example, interviewees viewed a variety of CPC precipitation and temperature outlook products, such as the 8-14 Day Precipitation Outlook below. The interviews were conducted from June to August 2016. Of the eight interviews, five were held in-person and three were held via web conference using Adobe Connect. The interviews ranged in length from 40 minutes to 1.5 hours. All interviewed experts were affiliated with Federal agencies. Seven participants were affiliated with NOAA, and one was affiliated with the United States Department of Agriculture. The individual interviews were transcribed and then sent to the interviewee via e-mail where they were provided with the opportunity to amend the transcript. The transcripts were then coded using NVivo software to identify information and themes. The majority of the participants volunteered to connect us to identified user communities, if it would be useful in future research phases. There was also general support for this research and the use of it to improve the outlook products. Key findings from Phase 1 are that boundary organizations and science translators are significant users of CPC outlooks and the current outlook products are not well understood by users.
Figure 1: 8–14 Day Precipitation Outlook
Phase 2: Identifying End-user Audiences
Key to improving the climate outlooks is identifying the end-user audiences and the potential or intended uses of the climate outlooks for their own understanding and use in a range of different decision-making contexts. Phase 2 will focus on the target end-users identified by CPC—the organizations that are between the level of the science translator, identified in Phase 1, and the level of the individual user. Specifically, we focused on the target end-users in four sectors identified in close collaboration with CPC. These sectors include emergency management, water resource management, agriculture, and energy. Through the interview processes of Phase 1, we have identified a small set of boundary organizations and science translators within these organizations that will provide initial contacts to the target end-users and their networks. Additional target end-users will be identified through a combination of methods such as: 1) snowball sampling of science translators identified in Phase 1, 2) secondary document analysis and Internet searches where decisions, information needs, or CPC products are presented and discussed or are eluded to. We completed a preliminary build-out of a database for all four sectors. Within all four sectors, we conducted pattern analyses of the identified entities to identify trends and gaps amongst characteristics for the entities including the types of decisions, timescale of decision-making, stakeholders, and point(s) of contact.
● Using a short survey (≤15 minutes) distributed widely to target end-users to the four focus sectors (using methods identified in Phase 2), we will assess both ends of this context. In addition to the survey, we will conduct one focus group with target end-users from each of the four sectors. The focus groups will emerge one decision type or decision context of interest to CPC (Phase 3, July 2017).
● The results of the CPC expert and user elicitations will provide the basis on which to diagnose any mismatch between existing visualizations and user needs (Phase 4, July 2017).
● After diagnosing visualization problems for different end-users, the climate outlooks will be redesigned for 1-3 user audiences (Phase 5, November 2017).
● After redesigning the visuals, CICS will test the effectiveness (both in terms of improved understandability and affect) of the redesigned climate outlooks (Phase 6, February 2018).
● We will develop the results both into peer-reviewed papers to be published in journals such as, Weather, Climate and Society, and recommendation for CPC in the form of technical reports (Phase 7, April 2018)
Publications (Publications Relevant or *Funded by this Grant)
1. Gerst, M.D., M.A. Kenney, A. Baer, J.F. Wolfinger et al. (2017) Effective Visual Communication of Climate Indicators and Scientific Information: Synthesis, Design Considerations, and Examples. A Technical Input Report to the 4th National Climate Assessment Report.http://www.umdindicators.org/?page_id=1028
2. Gerst, M.D., M.A. Kenney, J.F. Wolfinger, I. Feygina. Increasing understandability of visual climate indicator graphics. in prep
3. * Gerst, M.D., M.A. Kenney, A.E. Baer. Identifying Users, Diagnosing Understandability Challenges, and Developing Prototype Solutions for NOAA Climate Prediction Center’s Temperature and Precipitation Outlooks. in prep during Y2
1. Phase 1 Initial Scoping Think Aloud Interview Protocol used during semi-structured interviews with NOAA experts identified through CPC.
2. Phase 1: Identifying Users, Diagnosing Understandability Challenges, and Developing Prototype Solutions for NOAA Climate Prediction Center’s Temperature and Precipitation Outlooks memo
3. Options for Phases 2-4: Identifying Users, Diagnosing Understandability Challenges, and Developing Prototype Solutions for NOAA Climate Prediction Center’s Temperature and Precipitation Outlooks memo
4. Initial database for water, energy, agriculture, and emergency management end-user organizations
1. Kenney, M.A., M.D. Gerst, A.E. Baer. (2016) Phase 1 Results on the Understandability Challenges for Non-scientific Users of the Climate Temperature and Precipitation Outlooks. National Oceanic and Atmospheric Administration (NOAA) Climate Prediction Center, College Park, MD.
2. Kenney, M.A. (Fall 2016) Effective Scientific Information Translation to Support Environmental Decision-making. University of Maryland, College Park, MD. College of Information Studies.
3. Kenney, M.A. (Spring 2016) Environmental Decision Support Science to Support Complex Water Resources and Climate Decision-making. Tufts University, Medford, MA. Civil and Environmental Engineering.
4. Gerst, M.D., M.A. Kenney (November 14, 2016) Using visualization science to diagnose and improve global change indicator understandability. Institute for Operations Research and the Management Sciences (INFORMS), Nashville, TN.
Invited Workshop Presentations and Participation
1. Business Case for Resilience Roundtable. (November 2016) Invited Participant. Sponsored by White House Council on Environmental Quality. Washington, DC.
2. Water-Energy-Food Nexus Roundtable. (August 2016) Invited Participant. Sponsored by White House Office of Science, Technology, and Policy. Washington, DC.
3. Public Engagement Training for Leshner Leadership Institute Fellows. (June 2016). Sponsored by American Association for the Advancement of Science (AAAS) Leshner Leadership Institute. Washington, DC.http://www.aaas.org/pes/leshner-fellows
9-2-16 Kenney Attends White House Roundtable:
CICS-MD Scientist Melissa Kenney participated in the Water-Energy-Food Nexus Roundtable sponsored by the White House Office of Science and Technology.
It was held on August 25th in Washington, DC. The issues discussed included Alternative Water and Water Reuse, Biofuels, Flexible Grid Technologies, Food Waste, and Wastewater Treatment. Importance:Decision support science allows NOAA to disseminate its climate, weather, coastal, and marine data directly to policymakers. POC: M. Kenney
9-30-16 Kenney Quoted on Outreach to Students:
“AAAS Lessner Fellows help their communities to confront climate impact” is a news item in the September 29th issue of Science. CICS-MD Scientist Melissa Kenney (supporting OAR/CPO), a 2016 Lessner Fellow, is quoted as saying:
“Fellow Melissa Kenney, assistant research professor in environmental decision support science at the University of Maryland, worked with undergraduate students this summer to synthesize the challenges and informational needs of community representatives in the Chesapeake Bay by reviewing existing reports and other documents. "We wanted to do our homework so we didn't ask a lot of questions that were already answered," Kenney said. She and her team are looking for "research gaps and science translation opportunities" to approach community leaders with processes and tools that will "facilitate evidence-based decision-making….In a densely populated watershed, climate is amplifying existing impacts on our region," added Kenney. "So we don't have a choice; we have to make decisions to consider current and future climate impacts."
See the full article at http://science.sciencemag.org/content/353/6307/1508. Kenney (OAR/CPO/ASD) Kenney is the Task Leader on the Task entitled “Research, Development and Implementation of Physical, Ecological, and Societal Climate Change, Impact, Vulnerability and Preparedness Indicators for the United States.” Importance:Training NOAA's next generation of scientists helps NOAA sustain a world-class workforce. POC: M. Kenney
1-27-17 Kenney’s CICS Research Highlighted in Decision Analysis Today:
CICS-MD Scientist Melissa Kenney (OAR/CPO/ASD) and her NOAA-funded research on Climate Change Indicators were featured In the December 2016 issue of Decision Analysis Today, a publication of the INFORMS Decision Analysis Society.
Her work was the first in a new series “to highlight work at the intersection of practice and research, looking at decision analysis research that is impacting other fields or applications, fundamental research that can have an impact on practice, or great work in practice that can stimulate research ideas.” Merrick, Jason, 2016: President’s Letter, Decision Analysis Today, 35(3), 1–2, https://www.informs.org/Community/DAS/Newsletter. Importance:Applying NOAA’s scientific research to policy decision-making is an important element of “climate intelligence.” POC: M. Kenney
Advising and Mentoring: Graduate Students
1. Stärk, Johanna, Max-Planck Odense Center, 2016-present (dissertation committee)
2. Reyes, Julian, Washington State University, 2014-present (mentored dissertation chapter)
3. Anderson, Sarah, Washington State University, 2013-present (mentored dissertation chapter)
4. Prava, Venkat, Johns Hopkins University, 2010-2016 (co-advisor, dissertation committee, received two awards by the Decision Analysis committee for co-authored paper resulting from dissertation)
5. Salehittal, Akshata, University of Maryland, 2016-present
6. Larger, Eric, University of Maryland, 2015-2016
Advising and Mentoring: Undergraduate Students
1. Cavanagh, Meg, University of Maryland, 2016-present
2. Speciale, Amanda, University of Maryland, 2016-2017
3. Jaffee, Natalia, University of Maryland, 2016
4. Sproul, Dean, University of Maryland, 2016-present
5. Winik, Jason, University of Maryland, 2016-present
6. Veselka, Camille, University of Maryland, 2016
7. Bell, Julia, University of Maryland, 2016
8. O’Connor, Beatrice, University of Maryland, National Socio-Environmental Synthesis Center (SESYNC) Internship, 2016
9. Li, Angelica, University of Maryland, National Socio-Environmental Synthesis Center (SESYNC) Internship, 2016
10. Truitt, Elisabeth, University of Maryland, 2016
11. Sadecki, Nicole, University of Maryland, 2016