CENIC Members Doing the Impossible in Weather and Climate Forecasting with CENIC AIR and Advanced Networks

The West Coast of the US has a history of seemingly endless periods of drought that conclude suddenly with unpredictable torrential downpours. Today known as “atmospheric rivers,” these were seen as randomly occurring events that could only be recognized days after they had occurred.

Then about a decade ago, a group of climate scientists, physicists, data scientists, and computational scientists recognized that there had to be a better way to predict these seemingly random deluge events that could devastate unprepared communities.

Among that group was Scott Sellars, a postdoctoral scholar at the Center for Western Weather and Water Extremes (CW3E) at UC San Diego’s Scripps Institution of Oceanography (SIO). In 2017, while presenting at a CENIC conference, Sellars noted that it took weeks to months for scientists to analyze multi-Terabyte datasets.

However, what was once months has now shrunk to minutes, even seconds, and heralded a complete transformation in atmospheric and climate sciences and human preparedness in the face of a changing Earth.

From Treading Water to Swimming in a Sea of Data

Initially, the sheer volume of information was almost a liability for researchers. NASA could see and measure clouds forming over the Pacific Ocean, NOAA had data on the height and strength of those cloud formations, and the National Weather Service had data on wind speeds and direction.  Sellars and his colleagues from UC Irvine and UC San Diego’s California Institute for Telecommunications and Information Technology (Calit2) knew exactly where these vast datasets lived.

But those Terabytes of data were dispersed and disconnected, and required up to 20 days of compute time to download and process.  That they could be processed at all was something of a miracle, but a 20-day lag time was far too long to make meaningful weather forecasts. Predictions rapid enough to act on require advanced machine learning, cutting-edge AI tools, a powerful research and education network, and the compute and storage resources to pull it all together—and only the first steps were being taken to make them a reality.

Fortunately, California’s research institutions had access to CENIC’s California Research and Education Network (CalREN); the fledgling Pacific Research Platform, which launched in 2015; and many of the world’s most talented data and computational scientists.  At last, those steps were being taken.

“What Scott and his colleagues achieved by 2017 was to use machine learning to figure out starting points for precipitation objects,” said Larry Smarr, UC San Diego Professor Emeritus and Founding Director of Calit2. “And the big breakthroughs wouldn’t have been possible without CENIC’s network connecting these endpoints at UC San Diego, UCSD's Scripps Institution of Oceanography, and UC Irvine.”

20 Days to 20 Minutes with AI and Advanced Networks

As is always the case, true breakthroughs in innovative science were based on critical connections among not just equipment but also people—all facilitated by CENIC. The scientists involved at the time included Brian Kawzenuk, a hydro-meteorological research analyst at CW3E; Tom DeFanti, senior research scientist at Calit2; Phy Nguyen, adjunct professor of civil and environmental engineering at UC Irvine; John Graham, research engineer at Calit2; Larry Smarr, networking and data science luminary; and Sellars.

This multi-institutional group developed the CONNected objECT (CONNECT) algorithm, which was first applied to historical data to better understand why certain weather events had taken place in the past.

While the early Pacific Research Platform (PRP)—later the National Research Platform (NRP)—provided the needed compute power, the greatly improved speed of the data download challenged the algorithm itself. In fact, a dataset download that had once taken 20 days now took a mere 20 minutes, finally enabling predictions of atmospheric rivers.  The download speed had ceased to be the bottleneck in understanding to such an extent that the algorithms had to evolve to keep pace instead.

“What PRP and CENIC did [in 2017] is create a platform that sped up the data downloads to such an extent that our algorithm couldn’t keep up with it,” said Sellars. “We had to think about how we optimize the code to take full advantage of the data download speeds available to us. There are machine learning and artificial intelligence innovations coming online right now that should allow us to do just that.”

Those 2017 predictions are now coming true with the CENIC AI Resource (CENIC AIR), a powerful and growing distributed supercomputing cluster, participation in which is available as part of CENIC membership and which functions as the California portion of the NRP.

“When I think of the years I banged my head against walls developing the machine learning and data processing to predict weather events … well, AI would have sped all of that up,” said Sellars today. “The key breakthrough is the infrastructure to pull all of these resources together. The process of making predictions is so much better.”

Sellars moved on from California to serve as a senior policy adviser with the federal government, where he recognized even more fully the need to retain those infrastructure and human connections.

“In the future, the challenges will involve making the impact of weather and climate more relevant to those who need to make policy decisions,” Sellars said.

Watch for the next article that examines AI-powered tools and advanced networks in atmospheric and climate research.  We’ll introduce you to UC San Diego’s GAIA Initiative, founded by SIO assistant professor Duncan Watson-Parris. Bringing together more than 28 faculty across SIO, the Halıcıoğlu Data Science Institute, and other UC San Diego departments, GAIA will harness the power of artificial intelligence to transform how we understand, forecast, and steward Earth’s complex systems.

Related blog posts