Jan 17, 2019

Collaboration Kept Pepperdine Connected During Wildfires

As the Woolsey wildfire surrounded Pepperdine University in Malibu, the campus’s primary internet connections burned. “A lot of lines went down,” said Pepperdine Information Technology Director Rita Schnepp. “Although our routing protocols automatically redirected traffic and kept us connected, CENIC and Pepperdine staff still had to make quick adjustments to make the most use out of our remaining bandwidth during the disaster. It happened very fast. I was impressed.”

During this fall’s deadly and destructive wildfires, several CENIC members lost service entirely or lost circuit diversity at their sites. CENIC’s Network Operations Center staff worked with commercial partners to restore services to critical community anchor institutions, which are often central sites for first responders and community residents during emergency situations.

Engineers Responded Quickly

When Pepperdine’s primary fiber network connection went down, CENIC contacted Pepperdine engineers right away. Pepperdine had planned for such an emergency and had routing protocols in place that prevented the university from losing its internet connectivity entirely. Still, their bandwidth was about to saturate on the remaining alternate pathways.

Pepperdine IT staff put in an emergency request for increased capacity. With assistance from AT&T, CENIC engineers increased capacity from 100 Mbps to 850 Mbps within about three hours. Normally, it takes about 60 days after the execution of a contract to upgrade a circuit. “We've had a really good partnership with CENIC over the years because your engineers are very much like ours,” Schnepp said. “It takes no time for there to be a synergy.”

Meanwhile, many students, staff, and faculty who lived on campus remained there while others living in surrounding communities were ordered to evacuate. For some IT staff, that meant evacuating with their laptops and working remotely with CENIC and a VPN contractor. “We had IT staff collaborating together from all over the state including several who remained on the Malibu campus. They were listening for emergency evacuation messages as fires rage around their neighborhoods. They managed to get to safety and connect with one another,” Schnepp said.

Many Volunteered to Help

People did unusual things, Schnepp said. One Pepperdine IT executive, a father of six, evacuated with his family to a Los Angeles hotel. Yet he volunteered through the night to operate Pepperdine's main campus phone line and help desk. “He said, ‘I'm really happy to do it in the middle of the night, and I have a bunch of volunteers who want to keep these phones going 24/7 because we have a lot of concerned parents calling us, not just technical questions,’” Schnepp shared. Deans, faculty members, and IT staff whose jobs usually don't involve answering phones also took shifts to keep that communication going from locations both on and off campus.

Several emergency operations team members who lived on campus lost connections supplied by private internet service providers. Pepperdine IT staff brought them DSL equipment and hooked them up to the CENIC connection out of Calabasas. “Calabasas and West Los Angeles stayed up and that's what kept us going,” Schnepp said. Although Pepperdine’s IT department performs annual disaster drills, it still took collaboration and commitment from many people to keep the university connected during the wildfire, she said. “When there's an emergency, the CENIC and Pepperdine technology folks are immediately on the same page,” she said. “They just dive right in and tackle the problem.”

Schnepp also noted that the university was grateful that it had already moved many services, such as its Oracle Enterprise System, into the cloud, which helped people continue to get paid on time. “We’re also grateful to our local Spectrum service provider who spliced our main Malibu fiber connections back into service within nine days of the fire,” said Schnepp.

Community Partners Worked Together

In addition to Pepperdine, many other institutions lost connectivity in the fires: Oxnard Public Library, Butte County Library, Modoc County Office of Education, Modoc County Library, Ventura County Community College District Office, and California State University Channel Islands.

Several ISPs offered free assistance to affected communities. In Oxnard, GeoLinks deployed temporary microwave circuits to the main library and two branch sites. In addition to helping upgrade bandwidth at Pepperdine, AT&T’s Network and Disaster Recovery team deployed portable cell sites and recovery equipment statewide. CENIC helped restore connectivity to affected sites any way it could, offering to leverage its relationships with ISPs, provide temporary communications links, and donate decommissioned hardware.

In its response to Pepperdine’s urgent needs during the wildfires, CENIC showed its capacity to react quickly to support essential services in real time. Schnepp noted, “That’s an agile organization.”

Read more about the factors that support the success of CENIC's Network Operations Center.

Related blog posts

Solving Network Connections for USC on Catalina Island

CalREN connection and resiliency for USC's Wrigley Institute for Environmental Studies and its Marine Science Center on Catalina Island

Network Traffic Analysis Shows Changing Activity Patterns During COVID-19 Pandemic

Even as campuses, schools, libraries, and cultural institutions have closed amid stay-at-home orders, CENIC’s networks have remained a vital part of the Internet ecosystem for online learning and community engagement, remote access to research data and specialized computing facilities, academic medicine and clinical care, and work-from-home operations.