Apr 26, 2024

CENIC’s Integrated, Multi-Tool Approach to Network Configuration Management

The topic of network automation at CENIC has been treated before on our blog: a complex, ongoing program that enabled the creation of targeted network announcements in 2022 and the progress of which a team of CENIC network engineers presented to attendees at The Right Connection, our biennial conference that took place in March in Monterey, CA.

The focus of Vanessa Bernabe, Dennis Cagampan, Josh Dickman, Brian Ismay, and Cameron Nemo’s presentation was Network Configuration Management (NCM) – how and why it was implemented, the components that comprise it, how it’s used, and its potential for the future.

What CENIC and its Member Institutions Need from NCM and Why

Not only is CENIC’s California Research and Education Network (CalREN) one of the most extensive and complex regional networks in North America, but it facilitates collaborative research and education in the largest public education and library systems in the world as well as some of the most accomplished private, cultural, and scientific institutions.

And if California’s K-20 research and education community is to retain its leadership position, CalREN must not only remain in operation 24/7/365 no matter what but grow in bandwidth and services to exceed that community’s accomplishments. With a well-designed, well-integrated NCM system in place, achieving this becomes much more scalable, efficient, and cost-effective.

Initially, device configuration on CalREN was set manually via the devices’ command-line interfaces (CLI) using manually generated templates, an approach that is very difficult to scale as CalREN grows in size and complexity – especially when “no matter what” expanded in scope to include a global pandemic. In light of this, CENIC’s NCM project was inaugurated with the following goals:

1. The automation of network device configurations,

2. Standardized configuration management to ensure uniformity and consistency, and

3. Configuration audit capabilities for enhanced security and compliance.

Gathering the Pieces and Putting Them Together

Ensuring that CalREN stands ready to support and grow with its user communities means operating, maintaining, and upgrading a dizzying array of third-party equipment, connections, and configurations spread along 8,000 miles of fiber-optic cable. Not only must the state of hundreds of pieces of equipment and their settings be maintained, but the history of these settings and CENIC’s responses to network events must also be stored, along with what components comprise each individual member site’s connection to and through CalREN.

On top of this, all of this information must be securely accessible to and editable by CENIC network engineers and presented in a way that enables rapid, efficient evaluation and response.

This is plainly a tall order, and in response to it, CENIC engineering created a fully integrated NCM system composed of both third-party and in-house software tools, including the following:

• Cobber: This CENIC-created software tool maintains up-to-date awareness of the device components in the network alongside an exhaustive relational database containing full, standardized network circuit, equipment, and site contact information for the many thousands of complex components that comprise the CalREN backbone and all customer connections to it, from end to end.

• NetBox Labs’ NetBox Cloud service: This tool functions as a network source of truth (NSoT) with which CENIC accurately documents our physical infrastructure and other information across all 61 node sites and can share this information with other automation platforms.

• Cisco Network Services Orchestrator (NSO): This tool allows CENIC to seamlessly connect to network devices and manage their configurations. NSO ensures that changes are applied consistently across the entire network and supports over 1000 third-party device types and cloud services.

• Infoblox: This tool is used to plan, track, and manage the Internet Protocol (IP) addresses assigned and used over CalREN, a process known as IP Address Management (IPAM).

• Jira: This is the ticketing system used by CENIC to track network outages and CENIC’s response to them as well as to perform project tracking across engineering teams.

The last four tools listed above are brought together and the information they contain is integrated, presented, and made accessible and editable by Phoebe, a unified web application tailored precisely to CENIC’s operational processes.

Increased Security and Efficiency through Integration of Tools

CENIC engineers can automate services through Phoebe, and the entire system provides:

• Standardized definitions of services and device configurations,

• Minimized manual effort through doing away with the CLI,

• Logically grouped settings enabling faster, safer equipment configuration and service provision, and

• Historical tracking (what changes were made by whom and why) and audit, improving security, compliance, and event response.

Further, the system features several unique features and integrations, enabling the following:

• The history of proposed and applied configurations are preserved and displayed to users, enabling engineers to share configurations, revert changes quickly and completely, and compare service parameters.

• An auditing agent constantly checks the consistency of network device configurations.

• IPAM integration via Infoblox, provisioning IP address prefixes from a pool.

Workflows defined in Phoebe include new service deployments, service decommissioning, compliance auditing, and ticketing response.

The Future of NCM at CENIC

Future enhancements can include additional integrations such as automated access by one tool to another’s data, for example enabling other tools to access information stored in NetBox or adding a comment to a Jira ticket or in a Slack channel when a network change is committed. Onboarding devices can also be automated by importing information from NetBox. Closed-loop automations are another possibility – changes automatically initiated by monitoring which can result in a type of “self-managing” network.

Whatever the future of network automation may entail at CENIC, we’ll be sure to keep our members up to date via our website, events, and newsletters.

Related blog posts

Three Quick Guidelines to Help Your Organization Stay Safe from RPKI-Related Hacking

Using secure passwords, enabling MFA, and defining those authorized to make changes to your ARIN account can help your institution avoid RPKI-related hacking.

From the Ground to the Stars: Critical Big-Data Research in Africa

Learn about critical data-dependent research collaborations currently taking place in the nations of Africa, all of which require integration into the global high-performance network and cyberinfrastructure environments.