For most IT organizations, network automation boils down to change management. Network engineering teams implement tools that automate their processes for all manner of changes, including network software patches and upgrades, configuration changes, and security policy updates.
One area of change management that now receives increased attention is change validation. This is a two-stage process. It begins with validating that a proposed change will have the intended effect on the overall network without negative impacts. This is pre pre-change validation. The second stage is validating that an implemented change was done successfully without causing problems. This is post-change validation. Depending on whom one talks to, network teams also describe this practices as network verification and network assurance.
Regardless of what one calls it, very few IT organizations do it consistently. Enterprise Management Associates (EMA) recently surveyed 354 IT professionals about their network automation strategies (the results were published in “Enterprise Network Automation: Emerging from the Dark Ages and Reaching Toward NetDevOps”). In that survey, 98% claimed to have processes and tools for validating an automated change to the network before executing it, but only 11% applied pre-change validation to 100% of automated changes. Meanwhile, 99% had processes for post-change validation, but only 11% applied them 100% of the time.
“We have some validation, but it’s not enough,” a network engineer at a leading gaming company recently told me. "I would love to know if this change will make my traffic move in a different direction. Also, when I upgrade a device, what will happen? No one can do that in a container lab.”
And that’s the issue. For years, many network teams have had ad hoc approaches to change validation that are difficult to apply for a variety of reasons. For instance, they don’t scale, or they rely too much on manual processes.
For pre-change validation, network teams often build out a lab of physical network devices or virtual labs that simulate their network with virtual devices running on VMs (and more recently on containers).
“We have some containers we run in a lab environment to check changes and also some custom scripts,” a network engineer at a midmarket business services company told me. “It’s not a full topology of our network, just a reference model. We don’t have the resources to simulate our production environment.”
On the post-change side of network validation, network engineering teams have traditionally had more options, but few of them are very good. For instance, our research found that 45% do manual configuration checks on individual production devices to verify that their change went through. This manual process confirms that their config update matches what they pushed through their automation tool, but it doesn’t guarantee that the change is good. For that, you need an end-to-end view of the network. Unfortunately, many turn to their monitoring tools instead. More than 44% of network teams told us they rely on reporting and alerts from their network fault and performance monitoring tools for post-change validation. In other words, they wait for the NOC to tell them something is broken.
“We will get alerts form our network monitoring system that something is wrong,” a network automation engineer at a medical school and hospital network told me. “If it was not successful, then we roll back the change and document what went wrong and analyze why it didn’t work.”
Ideally, network validation should eliminate the possibility of changes leading to alerts. If a network team must roll back change it made via a network automation pipeline, then that automation pipeline is flawed.
There is an emerging class of vendors that automate the process of validation by discovering, documenting, and modeling production networks. The vendors often describe themselves as digital network twin providers. They offer visibility into the state of the network and allow engineers to simulate how changes will impact traffic flows, security policies, and more. Such vendors devote resources to building robust network discovery engines, scalable modeling engines, and powerful querying tools.
In fact, 58% of network teams told EMA they use a network modeling tool or digital twin for pre-change validation and 48% use such a tool for post-change validation. EMA believes that many of the respondents who claim to be using network modeling software are not using a true modern digital network twin solution. Many are probably using tools that discover and map networks. They show how things fit together, but they aren’t always dynamic enough to simulate all the possible outcomes of a network change. This is why digital twin technology tends to be more expensive than network mapping tools. They have the power to show all possible outcomes of a change and to discover and validate those impacts in the production network.
Regardless of what network teams use to validate network changes, they need to institute more scalable and automated tools and processes so that they can become more consistent with pre- and post-change validation. EMA’s research found that network automation strategies are usually more successful when they validate a higher percentage of the overall changes made through automation tools.
“The most challenging thing about network automaton is testing to make sure whatever you are doing will work,” said the network engineer at the medical school and hospital network. “Network automation is powerful, but it is also dangerous. You can make a change that can bring down the whole network.”
If you'd like to learn more about EMA's latest research on network automation but you can't afford to buy our new report, please check out our free on-demand webinar, which highlights our key research findings.