ARCHIVED: Completed project: IU core network upgrade (IUB/IUPUI)

Primary UITS contacts: David B. Hunter, Jason Mueller

Completed: December 29, 2011

Description: This project was completed on December 29, 2011 with the replacement of the IUB Data Center routers. All 18 new Juniper routers are now installed and in production across the IUB and IUPUI campuses.

Outcome:

• Increased reliability:
  • Control plane protection offered by the Juniper routers will be better than our current ones, which will protect CPU and packet forwarding during adverse traffic events
  • More ports available for dual-homing buildings
  • Additional routers will allow for reduced number of host adjacencies by distributing users and locations across devices
• Higher bandwidth capacity:
  • More 10 Gbps ports will be available, especially within the data centers, as those routers will have more card capacity
• Allow for growth of the network:
  • The routers should accommodate the ability to add more ports in the future without the need to upgrade the router chassis; this is especially important in the data centers, where larger routers will be installed; the larger data center routers will support roughly double the number of ports than initially installed
• Provide feature parity for hosts within the MPLS VPNs (VRFS):
  • IPv6 support
  • Multicast (IPv4/IPv6) support

Milestones and status:

• March-April 2011: Begin preparations for installation and migration to new equipment
• May 2011: Begin transition steps 1 and 2, bringing new border routers online and inserted into routing, as well as inserting new campus routers; no clients migrated yet
• June-August 2011: Begin migrating buildings, services, regionals, and GigaPOP connections to the new border and campus routers; also migrate the new data center routers into place

Benefits:

• Resiliency to adverse traffic events
• Increase in available router ports, and the ability to increase the number of dual-homed buildings on the IUB campus
• Distribution of user load across more devices
• A secondary border/backbone router at each campus, allowing for a more meshed and resilient design; campus routers will be dual-fed to each campus border router

Risks: Due to the size and scope of this project, there is a possibility of short-term service interruption. If outages or disruptions are anticipated, we will communicate them through Change Management and the LSP community with as much lead time as possible.

We will present all related changes, whether a disruption is expected or not, at Change Management. Primarily, we hope to minimize outages and interruptions, and when they are necessary, communicate them as much ahead of time as possible.

Issues: Time is major factor in this upgrade, with a goal for completion by fall semester. We began March 2011 by ordering equipment, lining up resources, such as cabling and power, and preparing to migrate configurations. Once the equipment arrives, we will undergo a transitional installation stage. This transitional stage (adding the new routers to the network, but not yet placing customers on them) will occur late April through the end of May.

The process of transitioning IU buildings and data centers to the new routing platform will likely start in June, and continue through the end of August, with completion before the fall semester. Scheduling of cutover times, communication to all stakeholders, and physical configuration and installations will be major components of this project.

Primary client: The primary client is each IU campus and its users. The core campus network provides Internet egress, and network services to buildings and data centers at IUB and IUPUI. Additionally, regional campuses are redundantly connected to IUB and IUPUI for access to central campus resources, and the Internet via the GigaPOP.

Client impact: Customers of the IU network should see a lesser or nonexistent impact to network services from UDP floods, DHCP storms, high traffic events, and looped switchports flooding router ports. Such events in the past have driven up our current router's CPU, and on occasion inhibited the forwarding of packets.

The new Juniper routers have faster processors, and better control plane policing functions to protect packet forwarding ability. Due to the addition of extra routers at each campus, we will be distributing the users among more devices, freeing up more processor time for forwarding packets, and adding diversity.

More routers on each campus increases the number of available customer-facing ports. We will be using the additional ports to dual-home more buildings on the IUB campus to provide redundancy to their connectivity. The additional routers will also better distribute clients and buildings across routers.

The Juniper routers have support for OSPFv3, and allow for IPv6 to be provided within an MPLS VRF, which allows for IPv6 availability within the data centers and behind the firewalls. Previously, we could not support MPLS in a VRF.

Multicast within a VRF also is a feature missing in our current routers. With this project, some users within VRFs, such as the Health Sciences Network, IU Health (Clarian), and users behind the firewall within the IUB Data Center will be able to use multicast internally (i.e., to support a load-balanced cluster).

Project team: The project team is composed of all CNI engineers at each campus, with two lead project members per campus. Due to the size and scope of this upgrade, we will need the assistance of every engineer at some time during this upgrade.

Management and design:

• Jason Mueller: Network design and project leader
• David B. Hunter: Campus network manager
• IUB:
  • Ken Dunn: Network engineer (lead)
  • Tony Brazzel: Network engineer (lead)
  • Chuck Escue: Wireless network engineer
  • Ian Washburn: Wireless network engineer
  • John Stigall: Network engineer
  • Dwight Hazen: Network engineer
• IUPUI:
  • Damon Beals: Network engineer (lead)
  • Seth Garrett: Network engineer (lead)
  • James Dickerson: Wireless network engineer
  • James Cormack: Wireless network engineer
  • Chris Gallardo: Network engineer

GRNOC research networks:

• Tom Johnson: Participating in support of research networks peering with campus, and providing consulting experience with JunOS

Governance:

• Dennis Cromwell
• Dave Jent
• Mike Lucas