An Inside Look: Midco’s Network Operations Center

Picture your average day. You wake up, get ready, drink coffee and fire up your laptop. Once you’re logged in, you read emails. You make calls and join video conference calls. You IM with your colleagues. Your internet and phone services just work.

Little did you know, your connection underwent maintenance yesterday. Not because it was broken, but because we identified that your local node – or network distribution point – was only performing at 70%.

That’s what my team does at Midco. As Director of Network Operations, my team is responsible for monitoring our 10,200 miles of fiber network 24/7 from Midco’s own Network Operations Center (NOC). We not only make sure your internet just works – we make sure that it performs at its optimum level.

Protecting a Fiber Optic Network

We monitor 30,000 network elements across five states and routinely complete scheduled maintenances to ensure uptime and optimal network performance. Let’s just say that my days rarely look the same. We have a lot of moving parts, but our team thrives on being agile – and pride ourselves on having the following superpowers:

• Hyper awareness: a sixth sense for danger
• Enhanced wits: a nimble and adaptive attitude
• Precognition: an ability to (try to) predict the future

Taking a Proactive Approach

In the past few years, our focus has been shifting from being reactive to proactive – to preventing issues before they happen.

Our team closely monitors online traffic patterns and continually refines the Midco Network’s capacity. The network is widely dispersed and designed to handle 20% above capacity peak usage times – meaning it’s always ready to handle emails, data transfers and video conference calls, whether you’re at your business or working from home.

We also use automated integrations to monitor both our customer care system and our network services – so we can cross-reference between what we’re seeing, and our customers are chatting about.

Here's a bird's eye view of our network – showing hardware connections between different areas. 

These systems also help us identify where our network is showing low quality of experience scores – or what we call Q scores. To give you a frame of reference, a Q score in the 30s or 40s, for instance, will cause delays getting onto Google or make your video calls cut in and out. Most of our nodes actually operate in the 90s, with overall node health rated at a 98.5% – meaning it’s unlikely that you’ll ever feel a normal delay. If and when you do experience downtime or slow connections, it’s more likely caused by:

Peak Utilization

This happens when a node is close to hitting capacity. To combat this, Midco proactively schedules node splits. We move customers to a new node and remap the route to provide a clearer connection.

Node Flapping

Our dashboards will show us when nodes are flapping (or bouncing), which means their connections are maintaining, then cutting out, but then coming back on again. When this happens, a technician will resolve the issue.

The following dashboard shows node health on a past date. Any network events (including proactive maintenance) are reflected so the NOC team can track online and offline statuses.

This is a detailed view of an individual node. Once a node appears in red on the overview dashboard, a tech will use this view to investigate why exactly it's been flagged. 

Recommended next: Evolving for Now: 2020 Business Needs 

Building in Redundancy

So, what happens when an outage does occur? First, we’re notified through multiple channels – alerts from our systems, dashboards and emails. Within seconds, customers with redundant fiber are switched over to their secondary workload to provide minimal interruption.

Then, we determine if there are high-impact, enterprise network customers in its path, specifically healthcare providers, financial institutions or government entities. We’ll often contact those customers directly and confirm that their redundant connection is working.

Midco strives to resolve any outage in less than four hours.

Cue problem-solving mode. Is it a commercial power outage? Is it located where there are many network transports? We’re searching for the “common upstream,” as I like to call it. This is when we test, test and test again to resolve the issue.

Countering Severe Weather

Days at the NOC can be unpredictable – and Mother Nature doesn’t always help. In the Midwest, we combat lots of kinds of possible severe weather. Last summer, for example, we had four tornadoes touch down in Sioux Falls. In that scenario, there’s little we can do to prevent outages, but we try instead to restore services as soon as possible. It was an all-hands-on-deck effort, and when we did identify outages, we sent our field staff out to make repairs on the spot.

Monitoring Network Partnerships

Midco partners with more than 40 other providers across the country to extend our reach – and we have the unique responsibility of working directly with other NOC teams to identify potential issues on their end. The NOC-to-NOC relationship allows us to work together to build, test, tear down and build again together – eventually pinpointing exact causes of an issue, even if they’re outside our own network.