Bad Listening Harms Relationships and Cyber Programs

By Chris Nyhuis, Vigilant Founder

A Vigilant Know More Series

In an IT role, you generally spend most of your time at a console building the future or saving the day. If you are on the cyber side of IT, that can turn into a 10x increase of time away from friends and family. For many of us that have been in the industry, this creates a strain on our relationships. I learned to maintain both secure networks and strong relationships even when time was not abundant early in my career. The secret is LISTENING. It won’t solve all your problems; however, by using some cheat codes of understanding you will build stronger and more resilient relationships and security. Think of it as Up, Up, Down, Down, Left Right, B, A, Select, Start to effective security and relationships. Creating a perfect time split between my kids schedules and my cyber career was a battle I felt like I would never win. I started to miss events and show up at the wrong place at the wrong time. I was late to the places that were right, and worse sometimes did not make it at all. Around the same time on the cybersecurity side of my life, my teams were tasked with protecting purchasing and financial information for the automotive industry. We did all the “right things”: logs, firewalls, and endpoints. UTM firewalls had just come out and moving to them seemed like a big jump forward. At the time, we used TAPs to collect data and basked in all the information we had to find threats; however, after investing in newer managed switches, we started using the mirror or spanning features of the switches. What was interesting is we saw a dramatic drop in verified true positives almost immediately after the new managed switches were installed. One would think that detections going down would be a good thing, especially since we had just added security technology; however, it didn’t sit right with me.

1. Attackers didn't change their cadence
2. We just put in “ better and more powerful” technology meaning that we should see more verified true positives than the previous systems, not less. That is why we upgraded to the new tech in the first place, right? How was I going to prove ROI to the org?

After digging into the data and loads of packet loss, we found that what we were experiencing in our network was caused by the same problem that I was having at home – a lack of good listening. At that point in my marriage, I was not a great listener. My wife is one of the most detailed people I’ve met. If there is a detail, she’s on it and she knows everything about it. I had been blaming my showing up at the wrong places on my wife’s communication. However, she was communicating; I was simply not listening. It wasn’t like I was ignoring her, I was just listening to part of the conversation. I was missing the whole story and made decisions with only partial data. This is a recipe for disaster in any relationship. From a cyber standpoint at work, we were having the same problem. However, my ears weren’t the issue. It was the transition from TAPs to span ports that caused our reduction in “hearing” and ultimately our ability to detect threats. Three important rules of listening to remember:

1. Collection – if your collection of information is flawed or diminished you are doomed from the start no matter how amazing your processing (AI, Machine Learning, Threat Hunting).
2. Priority – If you allow distraction and do not prioritize your attention to the people you are listening to, or the data you are gathering you will miss the intent of the conversation.
3. Processing – If your processing is not able to process the flow of information it will cause an attack point for a threat actor and it will cause you to miss threats.

TAPs vs. Span Ports - Collection

A TAP is a passive splitting mechanism installed between a device of interest and the network. A TAP copies the incoming network traffic and splits it. It passes the network traffic to the network and sends a copy of that traffic (both send and receive) to a monitoring device in real time.

Important Fact #1 – TAPs can collect traffic as line speed – you get 100% of the data passing across the TAP.

A SPAN/mirror port on a switch that copies traffic on a port or group of ports and sends the copied data to an analyzer. By its very nature it is half-duplex, which means that it cannot send all of the send and receive traffic it sees if traffic exceeds 50% of the bandwidth. Basically, it’s like us, when there is too much going on we can’t speak and comprehend at the same time so we miss things.

Important Fact #2 – SPANs will ALWAYS drop traffic sometimes under load up to 30% and more.

Can you imagine how badly a relationship would function if you didn’t listen to 30% of a conversation? I can tell you from experience as a former partial listener… it doesn’t pan out too well. Think about what having partial data about your environment does to your detection. I don’t care how great machine learning, AI or that brand new cool product on the market is, if it’s fueled by only 60% of the conversation the technology will learn wrong, create bad intel and will ultimately detect wrong. Not having all the information also wastes time as you must revisit conversations over and over. You may not miss all attacks however those partial detections will come at a price, you will miss attacks and your team will spend a lot of time dealing with false positives while missing false negatives.

Important Fact #3 - Architecture matters – you can’t expect technology to do something that it is not built to do.

Firewalls, Switches, Routers etc. primarily use ASIC chipsets that are not built for deep packet inspection let alone all the other features that they must do that most companies turn off anyway due to performance issues. ASIC chipsets are meant to move packets fast, not inspect them at deep levels in the way that security requirements need. ASIC chipsets are also cheaper which keeps the cost of the hardware technology lower for the service provider – security traded for cost. Going cheap always has a cost, this cost is usually seen in an attack or ransomware attack that was missed. This means that the systems that are deployed in most companies do not have the capacity to do its actual job. This trend is why security is failing repeatedly.

Going beyond partial listening and partial security

The first step to great relationships is the same as the first step to great security. You must put the right scenarios in place to capture all of what is being said, you must have the capacity to process what you heard, and you must take the time to think about what is needed from what you heard. If you don’t do these three first, you can invest in people and security. However, you have a high chance of missing what is really needed and your actions will most likely miss the mark. Actions taken out of context cause breakups and hacking events.

Vigilant’s Promise to our Clients

Vigilant is one of the only cyber companies in the industry that promises Unlimited Incident Response forensic services to our clients if they are ever attacked on our watch. When you guarantee your services, you must do what is right, it forces you to, and one of the first steps to that is starting in the right place which is data collection. In our MNDR services that starts with TAPS. Vigilant does not use span ports period unless it’s an additional data point.

1. We focus on using Network TAPs; sure, they are more expensive to us as the security provider; however, that is ok as they operate at line speed with no latency and no packet loss.
2. The next step is by providing detection systems that are built with the right processing ability to do the job and more.

At Vigilant we are in the relationship with you, we don’t leave the trench when it gets hard or ask any more in the relationship than we are willing to give either. We are great listeners, and we back it up by aligning risk with our clients. Let’s all commit to listening to each other. We'll have better relationships and better security.