Saturday, September 4, 2021

Event Suppression Sucks



I’ve always hated the concept of event suppression in security products.

Let’s start with some definitions of suppression, and where better than product documentation?

There’s two common reasons for this feature:

The first: “I don't want to see this thing in my console of actionable items because I don't have the time, knowledge, perspective, or priority to do anything with it right now.” There is nothing inaccurate in this behavior, but suppressed alerts are a mismatched solution to this problem. A better answer is multi-level event generation. The system should recognize that some events are not worthy of human attention. Low importance events should go into a statistical model or an audit log, not an analyst’s workbench. The further left (earlier) in the event creation and processing pipeline this happens, the better. This design results in better performance and scalability, which means more signal, less noise per dollar spent. Suppressing generated events at the end of the pipeline is wasteful design that throttles the system’s capacity.

The second is “This rule is wrong and I can't edit it, but I need to get rid of its alerts.” This situation is tougher because it’s where internal or external regulations are driving behaviors. The rule book says to generate alerts no one is going to look at, so we do. The proper answer is still multi level alerts. “Proper” in this case means most efficiently using human and compute time. However, vendor and customer will also need to explain and negotiate with the rule-quoting gatekeepers in order to demonstrate that the rules are not broken. Sending less important events into a separate channel is functionally no different than suppression, but the effect is far more efficient because the separate channel isn’t indexed for rapid and continuous human access. Instrument that channel and monitor it as a production service and you’re good to disable event generation from incorrect or non-actionable rules.

There is a sub-form of that second reason: the analysts don’t have permission to edit rules to fix them or change event generation. “I don’t want to look at this but I can’t stop it so suppression is the answer.”  Maybe this is product immaturity, maybe it’s organizational failure, but it’s still a problem to fix. As a vendor, if your product supports a better option that your customers won’t use, you’ve got a customer input collection tour to do. This is a case where technology can’t solve a people problem (Edwards’ Law), so it needs person to person communication and possibly a professional services engagement. Customers who can adjust to the better model will be more effective and efficient than those who don’t, but you may still need to offer support to those who can’t. The design job therefore is to discourage suppression without preventing it.

Security is full of whataboutism, but most of the desired data  is really low value — so pass it into a low value, low cost pipeline. Don’t leave it clogging up your SOC.