When I entered the world of threat hunting from a background of offensive security I came with a few misconceptions that I see commonly repeated. I obviously believed threat hunting was more important than some, hence my move, yet it wasn't until I actually saw it working before my eyes that I truly had to re-evaluate some of my previous assumptions. On reflection, the reality seems all too obvious, but then it always does doesn't it?
In this article then I'd like to run through some of the misconceptions I've either had myself, or commonly heard others repeating. Some of them may actually have some truth to them, but still serve as a good jumping off point for devils advocating, so don't take it all too literally. I do believe there is now a real shift in the way security is being approached that is very important, and this serves as a bit of a brain-dump of why I think that is.
Threat Hunting is just another empty buzzword, nothing new.
Security is full of buzzwords and snake oil, and unfortunately "threat hunting" is indeed used like this too. Many people use the term a little too freely, or let's face it, just to sound like they are cutting edge too. But all this bandwagon jumping doesn’t mean there isn’t something interesting going on underneath. Of course, threat hunting didn’t suddenly emerge fully formed from nowhere. It's the culmination of many years of thought on security and some people have been doing something akin to proto-threat hunting for quite a while.
People have always wanted to find hackers when they start messing with their stuff. This lead to anti-virus, IDS, and SoCs. Many of these things are purely reactive, waiting for an alert to be generated from some known bad, and then jumping in to action to analyse, contain, and maybe add some newly discovered known bad for next time. The best SoC analysts took the best data they had from their environments and trawled through it to find tell-tale signs of attack. Unfortunately, sometimes the data available was just a big pile, mostly of false-positive alerts, but the intelligent analyst could quickly sort the wheat from the chaff. Maybe they also had some netflow data or raw network data as well to make finding bad things a bit better. Threat hunting embraces this raw data mentality. Never mind the alerts, let's have detailed raw data about what is going on, on the network and on endpoints too. Now we can really hunt through and find unknown bad as well.
But perhaps what is new is the attitude. Hunting is not just a value add on top of day-to-day alert driven monitoring, it’s a primary activity. And because of that the tools and processes people use for it are improving rapidly as more and more people gain experience in what really works. So in a way it isn’t "new", but the pace of innovation and effective use of it is.
You're fighting a losing battle.
Ever the battle cry of attackers (authorised or not). And it's true that hackers will always be able to find a way in, or a way to bypass any measure you put in place.
It has often been said, attackers only need to be successful once, but defenders need to be successful every time. But that's an all-or-nothing attitude that assumes if anyone gets in at all its instantly game over. The reality is once in, attackers need to persist and move around, performing all kinds of malicious actions and they need to be successful at remaining undetected every time (illustrated by the "window of opportunity" on the blue team cyber kill chain). The longer they are in the more damaging it is likely to be for you, and some damage might occur quite quickly depending on the circumstances, but that doesn’t mean there's no point trying to hunt them out.
Good threat hunting is based on good data from within your environment. If you are able to execute a good set of hunting use cases, you begin to detect many different attacks using common factors they all must share. Many of these are based on nuances of how the platform being exploited works and are simply unavoidable if you want to do anything meaningful as an attacker. Imagine if you have visibility of unusual threads appearing, hooks in memory, executable pages that were writeable at one time, DLLs that were loaded from strange places, unexpected processes or process trees. Do you know how you will do something useful on a machine you compromise? Will you be able to hide all these things from the first time your fingerprints hit the box? The answer isn't a definitive "no", but your options certainly feel more limited.
By the time you catch them they probably did what they wanted anyway.
If they got in, you already lost.
If they got in, at least to some extent you did lose, a bit. After all no one wants to get hacked, and no compromise is lower risk than some compromise.
But getting in isn’t the end for an attack, it’s the beginning. Ask your red teams; if they are trying to execute a real world attack with an actual effect (money stolen, data obtained, etc.) then getting in is only the start. You then have to deal with lateral movement to get to the assets you want, which may mean performing internal recon to find out what technical assets are needed, what people to target, what internal processes will need to be accounted for. On a red team exercise it's not unusual to get in on day one and achieve your objective a month later. If you can find and evict attackers before this final point, your losses will be limited, and maybe almost nil.
Threat Hunting is just an excuse to be bad at prevention.
I'm sure some people out there are doing this. They are doing it wrong.
Because of course you don’t want them to get in at all. That's where all your other security efforts come into play: proper design, attack surface reduction, secure configuration, patching, pentesting and so on. All of that still matters. No one should be saying threat hunting removes those needs. But remember that losing battle defenders are supposedly fighting? This is that battles front line. It extends exactly as far as an 0day, or a mistake, and we certainly live in a world where both of those things exist! So if your security approach doesn’t extend beyond these things, you really are fighting a losing battle. Threat hunting is designed to extend beyond this, into the post-compromise world, but hopefully before it all goes BOOM. You better have some good IR too…
In fact, threat hunting is more effective when your general security is of a good standard. The low hanging fruit only serves to generate noise from attacks that shouldn't have happened in the first place.
Look, I can bypass this EDR!
You can't possibly rely on detection on a machine I already 0wned.
If you aren't in the kernel it's pointless.
Endpoint Detection and Response (EDR) software if often used by threat hunters. It consists of an agent running on an endpoint (a workstation or server) that sends data back for analysis to find attacks. Some EDR solutions have kernel components and some are purely user mode.
It's very tempting when you spend your life bypassing security to look at security software running on a machine you just popped a root/SYSTEM shell on and think, well this is fundamentally flawed. I can just turn it off, bypass it, feed it false data. Maybe if this isn’t a targeted attack it will be harder to do, but advanced attacks are often targeted and that's what threat hunting claims to be able to deal with right? Or maybe it's in the kernel, that's a bit harder, but look I have a kernel 0day, now it's definitely game over right?
The reality is though, even advanced targeted attacks don’t necessarily hit the kernel. If you're a pentester, how many times have you honestly gone into the kernel, even if you're a full on red teamer? Many times you might not even exploit a memory corruption vulnerability at all.
Maybe some of this is because you just don’t have to as a result of all the other easier security holes often available to you, because lots of people are still doing a bad job at even basic security. But even ignoring this, an attack rarely reaches the kernel, or even root, in one atomic step. There might be delivery steps, injected threads, or suspicious processes before you get to that point, all of which can be hunted out before you're able to subvert the agent. And realistically, there might be a lot of lateral movement before you even find a target you can get that level of access to.
So it's certainly possible to have an awesome 100% stealth rootkit in the kernel, or below. But there's usually some weird behaviour will reveal itself somewhere, and a huge number of those are accessible from user mode too. The reality is I've seen some otherwise pretty impressive attacks get detected in this way, things that weren't stopped or identified by any of the existing traditional security that was in place. Everything can be bypassed, but doing it perfectly doesn’t have to be easy, and one misstep in a thousand underlying actions to reach you goal is all a hunter needs to kick off a deeper investigation.
And EDR isn't the only data source for threat hunting anyway. There are still network taps and logs and other sources of information you may have less control over even with a popped kernel on a box.
Hunting is just filling in the gap while AV/next gen AV/IDS/IPS catches up.
It is tempting to think that all the efforts of threat hunting can and should be automated eventually. If you find something bad, you might as well signature it, and if you find some bad generic behaviour you might as well find that behaviour automatically in future. And yes, automation is important for effective threat hunting to avoid redoing work unnecessarily and to help pick out the signal from the noise. If something can be automatically detected with 99% confidence, by all means automate it, maybe even block it if safe to do so in your environment.
But not everything is 99% certain; let me rephrase that, not everything CAN be 99% certain. Take a silly example, a run key: there are loads of legitimate ones, and they will look largely the same as the bad ones. You might find a way to eliminate known goods from your list, and maybe have some indicators that make them look "more" suspicious, like random looking filenames or weird command line arguments. But for an unknown run key you really can't be completely sure if it's bad or not without some more investigation. They are simply not differentiated enough to stand out as definitely bad. But if you're looking for unknown bad stuff, you're certainly going to want to have this info available to you in an easy to use form.
EDR is Threat Hunting
Some people might not even realise they are effectively saying this. Endpoint detection is proving very useful in detecting attacks and is a vital data source for threat hunters. But that is all it is, a source of data. Threat hunting is not a software product, it's a human activity, you just need things like EDR, network data to do it really effectively.
The lines are also blurred in real life between EDR and other endpoint features more akin to next-gen AV or IPS where attacks can be blocked or some action can be taken. EDR software can also provide features for forensic investigation, and the line between threat hunting DFIR (Digital Forensics and Incident Response) can get very blurred at a technical level. All of this leads to lots of strange, misguided product comparisons and category errors. A product need not fit a category cleanly, but be sure you are comparing the right things when you do.
I'm sceptical this can actually work…
Me too. But we're doing it, and it does.