READ THIS Before You Get Into Vulnerability Detection, Management & Remediation

Why read this first? Aren't there tools that solve the vulnerability problem?

Yes there are tools, but the tools don't solve the problem by themselves. They move you forward, absolutely, but there is a web of labor and process to untangle here that needs some in depth thought and planning before you dive in and end up way over your head. A ~10min read here can put structure around this fog of vulnerability confusion and save you literally hundreds of hours, so lets get to it!

Okay, so what's the problem?

In todays world of cyber security & IT, hearing the word "vulnerability" has become the norm. Yes, vulnerabilities have existed since almost the beginning of software's existence. Yes malicious people wanting to make a buck or prove a point have always tried to exploit them... but things have changed. We're now up against nation state attacks, maybe entire malicious organizations with HR departments and benefits with a thousand+ employees all dedicated to one thing– exploiting you or your clients for money in some way through technology. Many of us think buying a piece of software and turning it on solves the problem, but it doesn't. It can't. You're going to lose unless you have a plan.

So we do need to do something about vulnerabilities, but we need to proceed with caution and do it in a scalable, manageable way.

The vulnerability detection process actually has three delivery areas you need to know about

So far this article has said "vulnerabilities" several times, but we actually need some definition around this because there's quite a bit to it. Lets start with the three major delivery areas of vulnerabilities:

Vulnerability detection: The process of a tool scanning endpoints (workstations and servers) and/or networks (internal and/or external) for vulnerabilities
Vulnerability management: The process of intaking the results of the vulnerability detection scans and compiling them into something readable / usable for the remediation and auditing teams to work from, and keeping track of mitigated / remediated / accepted vulnerabilities
Vulnerability remediation: The process of resolving or at least mitigating the vulnerabilities

Vulnerability detection

This is by far the easiest of the three, but you can still mess it up. The days of all computers being in an office or two are almost completely over, and you will almost always have at least some element of a remote workforce to deal with. This means that traditional vulnerability scanners that rely solely on local network scanners are no longer going to cut it. Surprisingly, that still represents most of them. That means you need to have some firm requirements in mind when deciding on what vulnerability detection tool you're going to use and make sure that it has "agent based" detection support (this is vulnerability detection software installed on each endpoint making the physical location of the endpoint insignificant).

Vulnerability management

Now things are starting to get tricky. Vulnerability management is what you probably expect– it's managing the state of vulnerability statuses. For example, if Adobe Reader is out of date and vulnerable in January, is it still out of date and vulnerable when we scan in February? Yes or no. Simple enough, right? Not quite.

If Adobe Reader shows out of date and vulnerable in February, is that because we still haven't updated Adobe Reader, or is it because a new vulnerability is already known on the new version of Adobe Reader and now we have a new vulnerability? In other words, how do we have any idea if we actually made progress and are now fixing a new problem, vs fixing technical debt from last month that no one took action on? Well we have to know the exact vulnerability that endpoint had and the version of Adobe Reader that the vulnerability was for, and we need to know when it was first detected. Crazy enough, most all of these vulnerability management platforms do not actually provide a way to track the first date a given vulnerability was detected. Worse yet, you've likely assumed (unless you learned otherwise through pain and suffering) that a vulnerability detection tool includes vulnerability management...it doesn't! At least, often it doesn't, but sometimes it does. It varies by tool and a higher price does not imply that it has more features so keep an eye out for that too.

Remember that there's varying degrees of usefulness of vulnerability management tools, so just because a tool says it has this feature doesn't mean it's any good... do your homework. Try to use it in real life.

So what does all of that mean? It means unless we have a very above average vulnerability management tool that we intentionally vetted to include these features, the only way for us to figure out if we're actually making progress and eliminating previous vulnerabilities is to manually check, balance, and consolidate our results month to month. This is a devastating task that I am going to call literally unmanageable. If you're using spreadsheets to compare month to month, you're missing things, you don't know what you're actually solving that's new vs technical debt, and you absolutely do not have real visibility and control over your vulnerabilities. Even if lets say you did manage to nail it... at what cost? This is extremely time consuming. Often times a single application out of date can create literally 10s of thousands of vulnerabilities. Imagine that but multiplied by total endpoints under your care, then multiplied by the hundreds of applications installed across all of them. "literally unmanageable" isn't dramatic, it's just the truth.

Vulnerability remediation...it's tough. Really tough.

This is where the rubber meets the road. You need to be really careful here, and if you're a service provider of any kind, you need to make sure you're not putting "remediation" in your services as "included". You will regret it, quickly. Why? Because it's complex and massively time consuming. Consider these points:

Windows Updates vs Out-of-Band: If you're like most, you'd see a vulnerability reporting that a Windows update is missing on 100 machines and think "okay great, just install the update and good to go!". This is however often not a reality at all thanks to several conditions:
- Out-of-band updates: Out of band updates are updates released by Microsoft that fall outside of their normal patch Tuesday release cycle. Essentially, it means the update was unplanned and created to solve a problem. That problem is a vulnerability... but how wide-spread or important that vulnerability is may vary so much that Microsoft actually decides to leave it as out-of-band forever and never roll it up into a standard Windows Update release. Why does that matter? Because out-of-band updates are not included in Windows updates. You have to go download these out of band updates yourself, apply them yourself, then verify it was successful. That's not always the case though! Sometimes Microsoft decides this was a worthy update that effects enough machines that it should be wide-spread released so they include it in the next major monthly rollup. You just don't and can't know when that will happen, and when it will just be an out-of-band update for forever. Depending on your number of endpoints, this means someone has to go manually install all of these (don't do this), or use a tool+scripting through an RMM or Intune (or similar) to get it deployed. Several of these out-of-band updates you would have never heard about, never known about, and never cared about if it weren't for the vulnerability scanner detecting it and deciding it needs to be resolved. In short: more work. A lot more work.

Accompanying registry: Oddly, Microsoft often requires accompanying registry keys to be set in addition to the out-of-band or just Windows updates being installed for it to actually fully resolve the vulnerability. This doesn't make much sense since Microsoft could have easily made the update set the registry key for you, but they don't. This means for several Windows updates and out-of-band updates you will also need to manually set or script a registry key to accompany that update or the machine will still show vulnerable in your vulnerability detection platform. An awful nuance here by the way is your vulnerability detection platforms (usually) do not tell you if it's the patch, the registry setting, or both– only that "you're vulnerable" so you just need to check all of these on all machines that are vulnerable and find which one it is.

Numerous OS versions: In addition to the complexities above, we also have to deal with numerous OS' and OS builds that are all effected by the same vulnerability. This means that when we manually install or script our out-of-band update installation, we can't just "download the update that's missing and install it", but rather we have to find the specific version of that update for each machine to find which one matches the machine. It's not uncommon for a single out-of-band update to have 15+ different KBs that you may use depending on the OS and OS build number, then have to match the right KB to the right version. Again, massively time consuming.
Superseded updates: This is the worst one by far. Microsoft regularly supersedes old out-of-band updates and Windows Updates by releasing a new update that resolves the same vulnerability. This means the newly released updates have new KBs, which means our last list of KBs we put together are no longer relevant. What makes this difficult is sometimes Microsoft will supersede an update so quickly that you weren't yet even finished with the rollout of the first set of KBs. Rinse and repeat and sometimes this happens 4-5 times+ before you're confident you don't need to be looking at it anymore, or until the vulnerability scanner stops telling you you're vulnerable. Oh yeah and lets not forget, vulnerability detections tools often fail to update the KB that needs to be installed to be the new superseding KB, so no help there either. This is massively time consuming and there's currently no automated or built in way for RMMs or Intune to solve this. WSUS and SCCM can do some look ups with superseding patches which is nice, but those are on-prem, dated, legacy systems that are largely on their way out and generally not used unless you're dealing in enterprise (which you may be!).

If Microsoft decided to add the out of band update into a monthly roll-up then you're golden, but again, you'll need to do the leg work to confirm if that happened.
Configuration: Depreciated protocols, open ports, general security configuration practices such as logs, firewall, access controls, password policies, and many more may come up

Non-Windows based vulnerabilities
- Network detections: So far we've only talked about Windows vulnerabilities which, to be fair, are the majority of the vulnerabilities you usually end up remediating, but that's just the start. We also have network firewalls, switches, APs, ESX hosts, appliances, IoT, phones, printers, PBXs, NASs, SANs, equipment, and much much more. Your networks you manage are a minefield of vulnerabilities you just don't know about until you flip the switch on that vulnerability detection software, and now you don't get to be ignorant and it's time to do something about it.
- MacOS: If you have Macs in your environment, chances are your management of them isn't great, so get ready to get better at them because you're going to have vulnerabilities here in the OS, software, protocols, etc. Generally there are far fewer vulnerabilities here to worry about than Windows, but it still does happen, and it's still work so we need to account for this.
- Linux: If you're like most, you're not doing a ton to manage your Linux devices out there... well again, get ready to. These are going to come up on scans too for OS, software, configurations, ports, etc.

Believe it or not there's even more but these are the big ones. This all represents tools, labor, process, management, and sometimes projects. Guard your time and your profits on this one. Understand the weight of this. Set expectations carefully. My advice: all remediation is billable outside of anything solved with an included update already managed by you. Registry key needs to be set? Billable. Third party application needs an update that's not on your usual list? Billable. You get the point.

Give up yet? Hang in there, here's some strategy to get through it

Endpoint based vulnerability detection: As I said earlier, make sure your vulnerability detection platform has an installable agent, and make sure that agent works on Windows, Mac, and Linux. There's nothing worse than settling in to your new tool after hard research and implementation just to find you have a gap that forces your hand to bring in another tool, meaning another evaluation, testing... you get the point.
Internal Network based vulnerability scanner: Remember that if you do have local networks you manage (and you likely do), you'll still also need a local network based vulnerability detection tool. Often the agents can double as the network scanner, but make sure to ask the question. You may end up needing to install a standalone appliance.
External vulnerability scanner: Don't assume your network vulnerability scanner is handling external vulnerabilities– that's likely separate! If you're going to do nothing else in all of vulnerability detection / management / remediation, make sure to nail it and have no exceptions on your external vulnerability scans to the best of your abilities.
Don't neglect the vulnerability management platform: I've seen too many businesses spend all of their time on selecting the right vulnerability detection platform and then either not have a vulnerability management platform at all, or have one that doesn't have the basic features required to make it usable– namely the "first detected" datapoint for any given vulnerability.
Configure whitelisting before you introduce vulnerability management/remediation: You're going to find a LOT of vulnerable applications when you begin this journey. Too much to effectively manage. In my opinion, the single most important thing to do before you introduce (endpoint based) vulnerability detection is to work with the client and determine a specific list of applications that the client will allow to be installed. Take this list, and run a project to uninstall all other applications that are not on this list, then apply your whitelisting policies to the entire client. This limits your vulnerable applications to a much smaller, more manageable list. This may seem extreme at first, but after your first adventure through vulnerability detection/management you're going to quickly understand why you should do this.
Only focus on the criticals and highs: You're not going to solve all vulnerabilities, and you really don't need to. Make sure your vulnerability detection tool is categorizing by severity, and focus on the two highest. At the end of the day, what we're really concerned with are "exploitable vulnerabilities" and those will fall in line with the critical and high classifications.
Plan maintenance schedules around vulnerability scans: A common scenario is you go update Google Chrome on all endpoints on a Friday, then on Thursday next week your vulnerability detection scan runs. The vulnerability scan finds Google Chrome is out of date and has vulnerabilities on all endpoints and you're thinking "but I just updated it!". This happens all of the time, and applications like Chrome just update constantly because it has vulnerabilities constantly. If you carefully plan your vulnerability scans to run immediately after OS and application maintenance, you can maximize your chances of having the lowest results possible. After all, the goal here is to find vulnerabilities that are active in the environment that represent risk we want to mitigate or resolve that need our attention, not to find a 5 day old version of Chrome that's going to be updated tomorrow automatically. That's called noise, and we don't want it.
Eliminate the junk: If you have IoT devices, old servers, old appliances, or anything similar on your network that are dirtying up your reports, see if you can just get them off the network. Throw them away, make a segmented network to keep those vulnerable devices away from your business infrastructure, look into general compensating controls if the updates are risky. Again, remember the goal here is removing risk from the business. If you have devices that don't need to be on the corporate network, take them off! We don't have to patch/update everything.
Keep remediation billable: I mentioned this above but it's a critical one not to miss. Sometimes you're going to spend an enormous amount of time resolving vulnerabilities. Ideally, you scope out the amount of work needed to "get to 0", treat this as a project to "level set", then it's ongoing detection / management from there and introduce billable remediations as they popup (which will be manageable after a level set project, good tooling, and using the strategies in this article).

Wrap up

This was long but there's a lot here to cover. I hope to put together another article to go through various tooling options to solve these problems, and I'd like to give some in depth comparisons between those tools. If this is something that would be valuable to you drop me a comment with some that you'd like to see!