Bullet Proof Windows Patching Verification (Plus KPIs)

Why is it so difficult to verify my endpoints are patched...that seems easy?

Windows patching is deceptively difficult, and the tools that are built to help you make sure nothing is broken are they themselves, broken. All of them. Don't believe me? Go dive into how your tools are recording patches installed vs patches missing. How does it determine it's "fully patched"? Simply if that machine doesn't see any patches missing from Windows update...but wait, what if the Windows Update components on the machine are broken so it's not successfully talking to Windows Update? Well that would be "no patches missing" and "100% patched". Crazy right?

We'd all like to imagine that management tools like RMMs and Intune have a database of what patches should be installed on each machine, then it checks those machines and verifies it matches up. That would make sense, and it would make the reporting checks and balances worth while. Unfortunately, that's not the case and a lot of us have assumed for a long time that's how it worked, just to eventually get a vulnerability scanner or maybe a third party audit to find we have several machines unpatched with thousands of vulnerabilities, or worse, we didn't notice until after an attack and then we're cleaning up the wreckage.

What is failing?

Is your team just failing to manage updates, or are their tools failing them? While it's possible we also have some failures in patch management, we know for sure that we have failure in the tools. Here's some scenarios that would make an endpoint show "no patches missing":

WSUS configured to a WSUS server that is offline or inaccessible
- Your RMM or Intune asks the machine to reach out to "Windows Update Servers" (the WSUS server), it doesn't reply with any patches missing, so...no patches missing. Keep in mind this could be because the WSUS server is no longer in service, or maybe the WSUS server is only available in the local office or when connected to VPN and this user rarely uses their VPN.
A "block all" 0 trust sort of configuration present at the network level or endpoint level that was never configured to allow Windows Update servers through
Windows Update components are broken in some way
Surely more...

That's a lot of loose ends and possible ways for endpoints to go unpatched for years, all the while reporting fully patched.

What to do about it

So what do we do? We need to look at the last date a patch was successfully installed. It seems too easy to boil the whole thing down to that, and I'm ashamed to admit the years it took to get to this conclusion, but it really is the most fool proof way to make sure things are in line and being successfully patched. It's easy to get tied up in looking at patch failures, communication verification to Windows Update servers, policy applied status to the endpoint from your RMM or Intune, the state of GPO settings in registry, and several others. I'm not saying those things aren't worth while, but none of them will so clearly and decisively tell you if patching is broken– they're all just small pieces and anyone of them "looking good" could still make you miss the problem: patching is not happening.

Note that a vulnerability detection solutions will do great job at quickly identifying endpoints that haven't patched in a long time, but now we're talking about more tools, money, implementation, monitoring, maintenance, etc. If you already have one in place then this can be a great verification method as well...although, the vulnerability detection platforms aren't going to allow you (some small exceptions) to plan automatic remediations or notifications.

Important KPIs

If you're in the market to ensure your patching is on point, then lets cover some other key performance indicators of successful patching while we're at it. While the "no patches installed in 30 days" is the single most important metric, it doesn't mean it's the only metric. Here is a solid go-to list of KPIs to track week-to-week:

Endpoints with no patches installed in 30+ days, should be 0
Endpoints with no reboot in 30+ days, should be 0
Total patch % of servers, should be 97%+
Total patch % of laptops, should be 97%+
Total patch % of desktops, should be 97%+

Technical implementation

Unfortunately, very few tools offer options to find this critical metric of just "last date a patch was installed", and even if they do, even fewer have a way to report on it. This means we need a consistent way to find this critical piece of information. Powershell time:

# Define the maximum number of days an endpoint can go without patching
$acceptableDays = 45 

# Function to get the latest patch installed via Windows Update API
function Get-LatestPatchviaAPI {
    $Session = New-Object -ComObject Microsoft.Update.Session
    $Searcher = $Session.CreateUpdateSearcher()
    $SearchResult = $Searcher.Search("IsInstalled=1").Updates
    $LatestPatchviaAPI = $SearchResult | Sort-Object -Property LastDeploymentChangeTime | Select-Object -Last 1
    return $LatestPatchviaAPI
}

# Function to get the latest patch installed via Get-hotFix/QuickFixEngineering
function Get-LatestPatchviaHotFix {
    $Hotfixes = Get-HotFix
    $LatestPatchviaHotFix = $Hotfixes | Sort-Object -Property InstalledOn -Descending | Select-Object -First 1
    return $LatestPatchviaHotFix
}

# Get the latest patch installed through API
$LatestPatchviaAPI = Get-LatestPatchviaAPI

# Get the latest patch installed through Get-HotFix
$LatestPatchviaHotFix = Get-LatestPatchviaHotFix

# Compare the installation dates to determine the newest patch
if ($LatestPatchviaAPI.LastDeploymentChangeTime -gt $LatestPatchviaHotFix.InstalledOn) {
    $NewestPatch = $LatestPatchviaAPI
    
    # Display information about the newest installed patch
    # if you want to display latest patch name use: $($NewestPatch.Title)"
    $lastPatched = ($NewestPatch.LastDeploymentChangeTime).ToString("MM/dd/yyyy")
} else {
    $NewestPatch = $LatestPatchviaHotFix

    # Display information about the newest installed patch
    # if you want to display latest patch name use: $($NewestPatch.HotFixID)
    $lastPatched = ($NewestPatch.Date).ToString("MM/dd/yyyy")
}

if ($lastPatched -lt $acceptableDate) {
    # Consider logging this information as well as throwing an exception
    throw "$env:computername has not been patched in the last $acceptableDays days."
} else {
    return "Last patch date: $lastPatched - System is within acceptable patch standards."
}

This will return whether or not you've been patched inside of the amount of days you deem acceptable. I believe 30 days to be a reasonable threshold, but tune to your preferences.

Alternative outputs

I often like to just return a boolean true/false if it's pass/fail so I have an option to use this as a monitor where I know what's failing to patch so I can take action. If you prefer that method, just replace the if statement at the bottom with this:

if ($lastPatched -lt $acceptableDate) {
    return $false
} else {
    return $true
}

If you need to return just the date of last patch so you can report off it easily, then replace the if statement with this:

return $lastPatched

These are all for single run scenarios. If you're more comfortable with Powershell there's a lot of ways you can tune this, but I personally like using a switch and usually wrapping in a function so I can determine if my condition is failed, then I can recall the same script and call it to action.

In this example, if I set $method to test then it will output truthy/falsy, and if I set $method to set, it would run the repair script (that still needs to be made)

switch ($method) {
    'test' {
        # Define the maximum number of days an endpoint can go without patching
        $acceptableDays = 45 

        # Function to get the latest patch installed via Windows Update API
        function Get-LatestPatchviaAPI {
            $Session = New-Object -ComObject Microsoft.Update.Session
            $Searcher = $Session.CreateUpdateSearcher()
            $SearchResult = $Searcher.Search("IsInstalled=1").Updates
            $LatestPatchviaAPI = $SearchResult | Sort-Object -Property LastDeploymentChangeTime | Select-Object -Last 1
            return $LatestPatchviaAPI
        }

        # Function to get the latest patch installed via Get-hotFix/QuickFixEngineering
        function Get-LatestPatchviaHotFix {
            $Hotfixes = Get-HotFix
            $LatestPatchviaHotFix = $Hotfixes | Sort-Object -Property Date -Descending | Select-Object -First 1
            return $LatestPatchviaHotFix
        }

        # Get the latest patch installed through API
        $LatestPatchviaAPI = Get-LatestPatchviaAPI

        # Get the latest patch installed through Get-HotFix
        $LatestPatchviaHotFix = Get-LatestPatchviaHotFix

        # Compare the installation dates to determine the newest patch
        if ($LatestPatchviaAPI.LastDeploymentChangeTime -gt $LatestPatchviaHotFix.Date) {
            $NewestPatch = $LatestPatchviaAPI
    
            # Display information about the newest installed patch
            # if you want to display latest patch name use: $($NewestPatch.Title)"
            $lastPatched = ($NewestPatch.LastDeploymentChangeTime).ToString("MM/dd/yyyy")
        } else {
            $NewestPatch = $LatestPatchviaHotFix

            # Display information about the newest installed patch
            # if you want to display latest patch name use: $($NewestPatch.HotFixID)
            $lastPatched = ($NewestPatch.Date).ToString("MM/dd/yyyy")
        }

        if ($lastPatched -lt $acceptableDate) {
            return $false
        } else {
            return $true
        }
    }
    'set' {
        # Run automated Windows Update repair script here
    }
}

Wrap up

Endpoints that have not been patched in 30 days or more really need to be found on a dashboard somewhere so you can report on them, and if you really want to track and make this actionable and alive in your organization, you should have a KPI where you track the number of endpoints that fail this test that is tracked on a weekly basis. Recording this data alone doesn't do anything– only measuring, taking action, improving, rinse and repeat will bring that change. What gets measured gets improved!

Hope this was helpful and improved your patching verification. If it did, leave a comment with your thoughts! If you have your own methods for patching verification, leave them in the comments!