Patching is Hard

I’m not being facetious with that title. Contrary to what many people would have you believe, patching machines in a corporate environment is not easy.

So the big story this week is WannaCry (and it’s variants), a ransomware attack that doesn’t just propagate via email, but also through a bug in the Windows SMB service. Yes it’s a worm, and once it’s in your environment people don’t actually have to do anything in order to be infected by it. Cool, right? Yeah, but a lot of organizations don’t think so at the moment. This is a piece of malware that just goes to show that we don’t learn well from lessons of the past and possibly the biggest finger is being pointed at infected companies because they hadn’t patched their systems for this vulnerability.

Here are a couple of questions to ask yourself:

1. Were your systems patched?
2. Do you actually know that all of your systems are patched?

This is where patch management starts to get difficult and why so many companies struggle with it. Keeping track of all of the various available patches, understanding the implications of patching your various systems, rolling out the patches in a controlled manner, and being able to tell the success of the patching process are only a few of the challenges. Since I’m making points in twos, here are the two primary things you need:

1. A patch management tool (it can be free, see Microsoft WSUS)
2. A plan. (Well, a plan, a process, a procedure, etc.)

Technology is easy, and in this case it can even be free. However, if you have a lot of non-Microsoft software you’ll need to buy something else. It’s the plan that gets bogged down because there are so many what ifs. What if Microsoft pushes a bad patch, what if a patch breaks an application, what if we reboot that server and it doesn’t come back up, and so on. These are all risks that have to be evaluated, but in today’s climate, most have be given very little weight in the risk assessment process. If you were to have gotten hit by WannaCry, how successful do you think the explanation of; “We were worried Microsoft would put out a bad patch.” would be with management? Likely not very successful. Does Microsoft put out bad patches? Sure, it has happened but it’s very infrequent and it’s usually caught within a day or so of the patch going out. Would all of your systems be patched with that bad patch in one day? Probably not, and that’s probably a bad idea anyway unless the risk is really high.

As an organization you need to determine what the risks are of patching and work out a plan that works for you. However, you have to be honest just like you would be in any other risk assessment. In reality, are you at a bigger risk of Microsoft releasing a bad patch or of malware exploiting that vulnerability? There’s plenty of evidence to show that the malware is a far more likely scenario. Here are some suggestions for a plan:

• Have a process for evaluating the risk of a patch. Maybe every patch Tuesday you meet to rate the risk for your organization. Use the vendors criticality rating as a start, but remember that just because it’s critical for them doesn’t mean it should be to you.
• Based on that assessment, make recommendations to your patching team(s) for whether patches can be applied during a regular patch cycle or out-of-band.
• The patching process doesn’t have to be complicated, but give the team an opportunity to find problems and quickly recover from them.
  • Maybe you start with a set of test machine on day 2 (one day after the patches are released)
  • Then move to a series of roll-outs (depending on how many machines you have) that hit all of your systems that are basically a match for your “gold standard” image. That way if it goes bad you just re-image.
  • Then you look at the harder to patch systems and roll those out in small batches, again, of similar systems. You should be able to do all of this within a few weeks of the patches being released.
• Make sure, at all steps, that a review of the patching process is done to determine if any systems are missed. It can be pretty much guaranteed that someone will have turned off their system or a patch fails for some reason. Always be able to tell what your risk level is by knowing which systems are up to date and which aren’t.

As I said at the beginning, this is not a simple thing. It takes time to build and refine the plan and ensure you have the right tools. However, it is becoming increasingly clear that the way most companies are patching today, isn’t working. I can almost guarantee that if they haven’t already, management will come to you asking if you’ve patched against this latest malware. At least be prepared to tell them that you’re building a plan to ensure that you are in the future.