Software Patching Best Practices - 18 Must Do Tips

Why Is Patching Software Important? 

Failure to Patch Can Be Very Costly

Failure to patch vulnerabilities in computer code can lead to losses of information that can cost more than your company can afford to pay.  A recent example is a small non-profit in Anchorage Alaska.  The small five facility nonprofit is paying federal regulators $150,000 in fines for allowing their system to be breached by not patching their software.  That penalty is on top of the costs that they will additionally incur in now patching the systems they failed to patch in the first place, the legal fees for defense, the costs associated with getting credit monitoring services for those affected by the breach, the on-going compliance monitoring by the Office of Civil Rights and all other costs incurred by this very public debacle.  Click here to go to the Resolution Agreement signed by the non-profit.  You don’t’ want to be forced to sign one of these, too.  If you need to patch the systems anyway then make sure you do it in a timely manner to avoid terrible impact on your business.  Go to this link to see an HHS Office of Civil Rights partial list of companies paying heavy fines, some in the millions of dollars, for breaches of their IT systems.

There are many other benefits to applying software patches including in some cases adding features, fixing bugs that make the software run slow or not work right.  All software needs to be patched. Whether the software sits on a disk and runs on a server, resides on a chip within a firewall, or is an app that is in your tablet devices, it all needs to periodically be updated and patched in order to be secure.

The following list of 18 software patching best practices is what we follow at Alvaka Networks when delivering on our Patchworx(SM) Patch Management Service.  It is important to note that all these steps are important, but not always are all them utilized or they can be utilized in different ways depending upon the needs of the client. Like us, you will need to decide what your patch management plan needs to look like to best suit your needs.

18 recommended best practices for patching your software:

1.       First identify all the software you are using.  Today’s IT systems present a challenge because most systems run dozens of different software titles.  You can’t know what you need to patch until you know what you have.  You have operating systems, server applications and desktop applications.

2.       You need to patch servers, PCs and mobile devices

3.       Remember, you need to have a strategy to patch many of your hardware based appliances like firewalls, routers, SANs and more.

4.       Set a regularly scheduled routine every month to patch your systems.  You can do it most efficiently all in one big event over a weekend where all systems are patched or you can elect to do 20% of them at a time over the course of the month to mitigate impacts from unexpected patching problems.  There are many other approaches you can take.  You need to decide what is best for your business.

5.       When dealing with multiple servers, you need to identify if you have dependencies that require a certain server reboot order for everything to work right on restart.  A good example is that you must restart an Exchange mail server before you restart your Blackberry Enterprise Server, otherwise at some point the following day you are going to get complaints from your Blackberry users that they are not getting e-mail.

6.       Read the release notes or ReadMe files to learn more about the implications of deploying a set of patches.  You should also review software user forums to see if anyone else is reporting problems with the new patches.

7.       It is good to apply patches in a timely manner, but unless there is an imminent threat, don’t rush to deploy the patches until there is an opportunity to see what affect it is having elsewhere in similar software user communities.  A good rule of thumb is to apply patches 30 days from their release.

8.       Before applying patches to your production system you should test the patches out on a test environment.  This can be difficult and expensive for most companies since it requires buying a lot of extra hardware and software to build the test environment.  If you don’t have a budget to do this, there are alternatives.  There are companies that provide patching services.  Testing those patches before deployment to production systems is one of the tasks they perform.

9.       During the testing process in the test environment, determine if the computers require a reboot or if they do so automatically.  If so, then you need to plan for a maintenance window in which to apply the patches to the production system so you don’t have unexpected system reboots that hurt your business operations or do damage to databases, etc.  You can expect 90+% of your patch deployments to require reboots.

10.   After you have applied patches, utilize a smoke testing procedure to make sure all applications and services are back online and running properly when servers and PCs restart.

11.   Change Management is important, but often overlooked.  You need to involve other stakeholders in the organization before making the changes.  Often they will let you know of system or organizational demands that will have an effect on your patch deployment task. Here is a link on What is Change Management?

12.   Notify your end-user community of your planned timeframe for patch deployment and let them know of the timeframe so they know what to expect.  Let them know what they should do if they encounter a problem after the patch deployment.

13.   Have a good roll-back plan.  A roll-back plan allows you to quickly reverse the patches and go back to the pre-patched if there is a significant problem with the deployment.   Good patching tools and procedures will allow for a roll-back of patches. 

14.   Have a good backup of all your systems and if possible, take an image snapshot of your servers right before your patch deployment.

15.   Are there any auto-scheduled maintenance jobs running to do maintenance such as for a SQL database?  If yes, be sure to put them on hold as they can really mess things up if left running.

16.   Use a service or automated tools whenever possible.  Don’t use tools like Auto-Update as you cannot control when patches are applied, you cannot test applications before patches are applied, there is no smoke testing procedure post patching to determine everything is running fine, and there is not patch deployment reporting that is required to show yourself, your management, auditors and regulators that you are running securely patched operation.

17.   Review the patching report after deployment and look for patches that failed to deploy.  Investigate why they failed to deploy, develop your remediation plan and then redeploy.

18.   Make sure you accommodate your exceptions.  Sometimes certain servers or applications cannot be upgraded or patched in order to maintain compatibility with a critical application that is in use.  When this happens, you need to make sure you have an alternative strategy for securing that system from the vulnerability left exposed by the inability to patch the software.

Go to this link to see an HHS Office of Civil Rights partial list of companies paying heavy fines, some in the millions of dollars, for breaches of their IT systems.  These companies were already victimized once by the breach.  I bet they felt victimized a second time when the fines were levied.  Don’t let that happen to you.  The moral of the blog is that you have to patch eventually one way or another so do it in a timely and professional manner.  It will bring you much peace of mind.

A recent Center for Strategic and International Studies (CSIS) study made a strong case for patch management. The study, conducted over a three-month period, found that simply applying the most recent patches to six software packages on Windows machines could prevent 99.8 percent of malware infections.