15 Jun (E)SAE DEEP DIVE SERIES PART 10 – Patch Management
This month our blog is about patch management. Today’s software is complex, the source code of Windows includes several million lines of code and vulnerabilities are discovered regularly. At the same time, the attacker scene has become much more professional, including state actors with access to immense resources. This makes it all the more important to close these vulnerabilities as quickly as possible with patches provided by the manufacturer.
An investigation by the security vendor FireEye of 60 vulnerabilities in 2018/2019 concludes that 58 percent of vulnerabilities were exploited before a patch was released and 27 percent were exploited within a month of a patch being released. At the same time, according to a study by the security vendor Edgescan, it takes an average of up to 73 days for organizations to close the vulnerability by installing the patch. The message is clear, the patch management process must be accelerated.
This sounds trivial at first glance, but customer experience shows that many organizations do not have a complete handle on the topic or simply take too long compared to the attackers. There are countless articles on the subject of patch management in the literature and on the web, which is why we will limit ourselves to describing our “standard process” in the blog article. We usually start the discussion with the customer with this standard process and adapt the process to the customer’s specific needs and technical requirements.
The patch management process includes the regular identification of existing patches, as well as the planning and installation of the patches in the company. The prerequisite for identifying patches is that the systems and software used in the company are known.
To shorten the time between becoming aware of a security vulnerability and closing it, it is important to automate this process as much as possible. Common management tools such as Windows Server Update Services (WSUS) orMicrosoft Endpoint Manager already offer a range of functions for automation:
- Provision of patch metadata
- Automatic inventory of systems
- Detection of uninstalled patches based on metadata
- Automatic installation of patches at defined times
- Reports on installed or uninstalled patches on all systems
Overview Patch Management-Process
Patches released by vendors need to be identified and evaluated on a regular basis. In the case of very urgent updates, such as this year’s remotely exploitable vulnerabilities for Microsoft Exchange, a monthly update cycle is not sufficient. In this case, an emergency patch process must be provided to install the patches as quickly as possible and, if necessary, manually on the affected systems. The remaining patches are installed as part of the normal patch process at regular intervals (in most cases every 4 weeks).
For the identification of the patches, management tools are very helpful, which provide and regularly update the metadata of the patches for Microsoft and ideally for widespread software manufacturers. Based on this metadata, these management tools can scan all managed systems and determine which patches are installed or not installed. In this way, extensive automation of many process steps can also be achieved.
For systems that are not automatically managed with management tools, such as servers in a DMZ or in Tier 0 in an ESAE environment, it must be determined what the procedure is there. In these cases, it often makes sense to deploy a separate WSUS environment.
The BSI also regularly issues cyber security alerts about new and threatening attack vectors.
In most cases, patches are rolled out on a monthly basis. In this case, it makes sense to bundle all patches and install them at the same time in a maintenance window. Patches as part of the emergency patch process are installed separately.
If ITIL-compliant change management is implemented, the installation of patches is approved by the Change Advisory Board (CAB). As the installation of patches is a regularly recurring activity, a standard Change is usually defined for this purpose, which is automatically approved.
For testing patches it is very helpful to have a test environment. This should ideally be as close as possible to the production environment. In the test environment, the essential infrastructure services such as Active Directory, Exchange or SharePoint should be installed, of course in a smaller version but still in a similar configuration. For Active Directory, this means at least two domain controllers or, in the case of SQL Server, in a cluster configuration if this is also the case in production (in test, two nodes are then usually sufficient).
In testing, the respective infrastructure service or application should be checked for functionality after the installation of patches. Here, automated tests help to reduce the effort and the critical time until the patches are installed. Monitoring tools such as System Center Operations Manager can be very helpful here because they regularly check whether a service is actually available.
Rolling out patches
To minimize risks, patches should not be rolled out to all systems simultaneously. For this purpose, different rollout groups are defined on which the patches are successively installed at different times. There is sufficient buffer between the installation times of the individual rollout groups to be able to react if problems occur.
The systems are categorized and assigned to the various rollout groups. Systems implementing a particular service should be assigned to different rollout groups. For example, Domain Controller 1 to Rollout Group 1 and Domain Controller 2 to Rollout Group 2.
To automate more complex dependencies, a runbook automation tool such as System Center Orchestrator can be used.
Tests are performed between the deployment of the different rollout groups. These tests are performed by the respective server or application managers according to the requirements of the application.
In a fully automated patch rollout process, there should be a way to prevent the installation of a patch for a specific system.
After installation, a check is made to ensure that the patches have actually been installed. In the case of common management tools, the check is performed automatically. The status is usually displayed in web-based reports. SQL Server Reporting Services offers the possibility to generate reports at specific times and send them to e-mail distribution lists. In addition, they contain an environment for developing reports that are adapted to the company’s requirements and prepare essential information for management.
Patch Compliance Overall
In addition, it can be very useful to use a vulnerability scanner such as Nessus. This scans the systems in the network and checks whether the vulnerability is exploitable.
Here we will outline a standard approach for implementing a patch management process.
It starts with an implementation project, the scope of which depends on the customer’s goals, the number of systems and applications, and the maturity of the existing process. A typical project structure may include the following activities:
The following describes the activities that occur in a patch cycle. In most companies, this is carried out on a monthly basis. In Microsoft-heavy companies, it has become customary to align the patch cycle with Microsoft’s “Patch Tuesday”. Microsoft always releases its patches on the second Tuesday of the month. In the meantime, a number of other manufacturers have adopted this approach.
First, the patches published by the manufacturers are identified. The patches are combined in a release (and, if necessary, in an emergency release). The patch release is installed on prepared waves for test systems. These systems are usually located in a separate test environment. In larger organizations, test environments are often multi-tiered. If one does not exist, dedicated test systems are used within the production environment. If a test environment exists, but it reflects the production environment only to a very limited extent, it is a good idea to include some production systems in the test wave.
The patches are rolled out in several waves. The systems are usually statically assigned to the individual waves. This means that this is configured once and no changes need to be made to this configuration during the regular patch cycle. In many patch management tools, the necessary steps can also be automated. The installation of the patches must be checked. For this purpose, the common patch management tools offer web-based reports.
The process presented here illustrates how the installation of the patches in the company can be carried out in a short time. Of course, this must be adapted to the requirements and framework conditions during an implementation.
In detail, there are certainly many more aspects that need to be considered, but we hope to have given you an idea of how a fast patch management process can look like. Feel free to leave us feedback.
Sieh dir diesen Beitrag auf Instagram an
Sieh dir diesen Beitrag auf Instagram an
In our January blog, we started an SAE deep dive series and explained how to use Hyper-V as a secure hypervisor in an (E)SAE scenario. Since by far not all our customers use Hyper-V, but many also use VMWare...15 June, 2020
(E) SAE Deep Dive Series Part 1: Hyper-V Host Guardian Service (HGS) and Shielded VMs in an EASE Environment
After the success of the first ESAE series, we decided to launch a deep dive series in which we go into a little more detail on various measures....16 January, 2020
After Hyper-V HGS and VM protection with VMWare, now the third part of our (E) SAE Deep Dive Series follows. Maybe you follow us on LinkedIn, Xing, Facebook, Instagram or Twitter and ...15 July, 2020