In our January blog, we started an SAE deep dive series and explained how to use Hyper-V as a secure hypervisor in an (E)SAE scenario. Since by far not all our customers use Hyper-V, but many also use VMWare, we would like to explain in this article how we have transferred the concepts of Hyper-V HGS and shielded VMs to the VMWare world. In the article we describe a solution that we developed and implemented together with a partner company for one of our SAE customers. As always, we cordially invite the community to discuss decisions that have been made and discuss improvements together. So, feel free to contact us at 😊.

As a reminder, here are the goals we want to achieve in the ESAE context:

• • VM Disks are encrypted
  • VMs can only be started on defined hosts
  • The host configuration is certified
  • Virtualization administrators cannot modify the VMs.
  • VM administrators cannot modify the hypervisor

In the project a new VMWare instance was implemented purely for the new ESAE infrastructure to be built. To not lose the focus on the security aspects, we do not describe some topics like high availability configuration or backup and restore or only to the extent that it is relevant to the topic of this article.

VMware encryption

Since vSphere 6.5, VMware can encrypt virtual machines “at rest” and on move with vMotion. Unlike Hyper-V, the encryption does not take place in the operating system (e.g. with bitlocker) but on the hypervisor level. The advantage of this is that it works independently of the operating system. A disadvantage of implementing VMWare is that an external key management server is required. It should be noted that the vCenter VM, which in our case should run on the same virtualization infrastructure, cannot be encrypted, because it is involved in the decryption process:

Unfortunately, VMWare does not provide a finished “architecture” in the sense of the Host Guardian Service as with Hyper-V.

Architecture

In the following picture we illustrate the project architecture. Later we will explain the relevant configuration.

Storage

One requirement of the architecture is that all (relevant) data is encrypted. Because both the KMS and vCenter VM are required for the decryption process, they cannot be encrypted with VMWare. Therefore, the project decided to store the ESXi installation, KMS, and vCenter on local self-encrypting SSDs. All other VMs are stored in the SAN. Thus, these components are also protected, should an attacker manage to get to one of the hard disks from the ESXi host.

ESXi Hosts

As described in the Storage chapter, ESXi was installed on encrypted SSDs. An image adapted by the hardware manufacturer was used for this purpose. From a security point of view, however, it is very interesting how the accesses to the ESXi hosts are configured.

Access Control

The following graphic provides an overview of the various access options

On the one hand, VMWare’s best practice was to have the hosts administered only by vCenter, but on the other hand, in the specific context, the decision was made to run vCenter as a VM on the ESXi servers that were managed with it. Thus, a direct access option is needed for emergencies. The access can be restricted by the so-called lockdown modes. In normal lockdown mode, no one has access except for users listed in the exception list or in the DCUI.access” parameter.

In strict mode, the DCUI service is also disabled, so access can only be configured using the exception list.

The following tables give an overview of the different configurations for root and local users:

Because vCenter Server is a virtual machine that resides in the single cluster of the inventory and uses encryption in the environment, strict lockdown mode is not an option. However, to have the ability to recover vCenter Server in an emergency, there must be a way to access the ESXi hosts. The solution to this requirement is normal lockdown mode with a configured DCUI.access setting for a user other than root. The ESXi Shell and SSH services remain disabled by default. This combination of settings results in a single access path to the environment in the event of a disaster. In an emergency, the non-root account is used to log in and disable lockdown mode in a dual control environment. Afterwards, the corresponding interfaces are then accessible again for the root user.

Further security controls

Furthermore, several security settings from VMware’s Security Configuration guide have been configured, which we will not go into here:

• Audit the status of the SSH and ESXi shell services
• Audit the “Exception user” list and the “DCUI.access” advanced setting
• Configure Syslog
• Configure a persistent log location (especially important for SD-booted hosts)
• Disable the managed object browser (MOB)
• Set session policies to project requirements
  • Auto-unlock-time
  • Account lockout policy
  • DCUI timeout
  • Password policies (for ESXI, vCenter OS and vSphere)
  • Shell timeout
  • Shell interactive timeout
• Verify Transparent Page Sharing setting
• Verify acceptance level of system is at least partner supported

x

vCenter

Access Controls

To limit access for the different administrator roles, there are, among other things, two central roles that an administrator can take on. These roles were defined by us during the project and form the basis for replicating the Hyper-V shielded VM feature.

• VMWare Admin: this user can create VMs, allocate more memory, etc. However, this user cannot establish a console connection to the virtual machine.
• VM Admin: In contrast, a VM Admin can establish console sessions to a VM, restart it etc. but he can not configure the Vcenter/ESXi itself.

This ensures that hypervisor administrators are separated from the virtual machines.nd.

Further security controls

The certificates used by vCenter for non-VMWare internal communications are replaced with certificates from the customer’s internal CA. This also meets a security requirement of the customer not to use self-signed certificates. In this case, it should not be forgotten that the certificates must then be included in the “normal” certificate lifecycle.

KMS

Unfortunately, VMWare needs an external Key Management Server solution for encryption. There are several providers on the market for such solutions. At our customer HyTrust KeyControl was used. The solution is based on a Free BSD hardened by the manufacturer, which is delivered as virtual appliance (OFA). The solution is operated as an active-active cluster.

Access control

Once the trust level has been configured with VMWare, no one actually needs to access the system, except when upgrading the solution. This is done by connecting the system to the ESAE Admin Forest and assigning the required HyTrust roles to a domain user. The use of the account is organizationally secured with the dual control principle, since the account also has the right to read and export keys.

VMs

Basically, not much configuration is required on the VMs. To ensure that all VMs are deployed identically, a VM template is used, and encryption is activated by a storage policy. All further hardening measures come from the Active Directory domain, as before via GPO.

Summary

VMWare also has many aspects to consider, but even here it is possible to effectively separate virtualization admins from workloads. The solution is almost identical to the security level of a Hyper-V HGS / shielded VMs system, but you have to improvise a bit at one point or another. There is not always best practice directly from VMWare, but what proved to be useful in the project context was a design review by an experienced VMWare consultant provided directly by the vendor. This review was extremely helpful, for example, to avoid making mistakes in the emergency access concepts. Furthermore, although the review did not give us an official “support stamp” (many lawyers would have to work together – you probably know what this is all about 😉), it did give us the certainty that an experienced consultant from the manufacturer found the solution to be complete and very good.