This article is intended to cover the “so what?” factor of the top VMware security features in modern versions of vSphere. This is not meant to be a technical deep dive; instead, we focus on the importance of each VMWare security feature using 3 basic criteria: What is it, why does it matter, and what should I do?
This article covers the following 5 VMware security features Trimarc recommends you configure in your vSphere environment:
Secure Boot with Trusted Platform Module for host security
ESXi host lockdown mode
Key Management Services & “new” vSphere Native Key Provider
VM Encryption
VMware Tools and VMware Hardware Version
Secure Boot with Trusted Platform Module for host security
What it is?
Starting with vSphere 6.5, VMWare has further developed the concept of “trust” within the vSphere technology stack. Enabling and leveraging UEFI Secure Boot on ESXi hosts with TPM (2.0) is a significant and meaningful way to secure the ESXi host stack from Power-On Self-Test (POST) to Operating Systems boot. With virtual infrastructure, the root of trust involves knowledge that every component in the stack is secure and has a known good value. Add to that the vCenter validation of that trusted boot stack to a reportable “attestation” and now you are looking at what VMware is branding “Trusted Platform” or the trust authority to operate a hardware root of trust all the way through the VM workload itself.
Why does it matter?
All workloads running on virtual machines (VMs) are affected by the security and trusted configurations of the underlying physical infrastructure. That said, it is not enough to only secure the POST, the drivers, or harden he OS of an ESXi hosts (all good things). There must be a way to establish the known configuration, to establish a root of trust along the entire vSphere stack from initial host power on to fully loaded VM. Establishing the root of trust and having vCenter report the attestation status of the hosts to administrators is critical to meeting this need.
What should I do?
Plan to establish the Trust Authority for your current vSphere (ESXi & vCenter) version. If your organization’s ESXi hosts do not currently have TPM chips, contact your hardware vendor or provider and order the appropriate item. It is then VERY IMPORTANT to read this article “Prepping an ESXi 6.7 host for Secure Boot“ (thank you Mike Foley). To start, run the secureBoot.py script (/usr/lib/vmware/secureboot/bin/secureBoot.py -c) BEFORE you switch on Secure Boot and enable the TPM on each host. The goal is that your ESXi hosts meet the minimum vSphere Installation Bundle (VIB) host acceptance level, in this case “Partner Supported”, to enable Secure Boot plus TPM. Plan to install them on your next maintenance window (thank you VMware clustering!).
Trimarc recommends the following steps:
Identify if existing and planned ESXi host servers include TPM chips and order hardware accordingly. All ESXi hosts should include TPM to enable enhanced security features.
Follow the guidance to prepare the ESXi host for Secure Boot and subsequent VMware KB articles for the modern vSphere versions.
Run a local command (/usr/lib/vmware/secureboot/bin/secureBoot.py -c) on each ESXi host to verify that the necessary VIB acceptance level is at minimum “PartnerSupported”.
Plan a maintenance window to perform the changes.
Migrate all VMs off the host.
Install TPM Chip in the ESXi hosts as appropriate.
Enable and configure Secure Boot and the TPM chip (guidance).
After configuring the environment using these steps, vCenter should be capable of reporting the attestation status of hosts.
Bonus
VMware has now updated their root of trust to extend even further and interconnect all of their concepts into a single framework “vSphere Trust Authority” for vSphere 7 and later. Also called the “Trusted Infrastructure”, it is a merging of the root of trust elements of Secure Boot plus TPM 2.0 with an external Key Management Service (KMS) and unique host cluster configuration. Though In my opinion, we are likely in the “wait and see how it goes” territory with regards to viable implementation. There appears to be great potential, but it is the first time that all of the pieces of the virtual stack have come together.
ESXi host lockdown mode
What is it?
Host lockdown mode is a security feature to restrict access to the hosts directly while being able to specify exceptions for individuals. This keeps direct access to hosts, depending on specific lockdown mode configured, largely blocked while allowing access to primarily be allowed through vCenter and its access controls, roles, and permissions. In lockdown mode, ESXi shell and SSH access are disabled with a specified exceptions list of accounts.
There are 2 primary lockdown modes:
Normal – The host is accessible through the local console or vCenter Server.
Strict – The host is accessible though vCenter Server. The Direct Console UI service is stopped.
The exceptions users list can be used for service accounts that require access to the hosts for various important line of business reasons, such as backup service accounts. This feature functions not to create a complete security solution but rather to reduce access to known accounts and allow flexibility for the planning of specific failure and recover scenarios.
Why does it matter?
One of the most often misunderstood or avoided security technologies is Host Lockdown mode. Some of this has to do with administrators not wanting to accidentally restrict regular administrative or troubleshooting break/fix access to hosts by “overly securing them”, often without understanding the implications. Thankfully VMware has realized the hesitation admins have and/or actual problematic scenarios that can occur with Host lockdown mode that they have written numerous articles and posted helpful video presentations to help clarify the white paper documentation that explains how this feature can really work and how to decide what is best for your organization.
Attackers often target direct host access to circumvent access controls normally enforced by a centrally managed vSphere environment using vCenter servers. Attack vectors for ESXi hosts include (but are not limited to) things such as rootkit exploits, accessing both local and network datastores, DoS attacks by compromising the host memory or device drivers, the nightmarish “escape the VM” and even ransomware attacks. ESXi hosts that are not regularly patched to the latest versions open themselves up to exploit for known bugfix issues and documented vulnerabilities.
What should I do?
Host Lockdown mode is an important layer of defense in depth to help protect the ESXi hosts from targeted attack since ESXi Shell and SSH access provides direct access that attackers and ransomware can easily leverage after compromising a privileged account. Get to understand the feature, evaluate the organizations administrative and recovery scenario needs, and enable Host Lockdown in the appropriate mode on all ESXi hosts. This article is where I would begin.
Trimarc recommends configuring ESXi Host Lockdown mode to “Strict” and add the accounts that require ESXi Shell and SSH access to the exceptions list. Regularly review this exceptions list to ensure only the accounts that require direct access are in this list.
Key Management Services & “new” vSphere Native Key Provider
What is it?
In order to access, enable, or perform cryptographic functions relating to features such as VM encryption a Key Management Service (KMS) is required. VMware currently supports 3 major approaches to KMS depending on the version of vSphere in the environment.
Standard key provider – independent and external (to the vSphere solution) KMS solutions are one way of enabling cryptographic ability in the vSphere environment. Once vCenter is configured with a KMS provider then new and existing virtual machines can be secured using VM Encryption.
Trusted Key provider – in vSphere 7 and newer, the key provider can be configured based on the “Trust Authority” if it is present in the environment. The Trust Authority makes access to the encryption keys conditional to the attestation state of a workload cluster. vSphere Trust Authority requires an external key server. While it may be the highest level of effort from a planning and implementation standpoint, this is also one of most complete configurations when considering the overall root of trust of the vSphere environment.
vSphere Native Key Provider – a new feature in vSphere 7 update 2 is the ability for vSphere to natively provide the keys for its own security features. This removes the original dependency on an external or independent KMS solution for vSphere to handle key-based security solutions such as enabling host cryptographic functions (Host encryption mode) or meeting the requirements for VM Encryption.
A good and relevant article describing key provider comparisons for deployment considerations can be found in the VMware Security guide.
Why does it matter?
This has been a challenging area to many customers as not all organizations have KMS implemented or have not prepared for the cost or labor necessary to configure one solely for the purposes of enabling vSphere security features. VMware has worked to meet these challenges with the introduction of a new feature in the latest version of vSphere 7, the Native Key Provider. As the name suggests, this is a vSphere “native” feature which is dedicated to the key provider needs of vSphere alone. It is also a great deal simpler from a planning and implementation standpoint than a traditional KMS solution. While adoption is in its early stages, this solution may enable organizations to proceed with the configuration of the security features required to safeguard workloads.
What should I do?
Review the key management options and weigh the benefits and drawbacks along with implementation and management overhead to determine the best deployment option for your organization. If you have an existing KMS that is redundant and resilient, then vSphere would directly benefit from that well deployed and maintained solution. If you have a slightly lower level of confidence in your KMS or as the vSphere administrator you have zero transparency into this very important prerequisite solution that could directly affect the stability and viability of your vSphere environment, then upgrade to vSphere 7u2 (7.0.2) and plan a vSphere native key provider integration.
Implementing a key provider allows for the activation of the following features:
Enabled VM Encryption. This provides encryption for newly deployed VMs and virtual disks as well as existing VMs and virtual disks.
Allow virtual Trusted Platform Module (vTPM) to be configured on virtual machines. This enables enhanced security capabilities that require or leverage TPM such as Windows Defender Credential Guard and Windows Bitlocker.
VM Encryption
What is it?
VM Encryption is, well, sort of just what it sounds like. It is the security feature that allows for the encryption of both the files that define the configuration of the virtual machine (VM) as well as the virtual disk(s) associated with a given VM. What makes this type of encryption “special” and not just another data at rest encryption solution (thank you storage providers everywhere for just taking care of that for me, signed a VM Admin) it is a very good way of securing VMs natively on vSphere as well as being the prerequisite for advanced VM security features that are also listed here in this article (see below).
Why does it matter?
vSphere native VM encryption is similar to other system encryption options that provide the ability to enable encryption on any/all VMs regardless of application or service (sound familiar?), but it’s more than that. VM Encryption is the prerequisite for advanced VM security features, particularly vTPM. To continue that root of trust that was begun with the ESXi hosts and carrying that all the way to the VM, you are going to need VM Encryption.
VM encryption is a huge step forward to protect sensitive server loads. The new role “No Cryptography Administrator” allows administrative privileges to be assigned to individuals while safeguarding against the ability to gain direct console access to VM Encrypted virtual machines. This also protects the VM storage files (Virtual Machine Disk or “VMDK” files) from being directly accessed (and potentially modified by an attacker) or downloaded which would provide an attacker all the data contained within the VM. Direct VM storage access (via VMDK) which has been one of the biggest security concerns with virtualization for years. If a VMWare administrator’s credentials were compromised, an attacker could download your Active Directory Domain Controller’s (DC) virtual disks compromising your entire environment (by extracting password data) or even temporarily take the virtual DC offline and modify highly privileged Active Directory group membership directly on disk before starting the DC’s VM again. Or even pick your scariest line of business application…or how about your backup servers! Yes, VM Encryption isn’t “just encryption”. You are not solving your problems just letting your backup solution or the Guest OS disk encryption just take care of it for you.
What should I do?
This is where it gets tough. VM Encryption has some pretty high level of effort (LOE) prerequisites. When you combine the strong security controls that VM Encryption offers along with the fact that VM Encryption is a key prerequisite for even more security features, I believe a strong case can be made for “it’s worth it”. There is good news on this front though, so keep reading. 😊 Please keep in mind that while there are several VM related security features below, not all of them have a hard requirement of enabling VM Encryption as a prerequisite. I have simply chosen to add them here, and comment on whether or not VM Encryption (or a KMS, which is a VM Encryption hard requirement) is relevant.
Enable Secure Boot
Just do it. It is the next link in the chain of root of trust and it is more secure than BIOS configuration for VMs. It does not require VM Encryption to enable; however, it does require the following to be considered:
Virtual hardware version 13 or higher
VMware Tools version 10.1 or higher
Enable EFI Firmware
Enabled Secureboot in Configuration Parameters
Is required to allow use of vTPM
Enable Virtual Trusted Platform Module
The ability to begin (or continue in the case of the vSphere stack) the hardware root of trust to the virtual hardware root of trust (I’ve never heard anyone say “virtual root of trust” #PatentPending 😊) is key for any organization that wants to level up their security posture. The virtual Trusted Platform Module (vTPM) is based on using VM Encryption to safeguard secrets for VM guest OS by replacing the function of a physical TPM chip with the VM Encrypted .nvram file for each virtual machine.
The reason the technology does not work like a physical TPM is that physical TPM’s are both is prohibitively small as well as serial based making it quite slow to keep secrets for, potentially, several hundreds of VMs on a given ESXi host. Another possible problem would be if the technology were to allow VMs to potentially be “hardware pinned” to those physical chips. If the technology were to allow this it would remove the ability to perform functions used in standard operation of a virtual machine (vMotion, cluster-based HA recovery, etc.). The following are considerations to enable vTPM:
Enabled EFI firmware
Enabled Secure Boot
Virtual Hardware version 14 or higher
VM Encryption (which itself requires KMS)
Is required or recommended to enable Windows 10 and Server 2012 security features (i.e., Credential Guard)
Enable Virtualization Based Security (support)
When you edit the VM and enable Virtualization Based Security (VBS) what is being enabled is just “support for VBS”. The necessary functionality is being exposed for the guest OS to then utilize and then the guest OS still needs to be configured to leverage it. The following are the considerations for enabling VBS Support on a given VM:
Enabled EFI firmware
Enabled Secure Boot
Virtual Hardware version 14 or higher
Enabled Support for VBS
Configure VBS backed features on the Guest OS
vTPM enabled (optional, but recommended)
The vTPM is not a hard requirement to enable VBS Support on a VMware virtual machine. Keep in mind though that if a virtualized TPM is not presented to the guest OS, some Windows security features like Windows Defender Credential Guard will be less secure (Microsoft KB article listed below).
VMware Tools and VMware Hardware Version
What is it?
VMware Tools is a suite of utilities that enhances the performance of the virtual machine's guest operating system and improves management of the virtual machine. The hardware version of a virtual machine reflects the supported virtual hardware features. These features correspond to the physical hardware available on a given ESXi host on which the virtual machine resides.
Why does it matter?
From a functionality perspective there are number of reasons why VMware tools should be installed and kept up to date on all individual VMs. The same goes for the Virtual Hardware version of a given VM. That said there are very specific reasons to meet the minimum requirements of the VMware tools and Virtual hardware versions from a security perspective, several of which are mentioned in the VM Encryption section above.
Not meeting the minimum Virtual Hardware version requirements means not being able to enable several key security features such as vTPM, VBS, as well as EFI firmware and Secure boot. Having VMs with older versions of VMware Tools version is actually quite a bit scarier. There are very specific exploits that are documented and actively used by attackers to compromise virtual machines with an old version of VMware Tools.
For example, if your organization has virtual machines operating with a VMware Tools version older than 10.1.0, all of these VMs are vulnerable to compromise using the “older” VIX API. This attack allows for direct guest access without being challenged for authentication.
Sounds bad right?
Trimarc detected VMWare Tools with versions older than 10.1.0 was in use at a customer environment during a Trimarc security assessment. Working with the customer’s operations and security teams, we were able to get to a “blinking cursor” and execute commands on the VM. Here is the worst part: the VM did not prompt for credentials, the account that was used only had vCenter privileges (and was an Active Directory domain user), and the VM was a Domain Controller! The Trimarc assessment team provided methods to immediately mitigate the issue as well as longer term strategies to facilitate business workflow securely. The important takeaway here is that VMware Tools versions and keeping them updated matters.
What should I do?
Keep your VMware Tools and Virtual Hardware versions up to date and ensure vCenter and ESXi host patches and versions are up to date as well (dependency). With vSphere version 6.5 and above, VMware has been working on releasing cross-version compatible VMware Tools versions as well as simplifying deployment and updates. Virtual Machines can be set to “update on reboot” if VMware Tools is configured to be managed by vCenter. Standalone installations of VMware Tools are more manual and can be challenging to keep updated. Trimarc recommends treating these updates as regular candidates for review at every Configuration Control Board (CCB) meeting or however your organization approves changes. After appropriate testing is performed, updates can be part of a regular maintenance/outage schedule like any other application or OS patch cycle. VM Snapshots are your friend and can help with any role back scenario in case VMware Tools updates go sideways (highly recommend using snapshots before changes like this are implemented).
A bit more care should be used in the Virtual Hardware Version updates. This is nearly a “click and reboot” update but VM snapshots will not help here as they do not affect this type of change in configuration/update. As with VMware Tools, it is recommended that the approach involves testing, review, and approval and use of maintenance windows for patch cycles.
Bonus
This helpful article describes how to use not only the straightforward PowerCLI commands to pull VMware Tools information but contributes additional information that allows collection of all related VMware Tools properties. Test extensively before using any scripts for production workloads.
Conclusion
While there are other security features that could merit mention, this article was not intended to be a complete list of all vSphere security, a vBrownBag session, or a white paper on the subject. Instead, these are features that Trimarc frequently sees in customer environments during security assessments that often go unaddressed or are not even implemented, configured, or planned for. Highlighting these 5 security features that organizations leveraging VMWare should be aware of and plan to configure was the goal.
Trimarc recommends implementing these security capabilities in the following order for ease of implementation:
Keep the VMWare Tools and VMWare hardware versions current.
Configure ESXi Host Lockdown mode to Strict.
Identify ESXi hosts that do not have TPM chips and order TPM for the servers that can accept them. After this, work to enable Secure Boot.
Review VM Encryption requirements (KMS) and configure on VMWare servers that support sensitive server loads such as Active Directory Domain Controllers.
References and Additional Information
Enable or Disable UEFI Secure Boot for a Virtual Machine https://cloudmaster.co.in/enable-or-disable-uefi-secure-boot-for-a-virtual-machine/
About vSphere Security (v6.7) https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-52188148-C579-4F6A-8335-CFBCE0DD2167.html
About vSphere Security (v7.0) https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-52188148-C579-4F6A-8335-CFBCE0DD2167.html
VMware vSphere 7 Update 2: New features and enhancements https://4sysops.com/archives/vmware-vsphere-7-update-two-new-features-and-enhancements/#vmware-vsphere-native-key-provider
VMware vSphere 7 Update 2: New features and enhancements https://core.vmware.com/blog/introducing-vsphere-native-key-provider
Introducing support for Virtualization Based Security and Credential Guard in vSphere 6.7 https://blogs.vmware.com/vsphere/2018/05/introducing-support-virtualization-based-security-credential-guard-vsphere-6-7.html
Windows Defender Credential Guard: Requirements https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-requirements
Protect derived domain credentials with Windows Defender Credential Guard https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard
VMware vSphere 7 and Encrypted vMotion – All You Need to Know https://www.starwindsoftware.com/blog/vmware-vsphere-7-and-encrypted-vmotion-all-you-need-to-know
VMware vSphere Security with Encrypted vMotion https://www.vembu.com/blog/vmware-vsphere-security-with-encrypted-vmotion/
VMware API Allows Limited vSphere Users to Access Guest OS https://www.securityweek.com/vmware-api-allows-limited-vsphere-users-access-guest-os
By: Demetrios Mustakas, Jr
Trimarc provides leading expertise in security solutions including security reviews, strategy, architecture, and implementation. Our methodology leverages our internal research and custom tooling which better discovers multiple security issues attackers could exploit to compromise the environment. Trimarc security services fit between traditional compliance/audit reviews and standard penetration testing/red teaming engagements, providing deep understanding of Microsoft and Virtualization technologies, typical security issues and misconfigurations, and provide recommendations based on our own best practices custom-tailored to balance operational and security challenges.
Trimarc performs security assessments that cover Active Directory, Azure AD & Microsoft Office 365, and VMWare. If you would like to improve the security of your VMWare infrastructure, let us know and we can discuss the Trimarc Virtual Infrastructure Security Assessment (VISA).
How to contact Trimarc
On Twitter @TrimarcSecurity
By email info@trimarcsecurity.com
By phone (202) 587-2735