Security, et al

I’ve been noticing a lot hype about security risks with the rise of virtualization and much of it vague and short on specifics.  Also much of it seems to assume all the security we’ve built up on a physical server goes out the window when you migrate to being a virtual machine.  And that’s just not true.  A virtual server is the SAME server it was before it was P2V’d from a physical server.  The same authentication, access control, audit, network controls are active as they were before.  The virtual server sits on hosts and SANs in the same datacenter as the physical server did before.  So what has changed?  What are the new risks?

It’s in the virtualization infrastructure layer.  (Given the ubiquity of VMWare and my personal knowledge of that solution I’m going to use it to give various virtualization components real world names.)  How secure is the host, the storage and the host control server.  In VMWare terms I mean how secure are the ESX/ESXi hosts, the SANS and the vCenter servers?  That is the real question here.  Now that we’ve inserted a new layer between the guest OS and the hardware that layer’s immediate components and other components they depend on (the Active Directory forest that the vCenter servers belong to) need the same security controls anything else on the work needs.  Virtualization security is even more critical in some respects because someone who compromises that layer has low level access equivalent to physical access to every guest server and its data.

I have observed in recent audits that some areas of security and control of virtualization components is immature and does reflect consciousness how critical the virtualization layer is to security.  Before I dive into those areas let’s first talk about network security.  In the hype I referred to in the introduction, much is made about network security risks associated with virtualization and I’m not really seeing that.  Most servers are not behind internal firewalls or on heavily restricted network segments in the first place so moving them to an ESX/i host on a virtual switch doesn’t expose the server to new network risks.  For servers that had such controls when they were physical, you can set up the very same thing with virtual switches and firewalls.  If you need to mix external facing and internal VMs on a host you can set up completely different virtual switches and that’s what I IT shops doing.  There are more advanced attack scenarios involving a compromised VM where the attacker breaks out of that VM and into the host and possibly back up into other VMs but right now that is one area where we are ahead of the bad guys in the arms race. That is something to stay vigilant about though.

The other area of network security is the protection of the visualization infrastructure itself.  Unless IT shops totally ignore virtualization best practices they are going to implement multiple network cards on ESX/i hosts that allow you to completely separate guest, live migration (aka vMotion), storage and management (connections from VMWare vCenter and clients to the host) traffic.  In the audits I’ve performed, SANs and the management interfaces on hosts are isolated from rest of the organization’s internal LAN. 

This brings me to the areas of virtualization where I have found serious risks: privileged access, Active Directory and auditing.  First privileged access.  Anyone reading this knows that allowing administrators (especially multiple administrators!) use the built-in root or admin account on operating system is bad practice and risky for all sorts of reasons.  Everyone should have their own account.  The same goes for virtualization hosts only more so.  However in my audits I’ve observed many hosts being managed by the built-in root account shared among multiple administrators.  With a virtualization host, the risk of shared and insecure root accounts is multiplied by number and criticality of all the guest VMs on that host.  There’s no good reason for this bad practice and there’s no need to create additional root accounts on hosts for each admin either.  That’s because best practice is to lock down ESX/i hosts so that even admins don’t directly access them and instead go through the central management server (called vCenter in the case of the VMWare environment).  vCenter doesn’t share the same prevalence of insecure root access because vCenter integrates with Active Directory and allows you to leverage the AD accounts admins already have.  If for some reason a shop can’t follow that important best practice of avoiding direct access to hosts then you can configure the hosts themselves to integrate with Active Directory.

Speaking of Active Directory, I’ve uncovered another prevalent risk associated with the dependence of virtualization infrastructures on AD.  Let me be clear that I think directory integration and unified authentication is definitely the way to go but here are some risk factors people have been missing.  First there’s the fact that I often find virtualization management servers like vCenter to be members of the main AD forest.  Here’s what that means.  A Windows server belonging to a domain is exposed to any all risks in Active Directory and all domain controllers within that forest (remember the security boundary in AD is the forest not the domain).  Building on that, a vCenter server is the “boss” of all the ESX/i hosts connected to it.  Ergo, everyone who has domain admin authority anywhere in the AD forest and anyone who compromises a domain controller in the AD forest can take the vCenter server and from there compromise the virtualization infrastructure and ultimately any guest VM and its data.  A point I have to make, sometimes to the chagrin of some folks in IT, is that any outstanding risks from previous AD audits must be carried forward to the virtualization infrastructure audit too.  For instance, at one bank, there was a problem of a very excessive number of IT folks having domain admin authority to the main AD forest.  That was problem enough as a risk to AD and the Windows systems within that forest.  But with the virtualization management server as a member of that forest, now every virtual machine – even those running Linux and Windows servers in other forests are now accessible to that same excessively large group of AD admins.  Worse, in this organization remote access was widely available with no strong authentication.  So the entire virtualization infrastructure and countless servers running on it were vulnerable to compromise by a successful password based attack against any one of many AD admins. 

The solution?  First, think about the AD forest(s) that hold either your virtualization management servers (e.g. vCenter) or those user accounts with privileged access to the virtualization infrastructure (i.e. users with the Administrator role in vCenter).  Those forests, including each domain and domain controller within them, must be locked down and secure to a level appropriate for the virtualization infrastructure itself.  There’s no avoiding that.  Organizations with outstanding AD security issues with no resolution in site should really look at implementing a small, separate AD forest for providing directory and authentication services to their infrastructure including virtualization, storage and network components.  This small AD forest would be much more locked down and protected and careful thought should be given before implementing synchronization or trust relationships between it and other forests.  This may be at the price of maintaining additional user accounts for infrastructure admins but that is the price of security in this instance.  If trust is implemented it should such that the infrastructure forest is trusted by the other forests not vice versa.   If synchronization is implemented, password changes or other authentication data should not flow from other forests or directories into the infrastructure forest.  

The final risk area I’ve observed with newly virtualized organizations is a lack of auditing and log management for virtualization infrastructure components.  And in this case there’s no easy solution.  Virtualization management servers (e.g. vCenter) and hosts (e.g. ESX/i) can generate audit logs and it is crucial to enable this feature and subsequently collect, archive alert and report on this log data the same as is necessary with any other security critical components on your network.  Virtualization hosts like ESX/i are simple to accommodate since they can send events via syslog.  But management servers like vCenter are more problematic.  vCenter creates a number of text log files named vpxd-1 through 9 but my research has proven them to omit very important data and fail to resolve other key ID codes.  That’s not to say the audit trails aren’t there.  They are but they are trapped inside database tables.  In the case of vCenter the audit trail is stored in VPX_EVENT and VPX_EVENT_ARG tables within the vCenter SQL database.   Incomprehensibly command line interfaces like Get-VIEvent that pull data from these leave out critical event arguments as well as the name or ID of the event itself!  So the final option seems to be direct query of the SQL tables themselves with the necessary resolution of foreign keys to rows in related tables.  This presents an opportunity to log management and SIEM vendors to distinguish themselves with enhanced support for collecting enriched audit trails from virtualization infrastructures.