Posted by: Greg Ness | November 12, 2007

Addressing VirtSec Challenges

In Weird Scenes I talked about the implications of virtualization and blade fabrics for the network security hardware industry.  This time let’s dig a little deeper into the core issues challenging security pros as they virtualize production environments, thanks again to Blue Lane CTO Allwyn Sequeira.  This entire series of blogs on virtsec has indeed been inspired by our Cupertino lunches over the last twelve or so months. 

The first issue is ignorance, in that many teams are just beginning to understand the new challenges and opportunities ushered in by the virtualization of production environments.  The business case is powerful, but the security case is usually misunderstood at best. A recent InformationWeek survey found that half of those questioned thought that virtual infrastructures were either no different or more secure than their physical counterparts.  In the rush to acquire the sizable benefits of virtualization, it is apparent that many teams have not done their homework.

Because virtualization creates a new layer between hardware and software, many new powerful capabilities are introduced.  Security and operations teams who understand these capabilities can take full advantage of them and enhance security – in fact, our belief is that virtualization will ultimately provide more effective and flexible security solutions than the status quo, provided organizations take advantage of the new virtsec capabilities; those who don’t either crimp back on the benefit/ROI or take additional security risks. 


One of these new capabilities is the ability to move virtual machines online and offline, unbeknownst to the rest of a team.  For example, when online images are patched offline images can be ignored, only to be moved back online and appear patched.  That vulnerable image can then be replicated and moved onto other hypervisors with different access privileges, and certain security assumptions.  It could be replicated again and moved and so on. 

Once security teams get burned -by an attack on any one of thousands of software vulnerabilities that already exist and by tools that are already in use by hackers- they may act to prohibit offline VMs.  Not only does that degrade flexibility and resource prioritization (eroding the business case), it also isn’t always an option to enforce a policy that can be so easily evaded. 


At my Archimedius blog one of the visitors shared his strategy, which involved limiting VMs to movement within specific security domains.  While this is a smart short term step, over the medium and long term it still represents limiting the infrastructure agility enabled by virtualization at the heart of the business case.  There is also the issue of lock-stepping server and network policies, at a time where virtualization is enabling more responsiveness. 

Even if you constrain movement the VMs behind well-tuned intrusion protection systems and firewalls may still be vulnerable.  Unpatched instances can appear minutes after signature tunes or vulnerability scans.  As I mentioned in The Beginning of the End, the heightened flexibility of virtualization introduces change factors that hasten the obsolescence of any static security measure.  You may limit VM traffic within security domains, but you still have the issue or percolating vulnerabilities within each domain microcosm.  Adding more domains means more management and observation resources and less flexibility. 


A similar notion is to architect the network around security (and other constraints).  This may be tolerable in the short term, but over time you could offset some of the benefits and flexibility of virtualization by constraining traffic to cordoned off VLANs.  In his NGDC deep dive presentation (which inspired this blog series about) Blue Lane CTO Allwyn Sequeira notes the eventual outcome “VLAN spaghetti”, a term used long before virtsec.  Taking it a step further you get heightened complexity and constrained mobility. 


Most network security appliances are very capable at inspecting traffic and spotting suspicious patterns or behaviors. They don’t have a reputation for accuracy because they have a critical, yet very broad mission to protect everything from anything.  Very few keep track of the software they’re protecting dynamically, without manual intervention.  Most of their policies are not specific to software, but rather classes of attacks or unusual traffic behaviors passing through their appliances. 

Because of their very broad (and challenging) mission, these appliances produce plenty of false alarms which also make them prone to suspicious traffic blocking and producing plenty of noise for security teams to decipher. Some are deployed in alert only while others turn on subsets of patterns (or signatures) in block mode.  According to one analyst firm, less than half of the known suspicious patterns delivered with NIPS are typically used to block attacks. 

Blocking good traffic isn’t good for maintaining server uptime or inspiring user confidence in the network.  That isn’t the worst, either.  A new class of threats emerged more than a year ago that is specifically designed to evade these static pattern matching systems.  Called polymorphic by some, these attacks mutate so that they cannot be recognized by static pattern matching technologies at the core of most IPS appliances.  VMs protected by these appliances, even if they’re in so-called secure zones, are still vulnerable to attacks that are already in use by hackers. 

Another issue is the inability to see inter-VM traffic.  VMs interacting with each other are a bit like a house of cards, in that one compromised VM can be used to compromise another.   Inter-VM traffic is not visible to NIPS products; a tainted VM can willy-nilly create havoc amongst the virtual servers, if an organization only relies on NIPS to safeguard their valued server resources. 

As I mentioned in “Weird Scenes” blade fabrics will present challenging deployment scenarios for NIPS.  Heavy reliance on manual tuning and “blindness” to certain kinds of traffic could create the equivalent of gridlock as server meshes get more powerful and traffic gets less predictable and enforcement policies get harder to implement. 


Host-based intrusion protection solutions (HIPS) actually make code changes directly on applications and operating systems- that make them more secure from attack.  They are also more operating system and applications savvy; many were architected with intimate knowledge about the inner workings and vulnerabilities of popular software programs.  They have their challenges, as the update cycle can introduce latency into the network and/or require reboots of updated machines. Apart from additional challenges like stealing significant CPU cycles, and having a hard time dealing with heterogeneous server OSs and distributions, the biggest issue is the same issue faced by Patch Management – anytime you require touching servers, you deal with significant availability, operations and business disruption challenges, especially in a virtual environment, given the proliferation, mobility, off lining & snapshotting of VMs.   

When servers are rebooted some don’t return to their previous working state.  Server downtime isn’t free or without its detractors. In many ways HIPS are like patch management solutions, which also change code within application or OS software. 

Both types of solutions are important in the new world of percolating vulnerabilities, but they do have their limits.  Offline (or suspended) images can be nearly impossible to update.  Availability is an important consideration for operations teams and/or a performance evaluation criterion for CIOs.  It isn’t taken lightly.  Yet the flexibility of virtualization also minimizes the downside risk of images not coming back to life.   

A previously saved image can be brought back online and updates can be tried again without the downside of a physical appliance going dead. So its very likely that HIPS and automated patch management will play an ever greater role in the virtsec world, yet won’t likely be able to keep up with the ever-moving and mutating world of VMs (without manual intervention). 

As the virtualized production environment moves to ever more powerful blade servers (blade fabrics) all of these shortcomings will become intensified.  The advantage of allowing uninhibited flexibility, online/offline and state changes without added security risk or operational friction will force security solutions from hardware appliance form factors into thin layer dynamic shields deployed on ever more powerful servers.   

Those shields will have to understand the traffic that passes through them in order to create highly intelligent partitions that enhance security without compromising availability or the very core of the business case that is driving virtualization into the data center.  Those who try to secure these fluid fabrics with manual labor and only partial visibility will likely be the slowest to virtualize production environments. 

It is inevitable that virtsec will replace netsec as virtualization forces enterprises to change their security postures in order to adapt to new challenges and new opportunities.  Highly intelligent shields can draw on the increasing power of the blade servers and scale accordingly without the need for incessant manual intervention and crude traffic inspection/blocking across fabrics of traffic pulsing between interdependent, moving, state-changing VMs. 


This post is one in a series about virtsec that started with my February 2007 Always on blog post about the impact of virtualization on the network security industry.=====================  

Disclosure: I’m the VP Marketing for Blue Lane Technologies, a winner of the 2007 InfoWorld Technology of the Year for security, Best of Interop 2007 in security and the AO 100 Top Private Company award for 2006 and 2007. Blue Lane is also a 2007 Best of VMworld Finalist in data protection. I’ve been a marketing executive at Juniper Networks, Redline Networks, IntruVert Networks and ShoreTel. I’ve been an Always On blogger/columnist since 2004. My recently launched personal blog is:    


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


%d bloggers like this: