Turning Failure Into Success

Using failure to succeed

Unfortunately, life does not always work out the way we plan. Security is no different. Whether it’s servers that just won’t stay up, hackers who find (and exploit) that hidden application vulnerability or Mother Nature who wipes out your internet presence, you will eventually suffer an outage. The question is, what are you going to do about it?

Being excellent at anything in life, whether it’s sports, business or security, is not about getting it right all the time. It’s about discovering your failures fast and responding to them nimbly. In sports we do that by scrimmaging against opponents early and often so we can figure out how quickly they’ll see through our new plays. In business it’s by developing rapid prototypes of our business ideas and getting them in front of customers for feedback early on (see the Lean Start-up to learn more). So how do we do this in security?

The goal is not to avoid failure, but to fail faster

As security leaders, we are always tempted to engage in the intellectual exercise of creating a nice defense in depth strategy, and calling it good. In that case, we’re just looking to confirm that what we did previously was a good idea. But what I’m suggesting here is that our goal should be to find ways to prove that our defenses are flawed.

The concept is not entirely new to us. Most organizations have a penetration testing and vulnerability scanning program, where we’re looking for systems that are vulnerable to compromise. These are a great start, but all too often they lead more to a false sense of security instead of a real assurance of security. We have system administrators focused on vanity metrics like percentage of systems patched and number of vulnerabilities patched rather than proactively seeking out new ways to make their systems secure.

Below are three steps that will help you move from vulnerability management that merely looks good, to vulnerability management that can make you secure.

Incentivize finding vulnerabilities. A well-known management truth is that you get what you incentivize. What do you base your promotions, raises and bonuses on? If it’s rolling out new systems, system enhancements and just getting through audits, then that’s exactly what your administrators will do. To create a culture that continuously improves security we must reward that behavior. If you tie spot bonuses, public acknowledgement, and even promotions to how effectively employees can identify security vulnerabilities with your systems, then you will be amazed at how quickly your employees become experts at finding those vulnerabilities.

Learn from the chaos monkey. If your disaster recovery testing is performed annually, with everyone sitting in a conference room deciding together how to take down systems and move them, then you don’t know how your organization will react during a real disaster. For years Netflix has run their Chaos Monkey against their systems. The Chaos Monkey is a utility that randomly turns off individual server instances. Just like a real disaster, there’s no warning. This may be too extreme for your organization and your recovery time objectives (not everyone needs to be 100% reliable 24×7 like Netflix does), but the concepts can apply to all organizations. To find out how you’d handle a real disaster, find ways to subject your systems to conditions that mirror a real life incident.

Understand better what failure looks like. While the majority of hacks are still preventable by relatively simple security controls (hat tip to the folks at the Verizon Data Breach Report), I’m going to make the assumption that you’ve already implemented the basics (check out the SANS Top 20 or better yet, the Australian Top 4 if you haven’t). The fact is, even the most comprehensive security programs have some holes. Since we know that we will always have some residual risk of breach, our primary goal should not always be to add yet another preventative control. Instead, focus more on getting better at detecting when something has gone wrong.

There is a spectrum of options for accomplishing this; from a simple installation of a file integrity monitoring program to advanced correlation of system, application and networking behaviors. The key is not in putting the perfect detection system in place, it’s in finding a detection system that you can afford, and then USING IT! If you are regularly owning your systems to see what real incidents look like, and ferociously tuning out false positives this tool can become an incredibly efficient means of finding out when you’ve been hacked, and responding to the incident in minutes or hours instead of days, weeks or even months.

Failure is not the enemy. Complacency, overconfidence and misaligned incentive packages are. This problem isn’t rocket science, but it won’t solve itself either. Start gradually and verify that you’re making regular progress, and you can ensure that every vulnerability you discovery will only make your organization more secure.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s