Author Archive

Application Intrusion Detection

Wednesday, August 19th, 2009 by Paul Tyma

I recently worked with a UNIX security expert setting up a small pile of servers. We hired him to handle the total system security of the servers as those servers would be charged with storing highly sensitive customer data. In fact, the vendor for this data had very strict requirements as to how we were allowed to store this data. The requirements (something similar to PCI Level I) were dictated in a 40 page document where one of the rules literally required a monitored camera to be shining directly on the primary database server at all times.

My job was rather easy as I just had to hire someone to get the servers secure - not actually do the work. I happily found a crackshot security guy that had done large installations at maximum security. And he didn’t disappoint. I know my way around UNIX systems pretty well but he worked magic I’ve surely never seen.

One of the last things he did was to install an intrusion detection system. That is, a system that monitors all interesting system activity and sends out (a lot) of email about everything it noticed. Change a UNIX config file? Get an email. Add a user? Get an email. Issue a command as root? Get an email. Needless to say I started getting a lot of email.

At one point during a discussion we had about the system I asked him why we had the system at all. I mean, if this guy was so good (and to this day, I still believe he is) - an intrusion detection system is sort of like a prenuptial agreement. You get one in the event of something bad happening - but at the same time you can’t really get one without sort of admitting you *expect* something bad to happen.

Wasn’t his system secure? Did he really personally think he left a hole somewhere that would allow a hacker to get into the system?

He responded that he was 100% confident that after his work, the system was as secure as possible. He left no holes and patched all known vulnerabilities. He followed his declaration that of course - regardless of all that - of course the system can still be broken into. His response was utterly matter of fact. Of *course* break in was still possible - even likely.

I suppose this wasn’t a huge surprise. Despite locking the doors to our houses and cars - we still have alarm intrusion detection systems that detect entry. No one is foolish enough to think a car or house door lock keeps out a real thief. A reasonable defense after that is to get immediate notification when an intrusion occurs. Another reasonable defense that many people follow is to hide valuables. You typically don’t leave your stash of cash on your dresser - you hide it somewhere. In your sock drawer or under your mattress - but typically, even though you have a front door lock, and maybe an alarm system, its clearly prudent to add time and difficulty to a prospective thief’s job.

After some more discussion my security guy invited me to attend Defcon. A yearly security/hacker conference held in Las Vegas. I was intrigued and agreed. When I went to pre-register online - I found out that there wasn’t any such thing. No pre-registration, on-site registration only. And - no credit cards - cash only $120.

I arrived early at the convention and gave a nice lady my $120. In return she gave me an anonymous badge. She didn’t want my name or my address or my email. Everyone at the convention was to remain as anonymous as they wanted to be. The convention itself was jam packed. Wall to wall humans mostly in black t-shirts wth typically funny hacker slogans on them.

In one room they had situated 5 teams with networked computers in a competition where they simultaneously defended their servers and attacked the other teams’ servers. Another room had a large projected screen showing passwords of people at the convention that sent them unencrypted over the wireless (I shut-off my laptop and phone for the rest of the weekend). They showed the IPhone hack that could own a remote iphone with a single sent SMS message. I saw talks on hacking websites with request forwarding and one by an MIT student that beat stock spammers at their own game.

Needless to say - the folks at this conference weren’t messing around. To this day I get into discussions with developers as to the appropriateness of code obfuscation. Obfuscation technology has come a very long way past simple identifier renaming. These days it’s more about things like control-flow obfuscation and opaque predicates. Their argument usually circles around to some variation of “If an expert really wants your code - they’re going to get it”.

I couldn’t agree more. In fact, after attending Defcon I believe it even more. The only place I start to disagree is when they contradict themselves with solutions that they think will actually work. If your code is reachable, in any form, by users - its vulnerable. Software-as-a-service is often cited as safety. If that was true, then I wouldn’t have needed to hire a security expert. And he wouldn’t have told me point blank that no matter what you do - an expert can hack your server.

Protecting your software is not about 100% sure solutions - because there are none. It’s about throwing as many obstacles in the way of attackers as you can. Like I said - your front door lock hasn’t a prayer at stopping a thief - but for some reason you still always lock your door. Alarm systems don’t stop thieves; they just let them know that they’ve only got a few minutes to get their job done. Hidden valuables will be found - but the act of hiding them just makes the job harder.

The name of the game is risk and reward. Simple security measures by you that cause major headaches for attackers are what you’re looking for. Obfuscation makes deciphering your IP harder. Software tampering detection lets you know someone is tinkering with your application. By nature, attack surfaces start out pretty large. The best you can do is reduce that area as much as possible.

Posted in Application Lifecycle Management | No Comments »

Jumping into the Clouds

Monday, April 27th, 2009 by Paul Tyma

Cloud computing is definitely hitting the news today. In many respects, it is a pretty tempting scenario. As I heard Jeff Bezos of Amazon describe it, its rather analogous to generating your own electricity. You might be a baker or butcher or brewery - and yes, you surely use electricity, but why would you want to generate it yourself? Why not let some experts do that. In fact, let the experts do it en masse, and by doing so, can do it cheaply and efficiently.

You get the wondrous luxury of simply using computing resources by the drink. That is, you only pay for precisely what you use. Sounds like a no brainer and its also a no brainer why every large computing company has (or soon wil be) jumping into the mix with their own cloud.

Sun Microsystems is always a fun example. They own Java (for the moment, Oracle will soon of course) but sadly failed to own the development tool, application server, or primary java-housed architecture markets. Even though a "java cloud" makes fantastic sense, it would seem that historically they should and apparently did come to the game very late. In their defense, Sun might have actually been "too early" to the game. They had the Sun Grid all through the birth of the cloud. The failure was partly because it was called "grid" and partly on how you used it. But at its core, it was a cloud.

Amazon, which surely isn’t given enough credit as a technology company, was really the one to popularize the cloud. Even now they’re the most mature, and usable system. The offer an entire computing solution from CPU to database - all, as Jeff Bezos is wont to say, by the drink. Microsoft is wisely creating their cloud to run with their technology.

Applications in the cloud need to be scalable. That’s not to be confused directly with performant. Performance is the web-sense is how fast a user gets a response. Scalability is more akin to how performance changes as you add more users and more resources. You could in theory have an unlikely server that could only serve one user at a time. That’s clearly not very performant. However, if you can keep adding one server and supporting yet another user, you are actually scalable.

In fact, that’s basically perfect scalability which isn’t common. Usually some bottleneck (which is often a synonym for "database") becomes a bottleneck. After that point, adding more servers just doesn’t help.

Scalability is a deep and interesting topic but needless to say, at some level, you need it in order to have any sort of web application. On the other hand, its surprising about how poor performance you can actually get away with in many applications. In some sense its sad, but in the true business sense, "throwing more hardware" at a problem is often a good business decision (as opposed to expensive developer time to rip out a bad architecture). And the cloud vendors have no problem with you giving them poor performing applications. Sure - use up all that CPU you want, they’ll send you the bill.

Given all that however, the cloud is quite a compelling story. Let’s say you develop your nice, scalable, semi-performant website and deploy it to the cloud. Then you get some press - and overnight, your 100 users goes to 100,000. Great work if you can get it. Unfortunately however, if your architecture isn’t ready for it, only the first few thousand of those users will actually get to see your site. Because after that it will come crashing down.

In the non-cloud world, you can then scramble to store, buy some servers, head over to the Colo, install them, configure them and bring them online. And of course by then, the users are gone anyway. Alternatively, you can buy plenty of hardware up front. That hardware will be sitting idle most of the time wasting power and rent space. Not to mention the time you take to maintain it, install it, and configure it.

The cloud does all this for you. If you get 99,000 new users, you simply hit the control panel of your cloud and add a few servers. Or add a few hundred servers - whatever you need. And when the onslaught is over, you drop those servers. Not bad, eh?

Looking at it this way - who wouldn’t use the cloud? Assuming its reasonably priced (and so far, they seem to be) - why not?

There are several reasons actually. One is that the tools for cloud computing is still catching up. As you can imagine, traditional systems for monitoring applications fall apart in many cases. You simply don’t know where your application will end up "truly" running. And its not uncommon that it performs different in the cloud than in your testing. The cloud is a wonderful tool to throw your code over a wall and let it just run, until you might need to actually know more detail (this gap was one impetus in Preemptive’s Runtime Intelligence software - let your applications phone home and tell you how they are - from anywhere).

Another point is if your data has privacy concerns. If your business pulls credit reports, its not likely you’ll be able to store it in the cloud. If for procedural reasons only - the cloud just won’t meet the standards set forth by many security standard solutions. Quite often they simply dictate - you must own the key to the machine with the data.

Notably, Sun is introducing their cloud with the ability for you to run it on your own machines. We’ll see how this pans out but of course that puts all the maintenance and configuration back on you. In addition, there are open source solutions appearing that come close to letting you setup your own cloud if you want to.

Without a doubt, a cloud is a great way to start a small business or new project. Effectively outsourcing all the computer details with (nearly infinite) room to grow. You’re somewhat agreeing to lock-in to a vendor but that’s not necessarily bad or even true depending on the cloud system. Also, tools to support your cloud are trailing the cloud itself - so make sure your system choice has all the pieces you’ll need to maintain your applications even though, they’re somewhere in the cloud.

Posted in Runtime Intelligence | 1 Comment »