Title: In Defense of Sys Admins

Yes, I know, no one likes sys admins. Collectively they are the porcupines of the computer world, and anyone interacting with them is likely to get a sharp jab. Part of the problem is that networking is still at a relatively primitive state, and it is true that all networks run better with no users. In the circumstances it's hardly surprising that most of them have an antipathy towards users - since all their problems are caused by users!

Most sys admins, though, are dedicated and hard working, witness the fact that their networks stay up despite the users. It's all the more unfortunate, therefore, that their collective image has made them the fall guys for the Code Red worm debacle. As I write this (early August) variant two of the worm is starting to appear and variant one is rearing its head again. Code Red is clearly out there in the wild and is not going to go away any time soon.

Microsoft is trumpeting the fact that a patch to fix this problem has been available for several months. It is, they claim, all the fault of the sys admins for not applying the patches. Likewise I've seen various pieces in the computer press fulminating against 'inept' sys admins. In the USA there have even been suggestions that it should be against the law not to apply patches to software! The common thread in these articles is that they are written by people who don't have to do the grunt work.

Think about it for a minute. Why would any sys admin, who after all is going to be the one dug out of bed at 3am to sort out the mess, knowingly leave his or her network vulnerable? It doesn't seem to make sense.

In fact, it doesn't make any sense at all if you just look at this one patch. So what's going on?

Well there are actually three different groups involved here and they all have very different reasons for not having installed patches to Microsoft's IIS. We'll look at each in turn.

The first group is easy to explain - they don't know they are running IIS! I know this sounds difficult to believe, but it turns out that it is the case. In certain installations IIS is installed as default and automatically started at boot up. The people running the machines have them set up for something else and, once having got their required application running, never looked any further. If you don't know you are running a vulnerable program, then obviously you aren't going to patch it.

The second group is sys admins of small companies. They know they are running IIS, but for some reason they just haven't got round to patching it - so why not? Well to understand this you have to look not just at the patch, but at the whole situation they face.

Since the start of this year Microsoft alone has issued something in the region of 50 patches for its Internet related software. That's an average of nearly two patches a week. And that's only for the Microsoft Internet related software. Even such old chestnuts as Bind, Sendmail and XNTP have shown up security problems recently, engendering even more patches. All this before you even start to look at non-security patches and patches for other applications. It's easy to envisage a situation where you are getting a new patch every day!

And then there is the actual situation of sys admins in small companies. Such companies can rarely afford to employ a full time system administrator, so it's usually done part time by 'Fred' who also has his 'real' work to do.

I'm sure you've all met Fred. He is the guy who had one of the first Sinclair Spectrums way back in the 80s. When the firm bought its first PC, he was the one who figured out how to get the printer working. As time went by Fred became the informal company computer help desk, computer fixer and only person who knew how to put new toner in the printer.

When networking came along Fred was the one who went and bought a book on LANs and NetBios and festooned the office with coax...

In the nineties - long before Web Hosting had been invented - management decided that the company needed a 'Web presence'. Who better than Fred to run it (as well as doing his regular work, of course)! If Fred was very lucky he managed to persuade management to send him on a training course for all this new stuff. Not very likely though, so it was off to the bookshop again to get a book to go with his nice new copy of Front Page.

Fred dutifully set up a small static web site and email system. It was quite a struggle and he's been relatively happy to leave it to run itself, apart from when management got persuaded they should upgrade everything to run the latest versions of Microsoft's software. You can see these sites all over the web - the design is clunky, and the mail is answered erratically and very late. That's because Fred doesn't have time to do the mail every day, and he certainly doesn't have time to acquire new content for the site.

In fact he doesn't really have time to administer the site. It was embarrassing when everything ground to a halt because the disk had filled up with uncleared logs, but at least he was able to make sure that didn't happen again by switching off the logs. And in mean time he keeps getting notifications of patches for his machine. Since it seems to be running alright and Fred is a firm believer in the adage 'If it ain't broke, don't fix it,' Fred leaves well alone.

Funnily enough, the other morning Fred came in to work and discovered that the server was running incredibly slowly, and while he was trying to find out what was wrong it seized up completely. Eventually he hit the reset button, and waited while the machine did some incomprehensible magic to the file system. Nothing was lost and it seemed to be working OK again.

Fred knows that Microsoft products sometimes freeze up for no apparent reason, so he mentally shrugged his shoulders and decided that this was just another of 'those' occasions.

Then he went back to his work.

And the third group? They're the professional sys admins of the large companies. They've obviously got the patch installed now, but a surprising number of them didn't install it till the original version of Code Red went on the rampage and infected over a third of a million computers. Surely they've got no excuse?

Well actually, yes, they do have fairly sound reasons.

Unlike Fred these sys admins are running systems whose functioning is crucial to their companies - 'mission critical' to use the American jargon. Their systems are supposed to be running continuously - 24 hours a day, seven days a week. You have to stop something running to patch it. And remember what I said earlier about the number of patches? That implies stopping some part of the application every day! Not exactly continuous...

But that's not the only problem. At least one Microsoft patch this year has contained a bug, which stopped previously running applications from working. What a nightmare - you patch a security hole in your server and half of your ancillary programs won't work with the server any more.

Of course, theoretically, you should be able to just back out the patch and have everything running again in short order. But we live in the real world, and we know that's not the way it works. It can take hours, sometimes days, to repair the damage. And all the time management are hovering around demanding to know why you were fiddling around with something critical that was already working properly.

Small wonder that there is an inclination to leave putting new patches in until other people have reported whether there are any problems.

The system administrator's lot is not a happy one...

Given the current set up, I frankly don't believe that it's good enough to just blame the sys admins for security holes in Internet aware software. There are plenty of security breaches that sys admins can take the blame for - mis-configured firewalls spring immediately to mind - but not this one.

Partly we as programmers have to shoulder the blame for writing insecure software. Partly our companies have to take the blame for refusing to allow sufficient time for security testing (or any other sort of testing for that matter). Ultimately, I guess, the users are partly responsible for accepting and putting up with buggy software. The problem is that they've been doing so for so long that buggy software has become the norm.

There are signs that there could be a change coming, though, surprisingly enough driven by the insurance industry. It has been the case for some time that it was possible to obtain insurance and damage caused by hackers breaking into your computer. Recently, though, one of the companies involved announced that they were going to start charging higher premiums for companies using Microsoft servers. This was as a result of analysing claims on their policies.

I suspect this is only the tip of the iceberg. Once the big insurance companies get their actuaries onto the issue they will have real analysis to price their policies with. At that stage the quality of implementation will start to be a real issue because it will have financial implications that even non-techies can see immediately.

I think we'd all do well to make sure we are not found wanting.

Well, I think that had better be all for this issue, since I'm out of space.

Have fun programming.