Title: Onward to American Online...

In last issue's article I talked about the evolution of the multi-player game program Federation from its inception until we got it running on General Electric's GEnie service. By the start of 1995 it was obvious that GEnie was on the way down. There were a number of reasons for this. The division of GE that ran it - GE Information Services (GEIS) - had never understood the service. Bill Loudon, GEnie's founder had long since left, and it was drifting about rudderless. GEIS had long been pulling money out of GEnie to bolster its bottom line while refusing to invest in updated equipment, condemning its customers to use 2400 modems while everyone else had 9600. In addition, like most established services it refused to acknowledge the importance of the nascent Web.

Our contract with GE was not an exclusive one, and so for some time we had been looking to run Federation on the other services. It was about this time that we got a positive response from America Online. At the time AOL was big, but nothing like the size it is now. Its subscriber base was less than a million, and it was still an hourly subscription service.

After discussion with AOL it was decided that we would port the game to a Hewlett Packard Series 9000 RISC machine which AOL would provide and host in their machine room. The interface to AOL would be through a gateway originally designed for remote use. This sounded just fine, and I came back to the UK with all the documentation for Federation's programmer to study.

This gave us two things to do to get running on AOL - port the code to HP-UX, and interface the program to AOL's gateway system. The main thing we were worried about at this stage was porting to HP-UX, which at that time had a reputation for being a somewhat idiosyncratic flavour of Unix. In the event this proved not to be a problem and the port itself went reasonably smoothly. (Incidentally, over the last couple of releases HP-UX has moved into the Unix mainstream.)

What did cause problems was the gateway interface. There were number of contributory factors to this:

All three of these things caused us problems both while we were porting and on an ongoing basis. In particular, it meant that a two-week trip to the US by Federation's programmer stretched to over six weeks! By the time the first royalty cheque arrived we literally had no money left in the bank!

Let's look at these problems one at a time...

First, the gateway code had only been used internally by the group that wrote it. I suspect that the gateway project was something that grew out of an internal project, rather than something designed as an external interface from the start. The routines used by the gateway group itself were very solid and very robust, their everyday use had long since ironed out the bugs. There were, however, a group of features that weren't used by the gateway programmers, and these proved to be much flakier. I suspect they were added as an afterthought when someone realised that the interface could be used externally. In effect we were debugging this code as we went along.

This is the perennial 'library' problem. A company writes a library for internal use (3D graphics libraries are the most obvious culprits) and finds it very useful. Then some bright spark says 'Hey! We can sell this library and make back some of the money it cost us to develop.' Then features are found to be missing so they are added and the whole package goes onto the market as a 'revolutionary, full-featured xxx library'.

Unfortunately, the joins show. If you want to develop a library for external use it needs to be specified as such right from the start, all the features need to be properly tested, and the interface needs to be complete. Take it from me - never buy a third party library except from a vendor who only produces libraries. This is one of the main reasons why PD/Shareware libraries are often so full of promise but so disappointing when you come to use them.

Second, the documentation was inadequate and/or incorrect. The documentation was obviously meant to be of a proper standard - it came in a loose-leaf folder with a couple of updates. I have no doubt that it was fully documented from the point of view of the gateway group - but the documentation was part of the body of their work, E-mail, source code comments, group discussion etc. Unfortunately all we had was the written specification, which didn't incorporate this material.

This is another aspect of third party libraries - the documentation must be comprehensive. (Oh and by the way - programmers should be able to browse through documentation in the bath, so trying to cut costs by providing electronic documentation is a big no-no.) Documentation should come with a lot of simple examples, each covering one, and only one, aspect of the library. Nothing clever, nothing complex. I seem to remember that Greenleaf's DOS comms library was a sheer delight from this point of view.

Third, the group producing the gateway code didn't seem to have a senior manager to whom they reported. They seemed to be completely autonomous. This meant that there was no one we could go to to iron out problems with the code. In practical terms we had to write work arounds. In the event this wasn't such a problem at the time of porting, because the other people we were dealing with at AOL were highly competent and helped sort things out. Where it did become a problem was later on.

We finally got up and running on AOL in mid-1995. It was magic - we had more people than we had ever had playing the game. It soon became obvious, though, that AOL users were a different breed to those we had previously been used to. For the first time in its history Federation needed a game management staff - what are now called hosts and navigators - but that lies outside of the scope of this article. It also became obvious that we needed to tweak the game to cope with the increased number players (up from 10-20 players to 50-60).

It was while we were doing this that the first problems emerged with the gateway to AOL. There was a general problem with the gateway group making ongoing changes to their code and not telling us, which meant that our code would mysteriously 'break'. Then there was an ongoing project of theirs to port the gateway to HP-UX. Since they swore there was 'no difference' they didn't give us an opportunity to test their code with ours. One day they simply put it in without telling us. Needless to say, our code stopped working properly.

But the worst problem was changes in the interface specs. At one stage they increased the maximum length of the messages that the gateway sent us. Buffer overruns are incredibly difficult to spot! Fortunately, we had a very good working relationship with the Games Channel people who did everything in their power to smooth things over. In late 1996 AOL announced that it was moving to a monthly subscription, instead of hourly charges.

Sadly, they hadn't thought through the implications of this for multi-player games, and failed to keep them charged on an hourly basis.

The results were pretty obvious - everyone wanted to play the games for hours on end. This caused both game design and coding design problems for us.

The game design problems were easy to define, but more difficult to correct. Federation was designed to be played in a pay per hour environment. There was an underlying assumption that cost would limit player's ability to stay on-line indefinitely. There were also problems with the puzzles that required the use of what had become, with several hundred simultaneous players, very scarce objects indeed! The resolution of these problems is outside the scope of this article.

Sorting out the coding design problems was more of an immediate problem. At this stage we were running flat out with about 200 simultaneous players, and hordes more knocking at the door. We approached AOL for a more powerful machine, but one was not forthcoming. Not surprising really, since more players would increase the load on their network without in-creasing their revenue in any way. We were obviously going to have to solve this at the software level.

At this stage a simplified view of what was happening looked like this:

AOL->Gateway (AOL)->Gateway(Fed)->Fed driver->Fed Server

Players came to us from AOL via the AOL part of the gateway. This was the gateway code running on multiple machines that multiplexed the users through to our machine. Our part of the gateway, running on our machine, was one process for each of AOL's machines. These processes de-multiplexed the input and passed it on to the appropriate driver for the player. The driver in turn communicated with the server. Remember that there was a driver for each player and that all the links between processes were using TCP/IP (the standard communications protocol for the Internet).

Now, TCP/IP is a very good protocol indeed when you are communicating over services where packets might be delayed or lost. But that robustness comes at a cost in processing, memory and time. With (say) 200 players you would have 400 TCP/IP connections inside the Fed host machine - 200 going into the drivers from the gateway, and 200 from the drivers to the host. Now, none of these connections was going out onto the network, so no packets were going to be lost, clearly TCP/IP for the internal communications was overkill, and we couldn't afford the cost. Various options were looked at and we settled on UNIX pipes as a replacement. The result of just a simple change was staggering - maximum usage shot up to 350 overnight.

Some of you might be wondering why we were using TCP/IP in the first place. There were two reasons - portability and scalability. TCP/IP was, when Fed was written, more readily available and at that time most implementations of pipes went through a disk file. Only relatively recently have pipes started to go through memory. Our original ideas on scalability suggested that the way to scale up the game to handle more players was to have the server on one machine and the drivers on another machine. Using TCP/IP meant this could be done with little or no re-coding.

But...

We still needed to get more players on to our host machine - 350, it was obvious, was still inadequate. Where to go next? Well, we decided to look at the problems caused by having all these different processes running on our host machine. In particular we looked at context switching. Context switching is what the processor does when it moves from one process to another. Before it switches it has to save all the information about the old task (the context) and load in all the information about the new task. This takes time, lots of it.

To give you a feel for this problem let's scale down our 100Mhz processor by 100 million to human time scales - 1 cycle a second. The following table indicates the problem:

As you can see context switching is expensive! The only way to reduce the context switching was to run fewer processes. Reluctantly, we decided to fold the driver processes into our end of the gateway code. Why reluctantly? Because this would massively hit the portability of the code by tying a significant part of the Fed code into AOL's gateway code. None the less, we couldn't see any option - we were caught in the oh so familiar portability v efficiency dilemma. Nothing comes without a price, and the price of portability is less efficient code. On the other hand we would also halve the number of UNIX pipes we were using.

The results of this exercise were dramatic to say the least - usage shot up to 800+ players with processor resources to spare! We heaved a sigh of relief, and went on to start making game design changes to cope with additional load.

It was at this stage that AOL decided that they didn't want multi-player games any more...

But yet again I'm out of time and space, so the final part (I hope) of this saga will have to wait for the next issue.

Have fun programming.