Title: Professionalism in Programming #15

I like sweeping generalisations and tenuous metaphors. Sue me. I've also been doing my research. I reckon there are over 40 churches in the city I live in. Now each one of these is subtly different; different types of people go to each, they do different things. They have different concerns and ways of working. They exist in different areas. However they're all doing roughly the same thing (or should be).

What on earth has this got to do with programming? Well by a somewhat tenuous link, software development works in pretty much the same way. Not in that we all file into a building every Sunday morning (well, most of us don't). Perhaps to outsiders we do both appear to engage in bizarre rituals and invoke arcane rites to get our own way with things that are out of the control of normal human beings.

But what I'm really trying to drive at here is that there is no single way to program, no one methodology that solves all problems. There is no one programming language. There are different classes of problems to be solved in many, many different arenas. In this article we'll investigate this, discover some of the common domains we program in, see how they differ, and learn what the particular problems and challenges are for each.

Some of these arenas overlap. That's natural. Nothing's ever quite as clear cut as you'd imagine. The following descriptions are necessarily general, since each of these are big fields with lots of variation within. Nonetheless, this should give you a flavour of what's going on out there.

This is what most non-techies would think of when you mention the word programming. It's probably the broadest category we'll consider here.

It is programming 'applications' - self-contained programs - for single user workstation-like computers. In this world of development the focus is on end-users and how they use their desktop machines. Almost exclusively these days it will be Win32 programming. Win32 is the Microsoft unified API for its 32 bit operating systems (Windows 95, Windows NT, upwards to Windows XP and whatever other daftly named 32-bit OSes they care to release in the future).

There are many common languages for this kind of work; our familiar C and C++ are in there. We also see common use of Visual Basic and Delphi, plus a number of libraries and frameworks like MFC, and Qt. Although recently you hear a lot about Linux programming, that's still not where the applications work is these days.

Modern application level programming has come on a long way in recent years. We now have rich environments to work in, threading support, good user interface libraries, and facilities for network transparency, for example. There is much more operating system support provided making applications programming easier, but also meaning there's a lot more to learn as you get started.

With all this extra support available we have seen the bar raised on what a 'good' application is. What was acceptable application behaviour a few years ago is not today. People expect standard interfaces and look-and-feel, good responsiveness, and feature packed programs (even if they will only use a fraction of the features available). The huge 'professional' applications marketed today are the result of large development teams, with groups within them specifically focusing on usability issues.

As the times change we are also seeing more of a move towards webbased systems, applications that run on browsers. This cuts into the enterprise or distributed programming arenas (described below) somewhat.

Figuratively speaking our applications sit on top of rich system libraries. If applications programming receives a lot of support from the underlying system then someone's got to supply an underlying system: this is 'systems programming'.

It is generally for workstation machines too, as above, but not aimed at the end-users so much. This concerns a lot of the low level logic that interacts with the computer at a very basic level, and also middle-level support frameworks that don't interface directly to hardware, but provide important services to the system.

The kinds of work in this arena are: writing device drivers (controlling devices such as printers, storage media, output devices, etc), writing common shared libraries and utilities for managing scarce resources, implementing the actual operating systems controlling the computer, and providing components such as filing systems and network stacks. Even compilers and installation tool suites can come under this heading. Each requires a different approach and presents its own subtle set of challenges.

Computer technology is showing up everywhere in our daily lives, whether we're aware of it or not. Consumer electronics products are requiring more and more software for control, operation and management. More often than not this software is invisible to the device's user.

Embedded software developers work under tight constraints. A user interface (in the commonly understood sense) may not exist; in fact there may be no direct interaction with a user at all. Usually there's not much memory, and the code is running on slow processors. Embedded systems are designed to do one job, and to do it reliably. Consumer electronics is built down to the lowest price possible, meaning there isn't a lot of bells and whistles available to use, and you have to shoehorn a lot of software into the available device space.

It should appear as if the software is not there, the embedded device should just work. All the time. Failure is rarely an option in this kind of work. Contrast this to a desktop computer - it's a general-purpose machine. It has to be able to word process, play movies, browse websites, read email, do your accounts, etc. As users we've been conditioned to accept the odd crash and bit of instability here and there. Embedded work is a totally different ballpark.

A good example of this is the modern car industry. We see vehicles manufactured with several embedded systems, controlling all sorts of things. However, the users (in this case the driver and/or passengers) don't have to be at all aware that there are any microprocessors whirring away under the bonnet. They expect the car to just work. When an engine management system fails, the users become acutely aware of the software! Think also about mobile phones. They are obviously computer driven devices, but not many consumers think of them as a computer.

An embedded system is basically a combination of a small computer and either a real-time operating system, or a simple controlling program. It will have direct control over the hardware on the device. Embedded systems are usually developed for specific purposes. In general, embedded systems have only one piece of software running on them, no highly complex threaded programming environments are used.

You might consider that programming applications for hand held devices like PDAs is embedded-level or applications-level work, depending on where you stand.

Distributed programming develops systems that comprise of more than just the one computer. The World Wide Web is effectively a huge distributed system with information being stored on many computers across the world, and applications being available remotely via your web browser (for example, eBay and Hotmail). It's not all about web browsers, though. Working with and designing distributed systems brings in whole world of new problems.

You might need to 'distribute' a system for a number of reasons. Perhaps some types of computer are more suited to particular tasks than others. Perhaps the system is in high demand, and you can share the workload out among many machines on a network to improve performance. Perhaps there are physical location restrictions for certain machines that mandate you must distribute the system.

Effectively, you design a system that comprises a number of programs on different machines that all work together as a cohesive whole. The RC5 and SETI screensaver projects are excellent examples of this kind of work. Each machine in the system is designated a particular task (or tasks), each task is developed for its target machine, and then brought together in the final system.

These disparate parts need to be glued together somehow; each of the programs need to be able to effectively call functions on remote machines as if they were locally linked to their code. This is known as remote procedure call (RPC), and such facilities are provided by a number of available middleware technologies. These act as brokers for data transfer between machines, they describe how you discover and talk to services on other machines and how you publish your services for other programs to call. There are security issues (whose allowed to call who?), network latency issues (what happens if a remote function call takes too long?), considerations for balancing synchronous remote function calls with asynchronous calls, and more.

Some of these middleware systems employ object-oriented technologies, some take more of a procedural approach. The middleware is simply "connectivity software" and allows some degree of platform neutrality. As long as the middleware runs on a given platform, the calling code shouldn't care what that platform is - it could be a ZX spectrum for all we care - you will call functions on it in the same way. Of course, in the design of a distributed system you will select the appropriate hardware for each task. It's doubtful you'll see any ZX spectrums knocking about!

Commonly used middlewares are CORBA, the Java RMI and Microsoft's DCOM. Using these we split the system between user interface elements, the 'business logic' (real workhorse code), and any storage required (e.g. a database and query engine). These days the user interface will often involve a web front end.

Enterprise is one of those cheesy buzzwords floating around at the moment, more management speak than any programmer dialect. An enterprise is literally a business organisation. So enterprise programming is providing systems for entire companies, gluing all their separate systems together to form a unified cohesive whole. Enterprise programming almost always means the development of large distributed systems.

They'll commonly be deployed on a company intranet (internal network), and join the different 'departments' of the business together to improve workflow. The systems may or may not be customer facing. Once the organisation is running a computer system it's generally not too hard to have automated customer interaction. Perhaps an enterprise system will need to interface to other companies' systems too, for example to track the delivery status of goods.

This kind of work involves scientific, highly technical tasks making heavy use of mathematics. This is another highly specialised area, which requires writing applications specifically targeted at particular numerical problems. The programs are often aimed at supercomputers, the fastest type of computer, capable of massive number crunching operations. Although we're still living in times where the 'fastest' type of computer is changing from year to year, these are very expensive platforms, employed for specialized applications that require immense amounts of mathematical calculations.

Weather forecasting, for example, requires a supercomputer (or perhaps a gift of prophesy!). We also see supercomputers used for animated graphics, fluid dynamic calculations, and other areas that require highly complex mathematical investigation and calculation. A supercomputer is not a mainframe. The latter is a high performance computer designed to execute as many programs as possible concurrently, often used as a centralised computing resource in a business setting. A supercomputer channels all its power into executing a few programs as fast as possible. There are a number of different supercomputer architectures exploiting different technological advances, each requiring different algorithmic approaches to fully exploit their power.

This kind of work requires high performance algorithms to get as many calculations done as possible, to really capitalise on the performance of the computing platform. It is common to make use of carefully designed, heavily optimised numerical libraries, and to make explicit use of parallel processing, designing this into the computational algorithms and processes. This will involve both task and data parallelism, either performing many similar tasks on many CPUs at once, or pipelining the algorithm, performing different parts of it on different CPUs.

Of course there are other areas than these, some pretty well defined, others more ephemeral. I can think of safety critical systems as one example.

There is a commonality we can observe: each of these different development realms requires fundamental design decisions to be made to suit software to them. Applications level code is not generally suited to an embedded environment, or a workstation application design may not scale when applied to a distributed system. This means that software developers tend to specialise in particular fields, and learn to think in particular patterns that suit their 'world'. Understanding the very real concerns of each environment will make you a more flexible and mature programmer.