Title: Building Java Applications

The most common answer I get when asking about a project build is, I'm sorry to admit, "Why don't you just use JBuilder?" - so my answer is practiced by now: JBuilder doesn't scale. I believe (I didn't try) that it may be useful for a home-based individual developer, but an environment which does not handle multiple languages, multiple users and variant (at least unit test vs. delivery) builds is simply not good enough for production work. That should not be taken as criticism of JBuilder (in fact, I'm told that some of its more irritating limitations have been fixed in latest versions), but rather as a criticism of "visual" environments in general. Large-scale software development requires programmable tools, going beyond "what you see is all you get".

And indeed there are programmable build tools. The problem of dependency management has been recognized (and solved) long ago, and its solution is now a de-facto standard build tool: make. From a popular manual [make]: "The make utility automatically determines which pieces of a large program needs to be recompiled, and issues commands to recompile them."

It would be perfect, if only it were true... Standard make does not automatically handle dependencies (not beyond "$NAME.o always depends on $NAME.c"), and in consequence, projects using make either generate most of the content of their makefiles by various add-ons (either make extensions, or external programs), or just ignore the problem and build from scratch every once in a while to flush the bugs out. A good overview of make problems can be found (perhaps unsurprisingly) in the documentation of an alternative build tool [CONS].

And if the general problems of make weren't enough to look for alternatives, there are also Java-specific ones: make's build model is based on C compilation (source files are compiled to object files, which are linked to make an executable), but Java doesn't work that way. Multiple class files can be generated from a single source, circular dependencies are quite common and the generation of jars is strictly optional.

Considering that the purpose of dependency analysis is to minimize compilation time, it is worth noting that javac is very slow to start up. For a few simple classes, it may not pay to check dependencies at all - the startup time (or perhaps it is time javac itself spends checking dependencies - see below) dwarfs time actually spent compiling, and so it saves practically no time to compile just part of the project.

On the other hand, there clearly are projects big enough that javac should not compile all their sources, every time - there's no justification for build times going up linearly with the size of the project. In theory, javac could handle dependency management internally. In practice, that functionality (the -depend flag of javac) existed in early versions of Sun's JDK, but it has been dropped in Java 2, and javac now recompiles only direct out-of-date dependencies, when their sources happen to be found on sourcepath or classpath. Sourcepath at least is an explicit command-line argument, but the recompilation of source files on classpath cannot even be turned off... Make has a particular problem with this "feature", because make builds normally create derived files in the same directory where the source is, and since javac must be able to find the class files (to compile other sources depending on them), it also checks the sources even when it shouldn't.

In the simplest case, a Java source file defines one class; if that class is public, it must have the same name as the file. But not all classes are public, and non-public (top-level) classes can be defined in any source file declaring classes of a given package. Java also has inner classes, defined inside a top level class and not necessarily having any name at all. As the JVM spec [JVM] says, "Typically, a class or interface will be represented using a file in a hierarchical file system. The name of the class or interface will usually be encoded in the pathname of the file." - but there are no guarantees, and certainly no published algorithm to derive the class file name. Sun's "anywhere" (as in "run anywhere") just doesn't seem to include Java environments from other providers...

But at least the package part is clear: a package corresponds to a relative directory. The absolute, top-level directories prepended to the relative ones are listed (together with jars - in this context, a jar is just a directory abstraction) in classpath, which is specified by a command-line argument (for command-line tools) or a user-settable option (for GUI applications), an environment variable or some hardcoded default (good only for the system classes). Pretty much everybody handles classpath the same way.

For the name part, let's get empirical: what class file names are actually created by Sun's javac? It appears that all inner classes have names constructed from multiple segments, separated by '$', where the first segment is the name of their enclosing class. Named inner classes (i.e. the only inner classes which can be referenced from files other than the one defining them) simply concatenate all names of their enclosing classes - for example, a class whose Java name is:

pkg.name.TopLevel.Inner.NestedInner.
                        DeeplyNestedInner

is compiled into a file

TopLevel$Inner$NestedInner$
                  DeeplyNestedInner.class

(in the pkg/name directory). Anonymous classes use two or three segments only. Unnamed classes have names constructed from two segments, while local classes (i.e. named classes declared in some method) need three segments (because local classes with the same name may be declared in different blocks). When there's no good name for a segment, a number is used instead; the numbers start from 1 and go up (when necessary to keep the whole name of every defined class unique) as the source is being parsed.

Overall, it seems possible to pry open the black box and get the class file names generated from a given source - provided we have a Java parser. Fortunately, Antlr is Java-based and (like any selfrespecting parser) parses its own programming language, so a utility for this task is within reach.

When parsing Java sources, it is naturally also possible to notice the used class names and construct a dependency graph. The hard part is determining what is a class name (as opposed to, say, field name - see Section 6.5 of the Java spec [JLS]). Basically, a dependency analyzer must know about all classes (on the classpath, and classes that may not exist yet, but are defined in the project's sources) and use that information to determine the meaning of a name. Admittedly, the most complicated scenarios should be rare - perhaps it would pay to cheat a bit... Also, since classes accessible across packages have one-to-one correspondence between their source file and class file, ignoring inter-package dependencies (and always recompiling all sources in an updated package) would considerably simplify the analysis.

A proof of this concept (including a demonstration of problems with file-level granularity) is werken.javad [Werken]. In my view, it suffers from being a make add-on, but it certainly can serve as a basis to build from - all code I wrote to research this article (and don't present it - it's just not good enough) is based on werken.javad.