Use your bonce

[fanf]

Vesta includes a purely functional programming language for specifying build rules. It has an interesting execution model which avoids unnecessary rebuilds. Unlike make, it automatically works out dependencies in a way that is independent of your programming language or tools - no manually maintained dependencies or parsing source for #include etc. Also unlike make, it doesn't use timestamps to decide if dependencies are still valid, but instead uses a hash of their contents; it can do this efficiently because of its underlying version control repository. Vesta assumes that build tools are essentially purely functional, i.e. that their output files depend only on their input files, and that any differences (e.g. embedded timestamps) don't affect the functioning of the output.

I've been wondering if Vesta's various parts can be unpicked. It occurred to me this morning that its build-once functionality could make a quite nice stand-alone tool. So here's an outline of a program called bonce that I don't have time to write.

bonce is an adverbial command, i.e. you use it like bonce gcc -c foo.c. It checks if the command has already been run, and if so it gets the results from its build results cache. It uses Vesta's dependency cache logic to decide if a command has been run. In the terminology of the paper, the primary key for the cache is a hash of the command line, and the secondary keys are all the command's dependencies as recorded in the cache. If there is a cache miss, the command is run in dependency-recording mode. (Vesta does this using its magic NFS server, which is the main interface to its repository.) This can be done using an LD_PRELOAD hack that intercepts system calls, e.g. open(O_RDONLY) is a dependency and open(O_WRONLY) is probably an output file, and exec() is modified to invoke bonce recursively. When the command completes, its dependencies and outputs are recorded in the cache.

bonce is likely to need some heuristic cleverness. For example, Vesta has some logic that simplifies the dependencies of higher-level build functions so that the dependency checking work for a top-level build invocation scales less than linearly with the size of the project. It could also be useful to look into git repositories to get SHA-1 hashes and avoid computing them.

It should then be reasonable to write very naive build scripts or makefiles, with simplified over-broad dependencies that would normally cause excessive rebuilds - e.g. every object file in a module depends on every source file - which bonce can reduce to the exact dependencies and thereby eliminate redundant work. No need for a special build language and no need to rewrite build scripts.

Comments

[simont]

In 1997 I had a summer job working with some code stored in ClearCase, and that did a very similar set of things: interface to source code repository via magic filesystem instead of checkouts in ordinary user filespace, special make utility automatically tracking dependencies, caching of build results so it could (in its terminology) "wink in" files that had already been built to the identical specification by somebody else.

Unfortunately it didn't speed matters up much in my case, since the back end of ClearCase is a big database file which, brilliantly, somebody thought it would be a good idea to mount via NFS from the other side of the Atlantic. A wink-in from that database took almost as long as a local compile would have!

I too have often thought that it would be neat to construct a make utility that automatically tracked dependencies via LD_PRELOAD, but never quite had the energy to try it.

[fanf]

Vesta has the advantage of a distributed replicated repository, so it should be local enough to be fast enough.

[hsenag.livejournal.com]

Clearcase has that too (you might have to pay more for it) but it's still slow as hell.

[ingulf.livejournal.com]

There is such a thing here: http://www.op59.net/yabs/readme.html

[fanf]

Yay lazyweb :-)