FOSS Build and Install Common Practices

The Free and Open Source Software common way of doing software project source build and installation.

Intro: | The Audience | The Problem | The Solution | Purpose
Source Structure and Usage | Installation Paths | Build Variables | FHS/GCS Decoder Ring

Introduction

The Audience

This document is for people who are creating projects intended to be built, installed, developed, and/or maintained on FOSS systems. Maybe you are someone who is new to FOSS and looking for "how is this done on FOSS" or you're someone who has been developing on FOSS for a while but want to make your source "work the same as everything else" so that it's easier for FOSS people to use, hack, and contribute to your project.

The Problem

One of the results of the modular and flexible nature of software designed according to the UNIX design philosophy is that there are many ways to do things. After some evolution, solutions to common problems are generalized and people developing new software learn to utilize the general solutions in order to same time and effort. But for some reason, after designing and writing software in this manner and the time comes for adding infrastructure to build and install the software, the same practice is not always followed. Build and install infrastructure is often an afterthought that is usually a "get it working well enough" effort, and usually reflects the habits of how the developer is used to doing things. It seems such a simple thing to write, and so there are many. In earlier days this was less of a concern, it's usually pretty easy to figure out how a project's build and install process works. But as the FOSS community has grown, the burden for end users to learn "yet another build and install system" increases with the exponential increase in projects.

The Solution

Fortunately some common practices and tools have emerged to solve these problems as well. These practices are primarily documented in the following standards and projects:

Purpose

People packaging software for distributions are the ones who most have to deal with the countless number of build and install systems, and are the ones that stand to benefit most from standardization and common tools. Also because of their experience in dealing with countless build and install systems across many projects, they have unique insight into the problem and the benefits of common solutions.

This document is intended to be a summary of the above resources, specific to build and install issues, covering the most important issues that FOSS projects should be aware of. It does not issues that relate to the implementation details of the software itself, such as C coding standards, documentation details, program behavior and interfaces, etc. except those related to the build/install. From an end user perspective, they care that the software builds properly and installs on the system in the correct locations, and not as much about the underlying details. That is not to say that FOSS projects shouldn't also care about those details (they are particularly important for developers that will modify and contribute to your software), just that this document doesn't cover those things.

Source Structure and Usage

The general practices based on the above standards (particularly GCS section 7.3) are to structure your source such that:

  1. The tarball: Users download a tarball with a name that is the name of the software, a dash, and the version. Example: foo-1.23 The software should unpack in, and only into, a directory of the same name.
  2. Helpful files: Once the tarball is unpacked, in the directory there should be
  3. Building: From the top level directory the user should be able to build the software on their system with something like "configure;make".
  4. Installing: From the top level directory the user should be able to install the software on their system with "make install". By default the software should install in the /usr/local hierarchy which is the location system administrators manage and expect the software to install (more on that below). Users should be able to adjust the install path by doing something like giving a flag to configure in the build step, setting a shell variable on the "make install" command line, or adjusting a variable in the top level Makefile. For example: Makefiles usually have something like:
    PREFIX=/usr/local
    BINDIR=$(PREFIX)/bin
    DOCDIR=$(PREFIX)/share/doc
    
    and autoconfiscated software has configure switches, for example
    $ ./configure --prefix=/usr
    
    It is expected that those packaging for an OS (install in /usr) or for a 3rd party application (install somewhere under /opt/) will set the install path this way. More details on this below.
  5. Permissions: When building is is always nice if the build (configure;make) can be done by a mortal user. The user running the install will need permissions to write to the install path and very often root permissions if installing in a system path and in order to chmod/chown files. Often, in the case of packaging, this can be accomplished with the fakeroot(1) tool so it's good if installing can work that way.
  6. When distributions or third parties package the software they will often modify the software to prepend the path of a pseudoroot within their build directory to the install path (but not the configure path, you don't want that location hard-coded in the build at all). Then they can run their packaging tools on the resulting directory. The usual way of doing this is adding a DESTDIR variable to the top level Makefile(but leaving it unset for the default case), and prepending it to any prefix variables. So adjusting our example above:
    DESTDIR=
    PREFIX=$(DESTDIR)/usr/local
    
    Then the packaging software can run,
    $(MAKE) install DESTDIR=$(CURDIR)/../pseudoroot
    
    or something similar.
  7. Packages should generally assume that the compilers will be able to find any needed libraries or header files in the standard search paths (so no hard-coded include paths) and shouldn't count on having addition software available nearby (like a source tree of a dependency at ../). Packages should also avoid shipping their own copies of build dependencies, and instead count on what they need to be on the system.

Installation Paths

The location files are installed on the system are a function of their purpose and who is installing them, and the document that describes this is the Filesystem Hierarchy Standard. Following the FHS does the following:

  1. logically and consistently organizes the files so users can find them, documentation, configuration files, user programs, etc all all in predictable places.
  2. prevents namespace collisions between O/S providers, system administrators, and 3rd party vendors. Concurrent versions can be installed simultaneously without conflict, with the users choosing the version they wish to use.
  3. allows for features such as:

Because of the above reasons, the Debian distribution considers any violation of the FHS it be a Release Critical bug and won't include any package that violates it. Fedora has a similar policy. If you want your software to be included in such distributions, it needs to "play well with others" by following the FHS.

Build Variables and Paths for use cases

FOSS projects generally use variables in their build systems to represent the install paths, to make it easy to change one place and have the change affect the whole build tree. The list is based on what autoconf uses, the GCS 7.2.5, the FHS, and general practice in the FOSS community.

NOTE: This is how I interpret the FHS and the variables and paths I think are correct for each use case. If you have a different interpretation let me know and maybe I'll agree and adjust.


FHS/GCS decoder ring (according to taggart)

File type/PathPurposeNameDefault valueParty Installing (with a $PROJECT of "foo")
sysadmin8distro3rd party
the name of the project, a convenience variablePROJECT(none)
pseudorootpseudoroot for testing/packaging purposes, a convenience variableDESTDIR""
base install prefixbase of the read-only program data tree, a convenience variablePREFIX/usr/local/usr/local/usr/opt/foo
user programsuser executable programsBINDIR$(PREFIX)/bin/usr/local/bin/usr/bin/opt/foo/bin6
admin programssysadmin executable programsSBINDIR$(PREFIX)/sbin/usr/local/sbin/usr/sbin/opt/foo/sbin6
config filesProgram configuration filesSYSCONFDIR$(PREFIX)/etc/etc/local or /usr/local/etc4/etc/etc/opt/foo6
init scripts10Program initialization script/etc/init.d/etc/init.d/etc/init.d/etc/init.d
default config11Program default configuration file/etc/default/etc/default/etc/default/etc/default
librariesRead-only, architecture-specific object code librariesLIBDIR$(PREFIX)/lib/usr/local/lib/usr/lib/opt/foo/lib6
program resourcesRead-only, arch-specific programs and libraries only used by the project's programsLIBEXECDIR$(PREFIX)/lib/$(PROJECT)2/usr/local/lib/foo/usr/lib/foo/opt/foo/lib6,9
header filesRead-only, arch-independent header filesINCLUDEDIR$(PREFIX)/include5/usr/local/include/usr/include/opt/foo/include6
program header filesRead-only, arch-independent, project specific header filesPROJECTINCLUDEDIR0$(PREFIX)/include/$(PROJECT)5/usr/local/include/foo7/usr/include/foo/opt/foo/include9
shared dataRead-only, arch-independent, program independent dataDATAROOTDIR
DATADIR1
$(PREFIX)/share
$(DATAROOTDIR)
/usr/local/share/usr/share/opt/foo/share
shared program dataRead-only, non-arch-specific, program specific dataPROJECTDATADIR$(DATADIR)/$(PROJECT)1/usr/local/share/foo/usr/share/foo/opt/foo/share9
man pagesManual pagesMANDIR$(DATAROOTDIR)/man/usr/local/share/man/usr/share/man/opt/foo/share/man
documentationproject documentationDOCDIR$(DATAROOTDIR)/doc/$(PROJECT)/usr/local/share/doc/foo/usr/share/doc/foo/opt/foo/share/doc9
local changing dataruntime dataLOCALSTATEDIR$(PREFIX)/var/var/local
(or /usr/local/var)3
/var/var/opt/foo
local program stateinstall specific, program specific state dataPROJECTSTATEDIR0$(LOCALSTATEDIR)/lib/$(PROJECT)/var/local/lib/foo/var/lib/foo/var/opt/foo/lib
local program cached datainstall specific, program specific cached dataPROJECTCACHEDIR0$(LOCALSTATEDIR)/cache/$(PROJECT)/var/local/cache/foo/var/cache/foo/var/opt/foo/cache9

If you providing an "agent" or "plugin" infrastructure, you probably want to have ways for them to "plug in" and one common way of doing this is by creating a ".d" directory that the installs can deliver files into where the main software can notice them.

NOTES:

0.) The GNU Coding standards doesn't define these, but projects definitely use them, so I invented names for them.

1.) The GNU Coding standards' $DATADIR and $DATAROOTDIR are the same by default, but defined in such a way that project data can be split from man/info/etc. Some projects define $DATADIR to the project specific $PROJECTDATADIR variable (invented here).

2.) Th GNU Coding standards says that $LIBEXECDIR should be $(BINDIR)/libexec, but very few projects use that.

3.) The GNU Coding standard says that $LOCALSTATEDIR would normally be /usr/local/var in the sysadmin case thus supporting the "can delete /usr/local to remove the program" rationale, but the FHS defines /var/local which supports the "mounts /usr read-only", "mounts /usr across multiple machines", or "performance tuned /usr and /var" rationale.

4.) The FHS allows either, due to conflicting goals. It also allows /usr/local/etc/ to be a symlink to /etc/local, so perhaps the latter is better.

5.) You might think that since they are arch independent that they might go in $DATADIR/include instead, but they don't and that directory doesn't exist.

6.) The FHS says that if the non-project specific version of this directory in exists in /opt (/opt/bin, /opt/doc, /opt/include, /opt/info, /opt/lib, and /opt/man), the project is allowed to install symlinks or copies in it, but the actual files should go under the project specific directory and the software should still continue to function normally in the absence of such links/copies.

7.) The FHS doesn't define, so this is a guess. It's hard to determined common practice without doing tons of "sysadmin" installs of projects, if you have data let me know.

8.) Many project install their own project directory under /usr/local. The FHS doesn't define if this is OK, but says

No other directories, except those listed below, may be in /usr/local after first installing a FHS-compliant system.
so that may leave the door open for installs after the initial system install.

9.) This /opt case is weird and doesn't fit the default variable value, so must be overridden.

10.) All the cases are the same. The LSB requires runtime implementation to provide install_initd and remove_initd scripts and recommends application use them, and defines how this namespace is managed.

11.) All the cases are the same. The FHS and LSB don't define /etc/default.


Matt Taggart <matt@lackof.org>
2008-04-09