FOSS Build and Install Common Practices

The Free and Open Source Software common way of doing software project source build and installation.

Introduction

The Audience

This document is for people who are creating projects intended to be built, installed, developed, and/or maintained on FOSS systems. Maybe you are someone who is new to FOSS and looking for "how is this done on FOSS" or you're someone who has been developing on FOSS for a while but want to make your source "work the same as everything else" so that it's easier for FOSS people to use, hack, and contribute to your project.

The Problem

One of the results of the modular and flexible nature of software designed according to the UNIX design philosophy is that there are many ways to do things. After some evolution, solutions to common problems are generalized and people developing new software learn to utilize the general solutions in order to same time and effort. But for some reason, after designing and writing software in this manner and the time comes for adding infrastructure to build and install the software, the same practice is not always followed. Build and install infrastructure is often an afterthought that is usually a "get it working well enough" effort, and usually reflects the habits of how the developer is used to doing things. It seems such a simple thing to write, and so there are many. In earlier days this was less of a concern, it's usually pretty easy to figure out how a project's build and install process works. But as the FOSS community has grown, the burden for end users to learn "yet another build and install system" increases with the exponential increase in projects.

The Solution

Fortunately some common practices and tools have emerged to solve these problems as well. These practices are primarily documented in the following standards and projects:

• GNU Make: Project and Manual - while it is a specific implementation, it is the canonical implementation and all FOSS projects use GNU make compliant syntax. Any competing implementation needs to follow the implementation exactly if they have any hope of being useful for any amount of FOSS source.
• GNU autoconf: Project and Manual - same comments as GNU make apply
• GNU Coding Standards
• Filesystem Hierarchy Standard

Purpose

People packaging software for distributions are the ones who most have to deal with the countless number of build and install systems, and are the ones that stand to benefit most from standardization and common tools. Also because of their experience in dealing with countless build and install systems across many projects, they have unique insight into the problem and the benefits of common solutions.

This document is intended to be a summary of the above resources, specific to build and install issues, covering the most important issues that FOSS projects should be aware of. It does not issues that relate to the implementation details of the software itself, such as C coding standards, documentation details, program behavior and interfaces, etc. except those related to the build/install. From an end user perspective, they care that the software builds properly and installs on the system in the correct locations, and not as much about the underlying details. That is not to say that FOSS projects shouldn't also care about those details (they are particularly important for developers that will modify and contribute to your software), just that this document doesn't cover those things.

Source Structure and Usage

The general practices based on the above standards (particularly GCS section 7.3) are to structure your source such that:

1. The tarball: Users download a tarball with a name that is the name of the software, a dash, and the version. Example: foo-1.23 The software should unpack in, and only into, a directory of the same name.
2. Helpful files: Once the tarball is unpacked, in the directory there should be
   • a README file that describes the software
   • an INSTALL file that explains how to install it
   • a file that describes the software's license, typically COPYING if using the GPL, or maybe LICENSE
3. Building: From the top level directory the user should be able to build the software on their system with something like "configure;make".
4. Installing: From the top level directory the user should be able to install the software on their system with "make install". By default the software should install in the /usr/local hierarchy which is the location system administrators manage and expect the software to install (more on that below). Users should be able to adjust the install path by doing something like giving a flag to configure in the build step, setting a shell variable on the "make install" command line, or adjusting a variable in the top level Makefile. For example: Makefiles usually have something like:

   PREFIX=/usr/local
   BINDIR=$(PREFIX)/bin
   DOCDIR=$(PREFIX)/share/doc
   

   and autoconfiscated software has configure switches, for example

   $ ./configure --prefix=/usr
   

   It is expected that those packaging for an OS (install in /usr) or for a 3rd party application (install somewhere under /opt/) will set the install path this way. More details on this below.
5. Permissions: When building is is always nice if the build (configure;make) can be done by a mortal user. The user running the install will need permissions to write to the install path and very often root permissions if installing in a system path and in order to chmod/chown files. Often, in the case of packaging, this can be accomplished with the fakeroot(1) tool so it's good if installing can work that way.
6. When distributions or third parties package the software they will often modify the software to prepend the path of a pseudoroot within their build directory to the install path (but not the configure path, you don't want that location hard-coded in the build at all). Then they can run their packaging tools on the resulting directory. The usual way of doing this is adding a DESTDIR variable to the top level Makefile(but leaving it unset for the default case), and prepending it to any prefix variables. So adjusting our example above:

   DESTDIR=
   PREFIX=$(DESTDIR)/usr/local
   

   Then the packaging software can run,

   $(MAKE) install DESTDIR=$(CURDIR)/../pseudoroot
   

   or something similar.
7. Packages should generally assume that the compilers will be able to find any needed libraries or header files in the standard search paths (so no hard-coded include paths) and shouldn't count on having addition software available nearby (like a source tree of a dependency at ../). Packages should also avoid shipping their own copies of build dependencies, and instead count on what they need to be on the system.

Installation Paths

The location files are installed on the system are a function of their purpose and who is installing them, and the document that describes this is the Filesystem Hierarchy Standard. Following the FHS does the following:

1. logically and consistently organizes the files so users can find them, documentation, configuration files, user programs, etc all all in predictable places.
2. prevents namespace collisions between O/S providers, system administrators, and 3rd party vendors. Concurrent versions can be installed simultaneously without conflict, with the users choosing the version they wish to use.
3. allows for features such as:
   • mounting programs (/usr) read-only for security and performance
   • sharing programs (/usr) across machines of the same architectures
   • sharing program data (/usr/share) across machines of different architectures
   • putting programs (/usr) and their data (/var) on separate disks, with difference sizes, configurations, and mount options, since they will have different usage patterns and performance needs
   • locally installed programs (/usr/local) can be shared across machines or safely deleted by the sysadmin without affecting the O/S

Because of the above reasons, the Debian distribution considers any violation of the FHS it be a Release Critical bug and won't include any package that violates it. Fedora has a similar policy. If you want your software to be included in such distributions, it needs to "play well with others" by following the FHS.

Build Variables and Paths for use cases

FOSS projects generally use variables in their build systems to represent the install paths, to make it easy to change one place and have the change affect the whole build tree. The list is based on what autoconf uses, the GCS 7.2.5, the FHS, and general practice in the FOSS community.

NOTE: This is how I interpret the FHS and the variables and paths I think are correct for each use case. If you have a different interpretation let me know and maybe I'll agree and adjust.

If you providing an "agent" or "plugin" infrastructure, you probably want to have ways for them to "plug in" and one common way of doing this is by creating a ".d" directory that the installs can deliver files into where the main software can notice them.

0.) The GNU Coding standards doesn't define these, but projects definitely use them, so I invented names for them.

1.) The GNU Coding standards' $DATADIR and $DATAROOTDIR are the same by default, but defined in such a way that project data can be split from man/info/etc. Some projects define $DATADIR to the project specific $PROJECTDATADIR variable (invented here).

2.) Th GNU Coding standards says that $LIBEXECDIR should be $(BINDIR)/libexec, but very few projects use that.

3.) The GNU Coding standard says that $LOCALSTATEDIR would normally be /usr/local/var in the sysadmin case thus supporting the "can delete /usr/local to remove the program" rationale, but the FHS defines /var/local which supports the "mounts /usr read-only", "mounts /usr across multiple machines", or "performance tuned /usr and /var" rationale.

4.) The FHS allows either, due to conflicting goals. It also allows /usr/local/etc/ to be a symlink to /etc/local, so perhaps the latter is better.

5.) You might think that since they are arch independent that they might go in $DATADIR/include instead, but they don't and that directory doesn't exist.

6.) The FHS says that if the non-project specific version of this directory in exists in /opt (/opt/bin, /opt/doc, /opt/include, /opt/info, /opt/lib, and /opt/man), the project is allowed to install symlinks or copies in it, but the actual files should go under the project specific directory and the software should still continue to function normally in the absence of such links/copies.

7.) The FHS doesn't define, so this is a guess. It's hard to determined common practice without doing tons of "sysadmin" installs of projects, if you have data let me know.

8.) Many project install their own project directory under /usr/local. The FHS doesn't define if this is OK, but says

No other directories, except those listed below, may be in /usr/local after first installing a FHS-compliant system.

9.) This /opt case is weird and doesn't fit the default variable value, so must be overridden.

10.) All the cases are the same. The LSB requires runtime implementation to provide install_initd and remove_initd scripts and recommends application use them, and defines how this namespace is managed.

11.) All the cases are the same. The FHS and LSB don't define /etc/default.