Abstract:
This document covers toc2, a build system for platforms hosting GNU versions of system tools, such as GNU bash, GNU find, GNU tar, and GNU Make (3.8.0 or higher).
This document is a work-in-progress. Please report errors and contribute feedback to the maintainer.
Maintainer: Stephan Beal (http://toc.sourceforge.net/contact/)
Table of Contents
1 Preliminaries |
1.1 License |
1.2 System and software requirements |
1.3 Typographical conventions |
2 Introduction |
2.1 Feature overview |
2.2 Source tree overview |
2.3 Three-minute mini-introduction |
2.4 Brief history of toc |
3 The configure script |
3.1 Arguments |
3.2 Running tests |
3.3 Exporting variables |
3.4 Filtering files |
3.5 toc2 shell API |
4 Makefiles |
4.1 toc2 makefile API |
4.2 Cleaning up |
4.3 Subdirectories |
4.4 Installation of files |
4.5 Distribution files |
5 Configure tests |
5.1 Running tests |
5.2 Writing tests |
5.2.1 Documenting tests |
6 Building C/C++ |
6.1 Dependencies |
6.2 Binaries |
6.3 Shared libraries |
6.4 Static libraries |
6.5 Creating MyLibrary-config helper script |
7 Miscellaneous tips and tricks |
7.1 Getting the name of the current makefile |
7.2 Symlinking files during build |
7.3 Sharing one toc2 installation amongst multiple source trees |
7.4 Skipping over ${TOC2_HOME} during builds |
You cannot guaranty freedom of speech and enforce copyright law."
Ian Clarke
''This [document] is encrypted with ROT26 encoding. Decoding it is in violation of the Digital Millennium Copyright Act.''
Anonymous Software Developer
The software described herein is released into the Public Domain, as is this documentation.
The toc2 software targets only platforms which host GNU versions of common system tools. As a rule, Linux platforms are supported without any special considerations, whereas Solaris, BSD, and similar platforms should work if they have GNU versions of their tools installed. By "tools" we mean sed, awk, find, tar, bash, and other low-level tools which are installed on all (or nearly all) Unix-like OSes. Vendor-specific versions of such tools which are not compatible with the GNU versions may cause unpredictable results or outright errors.
In particular, GNU Make version 3.8.0 or higher is required. Older versions of GNU Make will not work with toc2 because toc2 relies on the $(eval) function which was added in GNU Make 3.8.0.
Because the toc2 developer does not have an OS/X system, that operating environment cannot be tested. In particular, building C/C++ libraries on such systems reportedly does not work without modifying toc2 (what those changes need to be... no clue). If someone will donate such a system, or provide remote access to such a system, i will happily port toc2 to it.
The typographical conventions used in this document include:
Shell code is formatted like this.
Makefile code is formatted like this.
Shell variables are referred to in $THIS_FORMAT or ${THIS_FORMAT} and makefile variables use $(THIS_FORMAT).
This manual covers toc2, a rewrite of the toc (the other ./configure) tools. toc2 can be found on the web at http://toc.sourceforge.net/. This documentation assumes that the reader is familiar with writing shell code and makefile code, and will not make any effort to explain the idiosyncracies of either. Some useful resources include:
The Advanced Bash Scripting Guide covers Bash in detail and can be found at:
http://tldp.org/LDP/abs/html/
The GNU Make manual is an excellent resource for all of the features of GNU Make:
http://www.gnu.org/software/make/manual/
The book Managing Projects with GNU Make, 3.Xth Edition is a handy companion to the GNU Make manual and can be found at:
http://wanderinghorse.net/computing/make/
toc2 has the following notable features:
Designed to live completely in a project's source tree, making it immune to problems related to system-wide tools updates.
Builds run faster than with comparable Autotools trees, for reasons too complex to accurately summarize here. (Early tests showed speed increases of up to 20% for some C++ builds.)
Takes full advantage of GNU Make, the most feature-rich Make tool in the known universe.
While designed primarily for programming projects, it is just as suitable for managing trees of documentation and web site.
Is quite modular and very hackable.
Has been in daily use since 2003 and has proven to be a valuable, flexible tool.
And its mis-features include:
It does not support out-of-tree builds (e.g., read-only source trees which compile to another directory). Such support may be added in the future, but there are currently no concrete plans for doing so.
It targets ONLY Unix-like platforms which host GNU versions of the various common system tools, e.g. GNU find, GNU sed, GNU awk, GNU make, etc. It does not need any exotic tools, however - all of the tools it needs are installed on any sane *nix system. There are no plans whatsoever to change this limitation.
toc2 is unusual amongst build tools in that, instead of being installed system-wide, it is designed to live directly in your source tree. The main advantages of this approach include:
It is essentially unaffected by system-wide updates. Historically speaking, the system-wide tools which toc2 is built on top of are extremely stable, and toc has never been known to be inadvertently affected by upgrading the system or those tools.
It is not subject to backwards compatibility limitations. That is, when a new feature is added to toc2 or it undergoes a fundamental change, little or no attention must be paid to backwards compatibility because the changes will not trees using older versions of toc. Thus toc2 is fundamentally more flexible when it comes to introducing new features.
It can be hacked on a per-project basis without adversely affecting disparate projects (again, without compatibility concerns).
Most of its files (the project-independent ones) live in the toc2 subdirectory of your project tree. This directory is called the "toc home". In shell code it is referred to as $TOC2_HOME, and in Makefiles it is $(toc2.home).
The project-specific files which toc2 needs live in the root of your project's tree. These files are:
configure – this is a stub script which simple sets the project's name and version number and then calls the toc2 bootstrap script.
configure.PACKAGE_NAME – the main configure script. It is called automatically by the toc2 bootstrapping script.
postconfig.PACKAGE_NAME – this optional file contains project-specific help text. If it exists, it is shown when ./configure –help is run.
toc2.PACKAGE_NAME.make.at – this is a template makefile which should contain any makefile code intended to be included by all subdirectories of your project. It may include "at-variables", which get expanded during the configure process (see section ). At the end of the configure process, this file gets filtered and named toc2.PACKAGE_NAME.make.
toc2.PACKAGE_NAME.configure.make – this file is auto-generated during the configure phase and you should never need to touch it. It contains a list of all variables which are exported via toc2 during the configure phase. All of these variables are automatically imported into project makefiles during the build phase.
toc2.PACKAGE_NAME.help – if this optional file exists then it is shown when --help is passed to the configure script.
Makefile (or GNUmakefile) – while not strictly needed by toc2, in practice a top-level makefile is always used (and is required for some functions, like generating distribution tarballs (section )).
Where you see PACKAGE_NAME in the file names listed above, mentally substitute that with the name of your project. Throughout this document will will use either PACKAGE_NAME or simply PACKAGE to refer to the actual name of your source tree package.
Here we will give a very brief overview of what is necessary to create a toc2-based build tree from the ground up.
First, unpack the toc2 distribution file and rename it to whatever you like. Now edit the configure file in that directory so that it looks more or less like this:
#!/bin/bash
export PACKAGE_NAME=MyPackage
export PACKAGE_VERSION=0.0.1
export TOC2_HOME=${PWD}/toc2
. ${TOC2_HOME}/sbin/toconfigure
Note that the shell must be GNU bash, but the path to it may be different. The PACKAGE_NAME value must be a single token (no whitespace). The PACKAGE_VERSION must also be a single token, and need not conform to an X.Y.Z convention (e.g. "1" and "2.3.4-beta5" are both valid). TOC2_HOME must be set to an absolute path, not a relative path.
Now make the configure file executable by running chmod +x configure.
Next, create a file called configure.MyPackage and edit it to look more or less like this:
#!/do/not/bash
# ^^^ this helps text editors figure out that it's a shell script!
echo "Hello, world!"
toc2_test_require toc2_project_makefile
true
Create a file called toc2.MyPackage.make.at and edit it to look something like:
#!/do/not/make
# ^^^ this helps text editors figure out that it's a makefile
all: pre-build
pre-build:
@echo "building @PACKAGE_NAME@ @PACKAGE_VERSION@" \
"$(toc2.emoticons.okay)"
(Remember that if you copy/paste the makefile code from this document, the hard tab characters will get lost in the translation and you will have to replace them! Make is very picky about its tab characters!)
Now create a file called either Makefile or GNUmakefile and add the following:
#!/usr/bin/make
include toc2.make
We're now ready to try it out:
$ ./configure
...
$ make
...
building MyPackage 0.0.1 :-)
...
With that small amount of code in place, we're ready to move on to big and better things.
toc originally appeared in 2003 after i grew (extremely) frustrated with maintaining Autotools-based source trees. Nearly every time i upgraded my Linux distribution, i had to spend several hours going through my source trees and correcting new incompatibilities in my Autotools-based projects. When working on collaborative projects, this became more and more of a problem as my collaborators and i often had different Autotools versions on our systems. Thus, from the outset, toc was designed to be largely immune to side-effects involving system-wide upgrades. Historically speaking, toc has proven to be essentially immune to problems related to upgrading the host system.
Since its inception toc has seen heavy-duty use in nearly all of my C++ projects. In June 2007 i started work on an "underground edition of O'Reilly's book "Managing Projects with GNU Make, Third Edition". As part of that work, toc was rewritten and renamed to toc2. toc2 is fundamentally the same as toc1, but is more cleanly implemented and easier to hack on. A couple of the seldom-used features have been removed (e.g. qmake support), the dependencies generation support was reimplemented, and un/installation support has been improved, amongst other things.
toc gets its name from the conventional name of the source tree configuration script, "configure". The majority of C/C++ projects use a configure script which is generated by the GNU Autotools, so toc is known as "the other ./configure".
The file named configure is actually just a stub with a few lines in it. It defines the project's name and version number, then calls a toc-internal script which handles the rest of the process, including calling the "real" configure script, which is called configure.PACKAGE_NAME.
As a project grows, the only time you need to edit "configure" is when the version number changes, in which case you simply change the PACKAGE_VERSION variable. Both $PACKAGE_NAME and $PACKAGE_VERSION are important to the whole toc2 process, and may contain any non-space characters which are legal in file names.
Most of the work goes on in configure.PACKAGE_NAME, which contains all of the project-specific configuration code. It is a bash shell script which gets sourced by the toc2 bootstrapping process. The fact that it gets sourced, as opposed to run in a subshell, is important because it allows the configure script two-way communication with the toc2 framework. In other words, it allows the configure script to read variables exported by toc2 (or configure tests), as well as to export variables into that process. To avoid collisions with user-defined variables and shell functions, all toc2 shell variables are prefixed with TOC2_ and functions are prefixed with toc2_, e.g. $TOC2_HOME and toc2_test.
The configure.PACKAGE_NAME script has a small number of technical requirements which the user must guaranty not to violate:
It must use bash-compatible shell code.
exit must never be called. Doing so will unceremoniously end the configure process. If you feel that you need to call exit, try using toc2_die instead. If you want to end the configure process without killing it, then use return instead of toc2_die. If configure.PACKAGE_NAME returns a non-zero status code, then the framework will assume the configuration has died and will end with an error message.
To avoid hard-to-track problems related to the configure script ending with an error even though it doesn't seem that it should, the script should end with a command which returns a non-error exit status. Conventionally "true" is normally used for this purpose.
The configure script accepts a number of arguments and also parsed unknown arguments in a standardized way so that client-side code can easily check them. To see a full list of options which toc2 supports by default, run ./configure --help. Some of the options are briefly described here:
Argument | Behaviour |
--prefix=path | The prefix defines the installation path under which your software will be installed (if you use the installation features). If you are not going to install the package, this option is largely irrelevant, but it is particularly useful when building and installing software as a non-root user, in which case –prefix=$HOME is often useful. The prefix should always be an absolute path. |
--help-tests | Shows a selection list of the available toc2 tests, from which you can select a test to see its help text. |
--help-XXX | Shows the help text (if any) for the test named XXX. |
--enable-XXX=YYY | Sets the variable ${configure_enable_XXX} to YYY. |
--enable-XXX | Same as --enable-XXX=1 |
--disable-XXX | Same as --enable-XXX=0 |
--with-XXX=YYY | Sets the variable ${configure_with_XXX} to YYY. |
--with-XXX | Same as --with-XXX=1 |
--without-XXX | Same as --with-XXX=0 |
--XXX=YYY | Sets the variable ${XXX} to YYY. |
--XXX | Same as --XXX=1 |
--toc-quiet or --quiet | Makes the configure script and some makefile rules run "more quietly" (with less output). |
--toc-loud or --loud | configure script will output some otherwise superfluous information. |
--verbose | Disables --quiet and enables --loud |
--toc-debug | Enables debugging output in the configure script. Useful mostly for debugging toc itself. |
--toc-fail-fast | Causes configure script to exit if any toc2_test call returns a non-zero status code. |
It may seem rather anarchistic that toc2 exports any unknown arguments as variables of their own, but in practice this has proven to be very useful. Since toc2 cannot know which arguments you want to handle, it cannot choose to exit simply because someone passes an unknown (to toc2) argument. Client code is free to do custom argument processing in their configure script if needed, but in practice this has never been necessary.
When undefined arguments contain dashes, those dashes are converted to underscores. For example –my-option is exported to ${my_option} with the value of 1.
Historically there has been some confusion regarding when to use the "enable" flags as opposed to the "with" flags (both of which are adopted from Autotools conventions). The "enable" flag is normally used to mean enable/disable a particular feature of the current package, whereas "with" is used when referring to an external package. For example:
$ ./configure –-with-qt=/usr/lib/qt4 –-with-libz --enable-my-feature
It is up to the configure script writer to actually make use of the arguments. Some configure tests recognize their related enable/disable or with/without flags, and simply calling such a test will automatically enable/disable the feature based on those flags. Normally that means that the tests set some flag which can be used in makefiles or a config.h-style header file.
In toc2, tests are stored as individual files (shell scripts) under ${TOC2_HOME}/tests. Tests are not run directly, but are instead loaded via the toc2_test and toc2_test_require functions from inside configure.PACKAGE_NAME. For example:
toc2_test libz-dev
toc2_test_require libz-dev
The difference between those two is that the first returns the exit status of the test (zero on success, any other value on error), whereas toc2_test_require ends the configure process (does not return) if the test fails. The former allows client-side error handling and the latter does not. For example:
toc2_test libz-dev || {
toc2_boldcat <<EOF
zlib development files were not found! You can get zlib from:
http://www.zlib.net
EOF
toc2_die
}
if [[ ! toc2_test libbz2-dev ]]; then
toc2_die 127 "bz2lib development files not found!"
fi
The if/then block and the || operators perform the same job here, but if/then/else is easier to use if you need to handle both error and success cases:
if [[ ! toc2_test libbz2-dev ]]; then
echo "Non-fatal warning: bz2lib development files not found!"
else
... do something with vars exported by libbz2-dev test ...
fi
Internally, toc2 stores two types of variables for use in client makefiles and C code (normally used in a header named config.h, or similar). These variables are declared similarly to shell variables, using one of the various toc2 functions for exporting values. Here is how it is done:
toc2_add_make VAR=VALUE
toc2_add_config_h VAR=VALUE
toc2_add_config VAR=VALUE
toc2_export VAR=VALUE
Those functions behave as follows:
toc2_add_make adds the given key/value pair to the internal list of makefile variables.
toc2_add_config_h adds the key/value pair to the list of C-specific variables.
toc2_add_config adds the key/value pair to both the makefile and C-specific variables.
toc2_export works like toc2_add_config, plus is exports the variable into the current shell environment.
In practice, most toc code uses toc2_export for exportation. If, however, you want to export makefile variables which have names which are not valid shell variable names (e.g. a.var.with.dots) then you must use toc2_add_make to export those.
Exporting a variable more than once causes the most recent entry to be used, effectively overriding earlier entries.
How are these exported variables used?
First off, all of the make-specific variables are included in the file toc2.PACKAGE_NAME.configure.make, which is automatically created at the end of the configuration process and is included (indirectly) by client makefiles when they include toc2.make. That means those variables are immediately accessible via client makefile code. C-specific variables are not automatically exported, but must be "filtered" into a C file. How this is done is covered in section .
Users familiar with the GNU Autotools will immediately recognize that toc2 inherits its ability to filter certain files using variables embedded in the files in @THIS_SYNTAX@. In the Autotools such files typically have a file extension of ".in" (presumably meaning "input"), but in toc2 they have an extension of ".at" (derived from the embedded at-sign-delimited tokens).
During the configuration process, variables which get exported (section ) are accumulated for use in filtering files. Conventionally, certain makefiles and C/C++ header files get filtered this way, but in principal any text file can be filtered.
The framework provides several different ways to filter text files for so-called "at-tokens", as described below...
The most common use for at-filtering is to treat the input file as a makefile, which simply means that filtering it will replace any @TOKENS@ in the file which were exported using toc2_export or toc2_add_make. The simplest way to do this is to use the atfilter_file test:
toc2_test_require atfilter_file src/my_config.hpp.at src/my_config.hpp
The atfilter_file test is simply a small wrapper around the toc2_atfilter_as_makefile shell function. History has shown that there is rarely a need for a toc2_atfilter_as_header function (for C/C++ files), so there is no such function. The reason for this is that toc2_export is typically used in place of the more specific toc2_add_make and toc2_add_config_h functions, and that means that exported variables are all accumulated in the var list for makefiles, which means that toc2_atfilter_as_makefile is useful for both makefiles and C headers.
It is possible to filter files using arbitrary lists of @TOKEN@ values. Here is a very short demonstration:
propsfile=props.list
cat <<EOF > $propsfile
token_1=a b c
config_enable_foo=1
EOF
toc2_atfilter_file $propsfile INPUT_FILE OUTPUT_FILE
rm $propsfile
That will use the tokens defined in $propsfile for filtering INPUT_FILE, which should embed the tokens by surrounding them with @-signs, e.g. assume we have a header file we want to filter, which might look like this:
// This is a custom filtered header file. blah blah blah
#define config_enable_foo @config_enable_foo@
#define some_string_value "@token_1@"
Note the quoting of string tokens. The atfile parser (${TOC2_HOME}/bin/atsign_parse) has no knowledge of context, so it is up to you to ensure that the tokens are quoted properly for the given context. Note that the above sample input file does not quote the string for token_1. If it had, the quotes would have been expanded as part of the token and we would not need to quote it in the header file. But be aware that when passing token values on the command line to atsign_parser, the quotes around values are lost in translation. For example:
$ ${TOC2_HOME}/bin/atsign_parse token_1="a b c" < in > out
The quotes around "a b c" are lost when they are passed this way, so the header file above would need to have quotes around @token_1@ to make sure that the string is correct C code after filtering. This inconsistency may be considered to be a bug or a feature, depending on the needs of your project :/.
atsign_parse has several features not covered here. See the documentation (embedded in the program) for more information.
Aside from the functions covered above, toc2 provides a number of shell functions performing many operations related to configuring a project tree. Hard-core shell coders may want to browse the toc2 source code. The file ${TOC2_HOME}/sbin/toconfigure is a bootstrapping script which is called by the client-side "configure" file. toconfigure does some initial housework, but the "real code" lives in ${TOC2_HOME}/sbin/toc2_core.sh. Simply searching that file for "()" will lead you to the shell functions, all of which are documented.
This chapter goes into detail about how to use the more generic features of toc2 in your makefiles. To take advantage of toc2 you must include toc2.make in each makefile, like this:
#!/usr/bin/make -f
include toc2.make
That file is created by the configure script. Unlike most build tools, where one has to define the list of project directories in the top-level configure script, toc2 takes the anarchistic approach of searching your whole project tree (using GNU find) and creates the file toc2.make in each directory which contains a file named Makefile or GNUmakefile containing the text "include toc2.make".
Including toc2.make in a makefile has the following effects:
It includes the toc2 core framework makefiles, which provide common features such as cleanup, installation, and generation of distribution files (i.e., "make clean", "make install", and "make dist", respectively). It also provides a number of useful generic functions (see ${TOC2_HOME}/make/toc2-functions-core.make).
Includes all makefile variables which are exported by the configuration process (these live in $(toc2.top_srcdir)/toc2.PACKAGE.configure.make).
Includes $(toc2.top_srcdir)/toc2.PACKAGE.make. This is where project-global makefile code goes.
The file toc2.make is created from a template file, ${TOC2_HOME}/make/toc2.make.at, and can be customized on a per-project basis. The more generic (and recommended) approach, however, is to add all "global" makefile code to toc2.PACKAGE.make.at, which gets filtered during the configuration process to generate toc2.PACKAGE.make. If toc2 provides features which you do not want, or want to replace with your own, try looking in toc2.make.at and see if you can remove or replace the feature from there (e.g. by removing or replacing one of the included files).
toc2 provides a handful of core functions for performing common operations, such as finding files. These functions are documented in ${TOC2_HOME}/make/toc2-functions-core.make.
All APIs related to "extra" features (that is, most stuff) are implemented in feature-specific makefile snippets which either get included by toc2.make (if those features are globally useful) or can be included by specific client-side makefiles if needed (e.g. most of the basic C/C++ support lives in toc2-c.make).
For the list of makefiles which get included by default, see toc2.make. Then see the listed makefiles for their individual APIs.
The make-side functionality is made up of functions for use with $(call) and $(eval). By convention, code which should be used with $(call) is named toc2.call.XXX, whereas $(eval)-usable code is named toc2.eval.XXX. Many of the toc2.call functions simply forward their arguments to a toc2.eval snippet.
toc2 includes generic cleanup support. To add files or directories to the list of things which should be cleaned up (deleted) when make clean is called, simply:
package.clean_files += file1 ... fileN
package.distclean_files += fileA ... fileZ
Note that the += assignment is normally preferred. toc2 also keeps its own list of files to clean up, such as dependencies files, defined in $(toc2.clean_files), which also get removed during make clean. $(package.clean_files) should normally be defined on a per-directory basis, though it is often useful to add the following to $(toc2.top_srcdir)/toc2.PACKAGE.make.at:
package.clean_files += *~ *.o
The conventional make distclean is covered by $(package.distclean_files) and $(toc2.distclean_files). The difference between clean and distclean is that distclean should generally remove files which are created by the configuration process, and not those created during the build phase. That said, some developers feel that compiled binaries should always be cleaned up by distclean, and not clean.
Note that calling distclean also implicitly cleans up files which would be cleaned up by clean.
Client makefiles should never modify the $(toc2.xxx) variables, but it might be interesting to know about them, especially when writing framework-level makefile code.
If the list of files/directories to clean up is too long (it exceeds the command-line length arguments limit of the shell) then the cleanup rules may break. In practice this really never happens, but when it happens it can be fatal. One solution is to break your to-be-cleaned files into sets and define the new sets like this:
cleanset1.clean_files = file1 ... fileN
cleanset1.distclean_files = fileA ... fileZ
$(call toc2.call.define-cleanup-set,cleanset1)
With that in place, make clean will clean up the files listed in $(cleanset1.clean_files) during a separate step, and and distclean will clean those files in $(cleanset1.distclean_files). A restriction to be aware of is that your custom set may not have the name 'package' or 'toc2', as those names are already used by the framework.
Note that cleanset1.xxx does not need to be defined before the $(call...) is made, and it is not an error for either cleanset1.clean_files or cleanset1.distclean_files to be empty.
You may create an arbitrary number of custom cleanup sets, provided each has a unique name and you make the $(call) shown above for each one.
All but the most trivial projects have multiple directories, nested arbitrarily deep. To add generic support for building subdirectories, do one of the following:
If the subdirectories list is static (does not depend on any configuration information) then adding them is trivial: simply define package.subdirs before including toc2.make:
#!/usr/bin/make -f
package.subdirs := dir1 dir2 dir3
include toc2.make
all: subdirs
If your list of directories is dynamic (depends on configuration information), we have to do a small bit of work because the configuration information is not available until after toc2.make is included, but the subdirs support is generated during the toc2.make process. Simply do something the following:
package.subdirs := dir1 dir2
include toc2.make
ifeq (1,$(my_configure_option))
package.subdirs += dir3
subdirs: subdir-dir3
endif
all: subdirs
The reason that we need both an assignment to package.subdirs and a prerequisite to the subdirs target is to cover both the general case of make all (handled by make subdirs) and to help out targets which make use of the subdir list (e.g. make clean and make install), as those need a variable to hold the list of subdirectories (they cannot deduce it from the prerequisites).
Note that toc2 installs a generic rule called subdir-% which runs the all target in the subdirectory named %. Likewise, subdirs-% runs the target named % in all subdirectories defined by $(package.subdirs). Thus calling subdir-FOO is effectively the same as make -C FOO and subdirs-FOO runs make FOO for each $(package.subdirs) entry.
If you want to build your subdirectories in a very specific order, you can do so by ordering your subdir-xxx entries as prerequisites:
all: subdir-dir1 do_something subdir-dir2 something_else subdir-dir3
Whether or not subdirs need to be built before the current directory depends on the needs of the current directory, and this approach allows you to control that with fine granularity. As a general rule, subdirs need to be built first, but there are many cases where subdirs need to come last or be split up.
It is sometimes desirable to skip certain directories (or only traverse them) during the cleanup phase. toc2 has a special variable to check for this:
ifeq (1,$(toc2.flags.making_clean))
... we are doing 'clean' or 'distclean'
endif
Another common case is to only traverse certain directories when dist is running, in which case you can check to see if $(MAKECMDGOALS) equals dist.
toc2 provides a deceptively rich set of installation rules, allowing one to install an arbitrarily wide variety variety of file types, such as compiled binaries, shell scripts, DLLs, man pages, arbitrary documentation.
In this section we will just cover the basics, but if you are a makefile aficionado and want to take full advantage of the system, you can find all of the details in ${TOC2_HOME}/make/toc2-install.make.
toc2 divides files, perhaps quite artificially, into general categories based primarily on where they will be installed. Before we explain all of the sets, lets start off by assuming that we have sets called bins and libs, and we want to install a couple files using those rules. To do so, we simply:
package.install.bins := myApplication
package.install.libs := libmy.a
That will install myApplication to $(prefix)/bin and libmy.a to $(prefix)/lib. (Note that the libs set is intended for static libraries, not dynamic libraries, because dynamic libraries those require executable bits. More on this below).
That's all there is to it, but in fact you can customize the installation path and the installer flags for each set if you want to (or if you define new installation sets). See below for more info about this.
Here is an overview of the installation sets which toc2 supports by default, including the install path and any extra flags supplied to the installer:
Set name | Intended for | Default install path, relative to $(prefix)/ | Additional default install behaviour |
bins | Compiled application binaries | bin | Stripped and chmod 0755 |
sbins | Compiled "system" binaries | sbin | Stripped and chmod 0755 |
bin-scripts | Non-compiled application scripts | bin | chmod 0755 |
libs | Static libraries | lib |
|
package_libs | Static libs | lib/PACKAGE |
|
dlls | Dynamic libraries | lib | chmod 0755 |
package_dlls | Dynamic libraries | lib/PACKAGE | chmod 0755 |
headers | C/C++ headers | include |
|
package_headers | C/C++ headers | include/PACKAGE |
|
package_data | arbitrary data files | share/PACKAGE |
|
docs | arbitrary documentation | share/doc/PACKAGE |
|
man1 - man9 | man pages | man/man1 - man/man9 |
|
For most cases, these sets cover all of the necessary file sets. To use them, simply add files to $(package.install.SET_NAME). That will set up install and uninstall rules which will run during the make install and make uninstall phases, respectively.
If you want to change the default install path for a particular set, you can do so by setting package.install.SET_NAME.dest to a path. This path should be rooted at the configured $(prefix), e.g.:
package.install.package_dlls.dest := $(prefix)/lib/$(package.name)/plugins
package.install.headers.dest := $(prefix)/include/s11n.net/s11n
Do not add $(DESTDIR) to the path, as that is handled at a lower level by the core install/uninstall rules.
You can also modify the default flags passed on to the installation script (${TOC2_HOME}/bin/install-sh) by setting package.install.SET_NAME.install-flags. Be sure that you understand what arguments the install script accepts before changing these. The default behaviour, when no flags are defined for a given set, is effectively the same as calling 'cp'.
If you would like to create custom sets (e.g. sbin-scripts or myownthing), see ${TOC2_HOME}/make/toc2-install.make for how it is done.
toc2 provides the conventional make dist target to create a distribution file for your project. In the case of source projects, this distribution file normally contains all of the necessary sources for building your project, plus the toc2 files (remember, toc2 lives inside your source tree).
To add files to the distribution list, simply add a line to each subdirectory's makefile:
package.dist_files += file1 ... fileN
Be careful to use the += assignment because toc2 adds your Makefile (or GNUmakefile) to the dist files automatically. For the rare case where this is not desired, you can filter it out using $(filter-out), like this:
package.dist_files := $(filter-out $(firstword $(MAKEFILE_LIST)),\
$(package.dist_files))
We use $(firstword $(MAKEFILE_LIST)) because that will resolve to the real name of the current makefile, instead of hard-coding "Makefile" or "GNUmakefile".
Now running make dist will create a tarball containing all of the distribution files defined for your project, including all subdirs. The exact file(s) created depend on your build environment. The rules require GNU tar, and will fail if GNU tar is not available. They create a tar file of your dist files. If they find the gzip and bzip2 programs they will create gzipped/bzipped copies of the tarball, respectively. Additionally, if the zip tool is found then a zip file will be created. Which file you choose to distribute is up to you - you can delete the rest. As an additional feature, if the md5sum tool is found, the md5 value is calculated for distribution archive and is saved using the same name as the archive, plus an extension of .md5.
It is sometimes (though very rarely) useful to do your own dist processing, and this often has to happen only in the top-most source directory. You can do so by adding the code to your top-most makefile or to toc2.PACKAGE.make.at and checking for the top-most directory like this:
ifeq (.,$(toc2.top_srcdir))
... we are in the top source dir ...
subdirs-dist: my-presubdist
my-presubdist:
@echo "Do pre-subdir-dist processing here."
dist-.: my-predist
my-predist:
@echo "Do pre-dist processing here."
dist-postprocess:
@echo "Do pre-dist processing here."
endif
To understand the difference between "pre" and "pre-sub" it is helpful to understand the order of the dist processing:
dist: subdirs-dist dist-. dist-postprocess
In case it's not obvious, the dist-. target runs the dist rules for the current directory and subdirs-dist runs the dist target in all subdirs defined in $(package.subdirs). By adding dependencies for subdirs-dist and/or dist-., one can fine-tune the execution order of the dist process. Note that there are no default rules for dist-postprocess, so to implement it you must only define the target, instead of adding a prerequisite to it. That is:
# The preferred way:
dist-postprocess:
your rules go here...
# Unnecessary:
dist-postprocess: my-post
my-post:
you can do this, but it's unnecessary
In practice it is essentially never necessary to add pre- and post-processing, but if you need those features then there they are.
Keep in mind that it is necessary that, in the context of the dist target, subdirectories must be processed before the current directory because of the way the list of distribution files is generated (see ${TOC2_HOME}/make/toc2-dist.make if you want the gory details).
One caveat of the distribution support is that if ${PACKAGE_NAME} contains a plus symbol, e.g. mylib++, then that will probably confuse the tool which generates the tarball (it uses regexes, and treats the plus as a regex character). At some point this may be fixed, but so far it simply hasn't been problematic enough to warrant fixing it. If you're interested, the problematic code is in ${TOC2_HOME}/bin/makedist (a Perl script).
toc2 implements configuration scripts as shell code snippets which get sourced (not run in a subshell) by the configure script. Such scripts behave more or less like miniature applications, performing some arbitrary work and ending with a status code of zero on success and non-zero on error.
You can get help on using most tests by looking at the script code (see ${TOC2_HOME}/tests/TESTNAME.sh) or by running:
$ ./configure --help-TESTNAME
where TESTNAME is the name of the test you want help for (e.g. doxygen or gnu_cpp_tools). Alternately, use --help-tests to get a list of all test script, from which you can select a specific test to see the help text for (assuming it has help text).
Tests are normally run using one of two functions, toc2_test or toc2_test_require. The difference is that the former returns the error code of the test and the latter exits the configuration process if the test returns a non-zero exit status.
Test scripts live in ${TOC2_HOME}/tests and should have an extension of .sh. When calling a test, it is referred to simply by its base name, excluding the path and the extension. For example:
toc2_test my_test
toc2_test_require gnu_gcc_tools
Most tests search for a particular component and set one or more environment variables to flag whether or not it was found. Tests should document what variables they make use of so that configure script writers can easily figure out how to use them (see section ).
The main advantage of toc2_test_require is that if a test fails, it ends the configure process in a standardized manner, freeing the configure script author from having to deal with the error handling. The advantage of using toc2_test is just the opposite - the writer can control what happens in an error case:
if toc2_test check_for_xyz; then
echo "Got XYZ! ${TOC2_EMOTICON_OKAY}"
... do whatever here ...
else
toc2_die $? "Download XYZ from www.xyz.com and install it under ${prefix}"
fi
Of course, you need not exit in the case of an error, depending on the needs of your project. For example, a missing library might just trigger a flag to disable an associated feature of your software (e.g., if the libdl test fails then you might disable plugins support).
If you know how to write bash shell code then you know 90% of what you need to know in order to write configuration tests. First, you should make yourself familiar with the toc2 shell API (section ), then take a look at some of the test scripts under ${TOC2_HOME}/tests/*.sh. If you still have questions after that, get in touch with me and i'll write this documentation ;).
Test scripts should be documented so that they can more easily be integrated into projects. Doing so is trivial.
The first step is to add a very short description of the script near the top of the script file. This description will be shown when toc2_test or toc2_test_require are run. Simply add a line like the following to your test:
# toc2_run_description = looking for xyz
You may use the $@ environment variable in that description and it will be expanded. Most tests do not take arguments, but those that do may want to show them in the description text.
To add full-fledged help text to your test, simply add documentation text inside of comments, as one would normally do for shell scripts, and then add two special markers around the comments:
# toc2_begin_help =
# help text for the script goes here.
# It may span any number of lines and should describe what variables
# it expects (or optionally uses) from the configure process, as well
# as what variables it exports.
# = toc2_end_help
This text will be shown when the configure script is run with the --help-TESTNAME parameter. For example:
$ ./configure --help-libltdl
help for test: libltdl
Looks for libltdl. It calls toc2_export for the following variables:
HAVE_LIBLTDL = 0 or 1
LDADD_LIBLTDL = -lltdl or empty
LDADD_DL = empty or "-lltdl -rdynamic", possibly with -L/path...
Note that LDADD_LIBLTDL is specific to the libltdl test, whereas
LDADD_DL is used by both the libdl and libltdl tests.
The existing test scripts, under ${TOC2_HOME}/tests/*.sh, show many examples of good (or decent) documentation.
By far, the most common use for Make is the compilation of source trees, in particular C and C++ projects. Appropriately, toc2 provides add-on support for helping out such projects.
Because toc2 does not target only C/C++ projects, the core C/C++ support is not activated by default. Instead, it is implemented as a makefile which client makefiles should include. As a general rule it only needs to be included in directories which actually have C/C++ source code. Here is a snippet made for the toc2.PACKAGE.make.at file, which includes the core C/C++ support for all subdirectories which contain files which look like C/C++ source code:
ifneq (,$(wildcard *.c *.h *.cpp *.hpp *.flex *.y))
include $(toc2.dirs.makefiles)/toc2-c.make
endif
Of course, the wildcard may need to be modified for your tree.
Including toc2-c.make will add rules for compiling C/C++ source files to object files, transparent dependencies generation, and a handful of functions for compiling and linking.
C/C++ dependencies are automatically generated when the corresponding .o files are compiled. These files are called XXX.d, where XXX is an object or binary file name. All *.d files get included into the makefile and added to the cleanup rules automatically, so beware not to use files named *.d in your C/C++ directories!!!.
The tool used to perform the dependencies generation was adapted from the Linux kernel source tree, and can be found under ${TOC2_HOME}/bin/mkdep-toc2.c. That file is compiled automatically (to ${TOC2_TOP_SRCDIR}/mkdep-toc2) as needed. It compiles very quickly, has no external dependencies, and runs blazingly fast.
There are two approaches to building binaries. They are similar but incompatible, and should not be used in the same makefile. The first approach is to use the support in toc2-c.make, which can be set up like this:
myapp.BIN.OBJECTS = main.o argv_parser.o
myapp.BIN.LDADD = -rdynamic# optional linker args
$(call toc2.call.rules.c-bin,myapp)
all: myapp.BIN
That approach is really easy to use, requiring only:
Define xxx.BIN.OBJECTS
optionally define xxx.BIN.LDADD
$(call toc2.call.rules.c-bin,xxx)
add xxx.BIN to all target (if desired)
Another method uses yet another include file:
c-bins.list := myapp anotherapp
myapp.BIN.OBJECTS = main.o argv_parser.o
anotherapp.BIN.OBJECTS = foo.o
anotherapp.BIN.LDADD = -rdynamic
include $(toc2.dirs.makefiles)/c-bins.make
all: myapp.BIN anotherapp.BIN
The main difference between these two approaches is that the second one uses a more involved makefile code generation process and is more extensible.
Like building binaries, there are two different approaches to building shared libraries. The first, quick-n-easy way, is to use support available in toc2-c.make:
mylib.DLL.OBJECTS = foo.o bar.o
$(call toc2.call.rules.c-dll,mylib)
all: $(mylib.DLL)
package.install.dlls += $(mylib.DLL)
That creates rules for generating mylib.so (the file extension may differ on your platform), but does not set up any automatic installation rules. You may pass any number of library names to toc2.call.rules.c-dll, but xxx.DLL.OBJECTS must be defined for all of them.
The second approach is much more complete and includes proper install/uninstall support for the DLLs, as well as supporting DLLs with a version number:
c-dlls.list := mylib
mylib.DLL.VERSION := $(package.version)# optional, but see below!
mylib.DLL.OBJECTS = foo.o bar.o
mylib.DLL.LDADD = # optional list of linker flags$(LIBS11N_LDADD)
include $(toc2.dirs.makefiles)/c-dlls.make
all: $(mylib.DLL)
The version number of the DLL is optional, but if it is used it must be in X.Y.Z format. If it is not provided then these rules build mylib.so (again, the extension may be different). If the version number is provided then the library mylib.so.X.Y.Z is created, plus symlinks named mylib.so, mylib.so.X, and mylib.so.X.Y are created, all of which point to mylib.so.X.Y.Z. (This is a long-standing Unix convention.) Install rules are set up to handle the library as well. If you do not want the DLL to get installed then set mylib.dll.DO_INSTALL=0 before including c-dlls.make. That will disable the generation of installation rules for that target. If you would like a DLL to be installed in a different location than your other DLLs, set mylib.DLL.INSTALL_DEST=/path before including c-dlls.make.
Building static libraries is not nearly as intricate as building dynamic libraries, and it can be done like so:
mylib.LIB.OBJECTS = foo.o bar.o
$(call toc2.call.rules.c-lib,mylib)
package.install.libs += $(mylib.LIB)
Notice that the install rules are not automatically generated in this case, so we must use package.install.libs (or similar) if we want them installed. You may pass any number of library names to toc2.call.rules.c-lib, but xxx.LIB.OBJECTS must be defined for each one.
An increasingly common convention is for C/C++ libraries to ship with a script called LibName-config which provides the caller with information needed to compile and link clients of the software. Such a script can then be called from client makefiles (or configure scripts) to get the proper compile and link flags.
toc2 provides a test to generate a generic version of such a script. In brief, it is called as follows from the configure.PACKAGE script:
toc2_test_require PACKAGE_NAME-config \
PACKAGE_PREFIX=LIBMYLIB_ \
CLIENT_LDADD="-L${prefix}/lib -lmyLib" \
CLIENT_INCLUDES="-I${prefix}/include"
The exact arguments it expects are documented in full in ${TOC2_HOME}/tests/PACKAGE_NAME-config.sh, and that test uses the template file ${TOC2_HOME}/tests/PACKAGE_NAME-config.at to generate ${PACKAGE_NAME}-config in the top-most directory of the project.
When using this "test", the top-most makefile should get the following two lines:
package.distclean_files += $(package.name)-config
package.install.bin-scripts += $(package.name)-config
Run the generated script to see what arguments it supports. For example:
$ ./libs11n-config
./libs11n-config: shows information useful for configuring/building libs11n
client applications.
Usage:
./libs11n-config [options]
Options:
--libs : prints linker information about what libs clients should link to.
[-L/usr/local/lib -ls11n ]
--includes : prints INCLUDES information
[-I/usr/local/include -I/home/stephan/include ]
--prefix : prints the library's installation prefix
[/usr/local]
--version : prints the library's version
[1.3.0]
--toc2-make : prints a makefile snippet suitable for use with toc2 makefiles
--toc2-config : prints a snippet suitable for a toc2 configure script
--toc2-eval : like --toc2-config, but the output may be directly eval()ed.
This support "should" offer both an --includes and --cppflags option, but it currently does not because i fundamentally believe that clients of libraries should not require compiler flags specific to the library other than the includes path. You can of course modify the toc2 test (and the template file) to add arguments to your PACKAGE-config script (if you do, please send me your additions!).
This section imparts tidbits of knowledge related to solving specific problems which don't fall into a specific category.
While it may sound silly to ask, "what is the name of the current file", the question actually has several bad answers and one correct answer. The correct answer is:
this_makefile := $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
That resolves to the name of the current makefile, whether or not we are in the main makefile (which can be named Makefile or GNUmakefile) or an included file (which can have an arbitrary name). Be careful to use the := assignment operator, or you may get undesired results if you later include another makefile and then resolve the variable.
Because getting the name of the top-level makefile is so common in toc2 code, toc2 provides the toc2.files.makefile variable, which expands to the name of the current main makefile.
Getting the name of the current makefile is often useful, especially in the context of included files which themselves generate additional makefiles for inclusion. In those cases we normally want to make the generated makefiles depend upon the current makefile. To do so, we pick a unique variable name for our makefile and use it as a normal dependency:
feature-xxx.makefile := $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
$(feature-xxx.makefile):
generated_makefile = foo.make
$(generated_makefile): $(feature-xxx.makefile) $(toc2.files.makefile)
... generate content for $@ ...
-include $(generated_makefile)
Now $(generated_makefile) will be updated whenever $(feature-xxx.makefile) or $(toc2.files.makefile) are updated.
It is sometimes useful to symlink files, in particular header files, during a build. For example, my C++ trees always have the header files live in the same directory as the source files, but i set up $(toc2.top_srcdir)/include to contain symlinks to the headers, and then use "proper" #include lines in the source files. What that means is, e.g. foo.cpp uses #include <myproject/foo.hpp>, foo.hpp gets symlinked to $(toc2.top_srcdir)/include/myproject/foo.hpp, and the INCLUDES for my project have -I$(toc2.top_srcdir)/include added to them. This is easy to implement using toc2:
package.install.headers = foo.hpp bar.hpp
symlink-files.list := $(package.install.headers)
symlink-files.dest := $(toc2.top_srcdir)/include/myproject
include $(toc2.dirs.makefiles)/symlink-files.make
INCLUDES += -I$(toc2.top_srcdir)/include
all: symlink-files
The symlinks will be created when the symlink-files target is run and will be removed when the clean target is run.
Note that in practice the INCLUDES change is normally added to toc2.PACKAGE.make.at, as we normally want the INCLUDES path to affect all subdirectories.
While toc was designed to live inside the project tree which it manages, it can actually be shared amongst multiple project trees by installing it in a common location and setting ${TOC2_HOME} to the top-most directory of the toc2 tree (i.e. the directory containing the bin, sbin, make, and tests subdirectories). One technique which i've used several times is to host my projects in parallel source trees and changed their configure file to set TOC2_HOME=${PWD}/../toc2. (Remember, ${TOC2_HOME} must be an absolute path or things with break.)
The main benefit to this approach is, of course, saved space and less administrative hassle involved with having multiple copies.
The down-side to this approach is it makes creating a distribution tarball more difficult because the make dist feature does not support including files from outside the source tree into the distribution tarball. When working on projects where make dist is never used, or where the distributed version doesn't need toc2, then this is of course not a problem. This approach also makes the source trees more susceptible to compatibility bugs because hacking on the toc2 installation affects all of the projects which use it.
For the majority of source trees, traversing the toc2 subdirectories during a top-level make is a waste of time. Here is a snippet which you can put in your top-most makefile to cause toc2 to only be traversed during the distclean and distribution phases:
ifneq (,$(filter dist distclean,$(MAKECMDGOALS)))
package.subdirs += toc2
subdirs: subdir-toc2
endif
Note that traversing toc2 during the clean target is not necessary but during distclean it is because toc2.make is cleaned up during distclean and copies of it exist under ${TOC2_HOME}.