[[bbv2.extender]] = Extender Manual [[bbv2.extender.intro]] == Introduction This section explains how to extend B2 to accommodate your local requirements -- primarily to add support for non-standard tools you have. Before we start, be sure you have read and understood the concept of metatarget, <>, which is critical to understanding the remaining material. The current version of B2 has three levels of targets, listed below. metatarget:: Object that is created from declarations in Jamfiles. May be called with a set of properties to produce concrete targets. concrete target:: Object that corresponds to a file or an action. jam target:: Low-level concrete target that is specific to Boost.Jam build engine. Essentially a string -- most often a name of file. In most cases, you will only have to deal with concrete targets and the process that creates concrete targets from metatargets. Extending metatarget level is rarely required. The jam targets are typically only used inside the command line patterns. WARNING: All of the Boost.Jam target-related builtin functions, like `DEPENDS` or `ALWAYS` operate on jam targets. Applying them to metatargets or concrete targets has no effect. [[bbv2.extender.overview.metatargets]] === Metatargets Metatarget is an object that records information specified in Jamfile, such as metatarget kind, name, sources and properties, and can be called with specific properties to generate concrete targets. At the code level it is represented by an instance of class derived from link:#bbv2.reference.class.abstract-target[abstract-target]. footnote:[This name is historic, and will be eventually changed to `metatarget`] The link:#bbv2.reference.class.abstract-target.generate[generate] method takes the build properties (as an instance of the link:#bbv2.reference.class.property-set[property-set] class) and returns a list containing: * As front element -- Usage-requirements from this invocation (an instance of link:#bbv2.reference.class.property-set[property-set]) * As subsequent elements -- created concrete targets ( instances of the `virtual-target` class.) It's possible to lookup a metatarget by target-id using the `targets.resolve-reference` function, and the `targets.generate-from-reference` function can both lookup and generate a metatarget. The link:#bbv2.reference.class.abstract-target[abstract-target] class has three immediate derived classes: * link:#bbv2.reference.class.project-target[project-target] that corresponds to a project and is not intended for further subclassing. The link:#bbv2.reference.class.project-target.generate[generate] method of this class builds all targets in the project that are not marked as explicit. * link:#bbv2.reference.class.main-target[main-target] corresponds to a target in a project and contains one or more target alternatives. This class also should not be subclassed. The link:#bbv2.reference.class.main-target.generate[generate] method of this class selects an alternative to build, and calls the link:#bbv2.reference.class.basic-target.generate[generate] method of that alternative. * link:#bbv2.reference.class.basic-target[basic-target] corresponds to a specific target alternative. This is base class, with a number of derived classes. The link:#bbv2.reference.class.basic-target.generate[generate] method processes the target requirements and requested build properties to determine final properties for the target, builds all sources, and finally calls the abstract link:#bbv2.reference.class.basic-target.construct[construct] method with the list of source virtual targets, and the final properties. The instances of the link:#bbv2.reference.class.project-target[project-target] and link:#bbv2.reference.class.main-target[main-target] classes are created implicitly -- when loading a new Jamfiles, or when a new target alternative with as-yet unknown name is created. The instances of the classes derived from link:#bbv2.reference.class.basic-target[basic-target] are typically created when Jamfile calls a metatarget rule, such as such as `exe`. It it permissible to create a custom class derived from link:#bbv2.reference.class.basic-target[basic-target] and create new metatarget rule that creates instance of such target. However, in the majority of cases, a specific subclass of link:#bbv2.reference.class.basic-target[basic-target] -- link:#bbv2.reference.class.typed-target[typed-target] is used. That class is associated with a type and relays to generators to construct concrete targets of that type. This process will be explained below. When a new type is declared, a new metatarget rule is automatically defined. That rule creates new instance of type-target, associated with that type. [[bbv2.extender.overview.targets]] === Concrete targets Concrete targets are represented by instance of classes derived from `virtual-target`. The most commonly used subclass is `file-target`. A file target is associated with an action that creates it -- an instance of the `action` class. The action, in turn, hold a list of source targets. It also holds the link:#bbv2.reference.class.property-set[property-set] instance with the build properties that should be used for the action. Here's an example of creating a target from another target, `source` [source,jam] ---- local a = [ new action $(source) : common.copy : $(property-set) ] ; local t = [ new file-target $(name) : CPP : $(project) : $(a) ] ; ---- The first line creates an instance of the `action` class. The first parameter is the list of sources. The second parameter is the name a jam-level link:#bbv2.overview.jam_language.actions[action]. The third parameter is the property-set applying to this action. The second line creates a target. We specify a name, a type and a project. We also pass the action object created earlier. If the action creates several targets, we can repeat the second line several times. In some cases, code that creates concrete targets may be invoked more than once with the same properties. Returning two different instances of `file-target` that correspond to the same file clearly will result in problems. Therefore, whenever returning targets you should pass them via the `virtual-target.register` function, besides allowing B2 to track which virtual targets got created for each metatarget, this will also replace targets with previously created identical ones, as necessary.footnote:[This create-then-register pattern is caused by limitations of the Boost.Jam language. Python port is likely to never create duplicate targets.] Here are a couple of examples: [source,jam] ---- return [ virtual-target.register $(t) ] ; return [ sequence.transform virtual-target.register : $(targets) ] ; ---- [[bbv2.extender.overview.generators]] === Generators In theory, every kind of metatarget in B2 (like `exe`, `lib` or `obj`) could be implemented by writing a new metatarget class that, independently of the other code, figures what files to produce and what commands to use. However, that would be rather inflexible. For example, adding support for a new compiler would require editing several metatargets. In practice, most files have specific types, and most tools consume and produce files of specific type. To take advantage of this fact, B2 defines concept of target type and generators generators, and has special metatarget class link:#bbv2.reference.class.typed-target[typed-target]. Target type is merely an identifier. It is associated with a set of file extensions that correspond to that type. Generator is an abstraction of a tool. It advertises the types it produces and, if called with a set of input target, tries to construct output targets of the advertised types. Finally, link:#bbv2.reference.class.typed-target[typed-target] is associated with specific target type, and relays the generator (or generators) for that type. A generator is an instance of a class derived from `generator`. The `generator` class itself is suitable for common cases. You can define derived classes for custom scenarios. [[bbv2.extender.example]] == Example: 1-to-1 generator Say you're writing an application that generates {CPP} code. If you ever did this, you know that it's not nice. Embedding large portions of {CPP} code in string literals is very awkward. A much better solution is: 1. Write the template of the code to be generated, leaving placeholders at the points that will change 2. Access the template in your application and replace placeholders with appropriate text. 3. Write the result. It's quite easy to achieve. You write special verbatim files that are just {CPP}, except that the very first line of the file contains the name of a variable that should be generated. A simple tool is created that takes a verbatim file and creates a cpp file with a single `char*` variable whose name is taken from the first line of the verbatim file and whose value is the file's properly quoted content. Let's see what B2 can do. First off, B2 has no idea about "verbatim files". So, you must register a new target type. The following code does it: [source,jam] ---- import type ; type.register VERBATIM : verbatim ; ---- The first parameter to link:#bbv2.reference.modules.type.register[type.register] gives the name of the declared type. By convention, it's uppercase. The second parameter is the suffix for files of this type. So, if B2 sees `code.verbatim` in a list of sources, it knows that it's of type `VERBATIM`. Next, you tell B2 that the verbatim files can be transformed into {CPP} files in one build step. A generator is a template for a build step that transforms targets of one type (or set of types) into another. Our generator will be called `verbatim.inline-file`; it transforms `VERBATIM` files into `CPP` files: [source,jam] ---- import generators ; generators.register-standard verbatim.inline-file : VERBATIM : CPP ; ---- Lastly, you have to inform B2 about the shell commands used to make that transformation. That's done with an `actions` declaration. [source,jam] ---- actions inline-file { "./inline-file.py" $(<) $(>) } ---- Now, we're ready to tie it all together. Put all the code above in file `verbatim.jam`, add `import verbatim ;` to `Jamroot.jam`, and it's possible to write the following in your Jamfile: [source,jam] ---- exe codegen : codegen.cpp class_template.verbatim usage.verbatim ; ---- The listed verbatim files will be automatically converted into {CPP} source files, compiled and then linked to the `codegen` executable. In subsequent sections, we will extend this example, and review all the mechanisms in detail. The complete code is available in the `example/customization` directory. [[bbv2.extending.targets]] == Target types The first thing we did in the link:#bbv2.extender.intro[introduction] was declaring a new target type: [source,jam] ---- import type ; type.register VERBATIM : verbatim ; ---- The type is the most important property of a target. B2 can automatically generate necessary build actions only because you specify the desired type (using the different main target rules), and because B2 can guess the type of sources from their extensions. The first two parameters for the `type.register` rule are the name of new type and the list of extensions associated with it. A file with an extension from the list will have the given target type. In the case where a target of the declared type is generated from other sources, the first specified extension will be used. Sometimes you want to change the suffix used for generated targets depending on build properties, such as toolset. For example, some compiler uses extension `elf` for executable files. You can use the `type.set-generated-target-suffix` rule: [source,jam] ---- type.set-generated-target-suffix EXE : elf : elf ; ---- A new target type can be inherited from an existing one. [source,jam] ---- type.register PLUGIN : : SHARED_LIB ; ---- The above code defines a new type derived from `SHARED_LIB`. Initially, the new type inherits all the properties of the base type - in particular generators and suffix. Typically, you'll change the new type in some way. For example, using `type.set-generated-target-suffix` you can set the suffix for the new type. Or you can write a special generator for the new type. For example, it can generate additional meta-information for the plugin. In either way, the `PLUGIN` type can be used whenever `SHARED_LIB` can. For example, you can directly link plugins to an application. A type can be defined as "main", in which case B2 will automatically declare a main target rule for building targets of that type. More details can be found link:#bbv2.extending.rules.main-type[later]. [[bbv2.extending.scanners]] == Scanners Sometimes, a file can refer to other files via some include system. To make B2 track dependencies between included files, you need to provide a scanner. The primary limitation is that only one scanner can be assigned to a target type. First, we need to declare a new class for the scanner: [source,jam] ---- class verbatim-scanner : common-scanner { rule pattern ( ) { return "//###include[ ]*\"([^\"]*)\"" ; } } ---- All the complex logic is in the `common-scanner` class, and you only need to override the method that returns the regular expression to be used for scanning. The parentheses in the regular expression indicate which part of the string is the name of the included file. Only the first parenthesized group in the regular expression will be recognized; if you can't express everything you want that way, you can return multiple regular expressions, each of which contains a parenthesized group to be matched. After that, we need to register our scanner class: [source,jam] ---- scanner.register verbatim-scanner : include ; ---- The value of the second parameter, in this case `include`, specifies the properties that contain the list of paths that should be searched for the included files. Finally, we assign the new scanner to the `VERBATIM` target type: [source,jam] ---- type.set-scanner VERBATIM : verbatim-scanner ; ---- That's enough for scanning include dependencies. [[bbv2.extending.tools]] == Tools and generators This section will describe how B2 can be extended to support new tools. For each additional tool, a B2 object called generator must be created. That object has specific types of targets that it accepts and produces. Using that information, B2 is able to automatically invoke the generator. For example, if you declare a generator that takes a target of the type `D` and produces a target of the type `OBJ`, when placing a file with extension `.d` in a list of sources will cause B2 to invoke your generator, and then to link the resulting object file into an application. (Of course, this requires that you specify that the `.d` extension corresponds to the `D` type.) Each generator should be an instance of a class derived from the `generator` class. In the simplest case, you don't need to create a derived class, but simply create an instance of the `generator` class. Let's review the example we've seen in the link:#bbv2.extender.intro[introduction]. [source,jam] ---- import generators ; generators.register-standard verbatim.inline-file : VERBATIM : CPP ; actions inline-file { "./inline-file.py" $(<) $(>) } ---- We declare a standard generator, specifying its id, the source type and the target type. When invoked, the generator will create a target of type `CPP` with a source target of type `VERBATIM` as the only source. But what command will be used to actually generate the file? In B2, actions are specified using named "actions" blocks and the name of the action block should be specified when creating targets. By convention, generators use the same name of the action block as their own id. So, in above example, the "inline-file" actions block will be used to convert the source into the target. There are two primary kinds of generators: standard and composing, which are registered with the `generators.register-standard` and the `generators.register-composing` rules, respectively. For example: [source,jam] ---- generators.register-standard verbatim.inline-file : VERBATIM : CPP ; generators.register-composing mex.mex : CPP LIB : MEX ; ---- The first (standard) generator takes a _single_ source of type `VERBATIM` and produces a result. The second (composing) generator takes any number of sources, which can have either the `CPP` or the `LIB` type. Composing generators are typically used for generating top-level target type. For example, the first generator invoked when building an `exe` target is a composing generator corresponding to the proper linker. You should also know about two specific functions for registering generators: `generators.register-c-compiler` and `generators.register-linker`. The first sets up header dependency scanning for C files, and the seconds handles various complexities like searched libraries. For that reason, you should always use those functions when adding support for compilers and linkers. (Need a note about UNIX) *Custom generator classes* The standard generators allows you to specify source and target types, an action, and a set of flags. If you need anything more complex, you need to create a new generator class with your own logic. Then, you have to create an instance of that class and register it. Here's an example how you can create your own generator class: [source,jam] ---- class custom-generator : generator { rule __init__ ( * : * ) { generator.__init__ $(1) : $(2) : $(3) : $(4) : $(5) : $(6) : $(7) : $(8) : $(9) ; } } generators.register [ new custom-generator verbatim.inline-file : VERBATIM : CPP ] ; ---- This generator will work exactly like the `verbatim.inline-file` generator we've defined above, but it's possible to customize the behavior by overriding methods of the `generator` class. There are two methods of interest. The `run` method is responsible for the overall process - it takes a number of source targets, converts them to the right types, and creates the result. The `generated-targets` method is called when all sources are converted to the right types to actually create the result. The `generated-targets` method can be overridden when you want to add additional properties to the generated targets or use additional sources. For a real-life example, suppose you have a program analysis tool that should be given a name of executable and the list of all sources. Naturally, you don't want to list all source files manually. Here's how the `generated-targets` method can find the list of sources automatically: [source,jam] ---- class itrace-generator : generator { ... rule generated-targets ( sources + : property-set : project name ? ) { local leaves ; local temp = [ virtual-target.traverse $(sources[1]) : : include-sources ] ; for local t in $(temp) { if ! [ $(t).action ] { leaves += $(t) ; } } return [ generator.generated-targets $(sources) $(leafs) : $(property-set) : $(project) $(name) ] ; } } generators.register [ new itrace-generator nm.itrace : EXE : ITRACE ] ; ---- The `generated-targets` method will be called with a single source target of type `EXE`. The call to `virtual-target.traverse` will return all targets the executable depends on, and we further find files that are not produced from anything. The found targets are added to the sources. The `run` method can be overridden to completely customize the way the generator works. In particular, the conversion of sources to the desired types can be completely customized. Here's another real example. Tests for the Boost Python library usually consist of two parts: a Python program and a {CPP} file. The {CPP} file is compiled to Python extension that is loaded by the Python program. But in the likely case that both files have the same name, the created Python extension must be renamed. Otherwise, the Python program will import itself, not the extension. Here's how it can be done: [source,jam] ---- rule run ( project name ? : property-set : sources * ) { local python ; for local s in $(sources) { if [ $(s).type ] = PY { python = $(s) ; } } local libs ; for local s in $(sources) { if [ type.is-derived [ $(s).type ] LIB ] { libs += $(s) ; } } local new-sources ; for local s in $(sources) { if [ type.is-derived [ $(s).type ] CPP ] { local name = [ $(s).name ] ; # get the target's basename if $(name) = [ $(python).name ] { name = $(name)_ext ; # rename the target } new-sources += [ generators.construct $(project) $(name) : PYTHON_EXTENSION : $(property-set) : $(s) $(libs) ] ; } } result = [ construct-result $(python) $(new-sources) : $(project) $(name) : $(property-set) ] ; } ---- First, we separate all source into python files, libraries and {CPP} sources. For each {CPP} source we create a separate Python extension by calling `generators.construct` and passing the {CPP} source and the libraries. At this point, we also change the extension's name, if necessary. [[bbv2.extending.features]] == Features Often, we need to control the options passed the invoked tools. This is done with features. Consider an example: [source,jam] ---- # Declare a new free feature import feature : feature ; feature verbatim-options : : free ; # Cause the value of the 'verbatim-options' feature to be # available as 'OPTIONS' variable inside verbatim.inline-file import toolset : flags ; flags verbatim.inline-file OPTIONS ; # Use the "OPTIONS" variable actions inline-file { "./inline-file.py" $(OPTIONS) $(<) $(>) } ---- We first define a new feature. Then, the `flags` invocation says that whenever verbatim.inline-file action is run, the value of the `verbatim-options` feature will be added to the `OPTIONS` variable, and can be used inside the action body. You'd need to consult online help (--help) to find all the features of the `toolset.flags` rule. Although you can define any set of features and interpret their values in any way, B2 suggests the following coding standard for designing features. Most features should have a fixed set of values that is portable (tool neutral) across the class of tools they are designed to work with. The user does not have to adjust the values for a exact tool. For example, `speed` has the same meaning for all {CPP} compilers and the user does not have to worry about the exact options passed to the compiler's command line. Besides such portable features there are special 'raw' features that allow the user to pass any value to the command line parameters for a particular tool, if so desired. For example, the `` feature allows you to pass any command line options to a {CPP} compiler. The `` feature allows you to pass any string preceded by `-I` and the interpretation is tool-specific. (See <> for an example of very smart usage of that feature). Of course one should always strive to use portable features, but these are still be provided as a backdoor just to make sure B2 does not take away any control from the user. Using portable features is a good idea because: * When a portable feature is given a fixed set of values, you can build your project with two different settings of the feature and B2 will automatically use two different directories for generated files. B2 does not try to separate targets built with different raw options. * Unlike with “raw” features, you don't need to use specific command-line flags in your Jamfile, and it will be more likely to work with other tools. *Steps for adding a feature* Adding a feature requires three steps: 1. Declaring a feature. For that, the "feature.feature" rule is used. You have to decide on the set of link:#bbv2.reference.features.attributes[feature attributes]: * if you want a feature value set for one target to automatically propagate to its dependent targets then make it “propagated”. * if a feature does not have a fixed list of values, it must be “free.” For example, the `include` feature is a free feature. * if a feature is used to refer to a path relative to the Jamfile, it must be a “path” feature. Such features will also get their values automatically converted to B2's internal path representation. For example, `include` is a path feature. * if feature is used to refer to some target, it must be a “dependency” feature. 2. Representing the feature value in a target-specific variable. Build actions are command templates modified by Boost.Jam variable expansions. The `toolset.flags` rule sets a target-specific variable to the value of a feature. 3. Using the variable. The variable set in step 2 can be used in a build action to form command parameters or files. *Another example* Here's another example. Let's see how we can make a feature that refers to a target. For example, when linking dynamic libraries on Windows, one sometimes needs to specify a "DEF file", telling what functions should be exported. It would be nice to use this file like this: [source,jam] ---- lib a : a.cpp : a.def ; ---- Actually, this feature is already supported, but anyway... 1. Since the feature refers to a target, it must be "dependency". + ---- feature def-file : : free dependency ; ---- 2. One of the toolsets that cares about DEF files is msvc. The following line should be added to it. + ---- flags msvc.link DEF_FILE ; ---- 3. Since the DEF_FILE variable is not used by the msvc.link action, we need to modify it to be: + ---- actions link bind DEF_FILE { $(.LD) .... /DEF:$(DEF_FILE) .... } ---- + Note the `bind DEF_FILE` part. It tells B2 to translate the internal target name in `DEF_FILE` to a corresponding filename in the `link` action. Without it the expansion of `$(DEF_FILE)` would be a strange symbol that is not likely to make sense for the linker. + We are almost done, except for adding the following code to `msvc.jam`: + ---- rule link { DEPENDS $(<) : [ on $(<) return $(DEF_FILE) ] ; } ---- + This is a workaround for a bug in B2 engine, which will hopefully be fixed one day. *Variants and composite features.* Sometimes you want to create a shortcut for some set of features. For example, `release` is a value of `` and is a shortcut for a set of features. It is possible to define your own build variants. For example: [source,jam] ---- variant crazy : speed off on on ; ---- will define a new variant with the specified set of properties. You can also extend an existing variant: [source,jam] ---- variant super_release : release : USE_ASM ; ---- In this case, `super_release` will expand to all properties specified by `release`, and the additional one you've specified. You are not restricted to using the `variant` feature only. Here's example that defines a brand new feature: [source,jam] ---- feature parallelism : mpi fake none : composite link-incompatible ; feature.compose mpi : /mpi//mpi/none ; feature.compose fake : /mpi//fake/none ; ---- This will allow you to specify the value of feature `parallelism`, which will expand to link to the necessary library. [[bbv2.extending.rules]] == Main target rules A main target rule (e.g “link:#bbv2.tasks.programs[exe]” Or “link:#bbv2.tasks.libraries[lib]”) creates a top-level target. It's quite likely that you'll want to declare your own and there are two ways to do that. [[bbv2.extending.rules.main-type]]The first way applies when your target rule should just produce a target of specific type. In that case, a rule is already defined for you! When you define a new type, B2 automatically defines a corresponding rule. The name of the rule is obtained from the name of the type, by down-casing all letters and replacing underscores with dashes. For example, if you create a module `obfuscate.jam` containing: [source,jam] ---- import type ; type.register OBFUSCATED_CPP : ocpp ; import generators ; generators.register-standard obfuscate.file : CPP : OBFUSCATED_CPP ; ---- and import that module, you'll be able to use the rule "obfuscated-cpp" in Jamfiles, which will convert source to the OBFUSCATED_CPP type. The second way is to write a wrapper rule that calls any of the existing rules. For example, suppose you have only one library per directory and want all cpp files in the directory to be compiled into that library. You can achieve this effect using: [source,jam] ---- lib codegen : [ glob *.cpp ] ; ---- If you want to make it even simpler, you could add the following definition to the `Jamroot.jam` file: [source,jam] ---- rule glib ( name : extra-sources * : requirements * ) { lib $(name) : [ glob *.cpp ] $(extra-sources) : $(requirements) ; } ---- allowing you to reduce the Jamfile to just [source,jam] ---- glib codegen ; ---- Note that because you can associate a custom generator with a target type, the logic of building can be rather complicated. For example, the `boostbook` module declares a target type `BOOSTBOOK_MAIN` and a custom generator for that type. You can use that as example if your main target rule is non-trivial. [[bbv2.extending.toolset_modules]] == Toolset modules If your extensions will be used only on one project, they can be placed in a separate `.jam` file and imported by your `Jamroot.jam`. If the extensions will be used on many projects, users will thank you for a finishing touch. The `using` rule provides a standard mechanism for loading and configuring extensions. To make it work, your module should provide an `init` rule. The rule will be called with the same parameters that were passed to the `using` rule. The set of allowed parameters is determined by you. For example, you can allow the user to specify paths, tool versions, and other options. Here are some guidelines that help to make B2 more consistent: * The `init` rule should never fail. Even if the user provided an incorrect path, you should emit a warning and go on. Configuration may be shared between different machines, and wrong values on one machine can be OK on another. * Prefer specifying the command to be executed to specifying the tool's installation path. First of all, this gives more control: it's possible to specify + ---- /usr/bin/g++-snapshot time g++ ---- + as the command. Second, while some tools have a logical "installation root", it's better if the user doesn't have to remember whether a specific tool requires a full command or a path. * Check for multiple initialization. A user can try to initialize the module several times. You need to check for this and decide what to do. Typically, unless you support several versions of a tool, duplicate initialization is a user error. If the tool's version can be specified during initialization, make sure the version is either always specified, or never specified (in which case the tool is initialized only once). For example, if you allow: + ---- using yfc ; using yfc : 3.3 ; using yfc : 3.4 ; ---- + Then it's not clear if the first initialization corresponds to version 3.3 of the tool, version 3.4 of the tool, or some other version. This can lead to building twice with the same version. * If possible, `init` must be callable with no parameters. In which case, it should try to autodetect all the necessary information, for example, by looking for a tool in PATH or in common installation locations. Often this is possible and allows the user to simply write: + ---- using yfc ; ---- * Consider using facilities in the `tools/common` module. You can take a look at how `tools/gcc.jam` uses that module in the `init` rule.