Please enable JavaScript.
Coggle requires JavaScript to display documents.
Bazel (Bazel Commands (bazel build
bazel build //foo
Bazel prints the…
Bazel
Bazel Commands
bazel build
- bazel build //foo
- Bazel prints the progress messages as it loads all the packages in the transitive dependencies of the requested target, then analyzes them for correctness and to create the build actions, finally executing the compilers and other tools of the build.
- Bazel prints progress messages during the execution phase of the build, showing the current build step (compiler, linker, etc.) i.e. being started, and the number completed over the total number of build actions.
- As the build starts the number of total actions will often increase as Bazel discovers the entire action graph, but the number will usually stabilize within a few seconds.
- At the end of the build Bazel prints which targets were requested, whether or not they were successfully built, and if so, where the output files can be found.
- Bazel allows you to perform a build from a completely read-only volume
Specifying targets to build
- All target patterns starting with // are resolved relative to the current workspace
- //foo/bar:all -> All rules in the package foo/bar
- //foo/... or //foo/...:all -> All rules in all packages beneath the directory foo
- //foo/...:* or //foo/...:all-targets -> All targets (rules and files) in all packages beneath the directory foo
- Target patterns which do not begin with // are resolved relative to the current working directory
- These examples assume a working directory of foo
- :foo -> Equivalent to //foo:foo
- bar:wiz -> Equivalent to //foo/bar:wiz
- bar/wiz -> Equivalent to: //foo/bar/wiz:wiz if foo/bar/wiz is a package, //foo/bar:wiz if foo/bar is a package, //foo:bar/wiz otherwise
- :all -> //foo:all
- ...:all -> //foo/...:all
- ... -> //foo/...:all
- bar/...:all -> //foo/bar/...:all
Fetching external dependencies
- By default, Bazel will download and symlink external dependencies during the build
- If you would like to prevent new dependencies from being added during builds, you can specify the --fetch=false flag
- Note that this flag only applies to repository rules that do not point to a directory in the local file system
- If you disallow fetching during builds and Bazel finds new external dependencies, your build will fail.
- You can manually fetch dependencies by running bazel fetch
- fetch takes a list of targets to fetch dependencies for.
- bazel fetch //foo:bar //bar:baz
- To fetch all external dependencies for a workspace, run
Build configurations and cross-compilation
- All the inputs a build can be divided into two distinct categories.
- The first is the build rule, the values of its attributes, and the complete set of its transitive dependencies.
- The second is the external or environmental data, supplied by the user or by the build tool: the choice of target architecture, compilation and linking options, and other toolchain configuration options.
- We refer to a complete set of environmental data as a configuration
- In any given build, there may be more than one configuration
- the host configuration, which is used for building tools that run during the build
- the target configuration, which is used for building the binary you ultimately requested.
- Typically, there are many libraries that are prerequisites of both the requested build target and the host tools, for example some base libraries.
- Such libraries must be built twice, once for the host configuration, and once for the target configuration.
- Bazel takes care of ensuring that both variants are built, and that the derived files are kept separate to avoid interference; usually such targets can be built concurrently
- Bazel uses one of two ways to select the host configuration, based on the --distinct_host_configuration option.
- --distinct_host_configuration=false
- When this option is false, the host and target configurations are identical: all tools required during the build will be built in exactly the same way as target programs.
- This setting means that no libraries need to be built twice during a single build, so it keeps builds short
- --distinct_host_configuration=true (default)
- If this option is true, then instead of using the same configuration for the host and target, a completely distinct host configuration is used
Correct incremental rebuilds
- Bazel maintains a database of all work previously done, and will only omit a build step if it finds that the set of input files (and their timestamps) to that build step, and the compilation command for that build step, exactly match one in the database, and, that the set of output files (and their timestamps) for the database entry exactly match the timestamps of the files on disk.
- Any change to the input files or output files, or to the command itself, will cause re-execution of the build step.
Build consistency and incremental builds
- we define the state of a build as consistent when all the expected output files exist, and their contents are correct, as specified by the steps or rules required to create them
- When you edit a source file, the state of the build is said to be inconsistent, and remains inconsistent until you next run the build tool to successful completion. We describe this situation as unstable inconsistency
- There is another kind of inconsistency that is harmful: stable inconsistency. If the build reaches a stable inconsistent state, then repeated successful invocation of the build tool does not restore consistency: the build has gotten "stuck", and the outputs remain incorrect
- The build tool must be able to perform incremental builds without compromising consistency.
- Bazel offers the following guarantee: after a successful invocation of the build tool during which you made no edits, the build will be in a consistent state.
Sandboxed execution
- Hermeticity means that the action only uses its declared input files and no other files in the filesystem, and it only produces its declared output files
- Bazel uses sandboxes to guarantee that actions run hermetically and correctly.
- Bazel runs Spawns (actions) in sandboxes that only contain the minimal set of files the tool requires to do its job.
- Currently sandboxing works on Linux 3.12 or newer with the CONFIG_USER_NS option enabled, and also on macOS 10.11 or newer
Phases of a build
- In Bazel, a build occurs in three distinct phases
Loading phase
- The first is loading during which all the necessary BUILD files for the initial targets, and their transitive dependencies, are loaded, parsed, evaluated and cached.
- For the first build after a Bazel server is started, the loading phase typically takes many seconds as many BUILD files are loaded from the file system.
- In subsequent builds, especially if no BUILD files have changed, loading occurs very quickly
- Errors reported during this phase include: package not found, target not found, lexical and grammatical errors in a BUILD file, and evaluation errors
Analysis phase
- The second phase, analysis, involves the semantic analysis and validation of each build rule, the construction of a build dependency graph, and the determination of exactly what work is to be done in each step of the build
- analysis also takes several seconds when computed in its entirety.
- However, Bazel caches the dependency graph from one build to the next and only reanalyzes what it has to, which can make incremental builds extremely fast in the case where the packages haven't changed since the previous build.
- Errors reported at this stage include: inappropriate dependencies, invalid inputs to a rule, and all rule-specific error messages
Execution phase
- The third and final phase of the build is execution.
- This phase ensures that the outputs of each step in the build are consistent with its inputs, re-running compilation/linking/etc. tools as necessary.
- This step is where the build spends the majority of its time, ranging from a few seconds to over an hour for a large build.
- Errors reported during this phase include: missing source files, errors in a tool executed by some build action, or failure of a tool to produce the expected set of outputs.
Client/server implementation
- The Bazel system is implemented as a long-lived server process.
- This allows it to perform many optimizations, such as caching of BUILD files, dependency graphs, and other metadata from one build to the next.
- This improves the speed of incremental builds, and allows different commands, such as build and query to share the same cache of loaded packages, making queries very fast
- When you run bazel, you're running the client. The client finds the server based on the output base, which by default is determined by the path of the base workspace directory and your userid, so if you build in multiple workspaces, you'll have multiple output bases and thus multiple Bazel server processes.
- Multiple users on the same workstation can build concurrently in the same workspace because their output bases will differ (different userids).
- If the client cannot find a running server instance, it starts a new one.
- The server process will stop after a period of inactivity (3 hours, by default, which can be modified).
- Bazel servers can be stopped using the shutdown command.
- When running bazel, the client first checks that the server is the appropriate version; if not, the server is stopped and a new one started.
- This ensures that the use of a long-running server process doesn't interfere with proper versioning.
.bazelrc, the Bazel configuration file
- Bazel accepts many options. Typically, some of these are varied frequently while others stay the same across several builds. To avoid having to specify these unchanged options for every commands Bazel allows you to specify options in a configuration file.
- Where are the .bazelrc files?
- Bazel looks for optional configuration files in the following locations, in the order shown below.
- The system RC file:
- On Linux/macOS/Unixes: /etc/bazel.bazelrc
- On Windows: %ProgramData%\bazel.bazelrc
- The workspace RC file
- .bazelrc in your workspace directory
- The home RC file
- On Linux/macOS/Unixes: $HOME/.bazelrc
- On Windows: %USERPROFILE%.bazelrc if exists, or %HOME%/.bazelrc
- The user-specified RC file, if specified with -bazelrc=file
.bazelrc syntax and semantics
- Imports
- Lines that start with import or try-import: use these to load other "rc" files
- The difference between import and try-import is that Bazel fails if the import'ed file is missing, but not so for a try-import'ed file
- Option defaults
- Most lines of a bazelrc define default option values. The first word on each line specifies when these defaults are applied
- startup: startup options, which go before the command
- common: options that apply to all Bazel commands
- command: Bazel command, such as build or query to which the options apply. These options also apply to all commands that inherit from the specified command. (For example, test inherits from build.)
- Options on the command line always take precedence over those in rc files
--config
- the rc file can be used to group options and provide a shorthand for common groupings. This is done by adding a :name suffix to the command.
- These options are ignored by default, but will be included when the option --config=name is present, either on the command line or in a .bazelrc file
.bazelignore
- You can specify directories within the workspace that you want Bazel to ignore, such as related projects that use other build systems.
- Place a file called .bazelignore at the root of the workspace and add the directories you want Bazel to ignore, one per line. Entries are relative to the workspace root
Command log
- The Bazel output is also available in a command log file which you can find with the following command: bazel info command_log
- The command log file contains the interleaved stdout and stderr streams of the most recent Bazel command
Parsing output
- Two options that may be helpful for your script are --noshow_progress which suppresses progress messages, and --show_result n, which controls whether or not "build up-to-date" messages are printed; these messages may be parsed to discover which targets were successfully built, and the location of the output files they created.
- Be sure to specify a very large value of n if you rely on these messages
Troubleshooting performance by profiling
- The first step in analyzing the performance of your build is to profile your build with the --profile option.
- The file generated by the --profile command is a binary file.
- Once you have generated this binary profile, you can analyze it using Bazel's analyze-profile command.
- By default, it will print out summary analysis information for each of the specified profile datafiles. This includes cumulative statistics for different task types for each build phase and an analysis of the critical execution path.
- If Bazel appears to be hung, you can hit ctrl + \ or send Bazel a SIGQUIT signal (kill -3 $(bazel info server_pid)) to get a thread dump in the file $(bazel info output_base)/server/jvm.out.
- Ex: bazel build --profile=/tmp/blah //main:hello-world
- bazel analyze-profile /tmp/blah
Calling Bazel from scripts
- Bazel can be called from scripts in order to perform a build, run tests or query the dependency graph.
- Choosing the output base
- The --output_base option controls where the Bazel process should write the outputs of a build to, as well as various working files used internally by Bazel, one of which is a lock that guards against concurrent mutation of the output base by multiple Bazel processes
- if you need to put the build outputs in a specific location, this will dictate the output base you need to use.
- if you need to run multiple instances of your script concurrently, you will need to give each one a different (or random) output base
- What exit code will I get?
- Exit Codes common to all commands
- 0 -> Success
- 2 -> Command line Problem
- Refer doc for details
- Return codes for commands bazel build, bazel test
- 1 -> Build failed
3 -> Build OK, but some tests failed or timed out
- 4 -> Build successful but no tests were found even though testing was requested
Bazel Concepts
workspace
- A workspace is a directory on your filesystem that contains the source files & symbolic links to directories that contain the build outputs.
- Each workspace directory has a text file named WORKSPACE which may be empty, or may contain references to external dependencies required to build the outputs
- Directories containing a file WORKSPACE are considered the root of a workspace
Repositories
- Code is organized in repositories.
- The directory containing the WORKSPACE file is the root of the main repository, also called .
- Other, (external) repositories are defined in the WORKSPACE file using workspace rules.
- As external repositories are repositories themselves, they often contain a WORKSPACE file as well.
- However, these additional WORKSPACE files are ignored by Bazel.
- In particular, repositories depended upon transitively are not added automatically
packages
- A package is a collection of related files and a specification of the dependencies among them.
- A package is defined as a directory containing a file named BUILD or BUILD.bazel, residing beneath the top-level directory in the workspace.
- A package includes all files in its directory, plus all subdirectories beneath it, except those which themselves contain a BUILD file.
Targets
- The elements of a package are called targets
- The different kinds of targets are files, rules, package groups
Files
- Files are further divided into two kinds.
- Source files are usually written by people, and checked in to the repository.
- Generated files (derived files), are not checked in, but are generated by the build tool from source files according to specific rules.
rule
- A rule specifies the relationship between a set of input and a set of output files, including the necessary steps to derive the outputs from the inputs.
- The outputs of a rule are always generated files.
- The inputs to a rule may be source files, may be generated files, outputs of one rule may be the inputs to another
- The inputs to a rule may also include other rules.
- the files generated by a rule always belong to the same package as the rule itself; it is not possible to generate files into another package.
- It is not uncommon for a rule's inputs to come from another package, though
- Package groups are sets of packages whose purpose is to limit accessibility of certain rules.
- Package groups are defined by the package_group function.
- They have two properties: the list of packages they contain and their name.
- The only allowed ways to refer to them are from the visibility attribute of rules or from the default_visibility attribute of the package function; they do not generate or consume files
Labels
- All targets belong to exactly one package. The name of a target is called its label, and a typical label in canonical form looks like this: myrepo//my/app/main:app_binary
- Usually, a label refers to the same repository it occurs in, the repository name may be left out. So, inside myrepo this label is usually written as: //my/app/main:app_binary
- Each label has two parts, a package name (my/app/main) and a target name (app_binary)
- when the colon is omitted, the target name is assumed to be the same as the last component of the package name, so these two labels are equivalent: //my/app or //my/app:app
- Labels start with //, but package names never do, thus my/app is the package containing //my/app
- Within a BUILD file, the package-name part of label may be omitted, and optionally the colon too
- So within the BUILD file for package my/app (i.e. //my/app:BUILD), the following "relative" labels are all equivalent: //my/app:app or //my/app or :app or app
Rules
- A rule specifies the relationship between inputs and outputs, and the steps to build the outputs
- Every rule has a name, specified by the name attribute, of type string.
- for *_binary and *_test rules, the rule name determines the name of the executable produced by the build
- Every rule has a set of attributes
- Each attribute has a name and a type.
- The srcs attribute present in many rules has type "list of labels"; its value, if present, is a list of labels, each being the name of a target i.e. an input to this rule. Ex. srcs = ["my_app.cc"]
- The outs attribute present in many rules has type "list of output labels"
- The two types of label attributes thus assign direction to the edges between targets, giving rise to a dependency graph
- This directed acyclic graph over targets is called the "target graph" or "build dependency graph"
BUILD Files
- BUILD files are evaluated using Starlark.
- They are interpreted as a sequential list of statements
- In general, order does matter: variables must be defined before they are used
- However, most BUILD files consist only of declarations of build rules, and the relative order of these statements is immaterial
- To encourage a clean separation between code and data, BUILD files cannot contain function definitions, for statements or if statements (but list comprehensions and if expressions are allowed).
- Functions should be declared in .bzl files instead.
- Additionally, args and *kwargs arguments are not allowed in BUILD files; instead list all the arguments explicitly.
- dependent only on a known set of inputs, which is essential for ensuring that builds are reproducible
Loading an extension
- Bazel extensions are files ending in .bzl
- Use the load statement to import a symbol from an extension: load("//foo/bar:file.bzl", "some_library")
- This code will load the file foo/bar/file.bzl and add the some_library symbol to the environment.
- This can be used to load new rules, functions or constants (e.g. a string, a list, etc.)
- load statements must appear at top-level, i.e. they cannot be in a function body.
- The first argument of load is a label identifying a .bzl file.
- If it is a relative label, it is resolved with respect to the package (not directory) containing the current bzl file.
- Relative labels in load statements should use a leading :
- In a .bzl file, symbols starting with _ are not exported and cannot be loaded from another file.
Types of build rule
- The majority of build rules come in families, grouped together by language. For example, cc_binary, cc_library and cc_test are the build rules for C++ binaries, libraries, and tests, respectively.
- Other languages use the same naming scheme, with a different prefix, e.g. java_* for Java.
- *_binary rules build executable programs in a given language. After a build, the executable will reside in the build tool's binary output tree at the corresponding name for the rule's label, so //my:program would appear at (e.g.) $(BINDIR)/my/program.
- *_test rules are a specialization of a *_binary rule, used for automated testing. Tests are simply programs that return zero on success
- a program cc_test(name='x', data=['//foo:bar']) may open and read $TEST_SRCDIR/workspace/foo/bar during execution
- *_library rules specify separately-compiled modules in the given programming language. Libraries can depend on other libraries, and binaries and tests can depend on libraries, with the expected separate-compilation behavior
Dependencies
- A target A depends upon a target B if B is needed by A at build or execution time.
- The depends upon relation induces a Directed Acyclic Graph (DAG) over targets, and we call this a dependency graph.
- A target's direct dependencies are those other targets reachable by a path of length 1 in the dependency graph.
- A target's transitive dependencies are those targets upon which it depends via a path of any length through the graph.
Actual and declared dependencies
- A target X is actually dependent on target Y iff Y must be present, built and up-to-date in order for X to be built correctly.
- A target X has a declared dependency on target Y iff there is a dependency edge from X to Y in the package of X.
- every rule must explicitly declare all of its actual direct dependencies to the build system
- During a build of target X, the build tool inspects the entire transitive closure of dependencies of X to ensure that any changes in those targets are reflected in the final result, rebuilding intermediates as needed.
Types of dependencies
- Most build rules have three attributes for specifying different kinds of generic dependencies: srcs, deps and data
- Many rules also have additional attributes for rule-specific kinds of dependency, e.g. compiler, resources, etc.
srcs dependencies
- Files consumed directly by the rule or rules that output source files
deps dependencies
- Rule pointing to separately-compiled modules providing header files, symbols, libraries, data, etc.
data dependencies
- A build target might need some data files to run correctly. These data files aren't source code: they don't affect how the target is built.
- For example, a unit test might compare a function's output to the contents of a file. When we build the unit test, we don't need the file; but we do need it when we run the test.
- The build system runs tests in an isolated directory where only files listed as "data" are available.
- Thus, if a binary/library/test needs some files to run, specify them in data.