Please enable JavaScript.

Coggle requires JavaScript to display documents.

Bazel Remote Execution (Troubleshooting Remote Execution (Troubleshooting…

- - - - Managing implicit dependencies
        
        If a build tool can access dependencies across build actions, those actions will fail when remotely executed because each remote build action is executed separately from others.
        
        Some build tools retain state across build actions and access dependencies that have not been explicitly included in the tool invocation, which will cause remotely executed build actions to fail
        
        To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the local Docker sandbox
        
        Use the sandbox to prepare your build for remote execution by identifying and resolving dependency-related build errors
        
        Managing platform-dependent binaries
        
        Typically, a binary built on the host platform cannot safely execute on an remote execution platform due to potentially mismatched dependencies.
        
        For example, the SingleJar binary supplied with Bazel, targets the host platform. However, for remote execution, SingleJar must be compiled as part of the process of building your code so that it targets the remote execution platform
        
        Do not ship binaries of build tools required by your build with your source code unless you are sure they will safely run in your execution platform. Instead
        
        Ship or externally reference the source code for the tool so that it can be built for the remote execution platform
        
        Pre-install the tool into the remote execution environment (for example, a toolchain container) if it’s stable enough and use toolchain rules to run it in your build
        
        Managing configure-style WORKSPACE rules
        
        Bazel’s WORKSPACE rules can be used for probing the host platform for tools and libraries required by the build, which, for local builds, is also Bazel’s execution platform.
        
        If the build explicitly depends on local build tools and artifacts, it will fail during remote execution if the remote execution platform is not identical to the host platform.
        
        The following actions performed by WORKSPACE rules are not compatible with remote execution
        
        Building binaries. Executing compilation actions in WORKSPACE rules results in binaries that are incompatible with the remote execution platform if different from the host platform
        
        Installing pip packages. pip packages installed via WORKSPACE rules require that their dependencies be pre-installed on the host platform.
        
        Such packages, built specifically for the host platform, will be incompatible with the remote execution platform if different from the host platform
        
        Symlinking to local tools or artifacts. Symlinks to tools or libraries installed on the host platform created via WORKSPACE rules will cause the build to fail on the remote execution platform as Bazel will not be able to locate them.
        
        Instead, create symlinks using standard build actions so that the symlinked tools and libraries are accessible from Bazel’s runfiles tree.
        
        Mutating the host platform. Avoid creating files outside of the Bazel runfiles tree, creating environment variables, as they may behave unexpectedly on the remote execution platform
        
        If an external dependency executes specific operations dependent on the host platform, we recommend splitting those operations between WORKSPACE and build rules as follows
        
        Platform inspection and dependency enumeration. These operations are safe to execute locally via WORKSPACE rules, which can check which libraries are installed, download packages that must be built, and prepare required artifacts for compilation.
        
        For remote execution, these rules must also support using pre-checked artifacts to provide the information that would normally be obtained during host platform inspection.
        
        Pre-checked artifacts allow Bazel to describe dependencies as if they were local.
        
        Generating or compiling target-specific artifacts and platform mutation.
        
        These operations must be executed via regular build rules.
        
        Actions that produce target-specific artifacts for external dependencies must execute during the build
- - - - Prerequisites
        
        Install Docker and configure the permissions required to run it
        
        Install Bazel 0.14.1 or later
        
        Add the bazel-toolchains repo to your build’s WORKSPACE file as described in https://releases.bazel.build/bazel-toolchains.html
        
        Add flags to your .bazelrc file to enable the feature
        
        Troubleshooting natively
        
        This method executes Bazel and all of its build actions directly on the local machine and is a reliable way to confirm whether your build will succeed when executed remotely
        
        However, with this method, locally installed tools, binaries, and data may leak into into your build, especially if it uses configure-style WORKSPACE rules. Such leaks will cause problems with remote execution
        
        Step 1: Run the build
        
        Add the --config=docker-sandbox flag to the Bazel command that executes your build
        
        bazel --bazelrc=.bazelrc build --config=docker-sandbox <target>
        
        Run the build and wait for it to complete. The build will run up to four times slower than normal due to the Docker sandbox feature
        
        You may encounter the following error: ERROR: 'docker' is an invalid value for docker spawn strategy.
        
        If you do, run the build again with the --experimental_docker_verbose flag.
        
        This flag enables verbose error messages.
        
        This error is typically caused by a faulty Docker installation or lack of permissions to execute it under the current user account.
        
        Step 2: Resolve detected issues
        The following are the most commonly encountered issues and their workarounds
        
        A file, tool, binary, or resource referenced by the Bazel runfiles tree is missing..
        
        Confirm that all dependencies of the affected targets have been explicitly declared
        
        A file, tool, binary, or resource referenced by an absolute path or the PATH variable is missing.
        
        Confirm that all required tools are installed within the toolchain container and use toolchain rules to properly declare dependencies pointing to the missing resource
        
        A binary execution fails. One of the build rules is referencing a binary incompatible with the execution environment (the Docker container)
        
        A file from local-jdk is missing or causing errors.
        
        The Java binaries on your local machine are leaking into the build while being incompatible with it.
        
        Use java_toolchain in your rules and targets instead of local_jdk
        
        Troubleshooting in a Docker container
        
        With this method, Bazel runs inside a host Docker container, and Bazel’s build actions execute inside individual toolchain containers spawned by the Docker sandbox feature.
        
        The sandbox spawns a brand new toolchain container for each build action and only one action executes in each toolchain container
        
        This method provides more granular control of tools installed in the host environment.
        
        By separating the execution of the build from the execution of its build actions and keeping the installed tooling to a minimum, you can verify whether your build has any dependencies on the local execution environment.
        
        Step 1: Build the container
        
        Create a Dockerfile that creates the Docker container and installs Bazel with a minimal set of build tools
        
        Build the container as bazel_container
        
        Step 2: Start the container
        Step 3: Test the container
        
        docker ps
        
        bazel version
        
        Step 4: Run the build
        
        bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox <target>
        
        Step 5: Resolve detected issues
        
        If the build fails with an “out of disk space” error, you can increase this limit by starting the host container with the flag --memory=XX where XX is the allocated disk space in gigabytes.
        
        If the build fails during the analysis or loading phases, one or more of your build rules declared in the WORKSPACE file are not compatible with remote execution
  - - - To find what was executed during workspace initialization
        
        Run bazel clean --expunge. This command will clean your local cache and any cached repositories, ensuring that all initialization will be re-run
        
        Add --experimental_workspace_rules_log_file=/tmp/workspacelog to your Bazel command and run the build.
        
        Download the Bazel source code and navigate to the Bazel folder by using the command below.
        
        You need the source code to be able to parse the workspace log with the workspacelog parser
        
        git clone https://github.com/bazelbuild/bazel.git
        
        In the Bazel source code repo, convert the whole workspace log to text.
        
        bazel build src/tools/workspacelog:parser
        
        bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt
        
        To exclude specific rules from the output, use --exclude_rule option.
        
        Open /tmp/workspacelog.txt and check for unsafe operations.
  - - - Troubleshooting cache hits
        
        Ensure re-running the same build/test command produces cache hits