12 KiB

Raw Blame History Unescape Escape

Android Build System Concepts

This document provides high level explanations and mapping of the internal build system components and concepts of the Android build system and Bazel, and how components communicate with each other.

High level components

This table provides a high level overview of the components in the current Android Platform build system, and how each component maps to a concept in Bazel.

Android build system component	Description	Mapping to Bazel concepts
Kati	Make-compatible front-end. Encodes build logic in `.mk` scripts. Declares buildable units in `Android.mk`. Generates Ninja file directly.	Loading and analysis phase. Conceptually similar to `bazel build --nobuild`.
Blueprint	Build definition syntax. Build syntax parser. Internal data structures like Modules/Variations/Context/Scope. Ninja file generator.	Starlark.
Soong	Bazel-like front-end. Encodes build logic in Go. Declares build units in `Android.bp`, parsed by Blueprint. Uses Blueprint to generate Ninja file. Generates a `.mk` file with prebuilt module stubs to Kati.	Loading and analysis phase. Conceptually similar to `bazel build --nobuild command`.
Ninja	Serialized command line action graph executor. Executes Ninja graph generated from Kati and Soong.	Bazel's execution phase.
atest	Test executor and orchestrator.	Conceptually similar to `bazel test` command.
Blueprint + Kati + Soong + Ninja + atest	The entire build pipeline for Android.	Conceptually similar to `bazel build` or `bazel test` commands.
`<script>.sh`	Running arbitrary scripts in AOSP.	Conceptually similar to `bazel run` command.
Make (replaced in-place by Kati)	No longer in use. Entire build system, replaced by the tools above.	Loading, analysis, execution phases. Conceptually similar to `bazel build` command.
`Android.bp`	Build definition file for Soong.	`BUILD.bazel` or `BUILD`.
`Android.mk`	Build definition file for Kati.	`BUILD.bazel` or `BUILD`.

Communication between components

Kati product configuration component to generate config variables (config.mk, AndroidProducts.mk)
- “Kati-config” for the purpose of this document
Kati component to generate build actions in Ninja files (main.mk, Android.mk files)
- “Kati-build” for the purpose of this document
Kati component to generate packaging actions in Ninja files (packaging.mk file)
- “Kati-package” for the purpose of this document
Kati component to generate cleaning actions in Ninja files (cleanbuild.mk, CleanSpec.mk files)
- "Kati-cleanspec" for the purpose of this document
soong_build (and Blueprint) component to generate build actions (Android.bp, Blueprints files)
Ninja component to execute actions from Kati-build, Kati-package and soong_build
Bazel as the next generation of the entire build system, starting as a Ninja executor drop-in replacement
soong_ui as the tool to orchestrate all of the above, and with auxiliary tools like finder, path_interposer, soong_env, minibp and bpglob.

The current build system architecture primarily uses files as the medium for inter-process communication (IPC), with one known case of unix socket communication (e.g. path_interposer), and a fifo between Ninja and soong_ui for the Protobuf stream for build status reporting.

Component order

The build system components run in the following order, orchestrated by soong_ui:

soong_ui bootstraps itself with microfactory (go build replacement) and launches.
soong_ui runs auxiliary tools to aggregate files into filelists, for Android.mk, Android.bp, AndroidProducts.mk and several others.
soong_ui runs Kati-config with the config.mk entry point.
soong_ui orchestrates 3 Blueprint/Soong phases to generate the main out/soong/build.ninja file: minibootstrap, bootstrap, and primary.
1. Minibootstrap phase uses Blueprint/Microfactory to build itself (minibp) so that Android.bp and Blueprint files can be used to define Soong.
2. Bootstrap phase runs Ninja on a build.ninja file that runs minibp to read all Android.bp files across the source tree that describes Soong and plugins, and builds soong_build.
3. Primary phase runs Ninja on a build.ninja file that runs soong_build to generate the final out/soong/build.ninja file.
4. soong_build also runs its own tests alongside generating out/soong/build.ninja, which can be skipped with the --skip-soong-tests argument.
soong_ui runs Kati-cleanspec with the cleanbuild.mk entry point.
soong_ui runs Kati-build to generate a Ninja file, with the main.mk entry point.
soong_ui runs Kati-package to generate a Ninja file, with the packaging/main.mk entry point.
soong_ui generates a Ninja file to combine above Ninja files.
soong_ui runs either Ninja or Bazel to execute the build, with the combined Ninja file as entry point.

soong_ui has a --skip-make flag that will skip Kati-config, Kati cleanspec, Kati-build and Kati-package, used for Soong-only builds in NDK and some Mainline projects.

soong_ui

soong_ui is primarily responsible for orchestrating the build, cleaning the build environment, and running auxiliary tools. These tools (minibp, microfactory) can bootstrap other tools (soong_build), aggregate file lists (finder.go), improve hermeticity (path_interposer, nsjail) or perform checks against the environment (soong_env).

soong_ui uses finder.go to generate <filename>.list files for other tools. For example, it generates Android.mk.list for Kati-build, AndroidProducts.mk.list for Kati-config, and Android.bp.list for soong_build.

soong_ui uses path_interposer to prepare an hermetic $PATH with runtime checks against allowlisted system tools. The $PATH contains these system tools with checked-in prebuilts, and uses path_interposer to intercept calls and error out whenever non-allowlisted tools are used (see out/.path for directory of intercepted tool symlinks).

soong_ui generates a Kati suffix to ensure that Kati-generated files are regenerated if inputs to Kati have changed between builds.

soong_ui calls Soong and Kati to generate Ninja files, and eventually creates another Ninja file (out/combined-<product>.ninja) to combine the others, and executes either Ninja or Bazel to complete the build.

soong_ui sets up the sandbox and environment for the Ninja/Bazel process.

Kati-config

As a product configuration tool, soong_ui runs Kati-config in --dumpvars-mode to dump the values of specified Make variables at the end of an evaluation, with build/make/core/config.mk as the entry point. During this phase, Kati-config eventually evaluates soong_config.mk to generate the soong.variables JSON file. This way, Kati-config can communicate product configuration to soong_build, as soong_build parses the dumped variables from the JSON on startup, and stores them into an in-memory Config object.

To communicate dexpreopt variables to soong_build, dexpreopt.config is also generated as a $(shell) action and read by soong_build in a similar way as Kati-config evaluates dex_preopt_config.mk included in soong_config.mk.

soong_ui sets up a KatiReader to monitor Kati-config’s stdout/err for UI reporting and error handling purposes.

soong_build

soong_build’s primary role is to evaluate all Android.bp files, run a series of mutators, and generate out/soong/build.ninja file.

soong_build communicates with Kati-build by generating Make Vars and running the AndroidMk singleton to generate .mk files in the output directory (out/soong/{Android, late, make_vars}-<product>.mk).

Android-<product>.mk contains Soong modules as Make modules so Make modules can depend on Soong modules.
make_vars-<product>.mk contains Make variables for Kati-build, exported from Soong modules. There are also checks built into this .mk file to ensure that if a duplicate Make variable of the same name comes from another source, the Soong and Make variable values are identical.
late-<product>.mk contains Make variables that are not read while Kati-build parses the Android.mk file. (Late variables)
- soong_ui invokes Kati to parse make_vars .mk file earlier than the Android.mk files,and late.mk after parsing the Android.mk files.
- late.mk is used to define phony rules to take advantage of Make’s ability to add extra dependencies to an existing rule. late.mk is not strictly necessary to make this happen at this moment, since late.mk rules don’t currently depend on any variables defined during Android.mk processing (e.g. ALL_MODULES$(module).INSTALLED).

Kati-build / Kati-package

Kati-build’s primary role is to evaluate all Android.mk files with build/make/core/main.mk as entry point, and generate out/build-<product>.ninja. It also generates cleanspec.ninja for the product, containing statements on how to remove stale output files.

Kati-build’s primary role is to evaluate all packaging .mk files with build/make/packaging/main.mk as entry point, including build/make/packaging/distdir.mk for dist-for-goals calls, and generate out/package-<product>.ninja.

Kati-build/Kati-package’s stdout/stderr is monitored by soong_ui’s KatiReader to UI and error handling.

As Kati-build/Kati-package generates Ninja files, they also generate out/ninja-<product>.sh and out/env-<product>.sh. These scripts are wrappers for soong_ui to execute Ninja with the correct Ninja files, in a controlled environment.

Ninja

As Ninja executes files from Kati-build, Kati-package, soong_build and other bootstrapping tools like Blueprint, it writes to a fifo in a proto front end that soong_ui monitors with NinjaReader. NinjaReader ensures that the user interface for Ninja progress is consistent with the rest of the build.

Bazel

As more Soong modules are converted to BUILD files, soong_build serializes information about converted modules to BUILD/bzl files on disk. soong_build then consumes information about these targets from Bazel by directly calling the Bazel client to issue cquery calls about these targets.

12 KiB Raw Blame History Unescape Escape