You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
317 lines
14 KiB
317 lines
14 KiB
.. _docs-python-build:
|
|
|
|
======================
|
|
Pigweed's Python build
|
|
======================
|
|
Pigweed uses a custom GN-based build system to manage its Python code. The
|
|
Pigweed Python build supports packaging, installation, and distribution of
|
|
interdependent local Python packages. It also provides for fast, incremental
|
|
static analysis and test running suitable for live use during development (e.g.
|
|
with :ref:`module-pw_watch`) or in continuous integration.
|
|
|
|
Pigweed's Python code is exclusively managed by GN, but the GN-based build may
|
|
be used alongside CMake, Bazel, or any other build system. Pigweed's environment
|
|
setup uses GN to set up the initial Python environment, regardless of the final
|
|
build system. As needed, non-GN projects can declare just their Python packages
|
|
in GN.
|
|
|
|
Background
|
|
==========
|
|
Developing software involves much more than writing source code. Software needs
|
|
to be compiled, executed, tested, analyzed, packaged, and deployed. As projects
|
|
grow beyond a few files, these tasks become impractical to manage manually.
|
|
Build systems automate these auxiliary tasks of software development, making it
|
|
possible to build larger, more complex systems quickly and robustly.
|
|
|
|
Python is an interpreted language, but it shares most build automation concerns
|
|
with other languages. Pigweed uses Python extensively and must to address these
|
|
needs for itself and its users.
|
|
|
|
Existing solutions
|
|
==================
|
|
The Python programming langauge does not have an official build automation
|
|
system. However, there are numerous Python-focused build automation tools with
|
|
varying degrees of adoption. See the `Python Wiki
|
|
<https://wiki.python.org/moin/ConfigurationAndBuildTools>`_ for examples.
|
|
|
|
A few Python tools have become defacto standards, including `setuptools
|
|
<https://pypi.org/project/setuptools/>`_, `wheel
|
|
<https://pypi.org/project/wheel/>`_, and `pip <https://pypi.org/project/pip/>`_.
|
|
These essential tools address key aspects of Python packaging and distribution,
|
|
but are not intended for general build automation. Tools like `PyBuilder
|
|
<https://pybuilder.io/>`_ and `tox <https://tox.readthedocs.io/en/latest/>`_
|
|
provide more general build automation for Python.
|
|
|
|
The `Bazel <http://bazel.build/>`_ build system has first class support for
|
|
Python and other languages used by Pigweed, including protocol buffers.
|
|
|
|
Challenges
|
|
==========
|
|
Pigweed's use of Python is different from many other projects. Pigweed is a
|
|
multi-language, modular project. It serves both as a library or middleware and
|
|
as a development environment.
|
|
|
|
This section describes Python build automation challenges encountered by
|
|
Pigweed.
|
|
|
|
Dependencies
|
|
------------
|
|
Pigweed is organized into distinct modules. In Python, each module is a separate
|
|
package, potentially with dependencies on other local or `PyPI
|
|
<https://pypi.org/>`_ packages.
|
|
|
|
The basic Python packaging tools lack dependency tracking for local packages.
|
|
For example, a package's ``setup.py`` or ``setup.cfg`` lists all of
|
|
its dependencies, but ``pip`` is not aware of local packages until they are
|
|
installed. Packages must be installed with their dependencies taken into
|
|
account, in topological sorted order.
|
|
|
|
To work around this, one could set up a private `PyPI server
|
|
<https://pypi.org/project/pypiserver/>`_ instance, but this is too cumbersome
|
|
for daily development and incompatible with editable package installation.
|
|
|
|
Testing
|
|
-------
|
|
Tests are crucial to having a healthy, maintainable codebase. While they take
|
|
some initial work to write, the time investment pays for itself many times over
|
|
by contributing to the long-term resilience of a codebase. Despite their
|
|
benefit, developers don't always take the time to write tests. Any barriers to
|
|
writing and running tests result in fewer tests and consequently more fragile,
|
|
bug-prone codebases.
|
|
|
|
There are lots of great Python libraries for testing, such as
|
|
`unittest <https://docs.python.org/3/library/unittest.html>`_ and
|
|
`pytest <https://docs.pytest.org/en/stable/>`_. These tools make it easy to
|
|
write and execute individual Python tests, but are not well suited for managing
|
|
suites of interdependent tests in a large project. Writing a test with these
|
|
utilities does not automatically run them or keep running them as the codebase
|
|
changes.
|
|
|
|
Static analysis
|
|
---------------
|
|
Various static analysis tools exist for Python. Two widely used, powerful tools
|
|
are `Pylint <https://www.pylint.org/>`_ and `Mypy <http://mypy-lang.org/>`_.
|
|
Using these tools improves code quality, as they catch bugs, encourage good
|
|
design practices, and enforce a consistent coding style. As with testing,
|
|
barriers to running static analysis tools cause many developers to skip them.
|
|
Some developers may not even be aware of these tools.
|
|
|
|
Deploying static analysis tools to a codebase like Pigweed has some challenges.
|
|
Mypy and Pylint are simple to run, but they are extremely slow. Ideally, these
|
|
tools would be run constantly during development, but only on files that change.
|
|
These tools do not have built-in support for incremental runs or dependency
|
|
tracking.
|
|
|
|
Another challenge is configuration. Mypy and Pylint support using configuration
|
|
files to select which checks to run and how to apply them. Both tools only
|
|
support using a single configuration file for an entire run, which poses a
|
|
challenge to modular middleware systems where different parts of a project may
|
|
require different configurations.
|
|
|
|
Protocol buffers
|
|
----------------
|
|
`Protocol buffers <https://developers.google.com/protocol-buffers>`_ are an
|
|
efficient system for serializing structured data. They are widely used by Google
|
|
and other companies.
|
|
|
|
The protobuf compiler ``protoc`` generates Python modules from ``.proto`` files.
|
|
``protoc`` strictly generates protobuf modules according to their directory
|
|
structure. This works well in a monorepo, but poses a challenge to a middleware
|
|
system like Pigweed. Generating protobufs by path also makes integrating
|
|
protobufs with existing packages awkward.
|
|
|
|
Requirements
|
|
============
|
|
Pigweed aims to provide high quality software components and a fast, effective,
|
|
flexible development experience for its customers. Pigweed's high-level goals
|
|
and the `challenges`_ described above inform these requirements for the Pigweed
|
|
Python build.
|
|
|
|
- Integrate seamlessly with the other Pigweed build tools.
|
|
- Easy to use independently, even if primarily using a different build system.
|
|
- Support standard packaging and distribution with setuptools, wheel, and pip.
|
|
- Correctly manage interdependent local Python packages.
|
|
- Out-of-the-box support for writing and running tests.
|
|
- Preconfigured, trivial-to-run static analysis integration for Pylint and Mypy.
|
|
- Fast, dependency-aware incremental rebuilds and test execution, suitable for
|
|
use with :ref:`module-pw_watch`.
|
|
- Seamless protocol buffer support.
|
|
|
|
Detailed design
|
|
===============
|
|
|
|
Build automation tool
|
|
---------------------
|
|
Existing Python tools may be effective for Python codebases, but their utility
|
|
is more limited in a multi-language project like Pigweed. The cost of bringing
|
|
up and maintaining an additional build automation system for a single language
|
|
is high.
|
|
|
|
Pigweed uses GN as its primary build system for all languages. While GN does not
|
|
natively support Python, adding support is straightforward with GN templates.
|
|
|
|
GN has strong multi-toolchain and multi-language capabilities. In GN, it is
|
|
straightforward to share targets and artifacts between different languages. For
|
|
example, C++, Go, and Python targets can depend on the same protobuf
|
|
declaration. When using GN for multiple languages, Ninja schedules build steps
|
|
for all languages together, resulting in faster total build times.
|
|
|
|
Not all Pigweed users build with GN. Of Pigweed's three supported build systems,
|
|
GN is the fastest, lightest weight, and easiest to run. It also has simple,
|
|
clean syntax. This makes it feasible to use GN only for Python while building
|
|
primarily with a different system.
|
|
|
|
Given these considerations, GN is an ideal choice for Pigweed's Python build.
|
|
|
|
.. _docs-python-build-structure:
|
|
|
|
Module structure
|
|
----------------
|
|
Pigweed Python code is structured into standard Python packages. This makes it
|
|
simple to package and distribute Pigweed Python packages with common Python
|
|
tools.
|
|
|
|
Like all Pigweed source code, Python packages are organized into Pigweed
|
|
modules. A module's Python package is nested under a ``py/`` directory (see
|
|
:ref:`docs-module-structure`).
|
|
|
|
.. code-block::
|
|
|
|
module_name/
|
|
├── py/
|
|
│ ├── BUILD.gn
|
|
│ ├── setup.py
|
|
│ ├── package_name/
|
|
│ │ ├── module_a.py
|
|
│ │ ├── module_b.py
|
|
│ │ ├── py.typed
|
|
│ │ └── nested_package/
|
|
│ │ ├── py.typed
|
|
│ │ └── module_c.py
|
|
│ ├── module_a_test.py
|
|
│ └── module_c_test.py
|
|
└── ...
|
|
|
|
The ``BUILD.gn`` declares this package in GN. For upstream Pigweed, a presubmit
|
|
check in ensures that all Python files are listed in a ``BUILD.gn``.
|
|
|
|
.. _module-pw_build-python-target:
|
|
|
|
pw_python_package targets
|
|
-------------------------
|
|
The key abstraction in the Python build is the ``pw_python_package``.
|
|
A ``pw_python_package`` represents a Python package as a GN target. It is
|
|
implemented with a GN template. The ``pw_python_package`` template is documented
|
|
in :ref:`module-pw_build-python`.
|
|
|
|
The key attributes of a ``pw_python_package`` are
|
|
|
|
- a ``setup.py`` file,
|
|
- source files,
|
|
- test files,
|
|
- dependencies on other ``pw_python_package`` targets.
|
|
|
|
A ``pw_python_package`` target is composed of several GN subtargets. Each
|
|
subtarget represents different functionality in the Python build.
|
|
|
|
- ``<name>`` - Represents the Python files in the build, but does not take any
|
|
actions. All subtargets depend on this target.
|
|
- ``<name>.tests`` - Runs all tests for this package.
|
|
|
|
- ``<name>.tests.<test_file>`` - Runs the specified test.
|
|
|
|
- ``<name>.lint`` - Runs static analysis tools on the Python code. This is a
|
|
group of two subtargets:
|
|
|
|
- ``<name>.lint.mypy`` - Runs Mypy on all Python files, if enabled.
|
|
- ``<name>.lint.pylint`` - Runs Pylint on all Python files, if enabled.
|
|
|
|
- ``<name>.install`` - Installs the package in a Python virtual environment.
|
|
- ``<name>.wheel`` - Builds a Python wheel for this package.
|
|
|
|
To avoid unnecessary duplication, all Python actions are executed in the default
|
|
toolchain, even if they are referred to from other toolchains.
|
|
|
|
Testing
|
|
^^^^^^^
|
|
Tests for a Python package are listed in its ``pw_python_package`` target.
|
|
Adding a new test is simple: write the test file and list it in its accompanying
|
|
Python package. The build will run the it when the test, the package, or one
|
|
of its dependencies is updated.
|
|
|
|
Static analysis
|
|
^^^^^^^^^^^^^^^
|
|
``pw_python_package`` targets are preconfigured to run Pylint and Mypy on their
|
|
source and test files. Users may specify which ``pylintrc`` and ``mypy.ini``
|
|
files to
|
|
use on a per-package basis. The configuration files may also be provided in the
|
|
directory structure; the tools will locate them using their standard means. Like
|
|
tests, static analysis is only run when files or their dependencies change.
|
|
|
|
Packages may opt out of static analysis as necessary.
|
|
|
|
Installing packages in a virtual environment
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Python packages declared in the Python build may be installed in a specified
|
|
`virtual environment <https://docs.python.org/3/tutorial/venv.html>`_. The
|
|
default venv to use may be specified using a GN build arg. The venv may be
|
|
overridden for individual targets. The Python build tracks installation status
|
|
of packages based on which venv is in use.
|
|
|
|
The Python build examines the ``VIRTUAL_ENV`` environment variable to determine
|
|
the current venv. If the selected virtual environment is active, packages are
|
|
installed directly into it. If the venv is not active, it is activated before
|
|
installing the packages.
|
|
|
|
.. admonition:: Under construction
|
|
|
|
Pigweed Python targets are not yet aware of the virtual environment.
|
|
Currently, all actions occur in the active venv without consulting
|
|
``VIRTUAL_ENV``.
|
|
|
|
Python packages defined entirely in tree are installed with the ``--editable``
|
|
option. Partially or fully generated packages are installed without that option.
|
|
|
|
Building Python wheels
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
Wheels are the standard format for distributing Python packages. The Pigweed
|
|
Python build supports creating wheels for individual packages and groups of
|
|
packages. Building the ``.wheel`` subtarget creates a ``.whl`` file for the
|
|
package using the PyPA's `build <https://pypa-build.readthedocs.io/en/stable/>`_
|
|
tool.
|
|
|
|
The ``.wheel`` subtarget of ``pw_python_package`` records the location of
|
|
the generated wheel with `GN metadata
|
|
<https://gn.googlesource.com/gn/+/master/docs/reference.md#var_metadata>`_.
|
|
Wheels for a Python package and its transitive dependencies can be collected
|
|
from the ``pw_python_package_wheels`` key. See
|
|
:ref:`module-pw_build-python-wheels`.
|
|
|
|
Protocol buffers
|
|
^^^^^^^^^^^^^^^^
|
|
The Pigweed GN build supports protocol buffers with the ``pw_proto_library``
|
|
target (see :ref:`module-pw_protobuf_compiler`). Python protobuf modules are
|
|
generated as standalone Python packages by default. Protocol buffers may also be
|
|
nested within existing Python packages. In this case, the Python package in the
|
|
source tree is incomplete; the final Python package, including protobufs, is
|
|
generated in the output directory.
|
|
|
|
Generating setup.py
|
|
-------------------
|
|
The ``pw_python_package`` target in the ``BUILD.gn`` duplicates much of the
|
|
information in the ``setup.py`` or ``setup.cfg`` file. In many cases, it would
|
|
be possible to generate a ``setup.py`` file rather than including it in the
|
|
source tree. However, removing the ``setup.py`` would preclude using a direct,
|
|
editable installation from the source tree.
|
|
|
|
Pigweed packages containing protobufs are generated in full or in part. These
|
|
packages may use generated setup files, since they are always be packaged or
|
|
installed from the build output directory.
|
|
|
|
See also
|
|
========
|
|
|
|
- :ref:`module-pw_build-python`
|
|
- :ref:`module-pw_build`
|
|
- :ref:`docs-build-system`
|