You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
97 lines
4.6 KiB
97 lines
4.6 KiB
4 months ago
|
# ANGLE Testing Infrastructure
|
||
|
|
|
||
|
|
ANGLE runs hundreds of thousands of tests on every change before it lands in
|
||
|
|
the tree. We scale our pre-commit and post-commit testing to many machines
|
||
|
|
using [Chromium Swarming][Swarming]. Our testing setup heavily leverages
|
||
|
|
existing work in Chromium. We also run compile-only
|
||
|
|
[Standalone Testing][Standalone] that does not depend on a Chromium checkout.
|
||
|
|
|
||
|
|
Also see the documentation on [ANGLE Wrangling][Wrangling] for more info.
|
||
|
|
|
||
|
|
## Pre-Commit Testing
|
||
|
|
|
||
|
|
See the pre-commit try waterfall here:
|
||
|
|
|
||
|
|
[`https://ci.chromium.org/p/chromium/g/tryserver.chromium.angle/builders`](https://ci.chromium.org/p/chromium/g/tryserver.chromium.angle/builders)
|
||
|
|
|
||
|
|
We currently run pre-commit tests on:
|
||
|
|
|
||
|
|
* Windows 32-bit AMD and Windows 64-bit Intel and NVIDIA GPUs
|
||
|
|
* Linux 64-bit NVIDIA and Intel GPUs
|
||
|
|
* Mac NVIDIA, Intel and AMD GPUs
|
||
|
|
* Pixel 2 and Nexus 5X
|
||
|
|
* Fuchsia testing in a VM
|
||
|
|
|
||
|
|
Looking at an example build shows how tests are split up between machines. See for example:
|
||
|
|
|
||
|
|
[`https://ci.chromium.org/ui/p/chromium/builders/try/mac-angle-try/454/overview`](https://ci.chromium.org/ui/p/chromium/builders/try/mac-angle-try/454/overview)
|
||
|
|
|
||
|
|
This build ran 81 test steps across 3 GPU families. In some cases (e.g.
|
||
|
|
`angle_deqp_gles3_metal_tests`) the test is split up between multiple machines to
|
||
|
|
run faster (in this case 4 different machines at once). This build took 15
|
||
|
|
minutes to complete 1 hour of real automated testing.
|
||
|
|
|
||
|
|
For more details on running and working with our test sets see the docs in [Contributing Code][Contrib].
|
||
|
|
|
||
|
|
[Swarming]: https://chromium-swarm.appspot.com/
|
||
|
|
[Standalone]: #ANGLE-Standalone-Testing
|
||
|
|
[Contrib]: ../doc/ContributingCode.md#Testing
|
||
|
|
[Wrangling]: ANGLEWrangling.md
|
||
|
|
|
||
|
|
## ANGLE Standalone Testing
|
||
|
|
|
||
|
|
In addition to the ANGLE try bots using Chrome, and the GPU.FYI bots, ANGLE
|
||
|
|
has standalone testing on the Chrome infrastructure. Currently these tests are
|
||
|
|
compile-only. This page is for maintaining the configurations that don't use
|
||
|
|
Chromium. Also see the main instructions for [ANGLE Wrangling](ANGLEWrangling.md).
|
||
|
|
|
||
|
|
It's the ANGLE team's responsibility for maintaining this testing
|
||
|
|
infrastructure. The bot configurations live in four different repos and six
|
||
|
|
branches.
|
||
|
|
|
||
|
|
## Info Consoles
|
||
|
|
|
||
|
|
Continuous builders for every ANGLE revision are found on the CI console:
|
||
|
|
|
||
|
|
[https://ci.chromium.org/p/angle/g/ci/console](https://ci.chromium.org/p/angle/g/ci/console)
|
||
|
|
|
||
|
|
Try jobs from pre-commit builds are found on the builders console:
|
||
|
|
|
||
|
|
[https://ci.chromium.org/p/angle/g/try/builders](https://ci.chromium.org/p/angle/g/try/builders)
|
||
|
|
|
||
|
|
## How to add a new build configuration
|
||
|
|
|
||
|
|
1. [`bugs.chromium.org/p/chromium/issues/entry?template=Build+Infrastructure`](http://bugs.chromium.org/p/chromium/issues/entry?template=Build+Infrastructure):
|
||
|
|
|
||
|
|
* If adding a Mac bot, request new slaves by filing an infra issue.
|
||
|
|
|
||
|
|
1. [`chrome-internal.googlesource.com/infradata/config`](http://chrome-internal.googlesource.com/infradata/config):
|
||
|
|
|
||
|
|
* Update **`configs/chromium-swarm/starlark/bots/angle.star`** with either Mac slaves requested in the previous step or increase the amount of Windows or Linux GCEs.
|
||
|
|
|
||
|
|
1. [`chromium.googlesource.com/chromium/tools/build`](https://chromium.googlesource.com/chromium/tools/build):
|
||
|
|
|
||
|
|
* Update **`scripts/slave/recipes/angle.py`** with new the config.
|
||
|
|
* The recipe code requires 100% code coverage through mock bots, so add mock bot config to GenTests.
|
||
|
|
* Maybe run `./scripts/slave/recipes.py test train` to update checked-in golden files. This might no longer be necessary.
|
||
|
|
|
||
|
|
1. [`chromium.googlesource.com/angle/angle`](http://chromium.googlesource.com/angle/angle):
|
||
|
|
|
||
|
|
* Update **`infra/config/global/cr-buildbucket.cfg`** to add the new builder (to ci and try), and set the new config option.
|
||
|
|
* Update **`infra/config/global/luci-milo.cfg`** to make the builders show up on the ci and try waterfalls.
|
||
|
|
* Update **`infra/config/global/luci-scheduler.cfg`** to make the builders trigger on new commits or try jobs respectively.
|
||
|
|
* Update **`infra/config/global/commit-queue.cfg`** to add the builder to the default CQ jobs (if desired).
|
||
|
|
|
||
|
|
## Other Configuration
|
||
|
|
|
||
|
|
There are other places where configuration for ANGLE infra lives. These are files that we shouldn't need to modify very often:
|
||
|
|
|
||
|
|
1. [`chrome-internal.googlesource.com/infradata/config`](http://chrome-internal.googlesource.com/infradata/config):
|
||
|
|
|
||
|
|
* **`configs/luci-token-server/service_accounts.cfg`** (service account names)
|
||
|
|
* **`configs/chromium-swarm/pools.cfg`** (swarming pools)
|
||
|
|
|
||
|
|
1. [`chromium.googlesource.com/chromium/tools/depot_tools`](http://chromium.googlesource.com/chromium/tools/depot_tools):
|
||
|
|
|
||
|
|
* **`recipes/recipe_modules/gclient/config.py`** (gclient config)
|