From a66222c82bb9f378e7ef4f8f91fb192c404bef3d Mon Sep 17 00:00:00 2001 From: Carlos Amedee Date: Thu, 6 Jul 2023 13:32:15 -0400 Subject: [PATCH] This change adds the initial set of instructions for contributors to add builders to the LUCI ci system. They are not intended to be perfect but should provide contributors with enough information to get started and provide feedback to the team. --- DashboardBuilders.md | 51 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/DashboardBuilders.md b/DashboardBuilders.md index 4a5ba411..a15111e6 100644 --- a/DashboardBuilders.md +++ b/DashboardBuilders.md @@ -38,3 +38,54 @@ For WIP ports, the steps above can be done out of order as needed. But as a port Generally, community-run builders only run code that's already been reviewed & submitted. We only enable pre-submit testing for builders run by the Go team that have a lot of hardware available. However, the [Gomote tool](https://go.dev/wiki/Gomote) is available for a number of people on the Go team and in the Go community that lets them have arbitrary access to the builders for development & debugging. For paranoia reasons, you might want to run your builder in an isolated network that can't access any of your internal resources. + +# LUCI Builders + +The Go team is migrating the testing pipeline from a custom solution, the coordinator, to [LUCI](https://chromium.googlesource.com/chromium/src/+/master/docs/tour_of_luci_ui.md). [LUCI](https://chromium.googlesource.com/chromium/src/+/master/docs/tour_of_luci_ui.md) is an open source continuous integration system created by the Chrome open source team at Google. The Go team has adopted the use of LUCI in order to leverage a continuous integration solution which is used and supported by a larger group of developers. This should enable the team to provide a more featureful solution to the community. + +The LUCI system requires builders to run two applications which authenticate to LUCI and receive and process builds. LUCI token deamon generates a token needed to authenticate. The swarming bot uses the token to connect to LUCI and process builds. + +## How to set up a builder + + 1. [Create an issue](https://go.dev/issue/new?labels=new-builder&title=x%2Fbuild%3A+add+%3Cos-arch%3E+builder) on the Go Issue tracker requesting the addition of a new builder. + 1. Add the label `new-builder`. + 1. The title of the issue should be in the format: `x/build: add builder`. + 1. Choose a hostname. + + 1. Use `golang.org/x/build/cmd/genbotcert` to generate both a certificate signing request and a TLS private key using the hostname (chosen beforehand) as input. Send the Go team the certificate signing request. A team member will send you the resulting certificate. + 1. `genbotcert -bot-hostname ` + + 1. A Go team member will define your new builder in [LUCI](https://chromium.googlesource.com/chromium/src/+/master/docs/tour_of_luci_ui.md). + + 1. Install `go.chromium.org/luci/tokenserver/cmd/luci_machine_tokend` and configure to it to run every 10 minutes via cron as the root user. + The Machine Token Daemon communicates with the Token Server to generate and renew a LUCI machine token. The private key and the certificate should not be readable by the `swarming` user. + 1. `luci_machine_tokend -backend luci-token-server.appspot.com -cert-pem -pkey-pem ` + + 1. Install `golang.org/x/build/cmd/bootstrapswarm` and configure it to run in a loop under your operating system's process supervisor (systemd, etc) as the `swarming` user. `Bootstrapswarm` downloads the initial version of the swarming bot and ensures that it is always running. + 1. `bootstrapswarm -hostname ` + + 1. Verify the bot starts up without any errors in the logs. + +## Builder Requirements + + * An internet connection with the ability to connect to: + - https://proxy.golang.org (or an alternative proxy via GOPROXY). + - https://luci-token-server.appspot.com + - https://chromium-swarm.appspot.com + * Resources + - At least 512MB of memory. 1GB or more is highly recommended. + - 20GB disk space is ideal. + - Preferably with 2 or more (V)CPUs. + * Python3 installed and in the `PATH`. + * Permissions + - The bot should be run as the `swarming` user (without root rights). + - The bot automatically updates itself. It should have permissions to do so. + - The bot periodically restarts the machine. It should have permissions to do so (via sudo). + + + + + + + +