Hands on: Prow

Agenda

• What is Prow
• Prow Control Plane
• Deploy & Configure your Prow platform
• Prow Plugins
• Auto-management of your GitHub repo
• Testing with Prow
• Notification management
• References

What is Prow

• CI/CD System hosted by a Kubernetes/Openshift platform
• Management of Github repo via comments /lgtm or /retest
• Made in Golang
• Jobs manifests writed in YAML
• Similar behaviour than Operators
• This is the workflow diagram

What is Prow

• Stores the configuration on ConfigMaps and Secrets
• By default uses the test-pod namespace to trigger the pod jobs and the default namespace for Prow Control Plane
• An this is how looks like:

NAME                                READY   STATUS    RESTARTS   AGE
deck-f958755bc-skq64                1/1     Running   0          2d2h
deck-f958755bc-v5568                1/1     Running   0          2d2h
hook-57d5467cbc-6ndvg               1/1     Running   0          10d
hook-57d5467cbc-k22fm               1/1     Running   0          10d
horologium-79965ffb5-c29nt          1/1     Running   0          10d
plank-79646b78fb-mn796              1/1     Running   0          10d
sinker-796f7df8cb-tdj5b             1/1     Running   0          10d
statusreconciler-56b668448f-bm7f9   1/1     Running   0          10d
tide-6d66ddcb4c-gms2t               1/1     Running   0          10d

What is Prow

• Based on Deployments and Services

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/deck         NodePort    10.104.115.56    <none>        80:30000/TCP     10d
service/hook         NodePort    10.105.150.158   <none>        8888:30300/TCP   10d
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP          10d
service/tide         NodePort    10.109.88.186    <none>        80:30090/TCP     10d

• Where Deck is the UI component
• Hook is the key element for webhooks and redirections (to/from GitHub and to/from plugins)

Github /retest comment > Hook > Tide > Trigger jobs

• Tide will take care about testing and merging (once configured)

Prow Control Plane

Prow Control Plane: Hook

• Hook is the most important piece.
• It is a stateless server that listens for GitHub webhooks and dispatches them to the appropriate plugins.
• Hook's plugins are used to trigger jobs, implement 'slash' commands, post to Slack, and more...

Prow Control Plane: Plank

• Plank is the controller that manages the job execution and lifecycle for jobs that run in k8s pods.
• Decores some of the pod behaviour like Timeout, GracePeriod, etc...
• Also manages some components like Utility_Images
• Take care about spin up a pod that uploads the artifacts to GCS

Prow Control Plane: Deck

• Deck is the Prow's WebUI and presents a nice view of recent jobs, among other things

Prow Control Plane: Deck

• The PR current status

Prow Control Plane: Deck

• ...and history of merge automation

Prow Control Plane: Deck

• Also a dashboard for PR authors.

Prow Control Plane: Horologium and Synker

Both are very simple but useful components

• Horologium triggers periodic jobs when necessary.
• Sinker cleans up old jobs and pods. (works as a GC)
• Horologium does not have explicit config

Prow Control Plane: Tide

• Tide manages retesting and merging PRs once they meet the configured merge criteria.
• Ensure to sync the ConfigMap config with the running config and update it if applies
• Works with a pool of GitHub PRs that match a given set of criteria
• It will automatically retest PRs that meet the criteria and automatically merge them when they have up-to-date passing test results
• Serves live data to populate Tide Dashboard, PR Dashboard and also Tide History page
• Provides these merge modes: merge, squash and rebase

Prow Control Plane: Crier

• Crier manages the notifications against different providers like slack, github, etc..
• Reports your prowjobs on their status changes.
• Communication with providers through API tokens inside of k8s secrets

Those are the supported providers:

• Gerrit
• Pubsub
• Github
• Slack

Questions?

Deploying Prow

Deploying Prow

We have 2 ways to deploy Prow:

• Using Tackle: This will deploy a K8s cluster on GKE previously to deploy Prow
• Manually: You need to have at least a K8s working w/e you want to deploy, you could use this guide

We recommend to use Tackle because in some minutes you will have it Prow working, you will need to follow this procedure and some requirements:

• Bazel + Tackel + kubectl client + gcloud client
• Access to GCE account (Tackle will create a GKE cluster)
• GitHub Oauth token: To allow bot user to execute actions in your repo/org

This is a Sample Installation but without SSL (T_T)

Deploying Prow

Or Manual way:

With our new fresh K8s cluster we will start working to deploy Prow:

• Sample Prow Manual Installation (you could avoid the terraform part)

We will summarize all the nesessary steps to make it work (all explained in depth on the previous link):

• Get into your k8s instance and Install Golang, Bazel, Tackel and some dev requirements
• Create the proper RBAC and secrets that Prow needs
• Clone test-infra repository, create test-pods NS and apply the prow/cluster/starter.yaml manifest

Deploying Prow

Depending on How did you configure your K8s instances you will access to your Prow instances in a way or other:

• If you are working without GKE and you have all-in-one k8s instance, you could access via NodePorts

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/deck         NodePort    10.104.115.56    <none>        80:30000/TCP     10d
service/hook         NodePort    10.105.150.158   <none>        8888:30300/TCP   10d
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP          10d
service/tide         NodePort    10.109.88.186    <none>        80:30090/TCP     10d

• Otherwise you could access via Ingress through the LoadBalancer that GKE provides you.

Questions?

Configuring Prow

Configuring Prow - SSL

To deploy SSL in our cluster we will need some additional requirements

• Google Domain ".dev" (in my case external to GCE account)
• Some instructions to Deploy Cert-Manager
• Permissions to create LB's on GCE account

NOTE: depending on your decission you could go with Let's Encrypt or SelfSigned cert but if you're using a ".dev" domain you should use a real CA to validate your domain.

• After deploy the Cert-Manager you need to create the Issuer/ClusterIssuer
• Then the Certificate
• And at last the SSL Ingress

Check this detailed instructions!!

Configuring Prow

• Now we need to add the webhook to GitHub. Organization or Repository depending on the admin level that you want to give your bot.
• You will need a GitHub account to use it as a bot. Our stagging one Janitor.

You will need to spend some time preparing your config files, here you have our ones that maybe could be useful.

• Prepare the Config file for Prow
• Prepare the Plugins file for Prow
• Prepare the Labels file for Prow
• Create folder tree and their Jobs for Prow

Configuring Prow

Let's go into the different sections of the ConfigFile. We will not review all the possible options, we will work with our sample file ;)

• Main (no section):

prowjob_namespace: default
pod_namespace: test-pods
log_level: debug

• Sinker:

sinker:
  resync_period: 1m
  max_prowjob_age: 48h
  max_pod_age: 48h
  terminated_pod_ttl: 30m

Configuring Prow

• Plank:

plank:
  job_url_template: 'https://prow.kerbeross.dev/view/gcs/kerbeross-prow-storage/{{if eq .Spec.Type "presubmit"}}pr-logs/pull{{else if eq .Spec.Type "batch"}}pr-logs/pull{{else}}logs{{end}}{{if .Spec.Refs}}{{if ne .Spec.Refs.Org ""}}/{{.Spec.Refs.Org}}_{{.Spec.Refs.Repo}}{{end}}{{end}}{{if eq .Spec.Type "presubmit"}}/{{with index .Spec.Refs.Pulls 0}}{{.Number}}{{end}}{{else if eq .Spec.Type "batch"}}/batch{{end}}/{{.Spec.Job}}/{{.Status.BuildID}}/'
  job_url_prefix_config: 
    '*': "https://prow.kerbeross.dev/view/gcs/"
  pod_pending_timeout: 60m
  default_decoration_configs:
    '*':
      timeout: 2h
      grace_period: 15s
      utility_images:
        clonerefs: "gcr.io/k8s-prow/clonerefs:v20200117-903ce628f"
        initupload: "gcr.io/k8s-prow/initupload:v20200117-903ce628f"
        entrypoint: "gcr.io/k8s-prow/entrypoint:v20200117-903ce628f"
        sidecar: "gcr.io/k8s-prow/sidecar:v20200117-903ce628f"
      gcs_configuration:
        bucket: "kerbeross-prow-storage"
        path_strategy: "explicit"
      gcs_credentials_secret: "gcs"

Configuring Prow

• Plank needs more explanation here, as you see there are some fields that refers to GCS, this means that Prow takes care about to get the artifacts generated by the Jobs and uploads them to a GCS bucket previously configured

Note: Careful with the file name, must be service-account.json, the secret name is customizable.

• job_url_template: Points to the correct bucket path in order to store the artifacts
• pod_pending_timeout: The time that a pod will wait on Pending until fails
• default_decoration_config: section to extend the pod definition
• utility_images: Images that will take care about different tasks
• gcs_configuration.path_strategy: Explicit means that you set the concrete path with job_url_template

Configuring Prow

tide:
  sync_period: 2m
  merge_method:
    the-shadowmen/prow-devconf: squash
    the-shadowmen/kubevirt-tutorial: squash
    the-shadowmen/kubevirtci: squash
  queries:
    - repos:
        - the-shadowmen/prow-devconf
        - the-shadowmen/kubevirt-tutorial
        - the-shadowmen/kubevirtci
      labels:
        - lgtm
        - approved
      missingLabels:
        - do-not-merge
        - do-not-merge/hold

  pr_status_base_urls:
    '*': https://prow.kerbeross.dev/pr
  context_options:
    from-branch-protection: false
    skip-unknown-contexts: true

Configuring Prow

In Tide section you set the correct context to manages the automerge:

• On queries field we set the repositories to work with and the criteria to merge the PRs, in this example we need at least the lgtm and approve tags to perform an auto-merge.
• Tide will look for missingLabels field in order to check that those tags are not there, otherwise Tide will block the auto-merge.
• pr_status_base_url Is a PR page that shows you the status of your pendings ones. You need to configure this other thing in order to make it work (also have an SSL configured Prow instance)

Configuring Prow

Inside of Deck section and concretelly on spyglass one, we just are telling it that we will have 3 artifacts and are located on those paths

deck:
  spyglass:
    size_limit: 500000000 # 500MB
    viewers:
      "started.json|finished.json":
      - "metadata"
      "build-log.txt":
      - "buildlog"
      "artifacts/junit.*\\.xml":

Also on the config file we used to have the Notifiers and the Jobs, we will talk about those in their own section.

Configuring Prow

Now we need to verify that the ConfigFile and PluginFile is well formed with a Bazel task and then load it as ConfigMap on the default namespace.

• Go to the test-infra repo location and execute this command: bazel run //prow/cmd/checkconfig -- \ --plugin-config=$HOME/projects/prow-devconf/prow-config/plugins.yaml \ --config-path=$HOME/projects/prow-devconf/prow-config/config.yaml \ --job-config-path=$HOME/projects/prow-devconf/prow-config/jobs

• Then load them into K8s default namespace (full path needed): kubectl create configmap plugins --from-file=$HOME/prow_conf/plugins.yaml --dry-run -o yaml \ | kubectl replace configmap plugins -f - # If the CM already exists

kubectl create configmap config --from-file=config.yaml=config.yaml --dry-run -o yaml \
  | kubectl create configmap config -f -    # If the CM doesn't exists

kubectl create configmap label-config --from-file=$HOME/prow_conf/labels.yaml -o yaml

Configuring Prow

Ok, now we have all set to start working with a basic Prow platform, this allows you to manage and automate your Github repository with auto-labelling, approve merges (without auto-merging), etc...

The capabilities of this depend on what plugins did you activate on the PluginsConfig file
there are plenty of them, but some of them needs to add additional ConfigMaps/Sections in ConfigFile/etc...

Let's take a look to plugins...

Questions?

Prow Plugins

Prow Plugins

Well, Prow plugins a a bunch of resources that give you capabilities on Prow platform. As an example, assign plugin gives the functionality to assign a reviewer to a PR that is already on going.

Prow Plugins

This is how looks like part of PluginsConfig file, this one shows the enabled plugins for our organization:

plugins:
  the-shadowmen:        # Organization
  - size                # Manages the 'size/*' labels per PR
  - label               # Base plugin for labeling
  - hold                # Allows to add or remove the 'do-not-merge/hold'
  - assign              # Assigns or requests reviews from users.
  - blunderbuss         # Auto-Assign new PRs created to the reviewers.
  - lifecycle           # Close, reopen, flag and/or unflag an issue or PR as frozen/stale/rotten
  - verify-owners       # Validates OWNERS files if they are modified in a PR.
  - wip                 # Applies the 'do-not-merge/wip' Label to pull requests.

  the-shadowmen/prow-devconf:
  - trigger

As you see those plugins are enabled for all the organization and you doesn't need to add them for every repo you have. Instead some other plugins need to be enabled by repository to make them work

Prow Plugins

Inside of plugins: section we should add every repo and plugins that want to be managed by Prow:

  the-shadowmen/kubevirt.github.io:
  - trigger
  - owners-label

• And depending on the plugin, customize the config with their own configuration appart form plugins: section

lgtm:
- repos:
  - the-shadowmen/kubevirt-tutorial
  - the-shadowmen/kubevirtci
  - the-shadowmen/prow-devconf
  review_acts_as_lgtm: true

Prow Plugins

Some plugins needs additional Config sections on Prow ConfigFile or maybe needs Additional services to be working properly, like lgtm, approve, or trigger. All of them has a thing in common, are managed by the same service Tide, which needs to be properly configured.

As we said before:

• Tide manages retesting and merging PRs once they meet the configured merge criteria.

Also interacts with Plank in order to trigger Jobs

Prow plugins: config-updater

As you see, the config and the Jobs are mixed in the same ConfigMap/File, this is weird isn't it?
Well this is what you need:

Prow plugins: config-updater

Config updater redeploys configuration and plugin configuration files when they change and allows you to keep separated your Jobs from your config.

• Just add the configuration to your plugins.yaml:

config_updater:                 # Config Updater section
  maps:
    prow-config/config.yaml:    # It will take the Config from this path
      name: config
    prow-config/jobs/**/*.yaml: # It will take the Jobs from this path
      name: job-config
    prow-config/plugins.yaml:   # It will take the Plugins from this path
      name: plugins
    prow-config/labels.yaml:    # It will take the Labels from this path
      name: label-config

Note: Ensure you've been validated that the config, plugins and jobs are correctly set on their files with checkconfig.

Prow plugins: config-updater

After that you will need to add the config-updater to the repo that contains the data:

  the-shadowmen/prow-devconf:
  - trigger
  - owners-label
  - lgtm
  - approve
  - config-updater

This is the link between the repo and the config. Then the plugin will create the ConfigMaps:

NAME                     DATA   AGE
config                   1      2d2h
job-config               1      17h
label-config             1      20h
plugins                  1      2d2h

Prow plugins: config-updater

In order to let prow know that you are using a separated ConfigMap to the deployments, you need to patch the deck, hook, horologium, plank, sinker, tide and statusreconciler, with this changes:

• on container args section: - --job-config-path=/etc/job-config

• on volumeMounts section:

- mountPath: /etc/job-config
  name: job-config
  readOnly: true

• on Volume section: - configMap: defaultMode: 420 name: job-config name: job-config

Prow plugins: config-updater

Prow plugins: config-updater

• PR Sample

Prow plugins: Approvers/Owner-labels

Owners will allow you to set reviewers and approvers by folder or setting up somefilters in your managed repositories.

To make it work you only need to enable the plugin in the repo config and after that, create a OWNERS file in the desired folder, it will have this format:

    reviewers:
      - codificat
      - jparrill
      - iranzo
    approvers:
      - iranzo
      - ptrnull
    emeritus_approvers:
      - karmab # 10 july 2019
      - tripledes # 10 july 2019

Prow plugins: Approvers/Owner-labels

filters:
  ".*":
    reviewers:
      - codificat
      - jparrill
      - iranzo
    approvers:
      - iranzo
      - ptrnull
    emeritus_approvers:
      - karmab # 10 july 2019
      - tripledes # 10 july 2019
## This part is for owner-labels
  "^_posts/.*":
    labels:
      - kind/blog
  "^_(data|layouts|sass|css|js)/.*":
    labels:
      - kind/website

This is a real sample of filters

Prow plugins: Approvers/Owner-labels

This is a very good example about how to configure Prow Jobs among multiple work groups:

• Kubevirt Example

The folders have their own OWNERS file and every job file have the kind of job followed by the repo name: kubevirt-tutorial-presubmits.yaml
and the OWNERS file is like:

reviewers:
  - codificat
  - iranzo
  - ptrnull
  - jparrill
approvers:
  - jparrill
  - codificat
  - iranzo
  - ptrnull

Prow plugins: Others

• Trigger: This will allow you to trigger tests on desired repositories
• Lgtm: Using /lgtm act as review on a PR
• Approve: Approves a PR using /approve

Questions?

Auto-management of your GitHub repo

Auto-management of your GitHub repo

In order to manage out repo automatically, we need to go through our Config file. You need to ensure that you've filled these sections properly

• Plank: to have the utility images ready and GCS bucket linked to upload artifacts
• Tide: To manages automerge and testing jobs
• Deck: To describe the resultant artifacts to be uploaded to GCS

You also need to have the Github Webhook pointing properly to your Hook service on K8s on your Github Org

Auto-management of your GitHub repo

There is not a great way to bake the config, because the documentation is not good and you need to go through the code to know all the config statements, but you at least you have a clear workflow to follow.

• Prepare your new config
• Check it with the Bazel job checkconfig (as we saw before) bazel run //prow/cmd/checkconfig -- \ --plugin-config=$HOME/projects/prow-devconf/prow-config/plugins.yaml \ --config-path=$HOME/projects/prow-devconf/prow-config/config.yaml \ --job-config-path=$HOME/projects/prow-devconf/prow-config/jobs

Auto-management of your GitHub repo

• Upload the ConfigMap

kubectl create configmap plugins --from-file=plugins.yaml=$HOME/prow_conf/plugins.yaml \
  --dry-run -o yaml | kubectl replace configmap plugins -f -

kubectl create configmap config --from-file=config.yaml=$HOME/prow_conf/config.yaml \
  --dry-run -o yaml | kubectl replace configmap config -f -

kubectl create configmap label-config --from-file=labels.yaml=$HOME/prow_conf/labels.yaml \
  --dry-run -o yaml | kubectl replace configmap label-config -f -

This will allow you to start with Prow Jobs easily.

Questions?

Testing with Prow

Testing with Prow

If you'd been set the config as we followed, I've good news for you!

You should be able to execute testing right now! \o/

Testing with Prow

The Prow jobs works in the same way as you spin up an application in K8s, just select a Base Image to work with, the EntryPoint, Volumes, Env variables, NodeSelector (in order to use concrete nodes in a kind of test).

Let's see what kind of Jobs we have and how to configure them.

Periodic:

• The component that will take care about trigger this jobs is Horologium
• These are context free jobs.
• Will be executed in a pace that you decide.
• You cannot set a Repository to work with (as we do with others)

Testing with Prow

This is how looks like the Periodic Job spec:

periodics:
  - interval: 24h
    name: kubevirt-io-periodic-link-checker
    decorate: true      # The pod will be decorated with our special config
    always_run: true    # Tests always run
    extra_refs:
    - org: the-shadowmen
      repo: kubevirt.github.io
      base_ref: master
      path_alias: kubevirt.github.io skip_report: false  # Avoid to report the Job status
    spec:
      nodeSelector:
        region: primary
      containers:
        - image: docker.io/library/ruby
          env:
            - name: NOKOGIRI_USE_SYSTEM_LIBRARIES
              value: "true"
          command: ["/bin/sh", "-c"]
          args: ["/usr/local/bin/bundle install && bundle exec rake"]

Testing with Prow

Presubmit:

• The components that will take care about trigger this jobs are Hook, Plugins or|and Tide.
• This jobs are triggered when a PR is Created and will execute the tests directly on the proper branch BEFORE the merge happens.
• These jobs have context, then you need to put the repostitory to watch.
• Github will trigger a notification when the webhook criteria match, against the Hook service.
• From that Plugins comes in action and do their thing, after that sometimes the workflow ends here.
• Other times Tide will continue it managing the situation in case of /test or PR merge

Testing with Prow

This is how looks like the Presubmit Job spec:

presubmits:
  the-shadowmen/kubevirt.github.io:
    - name: kubevirt-io-presubmit-link-checker
      decorate: true      # The pod will be decorated with our special config
      always_run: true    # Tests always run
      skip_report: false  # Avoid to report the Job status
      spec:
        nodeSelector:
          region: primary
        containers:
          - image: docker.io/library/ruby
            env:
              - name: NOKOGIRI_USE_SYSTEM_LIBRARIES
                value: "true"
            command: ["/bin/sh", "-c"]
            args: ["/usr/local/bin/bundle install && bundle exec rake"]

Testing with Prow

Postsubmit:

• The components that will take care about trigger this jobs are Hook, Plugins and Tide.
• These jobs are triggered when a PR is Merged and will execute the tests directly on the proper branch AFTER the merge happens.
• These jobs have context, then you need to put the repostitory to watch.
• Github will trigger a notification when the webhook criteria match, against the Hook service.
• From that Plugins comes in action and do their thing, after that will notify Tide.
• Other times Tide will finish it executing the proper tests.

Testing with Prow

This is how looks like the Postsubmit Job spec:

postsubmits:
  the-shadowmen/kubevirt.github.io:
    - name: kubevirt-io-postsubmit-link-checker
      decorate: true      # The pod will be decorated with our special config
      always_run: true    # Tests always run
      skip_report: false  # Avoid to report the Job status
      spec:
        nodeSelector:
          region: primary
        containers:
          - image: docker.io/library/ruby
            env:
              - name: NOKOGIRI_USE_SYSTEM_LIBRARIES
                value: "true"
            command: ["/bin/sh", "-c"]
            args: ["/usr/local/bin/bundle install && bundle exec rake"]

Questions?

Prow Notifications

Prow Notifications

For Presubmits and Postsubmits jobs are not necessary to have these because GitHub manage the notifications, but what happens when a Periodic job fails? who will let me know about?, well Crier is here to save the day :).

This is not a mandatory service to work, but recommendable, as we said before:

• Crier manages the notifications against different providers like slack, github, etc..
• Reports your prowjobs on their status changes.
• Communication with providers through API tokens inside of k8s secrets

Those are the supported providers:

• Gerrit
• Pubsub
• Github
• Slack

Prow Notifications

To Configure Crier you could follow this guide or this one, but let's summarize:

• With test-infra repo, apply the RBAC manifest from test-infra/prow/cluster/crier_rbac.yaml
• Patch Crier manifest with --slack-workers=n and --slack-token-file=/etc/slack-api-token it is on test-infra/prow/cluster/crier_deployment.yaml
• Deploy Crier on your control plane
• Get a Slack API Token and put it inside of a secret. kubectl create secret generic slack-api-token \ --from-file=slack-api-token=$HOME/private/SLACK-API-TOKEN
• Mount the Slack API Token Secret on the Jobs directly, at /etc/slack-api-token path

Prow Notifications

• Modify the Prow's ConfigMap adding a slack_reporter section including the desired configuration:

slack_reporter:
  job_types_to_report:
  - presubmit
  - postsubmit
  - periodic
  - batch
  job_states_to_report:
  - triggered
  - pending
  - success
  - failure
  - aborted
  - error
  channel: team-kni-community
  # The template shown below is the default
  report_template: 'Job {{.Spec.Job}} of type {{.Spec.Type}} ended with state {{.Status.State}}. <{{.Status.URL}}|View logs>'

Questions?

References

References

Introduction:

• Readme file
• Command Help
• Prow Go Doc
• Mandatory Article to read
• Prow Quickstart
• Prow Images
• Prow PR Workflow
• Prow Best Practices

References

Prow Plugins:

• Prow Plugins
• Prow Code-Review process

Prow Jobs:

• Prow Jobs overview
• Life of a Prow Job
• Webhook Payload sample
• Deep dive Jobs
• Phaino
• Tide Config

References

Prow SSL:

• Cert-Manager Installation for this talk
• Prow-SSL Installation for this talk
• Configure SSL
• Cert Manager
• Cert Manager Tutorial

Others:

• Another useful Article
• Kubevirt Project-Infra A great example of how to manage Prow config and some config files to use it as reference

The end