Prow is a Kubernetes based CI/CD system. Jobs can be triggered by various types of events and report their status to many different services. In addition to job execution, Prow provides GitHub automation in the form of policy enforcement, chat-ops via /foo style commands, and automatic PR merging.
NOTE|WARNING: In order to make Prow work fine with your repo, the Kubernetes cluster MUST be reachable by GitHub Webhook. Then the most used option is to deploy it on GKE directly.
hook is the most important piece. It is a stateless server that listens for GitHub webhooks and dispatches them to the appropriate plugins. Hook's plugins are used to trigger jobs, implement 'slash' commands, post to Slack, and more. See the prow/plugins directory for more information on plugins.plank is the controller that manages the job execution and lifecycle for jobs that run in k8s pods.deck presents a nice view of recent jobs, command and plugin help information, the current status and (history)[https://prow.k8s.io/tide-history] of merge automation, and a dashboard for PR authors.horologium triggers periodic jobs when necessary.sinker cleans up old jobs and pods.tide manages retesting and merging PRs once they meet the configured merge criteria. See its README for more information.crier manages the notifications against different providers like slack, github, etc..Reference: https://raw.githubusercontent.com/kubernetes/test-infra/master/prow/cmd/README.md
Following https://github.com/kubernetes/test-infra/blob/master/prow/getting_started_deploy.md
Deploy instance on libvirt with terraform:
cd ~cnv/repos/kubevirt-tutorial/administrator/terraform/libvirt
terraform init -get -upgrade=true
terraform apply -var-file varfiles/jparrill.tf -refresh=true -auto-approve
cd ~ && curl -O https://dl.google.com/go/go1.11.5.linux-amd64.tar.gz
sha256sum go1.11.5.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.11.5.linux-amd64.tar.gz
cat <<EOF > $HOME/.bash_profile \
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
EOF
mkdir $HOME/go
source $HOME/.bash_profile
## Jobs by Bazel will need GCC
sudo yum groupinstall "development tools" -y
sudo yum install wget -y
sudo wget https://copr.fedorainfracloud.org/coprs/vbatts/bazel/repo/epel-7/vbatts-bazel-epel-7.repo -O /etc/yum.repos.d/bazel.repo
sudo yum install bazel -y
go get -u k8s.io/test-infra/prow/cmd/tackle
kubectl create clusterrolebinding cluster-admin-binding-kubernetes-admin --clusterrole=cluster-admin --user=kubernetes-admin
mkdir ~/private
openssl rand -hex 20 > $HOME/private/HMAC_TOKEN
kubectl create secret generic hmac-token --from-file=hmac=$HOME/private/HMAC_TOKEN
echo "f25cc009637532179fb2cdec2d888a39749ac067" > $HOME/private/OAUTH_SECRET
kubectl create secret generic oauth-token --from-file=oauth=$HOME/private/OAUTH_SECRET
cd $HOME && git clone https://github.com/kubernetes/test-infra.git && cd $HOME/test-infra
kubectl create namespace test-pods
kubectl config set-context $(kubectl config current-context) --namespace=default
kubectl apply -f prow/cluster/starter.yaml
deck deployment
# Use sshuttle to access the Prow interface
sshuttle -r jparrill@192.168.1.XXX 192.168.123.0/24 -v
[kubevirt@k8s-kubemaster test-infra]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deck NodePort 10.102.35.212 <none> 80:32494/TCP 2d21h
hook NodePort 10.101.54.234 <none> 8888:31050/TCP 2d21h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d15h
tide NodePort 10.100.34.208 <none> 80:31840/TCP 2d21h
In order to be more standard depending if you are using GKE, you just could use an ingress with a LB. If not you could use directly the NodePorts
Add WebHook to Github
# We need to update git in order to let Bazel to use "git -C ...." sentences
sudo sh -c "cat <<EOF > /etc/yum.repos.d/wandisco-git.repo
[wandisco-git]
name=Wandisco GIT Repository
baseurl=http://opensource.wandisco.com/centos/7/git/\$basearch/
enabled=1
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF"
sudo rpm --import http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
sudo yum update git -y
####
go get -u k8s.io/test-infra/experiment/add-hook
bazel run //experiment/add-hook -- \
--hmac-path=$HOME/private/HMAC_TOKEN \
--github-token-path=$HOME/private/OAUTH_SECRET \
--hook-url http://kubevirt-prow-0.gce.sexylinux.net:30300/hook \
--repo the-shadowmen \
--confirm=false
You also have a couple of make commands to work with, on Nostromo repo:
- make update-config
- make update-plugins
- make update-labels
NOTE: Depending on the plugin you will need to add some resources to the managed repo, like OWNERS file
cd $HOME/test-infra
bazel run //prow/cmd/checkconfig -- --plugin-config=$HOME/prow_conf/plugins.yaml --config-path=$HOME/prow_conf/config.yaml
kubectl create configmap plugins \
--from-file=$HOME/prow_conf/plugins.yaml --dry-run -o yaml \
| kubectl replace configmap plugins -f -
cd $HOME/prow_conf && kubectl create configmap config --from-file=config.yaml=config.yaml --dry-run -o yaml | kubectl replace configmap config -f -
cd $HOME/prow_conf && kubectl create configmap plugins --from-file=$HOME/prow_conf/plugins.yaml --dry-run -o yaml | kubectl replace configmap plugins -f -
cd $HOME/prow_conf && kubectl create configmap label-config --from-file=$HOME/prow_conf/labels.yaml -o yaml
We need to emulate what we're doing with Rake and TravisCI tool using Prow but adding some features. For this we will use:
The jobs are managed by more than one core components:
Plank as a job controller, this one manage the lifecycle of the jobsHorologium triggers periodic Jobs.Tide manages retesting and merging PRs.where you could make some kind of debugging of your jobs submission is in Tide component.
You need to upload the jobs to a config map by default called config. There you could put the Prow config (separated by component) and the jobs itself. In any case you could use a plugin called config-updater which allows you to maintain separated the config from the jobs itself.
As we shows before, this is an example of config file
In order to execute jobs you need to configure from this point, GCS to store the logs and artifacts (if applies), GitHub bot account credentials to manage the organization or repo.
You have 3 kind of jobs:
In the usual postsubmit and presubmit jobs there is not problems, Github will take care about the notifications but on periodic ones you need an additional component called Crier which will allow you to send notifications to external communication providers like Slack, Github, Gerrit, etc...
To do that, we need some things (sample Slack):
default namespace--slack-workers=n and --slack-token-file=path-to-tokenfileslack_reporter section including the desired configuration.