FOSDEM 2020

Abusing GitLab CI
to Test Kernel Patches

Michael
Hofmann

Sr. Software Engineer @ Red Hat

Debugging all the things

 

Nikolai
Kondrashov

Sr. Software Engineer @ Red Hat

Electronics/embedded enthusiast

DIGImend maintainer

spbnick

A team from Red Hat's
Base Operating Systems

and Global QE

China
Czechia
Finland
Norway
USA

"cookie"

Why Kernel CI?

Source:

Phoronix

Source:

Phoronix

Developers

Maintainers

 

 

 

 

 

 

 

 

 

 

Upstream

Git

Maintainers

Git

Release

QA

RH TESTS

Maillist

Maillist

Git

Maintainers

 

 

 

 

 

 

 

 

 

 

Red Hat

Developers

Maillist

Release

Developers

Maintainers

 

 

 

 

 

 

 

 

 

 

 

 

Upstream

Git

Maintainers

Git

Maillist

Maillist

RH TESTS

Release

RH TESTS

Release

RH TESTS

Release

Release

QA

Release

RH TESTS

Git

Maintainers

 

 

 

 

 

 

 

 

 

 

 

Red Hat

Developers

Maillist

Release

RH TESTS

Release

RH TESTS

What's So Hard?

Dmitry Vyukov

Reflections on kernel quality, development process and testing

What We've Built

Koji

COPR

GitHub

GitLab

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

Git Repo

Maillist

Git Repo

Baseline 🔔TRIGGER

master

🔔 Trigger

linux-5.2.y

rhel-8.1

rhel-7.7

 

 

 

Configuration

 

 

 

 

GitLab

Git Repo A

 

Git Repo B

 

Git Repo C

 

Merge

Build

Test

Koji

COPR

GitHub

GitLab

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

Maillist

Git Repo

Maillist

Patchwork 🔔TRIGGER

🔔 Trigger

 

 

 

 

 

Configuration

 

 

 

 

GitLab

Patchwork 1

 

linux-block

qemu-devel

Patchwork 2

 

rhel-8.1

rhel-7.7

Merge

Build

Test

master

master

rhel-8.1

rhel-7.7

Git Repo A

 

Git Repo B

 

Git Repo C

 

Lint

Koji

COPR

GitHub

GitLab

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

Git Repo

Maillist

Koji

COPR

Koji/COPR 🔔TRIGGER

🔔 Trigger

 

 

 

 

 

 

Configuration

 

 

 

GitLab

COPR

 

 

 

Test

kernel

KOJI

 

 

 

Fedora Infrastructure Message Bus (ZeroMQ)

kernel

mariadb

nodejs

bazel

qgis

python

tkn

kscreen

mame

ceph

ksh

Koji

COPR

GitHub

GitLab

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

Git Repo

Maillist

GitHub

GitLab

GitHub/GitLab Bot 🔔TRIGGER

 

 

 

 

 

 

 

 

 

 

GitHub/GitLab repo

PR/MR

 

 

 

 

 

 

 

 

 👩 fix foo

 🤖 ask me to test!

 👩 test please, 🤖!

 🤖 testing now!

 👩 ... 🍴 ...

 🤖 passed, ACK👍

RHEL7

 

Merge

Build

Test

RHEL8

 

Merge

Build

Test

stable

 

Merge

Build

Test

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

Kpet

Kpet-db

Koji

COPR

GitHub

GitLab

Git Repo

Maillist

 

 

KPET-DB

🔒

🔒

YAML

XML

 

 

 

 

 

 

 

 

 

 

KPET-DB

Arches

 

 

 

 

 

Host Types

 

 

 

 

 

x86_64

ppc64le

aarch64

+2

bare_metal

rdma

megaraid

+...

Suites

 

 

LTPLite

kdump

KVM

xfstests

USEX

firewall

stress-ng

realtime

blocktests

+89

Trees

 

 

 

 

 

Comps

 

 

 

 

 

Sets

 

 

 

 

 

rhel7

stable

rhel8

+16

headers

debug

devel

+3

net

mem

fs

+9

Suite data

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

Beaker

Koji

COPR

GitHub

GitLab

Git Repo

Maillist

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

Reporter

Mail

Koji

COPR

GitHub

GitLab

Git Repo

Maillist

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

WebUI

Data Warehouse

Koji

COPR

GitHub

GitLab

Git Repo

Maillist

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

Data Warehouse

Koji

COPR

GitHub

GitLab

Git Repo

Maillist

KCIDB

Reporter

WebUI

Mail

Data Warehouse

🔔Triggers

Lint

Merge

Build

Test

KCIDB

Kpet

Kpet-db

Beaker

 

 

 

GitLab

 

 

 

GitLab

🔔Triggers

Lint

Merge

Build

Test

Koji

COPR

GitHub

GitLab

Git Repo

Maillist

The Double-Repo Gambit

 

 

 

 

 

 

 

 

Pipeline Repo

 

 

 

 

 

 

 

 

Definition Repo

     rhel7

 

⚙️ rhel7.yml

📄 cki_pipeline.yml

⚙️ rhel8.yml

⚙️ ...

Trees

 

 

 

 

 

⚙️ stable.yml

⚙️ .gitlab-ci.yml

     rhel8

 

⚙️ .gitlab-ci.yml

    stable

 

⚙️ .gitlab-ci.yml

🔒

🔔TRIGGERing

 

 

 

 

 

 

 

 

 

 

Pipeline Repo

  stable

 

 

 

🔔 Baseline Trigger

         Stable queue: queue-5.2

🔔 Stable Queue Trigger

         Baseline: stable linux-5.2.y

         Retrigger: test-kpet-c1c4f1

  rhel7

 

 

 

         Baseline: rhel7 master

         Patch: rhel77: Fix that ...

         Retrigger: test-kpet-c1c4f1

🔔 Patchwork Trigger

🔔 GitHub Bot Trigger

🔒

🔔TRIGGERing

 

 

 

 

 

 

 

 

 

 

Pipeline Repo

  stable

 

 

 

         Stable queue: queue-5.2

         Baseline: stable linux-5.2.y

         Retrigger: test-kpet-c1c4f1

  rhel7

 

 

 

         Baseline: rhel7 master

         Patch: rhel77: Fix that ...

         Retrigger: test-kpet-c1c4f1

  stable

 

 

 

  rhel7

 

 

 

         Baseline: stable linux-5.2.y

         Stable queue: queue-5.2

         Retrigger: test-kpet-c1c4f1

         Baseline: rhel7 master

         Patch: rhel77: Fix that ...

         Retrigger: test-kpet-c1c4f1

🔒

🔔TRIGGERing

 

 

 

 

 

 

 

 

 

 

Pipeline Repo

  stable

 

 

 

         Stable queue: queue-5.2

         Baseline: stable linux-5.2.y

         Retrigger: test-kpet-c1c4f1

  rhel7

 

 

 

         Baseline: rhel7 master

         Patch: rhel77: Fix that ...

         Retrigger: test-kpet-c1c4f1

         Baseline: stable linux-5.2.y

         Baseline: rhel7 master

         Stable queue: queue-5.2

         Retrigger: test-kpet-c1c4f1

         Patch: rhel77: Fix that ...

         Retrigger: test-kpet-c1c4f1

🔒

📄 trees/rhel8-rt_brew.yml

📄 cki_pipeline.yml

📄 trees/rhel8-rt_brew.yml

📄 cki_pipeline.yml

📄 cki_pipeline.yml

📄 trees/rhel8-rt_brew.yml

📄 cki_pipeline.yml

📄 trees/rhel8-rt_brew.yml

📄 cki_pipeline.yml

Why GitLab for Kernel CI?

We Started Out With Jenkins...

Why Not GitLab?

Test System States

New

Waiting

Running

Pass

Canceled

Fail

Panic

Skip

GitLab Job States

Created

Pending

Running

Success

Canceled

Failed

 

Skipped

Limited CI Job States

  • Results of merge, build, publish jobs
  • at the moment: ~20GB/day
  • no per-project configuration (S3 etc)

Fixed Artifact Storage Location

Upgrades: No Running Pipelines

Limited Pipeline Customizations

KERNEL TESTS nEED REBOOTS

4119: Spurious K8s job terminations

  • No supervision of external kernel tests after killing

3376: graceful shutdown for jobs

Thank you!

https://cki-project.org/

cki-project@redhat.com

https://gitlab.com/cki-project/

https://github.com/cki-project/

Get our Stickers!

Icons from Font Awesome licensed under Creative Commons Attribution 4.0

Icons from Octicons (c) GitHub, Inc

"Self-Operating Napkin", Copyright Heirs of Rube Goldberg

Credits