162 lines · plain
1.. SPDX-License-Identifier: GPL-2.0+2 3=========================================4Automated testing of the DRM subsystem5=========================================6 7Introduction8============9 10Making sure that changes to the core or drivers don't introduce regressions can11be very time-consuming when lots of different hardware configurations need to12be tested. Moreover, it isn't practical for each person interested in this13testing to have to acquire and maintain what can be a considerable amount of14hardware.15 16Also, it is desirable for developers to check for regressions in their code by17themselves, instead of relying on the maintainers to find them and then18reporting back.19 20There are facilities in gitlab.freedesktop.org to automatically test Mesa that21can be used as well for testing the DRM subsystem. This document explains how22people interested in testing it can use this shared infrastructure to save23quite some time and effort.24 25 26Relevant files27==============28 29drivers/gpu/drm/ci/gitlab-ci.yml30--------------------------------31 32This is the root configuration file for GitLab CI. Among other less interesting33bits, it specifies the specific version of the scripts to be used. There are34some variables that can be modified to change the behavior of the pipeline:35 36DRM_CI_PROJECT_PATH37 Repository that contains the Mesa software infrastructure for CI38 39DRM_CI_COMMIT_SHA40 A particular revision to use from that repository41 42UPSTREAM_REPO43 URL to git repository containing the target branch44 45TARGET_BRANCH46 Branch to which this branch is to be merged into47 48IGT_VERSION49 Revision of igt-gpu-tools being used, from50 https://gitlab.freedesktop.org/drm/igt-gpu-tools51 52drivers/gpu/drm/ci/testlist.txt53-------------------------------54 55IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt56file, see below).57 58drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt59----------------------------------------------------------60 61Lists the known failures for a given driver on a specific hardware revision.62 63drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt64-----------------------------------------------------------65 66Lists the tests that for a given driver on a specific hardware revision are67known to behave unreliably. These tests won't cause a job to fail regardless of68the result. They will still be run.69 70Each new flake entry must be associated with a link to the email reporting the71bug to the author of the affected driver, the board name or Device Tree name of72the board, the first kernel version affected, the IGT version used for tests,73and an approximation of the failure rate.74 75They should be provided under the following format::76 77 # Bug Report: $LORE_OR_PATCHWORK_URL78 # Board Name: broken-board.dtb79 # Linux Version: 6.6-rc180 # IGT Version: 1.28-gd2af13d9f81 # Failure Rate: 10082 flaky-test83 84drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt85-----------------------------------------------------------86 87Lists the tests that won't be run for a given driver on a specific hardware88revision. These are usually tests that interfere with the running of the test89list due to hanging the machine, causing OOM, taking too long, etc.90 91 92How to enable automated testing on your tree93============================================94 951. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one96yet97 982. In your kernel repo's configuration (eg.99https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the100CI/CD configuration file from .gitlab-ci.yml to101drivers/gpu/drm/ci/gitlab-ci.yml.102 1033. Request to be added to the drm/ci-ok group so that your user has the104necessary privileges to run the CI on https://gitlab.freedesktop.org/drm/ci-ok105 1064. Next time you push to this repository, you will see a CI pipeline being107created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)108 1095. The various jobs will be run and when the pipeline is finished, all jobs110should be green unless a regression has been found.111 112 113How to update test expectations114===============================115 116If your changes to the code fix any tests, you will have to remove one or more117lines from one or more of the files in118drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms119affected by the change.120 121 122How to expand coverage123======================124 125If your code changes make it possible to run more tests (by solving reliability126issues, for example), you can remove tests from the flakes and/or skips lists,127and then the expected results if there are any known failures.128 129If there is a need for updating the version of IGT being used (maybe you have130added more tests to it), update the IGT_VERSION variable at the top of the131gitlab-ci.yml file.132 133 134How to test your changes to the scripts135=======================================136 137For testing changes to the scripts in the drm-ci repo, change the138DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in139drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.140janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.141 142 143How to incorporate external fixes in your testing144=================================================145 146Often, regressions in other trees will prevent testing changes local to the147tree under test. These fixes will be automatically merged in during the build148jobs from a branch in the target tree that is named as149${TARGET_BRANCH}-external-fixes.150 151If the pipeline is not in a merge request and a branch with the same name152exists in the local tree, commits from that branch will be merged in as well.153 154 155How to deal with automated testing labs that may be down156========================================================157 158If a hardware farm is down and thus causing pipelines to fail that would159otherwise pass, one can disable all jobs that would be submitted to that farm160by editing the file at161https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.162