How I do automated accessibility testing for my website

Follow along with this example of performing accessibility tests in GitLab with Pa11y and Cypress on a Jekyll website.
3 readers like this.
Digital creative of a browser on the internet

This article covers adding accessibility tests to your site using Pa11y (pa11y-ci with axe) and Cypress (with cypress-axe) in GitLab CI/CD. I use a Jekyll website as an example, but any website technology that runs in CI/CD can leverage this setup.

Prep your website

In addition to getting your website to run in CI/CD, I recommend enabling an XML sitemap feature. A sitemap allows the accessibility tests to parse all URLs to find accessibility issues across the site. I recommend the jekyll-sitemap plugin for Jekyll sites.

Collecting a list of all major URLs is a good alternate step if a sitemap is not possible. The URLs should cover all potential layouts of the website, such as pages with the highest traffic or the most landings. This approach won't catch all accessibility issues, especially content level concerns, but it will test the layout and main pages.

This scenario requires the npm or yarn package managers. I used npm for this article. If your project doesn't have npm initialized, run the npm init command to create the package.json file.

Begin with Pa11y

Pa11y is a free and open source software that tests websites for accessibility issues. Pa11y-ci is the command line utility geared towards continuous integration (CI). Install pa11y-ci as a development dependency with npm:

$ npm i --save-dev pa11y-ci

After you complete the installation, edit the package.json and add the following commands to the scripts section:

"start-detached": "bundle exec jekyll serve --detach",
"pa11y-ci:home": "pa11y-ci http://127.0.0.1:4000",
"pa11y-ci:sitemap": "pa11y-ci --sitemap http://127.0.0.1:4000/sitemap.xml --sitemap-find https://accessibility.civicactions.com --sitemap-replace http://127.0.0.1:4000 --sitemap-exclude \"/*.pdf\""
  • start-detached: Starts the web server that will run Jekyll for testing.
  • pa11y-ci:home: Runs pa11y-ci tests on the home page. Useful for troubleshooting.
  • pa11y-ci:sitemap: Runs pa11y-ci tests using the sitemap and excludes PDFs. The sitemap will refer to the live site URLs, so replace those with local URLs for testing in the CI pipeline.

Add a JSON file named .pa11yci that configures pa11y-ci with various options. Here is a sample file:

{
  "defaults": {
    "concurrency": 1,
    "standard": "WCAG2AA",
    "runners": ["axe", "htmlcs"],
    "ignore": [
      "color-contrast",
      "frame-tested"
    ],
    "chromeLaunchConfig": {
      "args": ["--disable-dev-shm-usage", "--no-sandbox", "--disable-gpu"]
    },
    "reporters": [
      "cli",
      ["./pa11y-reporter-junit.js", { "fileName": "./pa11y-report-junit.xml" }]
    ]
  }
}
  • concurrency: I reduced this set to 1 because increasing it caused errors (https://github.com/pa11y/pa11y-ci/issues/168 covers the bug, which might be fixed).
  • standard: I have stuck with the default WCAG2AA as the goal for this site.
  • runners: I ran axe (run tests using axe-core) and htmlcs (default, run tests using HTML CodeSniffer) to cover all potential accessibility issues.
  • ignore: With newer versions of axe and some of the changes to the site, I ran into color contrast false positives. I also have an embedded iframe that requires separate testing that axe will report about. I have follow-up issues to examine the axe results, so I am ignoring those criteria for now.
  • chromeLaunchConfig: pa11y-ci uses Chrome, and I found that the GitLab CI pipeline requires that the Chrome browser runs properly in the pipeline.
  • reports: I use the default command line reporter, but I also added a custom reporter that reports on the pa11y-ci results in a junit format. This came in handy for reporting the results in the GitLab CI pipeline.

That's it. Run this setup locally using npm, and you will see the following output (truncated for brevity):

dmundra in ~/workspace/accessibility/accessibility on branch main > npm run start-detached

> start-detached
> bundle exec jekyll serve --detach

Configuration file: /Users/dmundra/workspace/accessibility/accessibility/_config.yml
            Source: /Users/dmundra/workspace/accessibility/accessibility
       Destination: /Users/dmundra/workspace/accessibility/accessibility/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 8.217 seconds.
 Auto-regeneration: disabled when running server detached.
    Server address: http://127.0.0.1:4000
Server detached with pid '14850'. Run `pkill -f jekyll' or `kill -9 14850' to stop the server.
dmundra in ~/workspace/accessibility/accessibility on branch main > npm run pa11y-ci:sitemap

> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude "/*.pdf"

Running Pa11y on 110 URLs:
 > http://localhost:4000/guide/glossary - 0 errors
 > http://localhost:4000/guide/introduction - 0 errors
 > http://localhost:4000/guide/history - 0 errors
 > http://localhost:4000/guide/design - 0 errors
...

✔ 110/110 URLs passed

The site passes the tests. Here is an example job running in GitLab. The pa11y configuration continues to test all site pages for accessibility issues and report on them.

What does an error look like? Here is an example:

 > http://localhost:4000/guide/introduction - 1 errors

Errors in http://localhost:4000/guide/introduction:

 • <ul> and <ol> must only directly contain <li>, <script> or <template>
   elements (https://dequeuniversity.com/rules/axe/3.5/list?application=axeAPI)

   (#main-content > div:nth-child(2) > div > div > div > div:nth-child(1) > nav
   > ul)

   <ul class="usa-sidenav">

You get a count of the number of errors at a given URL and then details on the accessibility issue. It also displays a link to the criteria being violated and the location in the HTML of the issue.

Try Cypress

Cypress is a JavaScript testing framework and is very helpful in writing tests that interact with the site and assert that features work as expected. The setup for Cypress is very similar to pa11y-ci in terms of installation with npm.

$ npm i --save-dev cypress cypress-axe cypress-real-events

After the installation is complete, edit the package.json and add the following commands to the scripts section:

"cypress-tests": "cypress run --browser chrome --headless"
  • cypress-tests: Run the Cypress tests with a headless Chrome browser.

When launching Cypress for the first time, you get a wizard to create the configuration file. Here is a sample file:

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  video: true,
  videosFolder: 'cypress/results',
  reporter: 'junit',
  reporterOptions: {
    mochaFile: 'cypress/results/junit.[hash].xml',
    toConsole: false,
  },
  screenshotsFolder: 'cypress/results/screenshots',
  e2e: {
    // We've imported your old cypress plugins here.
    // You may want to clean this up later by importing these.
    setupNodeEvents(on, config) {
      return require('./cypress/plugins/index.js')(on, config)
    },
    baseUrl: 'http://localhost:4000',
  },
})
  • video: Take videos of the tests, which are helpful for troubleshooting.
  • videosFolder: Defines the video storage folder.
  • reporter: Set to junit to make it easier to report the results in the GitLab CI pipeline.
  • reporterOptions: Includes a path for the junit files and the keyword [hash] to preserve unique reports for each test file (otherwise, the file is overwritten). Skip the console output for the reporter and use the default output.
  • screenshotsFolder: Defines the screenshot storage folder (useful for troubleshooting).
  • e2e: References the local URL of the site and the plugins.

After setting up Cypress and writing some tests (see below for examples), run the tests locally using npm. You will see the following output (truncated for brevity):

dmundra in ~/workspace/accessibility/accessibility on branch main > npm run cypress-tests

> cypress-tests
> cypress run --browser chrome --headless

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        11.2.0                                                                         │
  │ Browser:        Chrome 109 (headless)                                                          │
  │ Node Version:   v18.10.0 (/usr/local/Cellar/node/18.10.0/bin/node)                             │
  │ Specs:          5 found (accordion.cy.js, home.cy.js, images.cy.js, menu.cy.js, search.cy.js)  │
  │ Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  search.cy.js                                                                    (5 of 5)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     2 seconds                                                                        │
  │ Spec Ran:     search.cy.js                                                                     │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /Users/dmundra/workspace/accessibility/accessibility/cypres    (0 seconds)
                          s/results/search.cy.js.mp4     
 
...
  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  search.cy.js                             00:02        1        1        -        -        - │

...

While Pa11y-ci can test interactivity, Cypress and its plugins can do much more. For a Jekyll site, I found that pa11y-ci did not catch any accessibility issues in mobile drop-down menu, dynamic search, or accordion features. I ran Cypress tests to interact with the elements (like performing searches, clicking menus, or clicking the accordion) and then checked if the results still passed accessibility tests. Here is the search example:

describe('Search', () => {
  it('should be accessible', () => {
    cy.visit('/search')
    cy.get('#search-input').type('accessibility')
    cy.checkA11yWithMultipleViewPorts()
  })
})

Here is a quick video of the running test.

The above test visits the search page, types the word "accessibility" in the search field, and then checks the results for accessibility issues. I use the cypress-axe plugin to check accessibility issues with axe core, just like pa11y-ci. I have wrapped the cypress-axe functions in a function to test multiple window sizes and report on the issues in a table format.

I also use the plugin cypress-real-events to interact with the site with a keyboard to check that the features are keyboard-accessible. Keyboard accessibility is a critical consideration (Operable principle of WCAG), and having an automated test that can confirm the features are keyboard accessible means that, maybe, there is one less test to run manually. You can see an example of the test here.

Here is an example of what an error looks like:

  Running:  a11y/anonymous_a11y.cy.js                                                      (1 of 36)
cy.log(): Accessibility scanning: Home (/)
cy.log(): 4 accessibility violations were detected
┌─────────┬────────────────────────┬────────────┬────────────────────────────────────────────────────────────────────────────────────┬───────┐
│ (index) │           id           │   impact   │                                    description                                     │ nodes │
├─────────┼────────────────────────┼────────────┼────────────────────────────────────────────────────────────────────────────────────┼───────┤
│    0    │      'image-alt'       │ 'critical' │   'Ensures <img> elements have alternate text or a role of none or presentation'   │   4   │
│    1    │      'link-name'       │ 'serious'  │                       'Ensures links have discernible text'                        │   3   │
│    2    │ 'page-has-heading-one' │ 'moderate' │ 'Ensure that the page, or at least one of its frames contains a level-one heading' │   1   │
│    3    │        'region'        │ 'moderate' │                'Ensures all page content is contained by landmarks'                │   2   │
└─────────┴────────────────────────┴────────────┴────────────────────────────────────────────────────────────────────────────────────┴───────┘

Cypress logs provide a count of the number of errors at a given URL and then details on what the accessibility issue is, the impact, and its location.

You can find additional details and examples in the Cypress folder.

Use the GitLab CI/CD

Now that you have pa11y-ci and Cypress running locally, see how to run automated accessibility tests in GitLab using CI/CD features. The GitLab repository is available here. Here is the .gitlab-ci.yml file setup:

stages:
- test

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .npm/
    - vendor/ruby

default:
  image: ruby:2
  before_script:
    - apt-get update
    - apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libnss3 lsb-release xdg-utils wget libgbm1 xvfb
    - apt-get install -y nodejs npm
    - bundle install -j $(nproc) --path vendor/ruby
    - npm ci --cache .npm --prefer-offline
    - npm run start-detached

pa11y-tests:
  stage: test
  script:
    - npm run pa11y-ci:sitemap
  artifacts:
    when: always
    reports:
      junit:
        - pa11y-report-junit.xml
    expire_in: 1 day

cypress-tests:
  stage: test
  script:
    # Install chrome browser manually, taken from https://github.com/cypress-io/cypress-docker-images/blob/master/browsers/node16.14.2-slim-chrome100-ff99-edge/Dockerfile#L48
    - wget --no-verbose -O /usr/src/google-chrome-stable_current_amd64.deb "http://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_105.0.5195.125-1_amd64.deb"
    - dpkg -i /usr/src/google-chrome-stable_current_amd64.deb
    - rm -f /usr/src/google-chrome-stable_current_amd64.deb
    - npm run cypress-tests
  artifacts:
    when: always
    paths:
      - cypress/results/
    reports:
      junit:
        - cypress/results/*.xml
    expire_in: 1 day

The file currently defines only one stage test and caches folders that store dependencies when installed. Then:

  1. Steps used by all stages:
    1. Use the Ruby version 2 image because it is compatible with the current Jekyll installation.
    2. I install many dependencies based on the documentation at running puppeteer on GitLab. Install node and npm to install site dependencies.
    3. Install the Jekyll Ruby dependencies.
    4. Install the Cypress and pa11y-ci dependencies via npm.
    5. Start the web server.
  2. Run the pa11y-ci to test the site and capture the output to a file.*
  3. Install the Chrome browser dependencies in cypress-tests using steps provided by Cypress in their Docker image configurations. Run the Cypress tests and capture the output to files.*

* Capture the output of Cypress and pa11y-ci tests as junit XML files.

Here is an example screenshot of the GitLab pipeline (taken from https://gitlab.com/civicactions/accessibility/-/pipelines/744894072):

GitLab pipeline

(Daniel Mundra, CC BY-SA 4.0)

Here is an example of the test results in the same pipeline:

Test results

(Daniel Mundra, CC BY-SA 4.0)

GitLab CI/CD automatically take junit XML files and outputs them in a clear format. Cypress tests provide the junit XML output as part of their features (see above). I created a custom reporter for pa11y-ci to output the format in junit (credit to macieklewkowicz/pa11y-reporter-junit).

Note: GitLab version 12.8+ supports Pa11y accessibility tests (see https://docs.gitlab.com/ee/ci/testing/accessibility_testing.html for details). The above setup allows for customization of the pa11y-ci and also targeting of local URLs. I recommend using their options for live sites.

Wrap up

Using the above steps, you can provide accessibility testing for your site locally and in CI. This process helps you track and fix accessibility issues on your site and in the content. An important caveat about automated testing is that it only catches 57% of issues, so you definitely want to include manual testing with your accessibility testing.

Further reading and examples

Thank you to Marissa Fox and Mike Gifford for your support, thoughts, and feedback.

Headshot of Daniel Mundra
Daniel is an Associate Director and Drupal back-end engineer at CivicActions, Inc. Daniel participates in the CivicActions accessibility practice area and is interested in improving how accessibility testing is included in our projects and community.

1 Comment

Great article, Daniel! It's always important to prioritize accessibility, and your detailed guide on using Pa11y and Cypress for automated testing in GitLab CI/CD with Jekyll websites is invaluable. This process helps website owners track and fix issues more efficiently while remembering that manual testing should still be part of the overall strategy. Keep up the good work!

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.