One of the most common questions regarding accessibility testing is: what testing method should I use? The second most common one is: how much coverage of WCAG criteria do various testing methods have?
Not sure what WCAG is? Take a look at our video & article Introduction to Accessibility.
To answer this we need to discuss how often various accessibility issues occur on websites. In life, we commonly refer to the 80/20 rule. That applies to web accessibility as well. About 80% of issues come from about 20% of the WCAG criteria.
This means that a small number of tests can find the bulk of total accessibility issues on a website.
While it would be great if these key items could be tested by automation, the reality is they can’t. Automation can partially test about 25-30% of WCAG criteria. Since a small number of criteria make up the bulk of issues, automation can find about 50% of issues on a given web page. Unfortunately, many of the key blocking issues for people, automation is unable to test.
So where is the middle ground? How do you get enough coverage with your tests without going into a full manual audit? You do a shortened audit, focusing on a mix of top issues that occur and top blocking issues that prevent users from using a web page. We call it the accessibility quick audit.
How much WCAG coverage?
Deque did a detailed study looking into what issues came up the most often and how much automation could test them. Their tests used the Axe library which is open source and powers many tools around the web, including Google’s Lighthouse. In their testing, they found 12 WCAG 2.1 level AA success criteria account for 92% of all errors. To keep with our 80/20 example, 80% of WCAG issues came from 15% of the WCAG criteria.
From their data, we can see that automation did very poorly on issues that affect keyboard users. Focus Order, Keyboard, and Focus Visible are very important to people who use keyboard navigation. Automation was able to find almost none of these issues. In our quick audit process, we have 3 manual tests to check for these issues as they can make a page completely unusable for keyboard users.
| WCAG Criteria | Quick Audit | % Auto Issues | % of all issues |
|---|---|---|---|
| 1.4.3 Contrast (Minimum) | Fully | 83% | 30% |
| 4.1.2 Name, Role, Value | 80% | 54% | 16% |
| 1.3.1 Info and Relationships | 90% | 45% | 12% |
| 4.1.1 Parsing | 90% | 90% | 12% |
| 1.1.1 Non-Text Content | 80% | 68% | 8% |
| 2.4.3 Focus Order | Fully | 0% | 3% |
| 2.1.1 Keyboard | Fully | 2% | 3% |
| 2.4.7 Focus Visible | Fully | 0% | 3% |
| 1.4.11 Non-text Contrast | Fully | 0% | 2% |
| 1.4.1 Use of Color | Fully | 12% | 1% |
| 1.3.2 Meaningful Sequence | 90% | 0% | 1% |
| 3.3.2 Labels for Instructions | Fully | 20% | 1% |
| Total % of errors found | 85% | 56% | 92% |
What testing method should you use?
I recommend starting with a single-page quick audit. It’s an affordable way to get a good picture of your accessibility. In addition to the items above in the table (showing about 85% WCAG coverage on the quick audit), it also tests for an accessibility statement, cookie & privacy policies, text resizing, autoplay, and captions/transcripts for your videos. Finally, the quick audit includes a short screen reader test (see next section for full screen reader testing).
The report from the quick audit includes recommended steps to take to remedy the issues. After the quick audit, you can schedule the included Zoom call to review the report and ask questions.
If you have your own accessibility testing staff, you can do it yourself using our quick audit guide.
Hiring screen reader users to test
If you want to ensure that your website is accessible, you need screen reader testing. Contact us to schedule a screen reader user test. We have experienced screen reader users with certifications from the IAAP. We can also refer you to other screen reader users who are available for direct pay testing.
Watching a screen reader user go through your website over a Zoom call is one of the most educational experiences you will have in web accessibility.
In terms of testing coverage, a person who is blind using a screen will not be able to test for visual issues. When we do screen reader tests for clients, we make sure that a sighted person is available on the call to point out any sections of the page that are unreachable by our screen reader testers.
If you are going to hire your own screen reader tester, ask to be on a screen sharing zoom call with them so that you can watch them navigate your site and see if there are any areas they can not reach. And always, always, make sure that you are fairly compensating your testers. They have put thousands of hours into learning how to use screen readers and deserve to be compensated fairly.
Full manual testing
We always recommend that you start with limited scope testing, like our quick audit process. This allows you to the major blocking issues and resolve them. You should also get a screen reader tester involved early to ensure there are no major blocking issues for them.
We recommend getting a full manual test (which includes screen reader users) for a section of your key pages. If you have a website such as eCommerce, we always recommend getting a user flow test of the process. It normally starts with your search, includes adding products to the cart, navigating to and from the cart, and then the checkout page and finally thank you page. It also includes checking the thank you email.
How often to test
If you are doing active remediation work on your site, you should get regular testing to ensure the process is going well. This can be via live testing with screen reader users, or via live testing with experienced sighted tester.
Once your site is at a good point for accessibility, we recommend regular automated testing to alert you to issues. The two platforms that we work with, Accessibility Cloud and Equalify, include weekly monitoring of sites.
In addition to automated monitoring, you want to run periodic manual testing. For low change websites, this can be done quarterly. If you are have a higher change site, like a busy eCommerce site, we encourage monthly manual testing.
How to report your testing
The best way to report your testing is to list the types of testing and frequency in your accessibility statement.
This lets customers know that you care about accessibility and that if they have issues and contact you about them, you will act. It can be a deterrent to law suits, and regular testing is a requirement in some accessibility law suit settlements.
Some accessibility laws require certain disclosures on testing status in accessibility statements. These are most common in Europe.