To answer this question, we looked at data from a study by Deque. The below table shows the WCAG 2.1 level AA criteria that Deque’s Axe product was able to test for. It also shows what percentage of errors the automated testing was able to find, and what percentage of total errors that criteria accounted for.
For example, on 1.4.3 contrast, Axe found 83% of the total errors and human testers found the remaining contrast errors. Contrast accounted for 30% of the total errors on the page. This tells us that contrast errors are extremely common and often show up many times on a specific page.
| WCAG Criteria | % of errors automated testing found | Total % of errors |
|---|---|---|
| 3.1.1 Language of Page | 91.81% | 0.74% |
| 4.1.1 Parsing | 90.28% | 11.69% |
| 1.4.3 Contrast (Minimum) | 83.11% | 30.08% |
| 2.4.1 Bypass Blocks | 79.00% | 0.86% |
| 1.1.1 Non-Text Content | 67.57% | 8.04% |
| 4.1.2 Name, Role, Value | 54.42% | 16.37% |
| 1.3.1 Info and Relationships | 45.17% | 12.33% |
| 3.3.2 Labels or Instructions | 20.42% | 0.86% |
| 1.3.5 Identify Input Purpose | 15.31% | 0.29% |
| 1.4.1 Use of Color | 12.17% | 1.26% |
| 2.4.2 Page Titled | 11.26% | 0.75% |
| 2.5.3 Label in Name | 6.07% | 0.18% |
| 2.2.1 Timing Adjustable | 5.46% | 0.14% |
| 2.1.1 Keyboard | 2.49% | 3.19% |
| 1.4.12 Text Spacing | 2.23% | 0.23% |
How much can automation help with accessibility?
This is a really common question, as more and more automation and AI powered tools are entering our digital lives. To cut through all of the marketing noise and promises many of these companies are making, I looked for some real data. I found a study by Deque regarding their Axe product.
If you are unfamiliar with those names, you’ve probably heard of Google and their Lighthouse tool. Lighthouse is powered by Axe. It’s not exactly the same as Lighthouse filters some of the tests that Axe uses. Before we get into the data, I’m going to use the phrase success criteria a lot. This reference is WCAG, or Web Content Accessibility Guidelines, and their success criteria.
WCAG guidelines are the most often ones used for testing, and some have even been codified into some laws. Success criteria basically just means how to pass or fail each of those individual guidelines. More information on the source data I’m using here can be found in the video description and accompanying blog post.
Data from the study
Deque found that Axe was able to partially test 15 out of the 50 success criteria in WCAG. Or about 30 percent of the success criteria could be tested. So, that’s not ideal since that leaves about 70 percent of them untested completely. But, let’s see how well Axe did at testing the 30 percent that it could test.
For two of these success criteria, Axe was able to find more than 90 percent of the total issues. But for four of them, it was able to find less than 7%, not great. From this data, we can see that the automated testing did a pretty good job in some cases, and did pretty much terrible in others. And of course, there’s a 70 percent of criteria that it simply couldn’t even touch.
Now that we’ve looked at what automation is capable of partially testing for, let’s see how often those specific criteria come up. Think of it this way. When you drive a car, there are some driving laws that you encounter all the time, and some almost never. For example, stopping at a stop sign. This is frequently encountered a lot.
Whereas an esoteric law regarding a left hand turn from a one way street to a one way street. Eh, some people may never encounter that. Web accessibility testing is similar. Some success criteria, like color contrast, you will encounter on pretty much every website. Whereas some others, like adjusting a timer limit, you may never encounter.
How many issues were found
Let’s look at color contrast a little more, since that comes up so often in web accessibility. Color contrast errors accounted for about 30 percent of all the errors. That’s a lot.
Looking at the color contrast a little more automated testing found 83 percent of text contrast issues, leaving 17 percent to be found by manual testing. So it was definitely helpful and beneficial clearing a whole bunch of the testing for us. But it was definitely not comprehensive.
Color contrast can be particularly tricky since there’s so many CSS rules and JavaScript rules which can apply to it. From the Deque study, Axe found a total of 57 percent of the actual errors that occurred on webpages. Again, this is because some success criteria happen a lot and some happen very rarely.
What lighthouse scores mean
So when we look at scores like Lighthouse, which is powered by Axe, it will tell you about how many tests you passed. Again, only able to test for a certain subset of the success criteria. So if you have a score of 100 from Lighthouse, it means 100 percent of the tests were passed. These tests can find about 57 percent of issues on a typical page.
So a score of 100 means you still have 43 percent of issues that may be missed on a typical page. And if your page has a lot of functionality on it, probably more. So what does all this data tell us? It tells us that automated testing is able to test for less than half of the success criteria. But many of those criteria don’t show up that often.
So automated testing is able to find about 57 percent of issues on a typical page. If your page has things like videos, pop up elements, fixed position elements, calendars, e commerce, membership, eLearning, or any of the other multitudes of advanced functionality, it’s probably going to find fewer of your actual issues.
Summary
Automated testing cannot comprehensively test for any of the criteria, which means that manual testing is always important to your process and cannot be skipped. So the next time you see marketing information from a company, which promises to fully test your page with automation or AI, I wouldn’t believe them.
The data says automation can partially test, and the data shows that there’s a lot that automation simply cannot test for. Thanks for watching. I’m Gen Herres from The Easy A11y Guide, where we try to make web accessibility easier to implement through done-for-you processes and tools. If you enjoyed this video, please give it a thumbs up, and you can of course subscribe to our channel for more accessibility tips.
For more information on web accessibility, including tools, tutorials, and more, please visit easya11yguide.com. I look forward to seeing you again soon.