Automated Tools & Their Limits (Web Accessibility Tutorial)

Automated accessibility tools scan a page and report problems in seconds: missing alt attributes, unlabelled fields, low contrast, empty buttons. They are fast, free and repeatable, so every developer should use them. They also have a hard limit: a tool can check whether something exists, but rarely whether it is correct or makes sense to a person.

In this lesson you will meet the most common tools, run the axe engine inside a page, add automated checks to a test suite, and learn exactly which problems still need human testing.

The main tools

Tool	What it is	Where you use it
Lighthouse	Audits built into Chrome DevTools, with an accessibility score from 0 to 100	DevTools, Lighthouse panel; or the `lighthouse` command-line tool
axe DevTools	Browser extension by Deque, built on the open-source axe-core engine	Chrome, Edge and Firefox developer tools
axe-core	The open-source rules engine itself, a JavaScript library	Inside pages, test scripts and CI pipelines
WAVE	Browser extension by WebAIM that marks problems directly on the page	Chrome, Edge and Firefox
Accessibility Insights for Web	Free extension by Microsoft with a quick automated check and a guided manual assessment	Chrome and Edge
W3C Markup Validation Service	Checks HTML syntax: duplicate ids, wrong nesting	validator.w3.org

Lighthouse's accessibility audits use axe-core rules. Each audit simply passes or fails, with no partial credit, and the score is a weighted average of those audits based on how much each problem affects users. Lighthouse also lists "additional items to manually check", which do not affect the score at all.

What a tool can and cannot find

A tool can find	A tool cannot judge
An image with no `alt` attribute	Whether the `alt` text actually describes the image
A form field with no label	Whether the label makes sense
Text with contrast below 4.5:1 on a solid background	Contrast of text over photos and gradients (often "needs review")
A button with no accessible name	Whether "Submit form 7B" is a sensible name
Duplicate ids, missing `lang`, a skipped heading level	Whether headings describe their sections
`aria-*` attributes that are invalid for a role	Whether the keyboard order is logical, or focus is lost after an action
Missing `<title>`	Whether captions are accurate, or motion makes someone dizzy

The W3C is clear that evaluation tools can only assist: they cannot decide whether a page is accessible, and some of their results can be wrong. A Lighthouse score of 100 does not mean a page is accessible. It means the page passed the checks that a machine can run.

Run axe inside a page

You can load axe-core into any page and run it with one function call. This example loads version 4.14.0 from a CDN and lists the problems it finds in a small, deliberately broken page.

HTMLTry it yourself

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>axe-core Demo</title>
  <style>
    .pale { color: #aaaaaa; }
    #report { background: #f2f6fb; padding: 8px 12px; }
  </style>
</head>
<body>
  <main>
    <h1>Pune Cycle Rentals</h1>
    <img src="https://course.ethicscomputer.in/assets/samples/landscape-400.svg" width="200" height="125">
    <p class="pale">Rent a cycle from ₹40 per hour.</p>
    <input type="text" placeholder="Your mobile number">
    <button type="button"></button>
    <a href="#">Click here</a>
  </main>

  <section aria-labelledby="report-title">
    <h2 id="report-title">axe report</h2>
    <div id="report" role="status">Running checks...</div>
  </section>

  <script src="https://cdn.jsdelivr.net/npm/axe-core@4.14.0/axe.min.js"></script>
  <script>
    axe.run(document.querySelector("main"), { runOnly: ["wcag2a", "wcag2aa", "wcag21aa", "wcag22aa"] })
      .then((results) => {
        const list = document.createElement("ul");
        for (const v of results.violations) {
          const li = document.createElement("li");
          li.textContent = `${v.id} (${v.impact}): ${v.help} - ${v.nodes.length} element(s)`;
          list.append(li);
        }
        const report = document.querySelector("#report");
        report.textContent = `${results.violations.length} rule(s) failed, ${results.incomplete.length} need manual review.`;
        report.append(list);
      });
  </script>
</body>
</html>

axe reports three failed rules: the image without alt, the pale text and the empty button. Look at what it does not report. The field with only a placeholder passes, because the browser uses the placeholder as a fallback name, even though that name vanishes as soon as the user types. The "Click here" link also passes, because it has a name, even though "Click here" fails the purpose of 2.4.4 for real users. That is the limit of automation in two examples.

The runOnly option limits the check to rules tagged with WCAG levels A and AA. Without it, axe also runs "best practice" rules that are not WCAG requirements; they are worth reading, but keep them separate in reports.

Automated checks in your test suite

Running checks by hand is easy to forget. Teams add them to their end-to-end tests so every change is scanned. With Playwright and the @axe-core/playwright package, a test looks like this:

JavaScript

import { test, expect } from "@playwright/test";
import AxeBuilder from "@axe-core/playwright";

test("booking page has no detectable WCAG A/AA violations", async ({ page }) => {
  await page.goto("http://localhost:3000/booking");
  const results = await new AxeBuilder({ page })
    .withTags(["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"])
    .analyze();
  expect(results.violations).toEqual([]);
});

Run the same scan after opening menus and dialogs, and on error states, not only on the first view of the page. A closed dialog cannot be checked. You can also run Lighthouse from the command line, for example npx lighthouse https://localhost:3000 --only-categories=accessibility --view, to get the same report you see in DevTools.

A sensible testing routine

Combine automated and manual checks; each catches what the other misses:

While coding: an editor linter or the axe extension on the component you are building.
On every pull request: automated axe checks in the test suite, for every page and important state.
Before release: a keyboard-only pass, a 200% and 400% zoom check, and a screen reader pass with the script from Testing with Screen Readers.
Regularly: testing with disabled users, who find problems that neither tools nor developers notice.

Common mistakes

Treating the score as the goal. Teams chase 100 and stop testing.
Scanning only the home page in its first state, never forms, errors, dialogs or logged-in pages.
Ignoring "needs review" items, such as contrast over images.
Silencing rules to make the build pass instead of fixing the cause.
No manual testing at all. Many WCAG criteria, such as meaningful link text, logical focus order and caption accuracy, need human judgement.

Key points

Lighthouse, axe, WAVE and Accessibility Insights find machine-testable problems quickly.
Lighthouse's score is a weighted pass/fail of axe-based audits; manual items do not count.
Tools check that things exist, not that they make sense; a score of 100 is not proof of accessibility.
Run axe in your test suite on every important page and state.
Always add keyboard, zoom and screen reader testing, and test with real users.

Exercise

Copy the axe Try-it example and fix the page until axe reports 0 failed rules. Then fix the one problem that axe did not report, and explain which WCAG criterion it relates to.

Show solution

HTML

<main>
  <h1>Pune Cycle Rentals</h1>
  <img src="landscape-400.svg" width="200" height="125" alt="Hills at sunrise beside a river, on the city cycle trail">
  <p class="pale">Rent a cycle from ₹40 per hour.</p>
  <label for="mobile">Your mobile number</label>
  <input id="mobile" type="tel" autocomplete="tel-national">
  <button type="button">Book a cycle</button>
  <a href="#prices">See all rental prices</a>
</main>
<style>
  .pale { color: #595959; }
</style>

The image has alt, the text colour now has a 7:1 ratio, the field has a real label and the button has a name, so axe passes. The "Click here" link passed axe but failed users; its new text "See all rental prices" meets 2.4.4 Link Purpose (In Context).

Quick Quiz

1. What does a Lighthouse accessibility score of 100 tell you?

2. Which problem can an automated tool reliably detect?

3. Which open-source engine do Lighthouse's accessibility audits use?

4. Why did axe not report the "Click here" link?