How AI Agents Use Websites, Where They Fail, and What to Fix

Written by AI

TLDR: MERJ tested browser agents on 100 purpose-built tasks and found that accessibility defects often cause commercially important failures.
Best for: Teams testing whether AI agents can complete critical website journeys
The tip: Measure task completion on your own site instead of relying only on public benchmarks or readiness scanners.
Time: 44 min
What you need: Critical website journeys to test
Start here: Define one real task on your site
Let an agent choose its own steps
Record whether the intended state change occurred
Filed note: MERJ tested browser agents across 100 tasks and found that accessibility defects cause many failures, while only task-level testing reveals whether critical journeys actually work.

Highlights: A large and commercially important subset of browser-agent failures originates in the same structural, naming, state and feedback defects that accessibility engineering already addresses.
Accessibility work gives browser agents a stronger interaction layer.
Only task-level measurement on your own site does.
Public benchmarks cannot prove that critical journeys work.
Excerpt: Agents are a new actor on the web, but businesses have little visibility into what they do on their websites. In one of our tests, we deliberately gave two buttons misleading ARIA labels. When asked to save a document, one agent clicked Cancel first in 25 out of 25 runs.
From merj.com