AI agents that browse the web, fill in forms, and act on a person’s behalf have gone from novelty to boardroom talking point in about a year. Most major AI vendors now sell some version of a browser agent. The pitch is always the same: hand it a task, walk away, come back to it done.
Decodo, a web data infrastructure company, decided to test that pitch. Its Agentic Capability Report scored 45 AI browser agents against 10 capabilities that come up constantly in business use: filling in forms, completing a purchase, remembering what happened earlier in a session, chaining several steps together without being re-prompted. The team didn’t stop at reading the documentation. They ran the tools themselves.
Two numbers for every tool
Each tool got a documented score first, built from how clearly its own docs, pricing pages, and changelogs described a given capability. Zero meant the capability was never mentioned. Two meant the vendor stated it in plain language. Ten capabilities, two points each, twenty possible.
Then Decodo tested the highest-stakes claims by hand. A tool that claimed full marks for chaining tasks without re-prompting had to actually chain tasks without re-prompting. A crash or a dropped session cost a point. A claim that turned out to be staged, like a demo of a bug fix for a bug that didn’t exist, zeroed the whole category out.
Fellou’s documentation claimed a near-perfect score for multi-step tasks, 19 out of 20. In testing, it got stuck on a complex page layout, looped on the same action, and never got past a CAPTCHA, which took the multi-step score to zero and its total down to 17. Devin fared worse. Its docs promised full multi-step support, but across ten real tasks it finished two. Its own promotional demo had shown it fixing bugs in files that, as it turned out, didn’t exist in the referenced repository. That took its score from 13 down to 9.
Claude for Chrome and Amazon Nova Act came out on top, barely
Claude for Chrome and Amazon Nova Act tied for first with 18 out of 20. Nobody in the report hit 20. Where tools lost points tended to cluster in the same two places: completing a purchase, and running unsupervised in the background. Those are also the two places where a mistake is hardest to undo.
Cross-tab awareness split along a clean structural line. Browser extensions like Claude for Chrome, Chrome Gemini, and Brave Leo scored well, because they live inside a real browser window with real open tabs to look at. Cloud-based automation platforms scored badly, because a headless browser session run on a server has no tabs to be aware of in the first place. Nobody expects that to change soon; it’s built into how the two kinds of tools work.
Almost nothing claims to handle checkout
Transactional actions, meaning completing a purchase rather than just filling a basket, scored 0.43 out of 2 on average, the lowest of the ten. Six tools out of 45 claimed to do it fully: Amazon Nova Act, BrowserOS, ChatGPT Desktop, Minded, Sigma Browser, Skyvern.
OpenAI and Stripe built the Agentic Commerce Protocol, which already runs Instant Checkout inside ChatGPT. Google has its own competing cart standard, with Target, Walmart and Wayfair signed on as retail partners. Even with that infrastructure in place, most vendors would rather publish a claim about autofilling a form than a claim about clicking “pay.”
Gabriele Vitke, who leads product marketing at Decodo, doesn’t see that hesitation as a failure. “It isn’t a bad thing that some of the agentic tools aren’t capable of completing the purchase,” she said. “With the growing skepticism around the safety of AI tools, complaints about an agent purchasing the wrong product wouldn’t help this case. It’s better to improve the flows of the tools and only then roll out such features.”
Half the field can chain a task and never asks first
Thirty-five tools fully document multi-step task completion. Eighteen of them document no confirmation step before an irreversible action. More than half the field can run a task start to finish on its own. Fewer than half say anywhere in their documentation that they’ll stop and check with a person first.
Background tasks make this worse. Seventeen tools document the ability to run without an active session. Seven of those pair it with zero documented confirmation step: Airtop, Axiom AI, BrowserOS, Browserbase, HARPA AI, OpenHands, Opera Neon. An agent from any of those seven can act while the person who set it running is asleep or in a meeting, and nothing in the vendor’s own materials mentions a check-in point. Only two tools in the whole report, ChatGPT Desktop and Amazon Nova Act, document both a completed purchase and a pause-and-ask step.
Anthropic has published its own research on prompt injection against Claude for Chrome, showing how a webpage can carry hidden instructions that hijack an agent mid-task without the person watching ever noticing. OpenAI describes something similar in its write-up on hardening ChatGPT Atlas against prompt injection, where an attacker hides instructions inside an email to get an agent to forward private documents. Brave’s security team published a proof-of-concept attack against Perplexity’s Comet browser, where hidden text on a Reddit page made Comet navigate to Gmail on its own and pull a one-time password straight out of the inbox.
Picking a tool for the job you actually have
Decodo’s advice for businesses shopping for an agent is to stop looking at the overall score. A tool that scores well across all ten categories is optimized for breadth, and most companies only care about one or two of them.
Back-office data entry needs form filling and reliability under real conditions more than anything else on the list. Scheduled reporting needs background task support alongside confirmation steps, since an agent working unsupervised has to be able to flag a problem instead of failing quietly. Procurement needs transactional actions paired with confirmation before irreversible actions, and few tools manage both. Research spanning several tabs needs cross-tab awareness and multi-step completion, full stop.
Vitke put the risk in plain terms. “Someone on your team will still have to own what happens when the agent gets it wrong,” she said. “An agent buying the wrong inventory or emailing the wrong client won’t announce itself. And don’t assume the most capable-sounding tool is the cheapest one to deploy. A tool that fails and needs constant babysitting costs more in staff time than a narrower tool that does one thing reliably.”
Where this leaves things
The industry has worked out how to chain actions together and connect an agent to other tools. It hasn’t worked out when an agent should stop and ask a person first. Eighteen tools in this report can run a multi-step task entirely on their own, with no documented moment where they check in. The Agentic Commerce Protocol and Google’s cart standard are starting to standardize the payments side of agentic AI. Nothing equivalent exists yet for oversight, so each vendor is still deciding for itself what “safe enough” means.
The same question extends past browser agents, to any system built on live web data. An agent completing a task and a pipeline scraping a website at scale both depend on the infrastructure underneath them. Decodo builds part of that infrastructure itself, including its Web Scraping API, and argues that reliable access to web data now needs the same scrutiny as the model sitting on top of it.
Decodo’s full Agentic Capability Report, with the complete scoring table for all 45 tools, is on its website:
https://decodo.com/blog/agentic-capability-report