时地方
This commit is contained in:
@@ -0,0 +1,376 @@
|
||||
---
|
||||
title: "Hermes Has a Browser Escape Hatch Most Agents Don’t"
|
||||
source: "https://x.com/NeoAIForecast/status/2047521653954089113"
|
||||
author:
|
||||
- "[[@NeoAIForecast]]"
|
||||
published: 2026-04-24
|
||||
created: 2026-04-24
|
||||
description: "Most AI browser agents break when the page gets weird.A button hides inside an iframe. A native dialog blocks execution. A cookie issue ruin..."
|
||||
tags:
|
||||
- "clippings"
|
||||
---
|
||||

|
||||
|
||||
Most AI browser agents break when the page gets weird.
|
||||
|
||||
A button hides inside an iframe. A native dialog blocks execution. A cookie issue ruins the session. A JS-heavy app does not show the state you need.
|
||||
|
||||
Hermes has a technical answer for this: **browser\_cdp.**
|
||||
|
||||
It is not the normal browser tool. It is the escape hatch underneath it.
|
||||
|
||||
Most browser agents are built around high-level actions:
|
||||
|
||||
- navigate to a page
|
||||
- click a button
|
||||
- type into a field
|
||||
- scroll
|
||||
- read the page
|
||||
- take a screenshot
|
||||
|
||||
That works until it doesn’t.
|
||||
|
||||
Modern websites are messy. They have shadow DOMs, iframes, native dialogs, hidden state, client-side routing, weird auth flows, cookie problems, dynamic rendering, and JavaScript errors that never appear in the visible page.
|
||||
|
||||
Hermes Agent handles normal browser work with high-level tools like:
|
||||
|
||||
```plaintext
|
||||
browser_navigate
|
||||
browser_snapshot
|
||||
browser_click
|
||||
browser_type
|
||||
browser_console
|
||||
browser_vision
|
||||
```
|
||||
|
||||
But when those are not enough, Hermes exposes something much lower level:
|
||||
|
||||
```plaintext
|
||||
browser_cdp
|
||||
```
|
||||
|
||||
That stands for:
|
||||
|
||||
**Chrome DevTools Protocol.**
|
||||
|
||||
It lets Hermes send raw CDP commands directly to a connected Chrome instance.
|
||||
|
||||
In plain English:
|
||||
|
||||
Hermes can drop below “AI browser control” and talk to the browser like DevTools does. That is a serious power-user feature.
|
||||
|
||||
## Why high-level browser tools are not enough
|
||||
|
||||
High-level browser tools are great for most agent workflows.
|
||||
|
||||
They give the model a clean accessibility-tree view of the page. Hermes can see interactive elements, assign refs like [@e1](https://x.com/@e1) or [@e2](https://x.com/@e2), click buttons, fill forms, scroll, and inspect page text.
|
||||
|
||||
That is exactly what you want for normal browsing. But the abstraction can hide important details.
|
||||
|
||||
For example:
|
||||
|
||||
- the accessibility tree may not expose the element you need
|
||||
- an iframe may require target-specific inspection
|
||||
- a native JavaScript dialog may block the page
|
||||
- a frontend error may only appear in the console
|
||||
- a site may store critical state in cookies or local storage
|
||||
- a SPA may render data after network calls complete
|
||||
- visual state may differ from text state
|
||||
|
||||
Most agents either fail here or hallucinate around it.
|
||||
|
||||
Hermes gives the agent a lower-level path.
|
||||
|
||||
Takeaway: High-level browser tools are the default. **browser\_cdp** is what you use when the default abstraction leaks.
|
||||
|
||||
## What browser\_cdp actually does
|
||||
|
||||
browser\_cdp is a raw Chrome DevTools Protocol passthrough.
|
||||
|
||||
That means Hermes can call CDP methods such as:
|
||||
|
||||
```plaintext
|
||||
Target.getTargets
|
||||
Runtime.evaluate
|
||||
Page.handleJavaScriptDialog
|
||||
Network.getAllCookies
|
||||
DOM.*
|
||||
Emulation.*
|
||||
Storage.*
|
||||
Browser.*
|
||||
```
|
||||
|
||||
The official Hermes docs describe it as an escape hatch for browser operations not covered by normal browser tools.
|
||||
|
||||
Common uses include:
|
||||
|
||||
- native dialog handling
|
||||
- iframe-scoped evaluation
|
||||
- cookie inspection
|
||||
- network control
|
||||
- direct JavaScript evaluation
|
||||
- browser/tab target inspection
|
||||
- low-level DOM operations
|
||||
- debugging page state
|
||||
|
||||
Example:
|
||||
|
||||
```plaintext
|
||||
browser_cdp(method="Target.getTargets")
|
||||
```
|
||||
|
||||
That lists available browser targets/tabs.
|
||||
|
||||
Then a page-level command can target a specific tab:
|
||||
|
||||
```plaintext
|
||||
browser_cdp(
|
||||
method="Runtime.evaluate",
|
||||
params={
|
||||
"expression": "document.title",
|
||||
"returnByValue": true
|
||||
},
|
||||
target_id="<tabId>"
|
||||
)
|
||||
```
|
||||
|
||||
This is not “click the button.”
|
||||
|
||||
This is “ask Chrome directly what the page is doing.”
|
||||
|
||||
Takeaway: browser\_cdp turns Hermes from a browser user into a browser debugger.
|
||||
|
||||
## The setup is intentionally explicit
|
||||
|
||||
browser\_cdp is not always available.
|
||||
|
||||
Hermes only exposes it when a reachable CDP endpoint exists at session start.
|
||||
|
||||
The clean path is local Chrome via:
|
||||
|
||||
```plaintext
|
||||
/browser connect
|
||||
```
|
||||
|
||||
Inside the Hermes CLI, you can run:
|
||||
|
||||
```plaintext
|
||||
/browser connect
|
||||
/browser status
|
||||
/browser disconnect
|
||||
```
|
||||
|
||||
By default, Hermes connects to:
|
||||
|
||||
```plaintext
|
||||
ws://localhost:9222
|
||||
```
|
||||
|
||||
You can also provide a specific CDP endpoint:
|
||||
|
||||
```plaintext
|
||||
/browser connect ws://host:port
|
||||
```
|
||||
|
||||
If Chrome is not already running with remote debugging, Hermes can attempt to launch it.
|
||||
|
||||
The manual Linux path looks like this:
|
||||
|
||||
```bash
|
||||
google-chrome \
|
||||
--remote-debugging-port=9222 \
|
||||
--user-data-dir=$HOME/.hermes/chrome-debug \
|
||||
--no-first-run \
|
||||
--no-default-browser-check &
|
||||
```
|
||||
|
||||
Then launch Hermes from the terminal and run:
|
||||
|
||||
```plaintext
|
||||
/browser connect
|
||||
```
|
||||
|
||||
Important detail:
|
||||
|
||||
**/browser connect** is an interactive CLI slash command.
|
||||
|
||||
It is not dispatched by Telegram, Discord, WebUI, or other gateway chats. If you type it into a gateway chat, it will be treated like normal text instead of executing the slash command.
|
||||
|
||||
Takeaway: browser\_cdp is a CLI/operator feature. Start Hermes locally when you want raw browser control.
|
||||
|
||||
## The mental model: two browser layers
|
||||
|
||||
Hermes browser automation has two layers.
|
||||
|
||||
**Layer 1: Agent-friendly browser tools**
|
||||
|
||||
Use these for normal workflows:
|
||||
|
||||
```plaintext
|
||||
browser_navigate
|
||||
browser_snapshot
|
||||
browser_click
|
||||
browser_type
|
||||
browser_scroll
|
||||
browser_press
|
||||
browser_console
|
||||
browser_vision
|
||||
```
|
||||
|
||||
This layer is optimized for LLMs.
|
||||
|
||||
Hermes converts the page into a text-based accessibility tree. The agent sees refs like [@e3](https://x.com/@e3), clicks them, types into fields, and verifies results.
|
||||
|
||||
This is the default path.
|
||||
|
||||
**Layer 2: Raw CDP passthrough**
|
||||
|
||||
Use this when the first layer cannot see or control what you need:
|
||||
|
||||
**browser\_cdp**
|
||||
|
||||
This layer is optimized for power users. It exposes the Chrome DevTools Protocol directly.
|
||||
|
||||
The agent can inspect targets, evaluate JavaScript, read cookies, handle dialogs, and interact with browser internals.
|
||||
|
||||
Takeaway: Hermes does not force every browser task through one abstraction. It gives you a clean path and a low-level fallback.
|
||||
|
||||

|
||||
|
||||
## When I would use browser\_cdp
|
||||
|
||||
**Good use cases**
|
||||
|
||||
1\. Native JavaScript dialogs
|
||||
|
||||
If a site opens a blocking alert, confirm, or prompt, the normal click/type loop may get stuck.
|
||||
|
||||
CDP can handle it directly:
|
||||
|
||||
```plaintext
|
||||
Page.handleJavaScriptDialog
|
||||
```
|
||||
|
||||
2\. Debugging frontend state
|
||||
|
||||
If the page looks wrong, ask the browser directly:
|
||||
|
||||
```plaintext
|
||||
Runtime.evaluate
|
||||
```
|
||||
|
||||
Example expression:
|
||||
|
||||
```plaintext
|
||||
document.title
|
||||
```
|
||||
|
||||
Or:
|
||||
|
||||
```plaintext
|
||||
window.location.href
|
||||
```
|
||||
|
||||
Or app-specific state if exposed globally.
|
||||
|
||||
3\. Cookie/session inspection
|
||||
|
||||
For auth issues, CDP can inspect cookies:
|
||||
|
||||
```plaintext
|
||||
Network.getAllCookies
|
||||
```
|
||||
|
||||
That is much more precise than guessing whether a login “worked.”
|
||||
|
||||
4\. Iframe problems
|
||||
|
||||
Some pages hide the useful interaction inside nested frames.
|
||||
|
||||
CDP gives the agent a way to inspect targets and work closer to the browser’s real structure.
|
||||
|
||||
5\. Frontend QA
|
||||
|
||||
For local web-app testing, Hermes can combine:
|
||||
|
||||
```plaintext
|
||||
browser_console
|
||||
browser_snapshot
|
||||
browser_cdp
|
||||
```
|
||||
|
||||
That means it can see the page, check the console, and query browser internals.
|
||||
|
||||
Takeaway: Use browser\_cdp when the browser state matters more than the visible page text.
|
||||
|
||||
## The workflow I’d teach operators
|
||||
|
||||
Here is the practical Hermes browser-debug workflow.
|
||||
|
||||
Step 1: Start Chrome with CDP
|
||||
|
||||
```bash
|
||||
google-chrome \
|
||||
--remote-debugging-port=9222 \
|
||||
--user-data-dir=$HOME/.hermes/chrome-debug \
|
||||
--no-first-run \
|
||||
--no-default-browser-check &
|
||||
```
|
||||
|
||||
Step 2: Open Hermes locally
|
||||
|
||||
```plaintext
|
||||
hermes
|
||||
```
|
||||
|
||||
Step 3: Connect the browser
|
||||
|
||||
```plaintext
|
||||
/browser connect
|
||||
```
|
||||
|
||||
Step 4: Verify the connection
|
||||
|
||||
```plaintext
|
||||
/browser status
|
||||
```
|
||||
|
||||
Step 5: Use normal browser tools first
|
||||
|
||||
Ask Hermes to navigate, inspect, click, type, and read the page normally.
|
||||
|
||||
Example:
|
||||
|
||||
```plaintext
|
||||
Open my local app at http://localhost:3000, inspect the login flow, and check for JS errors.
|
||||
```
|
||||
|
||||
Step 6: Drop to CDP only when needed
|
||||
|
||||
If the normal browser tools cannot reach the right state, ask Hermes to use browser\_cdp.
|
||||
|
||||
Example:
|
||||
|
||||
```plaintext
|
||||
Use browser_cdp to list targets, identify the active tab, and evaluate document.title on that tab.
|
||||
```
|
||||
|
||||
Step 7: Disconnect when done
|
||||
|
||||
```plaintext
|
||||
/browser disconnect
|
||||
```
|
||||
|
||||
Takeaway: Treat CDP as the escalation path, not the first move.
|
||||
|
||||
## Final takeaway
|
||||
|
||||
Most browser agents give you a steering wheel. Hermes also gives you access to the diagnostic port.
|
||||
|
||||
That is the difference.
|
||||
|
||||
For normal tasks, use the clean high-level browser tools. But when the page gets weird, Hermes can drop into Chrome DevTools Protocol and debug the browser directly.
|
||||
|
||||
That is why browser\_cdp is one of the most technical and underrated Hermes features.
|
||||
|
||||
It is not flashy, It is better than flashy, It is the escape hatch you want when real web automation breaks.
|
||||
Reference in New Issue
Block a user