Show HN: Libretto – Making AI browser automations deterministic

"Claude, reverse engineer the APIs of this website and build a client. Use Dev Tools."

I have succeed 8/8 websites with this.

Sites like Booking.com, Hotels.com, try to identify real humans with their AWS solution and Cloudflare, but you can just solve the captcha yourself, login and the session is in disguishable from a human. Playwright is detected and often blocked.

The playwright codegen tool exists, but the script it generates is super simple and it can't handle loops or data extraction.

So for libretto we often use a mix of instructions + recording my actions for the agent. Makes the process faster than just relying on a description and waiting for the agent to figure out the whole flow

Our healthcare workflows are sensitive so if there happens to be some website refactor or something, we want to be kept in the loop and not have AI just try and figure it out.

At the end of the day this all lives as code though, so if you you needed to use the DOM and wanted a workflow that would naturally handle any handle DOM change without escalating to you, then you could just throw a computer use agent at it.

1. Noticed that the API was a couple seconds faster than spinning up the coding agent

2. Spinning up a separate agent you can't guarantee its behavior, and we wanted to enforce that only a single LLM call was run to read the snapshot and analyze the selector. You can guarantee this with an API call but not with a local coding agent

Right now we kind of have a mixture of the 2 approaches, but there's a large room for improvement.

- When libretto performs the code generation it initially inspects the page and sends the network calls/playwright actions using `snapshot` and `exec` tools to test them individually. After it's tested all of individual selectors and thinks it's finished, it creates a script and then runs the script from scratch. Oftentimes the generated script will fail, and that will trigger libretto to identify the failure and update the code and repeat this process until the script works. That iteration process helps make the scripts much more reliable.

- The way our `snapshot` command works is that we send a screenshot + DOM (depending on size may be condensed) to a separate LLM and ask it to figure out the relevant selectors. We do this to not pollute context of main agent with the DOM + lots of screenshots. As a part of that analyzers prompt we tell it to prefer selectors using: data-testid, data-test, aria-label, name, id, role. This just lives in the analyzer prompt and is not deterministic though. It'd be interesting to see if we can improve script quality if we add a hard constraint on the selectors or with different prompting.

I'm also curious if you have any guidance for prompt improvements we can give the snapshot analyzer LLM to help it pick more robust selectors right off the bat.

We can update the config though to allow you to set up snapshot through the CLI instead of going through the API!

- We use runtime agents in very specific places. For example on Availity they frequently have popups right after you login, so if there's a failure right after signup we spin up an agent to close it and then resume the flow with basically a try/catch

- We wait for it to fail and then tell the agent to look at the error logs and use `libretto run` command to rerun the workflow and fix the error

We're thinking of extending libretto to handle these better though. Some of our ideas:

- Adding a global/custom fallback steps to every page action. This way we could for example add our popup handler error recovery to all page actions or some subset of them

- Having a hosted version which flags errors and has a coding agent examine the issue and open a PR with the fix

Curious if you have any other ideas!

- Libretto prefers network requests over DOM interaction when possible, so this will circumvent a lot of complex JS rendering issues

- When you do need the DOM, playwright can handle a lot of the complexity out of the box: playwright will re-query the live DOM at action time and automatically wait for elements to populate. Libretto is also set up to pick selectors like data-testid, aria-label, role, id over class names or positional stuff that's likely to be dynamic.

- At the end of the day the files still live as code so you could always just throw a browser agent at it to handle a part of a workflow if nothing else works

The implementation is also pretty different:

- libretto gives your agent a single exec tool (instead of different tools for each action) so it can write arbitrary playwright/javascript and is more context efficient

- Also we gave libretto instructions on bot detection avoidance so that it will prefer using network requests for automation (something that other tools don’t support), but will fall back to playwright if it identifies network requests as too risky

libretto gives a similar ability for agents for building scripts but:

- agents automatically run, debug, and test the integrations they write - they have a much better understanding of the semantics of the actions you take (vs. playwright auto-assuming based on where you clicked) - they can parse network requests and use those to make direct API calls instead

there's fundamentally a mismatch where playwright-cli is for building e2e test scripts for your own app but libretto is for building robust web automations

// Let AI click await stagehand.act("click on the comments link for the top story");

the issue with this is that there's now runtime non-determinism. We move the AI work during dev-time: AI explores and crawls the website first, and generates a deterministic legible script.

Tangentially, Stagehand's model may have worked 2 years ago when humans still wrote the code, but it's no longer the case. We want to empower agents to do the heavy lifting of building a browser automation for us but reap the benefits of running deterministic, fast, cheap, straightforward code.

For more complex cases where libretto can't validate that the network approach would produce the right data (like sites that rely on WebSockets or heavy client-side logic) it falls back to using the DOM with playwright