"If you see a CAPTCHA or similar test, just wait for it to get solved automatically." That works, but the model decides for itself how long to wait and how to tell a solve is still running, from a screenshot alone.
Captcha telemetry gives you the signal directly. This cookbook wires it into a @onkernel/browser-loop agent so that browser actions are held while a solve is outstanding, and the agent is told the outcome in terms it can act on.
The design follows one split: telemetry decides what to say, while the live page decides whether to interrupt the agent at all. A solver task succeeding and a challenge clearing are different facts, so the gate never reports one as the other.
What the events tell you
Three rules from Correlate captcha tasks and challenges shape everything below:
task_idis the only join. Pair a start with its result on it.challenge_idgroups tasks for one visible challenge and is present only when Kernel tracked the widget — tasks without one can never be joined to acaptcha_challenge_result, even when one is emitted for the same page.- Delivery is best-effort and unordered. A start can arrive after its result, and any event can be absent. Nothing may depend on arrival order, and every wait needs a deadline.
- Fall back to the page. When you need a challenge-level outcome and don’t have one, use the available task results and the current page state.
Setup
@earendil-works/pi-agent-core ships ESM only. Without type: module in package.json, tsx resolves captcha-gate.ts’s import of it as CommonJS and fails immediately with ERR_PACKAGE_PATH_NOT_EXPORTED.
Every run needs a KERNEL_API_KEY and a provider key for whichever model LOOP_MODEL points at (anthropic:claude-sonnet-5 by default, so ANTHROPIC_API_KEY):
Build the gate
The four snippets below are one file,captcha-gate.ts, in order.
1
Read the events
Three maps, one per thing the telemetry can tell you, all keyed so nothing depends on arrival order.
joinable is the important one: it holds only the challenge_ids that actually appeared on a task event, which are the only ones a challenge result may be attributed to.A task record created by a result already has a status, so a captcha_solve_started that arrives afterwards finds a closed task and leaves it closed. That only works while the task is still in tasks, though — once reset has cleared it, a late start for the same task_id would otherwise look like a brand-new one. settledAt remembers when reset cleared each ID, so a late arrival within TASK_SETTLE_MS of that is dropped instead of reopening an episode the agent has already been told about. Past that window the same ID is treated as fresh again, since Kernel can reuse a challenge_id when an episode continues across a page reload.The stream’s AbortController is saved so the gate can close it later — the loop runs detached from the caller, and leaving it open past the browser’s deletion keeps the process alive waiting on a connection that will never produce another event.captcha-gate.ts
2
Decide when to hold
A task counts as open while it has no terminal status and is still inside its deadline — that deadline is what stops a missing
captcha_solve_result from holding the agent forever. until is the only waiting primitive, and it always takes a timeout.captcha-gate.ts
3
Resolve an outcome
resolve settles what it can, waits a bounded interval for a challenge-level result only when a task actually carried a challenge_id, then reads the page and picks the best available source: an attributable challenge result, then any challenge result (labelled as a page observation rather than a join), then task results, then the page alone.holding and pending are separate on purpose. pending covers a terminal outcome that lands while no action is in flight — without it, a challenge result arriving between tool calls is recorded and never told to anyone. It has to catch every settled task, not just failures: a successful task-only result (Turnstile’s usual path) and a task whose deadline lapsed with no result at all are both settled outcomes the agent needs to hear about.reset only clears a task once something actually covers it — a terminal status of its own, or a challenge result for the episode it belongs to. A task the solver hasn’t finished with survives, so holding stays true and the next tool call keeps waiting on it instead of losing track of a solve that’s still in progress.captcha-gate.ts
4
Wire it to the agent
harness.on("tool_call", …) is awaited before the tool is dispatched, so returning from it late holds the action and returning { block: true, reason } replaces it with a message the model reads. That is cheaper than aborting the turn and it stops the action before it reaches the page.The page probe is a read-only Playwright snippet. Its result is what decides whether the agent is interrupted at all — a cleared page just resumes with no extra turn. That decision checks tokenPresent, not just widgets: a solved reCAPTCHA or hCaptcha widget commonly stays rendered on the page, so a response token is the signal that the challenge is actually behind the agent, not the widget disappearing.gate.close() runs before the browser is deleted, so the detached telemetry loop from step one gets torn down instead of hanging on a connection to a session that no longer exists.The gate only speaks up when telemetry gives it something to report. Since any event can be absent, the system prompt keeps a fallback instruction for the case where nothing ever arrives — the model still knows to wait out a captcha it can see but the gate never heard about.captcha-gate.ts
The complete captcha-gate.ts
The complete captcha-gate.ts
captcha-gate.ts
What the agent is told
Every verdict pairs a telemetry claim with the page state it was checked against:
Durations come straight from each event’s
duration_ms, which is authoritative; don’t compute them from event timestamps.
Limits
TASK_SETTLE_MSandCHALLENGE_GRACE_MSare the safety net. Events can be absent, so both waits are bounded and the gate falls through to the page rather than stalling. RaiseTASK_SETTLE_MSif your solves routinely run longer than 20s.- Challenge results aren’t emitted for every widget type. Turnstile, for instance, reports task events only, so the task-plus-page path is the one that runs there.
- One episode at a time. The gate resets after each verdict; overlapping visible challenges from the same provider aren’t split apart, and the telemetry docs note Kernel’s own event model can’t always attribute a result in that case either.
- The page probe is best-effort. It reads the DOM for known widget markers and a response token. A site that renders its challenge somewhere those selectors miss won’t be held past the logged message alone.
- Press-and-hold has no selector at all. It emits
captcha_challenge_resulttelemetry like reCAPTCHA and hCaptcha, butPROBE’sgroupshas no entry for it, sowidgetsnever reports one — the tool-call handler treats that as clear and lets the agent proceed even while the challenge is still open. Add a selector for your target’s specific implementation if you need it to actually hold. - A reload within
TASK_SETTLE_MSof the last verdict can still be missed.settledAttreats an ID as fresh again once that window passes, so a genuinely new outcome for a reusedchallenge_ideventually gets through — but if the reload happens sooner than that, its result is indistinguishable from a stale duplicate and gets dropped.
Next steps
- Telemetry Categories — the full captcha event schema and correlation rules
- Stream Telemetry — resuming a dropped stream, filtering by category
- Stealth mode — what the automatic captcha solver covers
- Playwright with Computer Use Fallback — holding and redirecting an agent mid-run for a different reason