rb.ai

rb.ai asks a model about the screen: decide picks one of the script’s choices, find locates what a description names, and read returns the text it names. It is part of Rebind Pro: the first call from a signed-in account without Pro starts its free 7-day trial, and each call’s cost counts against the month’s AI allowance, then the account’s credits. A marketplace listing that uses it is marked Pro. Needs: the ai permission, or ai:decide, ai:find or ai:read for one call, in a script that declares permissions (a script with no permission= line may call all three), plus screen or screen:target for every call that looks at the screen; a coroutine for every call, so call decide, find and read inside rb.run or rb.async (see Where code runs); min_sdk=3.10.0. This script moves the pointer to the Store button when F7 is pressed, then asks whether a dialog is open:

-- rebind: min_sdk=3.10.0 name=example version=1.0.0
rb.bind("F7", rb.async(function()
  local store = rb.ai.find("the Store button")
  if store then
    rb.hid.move_to(store.x, store.y)
  end
  local answer, confidence = rb.ai.decide("Is a dialog open?", { screen = true, choices = { "yes", "no" } })
  rb.log.info(answer, confidence)
end))

Signature	Returns
`rb.ai.decide(question: string, opts: AiDecideOptions) -> (string, number, AiDetail)`	one of `opts.choices`, the model’s confidence from 0 to 1, and the whole answer; the third value `min_sdk` 3.22.0
`rb.ai.find(what: string, opts: AiFindOptions?) -> AiPoint?`	`{ x, y }`, the center of what is described, in screen coordinates, or `nil` when it is not visible
`rb.ai.read(what: string, opts: AiReadOptions?) -> string`	the described text as it appears, or `""`

What a call looks at:

window: a window title (on macOS, a title or app name). The call reads the screen inside that window’s rectangle, including anything drawn over it. No match, or a minimized or off-screen window, raises.
Without window, find and read look at the display under the cursor, or at the target window when the script declares screen:target. decide looks only with screen = true, a window or a region; otherwise it answers from the question and state.
at (for read, and for decide together with screen = true or a window): a point, such as one find returned. The call reads a 400 by 160 box centered on it.
region (for decide; min_sdk 3.22.0): a rectangle { x, y, w, h } in screen coordinates. The call reads only that rectangle, so small detail such as a health bar stays legible. A rectangle is shrunk only when it is over 1,024 pixels of the display on its long edge (512 points on a 2x display). Pass at or region, not both.

decide takes 2 to 32 distinct choices of 1 to 200 characters and a question of up to 4,000; find and read take a description of up to 1,000 characters. No call is retried. A call ends by itself after about 10 seconds with the timeout error in the list below. Errors:

rb.ai needs Rebind Pro: AI is part of Rebind Pro. Sign in to start your free 7-day trial. when signed out, or rb.ai needs Rebind Pro: AI is part of Rebind Pro, and this account's trial has been used.
rb.ai: This month's AI allowance is used up. Add credits to keep going; it resets on the 1st.
rb.ai: the model took too long; try again
rb.ai.find: needs the 'ai' permission: add -- rebind: permission=ai
rb.ai.find waits for an answer: call it inside rb.run (with the call’s name), for a call made outside a coroutine.

Facts and choices

Needs: min_sdk=3.22.0. decide takes what the script already knows apart from the question:

state: the facts, as a string or a table, which is sent as JSON. An empty string or table raises. Keep the question for the rule and put the facts here. With state and no look, the call sends no screenshot and needs no screen permission. A value JSON cannot hold raises with its key, such as rb.ai.decide: opts.state.target.on_hit cannot be sent: it is a function.
descriptions: a table from a choice to what it means, up to 300 characters each. Every key must be one of choices.

Each call sends the question or description, the choices, state and descriptions when given, and the screenshot when the call looks. Nothing else from the script is sent.

The whole answer

Needs: min_sdk=3.22.0. decide returns a third value, a table:

Field	Holds
`probabilities`	each choice’s probability, by choice; `nil` when the answer came without them
`usage`	`input_tokens`, `cost_usd` (what the call counted against the AI allowance), `request_id`, `upstream_ms` and `model`; `nil` when the provider reported no cost
`ms`	milliseconds the call took from this computer

To act only on a sure answer, compare detail.probabilities[choice] with a threshold you have checked against your own cases. confidence is not the picked choice’s probability. This script asks which kind of text is on the clipboard, passing it as state, and logs the choice with its probability and the call’s cost:

-- rebind: min_sdk=3.22.0 name=example version=1.0.0
rb.bind("F7", rb.async(function()
  local kind, _, detail = rb.ai.decide("Which kind of text is this?", {
    choices = { "error message", "address", "other" },
    state = { text = rb.clipboard.get() or "" },
  })
  rb.log.info(kind, detail.probabilities and detail.probabilities[kind], detail.usage and detail.usage.cost_usd)
end))

Overview

Script basics

Input and output

Timing and script control

Panels, overlays and dialogs

Screen, windows and apps

Operating system

Files and data

Network and pipes

Rebind Pro

Lookup

Facts and choices

The whole answer