> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rebind.gg/llms.txt
> Use this file to discover all available pages before exploring further.

# rb.ai

> Ask a model to pick a choice, find something on the screen, or read text from it.

`rb.ai` asks a model about the screen: `decide` picks one of the script's
choices, `find` locates what a description names, and `read` returns the text
it names. It is part of [Rebind Pro](https://docs.rebind.gg/account/pro.md): the first call from a
signed-in account without Pro starts its free 7-day trial, and each call's cost
counts against the month's AI allowance, then the account's credits. A
marketplace listing that uses it is
[marked Pro](https://docs.rebind.gg/reference/packages.md#the-marketplace-listing).

**Needs:** the `ai` [permission](https://docs.rebind.gg/reference/sdk/modeline.md#permissions), or
`ai:decide`, `ai:find` or `ai:read` for one call, in a script that declares
permissions (a script with no `permission=` line may call all three), plus
`screen` or [`screen:target`](https://docs.rebind.gg/reference/sdk/modeline#scoped-permissions) for
every call that looks at the screen; a coroutine for every call, so call
`decide`, `find` and `read` inside `rb.run` or `rb.async` (see
[Where code runs](https://docs.rebind.gg/reference/sdk/globals.md#where-code-runs)); `min_sdk=3.10.0`.

This script moves the pointer to the Store button when F7 is pressed, then asks
whether a dialog is open:

```luau theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
-- rebind: min_sdk=3.10.0 name=example version=1.0.0
rb.bind("F7", rb.async(function()
  local store = rb.ai.find("the Store button")
  if store then
    rb.hid.move_to(store.x, store.y)
  end
  local answer, confidence = rb.ai.decide("Is a dialog open?", { screen = true, choices = { "yes", "no" } })
  rb.log.info(answer, confidence)
end))
```

| Signature | Returns |
| - | - |
| `rb.ai.decide(question: string, opts: AiDecideOptions) -> (string, number, AiDetail)` | one of `opts.choices`, the model's confidence from 0 to 1, and [the whole answer](#the-whole-answer); the third value `min_sdk` 3.22.0 |
| `rb.ai.find(what: string, opts: AiFindOptions?) -> AiPoint?` | `{ x, y }`, the center of what is described, in screen coordinates, or `nil` when it is not visible |
| `rb.ai.read(what: string, opts: AiReadOptions?) -> string` | the described text as it appears, or `""` |

What a call looks at:

* `window`: a window title (on macOS, a title or app name). The call reads the
  screen inside that window's rectangle, including anything drawn over it. No
  match, or a minimized or off-screen window, raises.
* Without `window`, `find` and `read` look at the display under the cursor, or at
  the target window when the script declares `screen:target`. `decide` looks only
  with `screen = true`, a `window` or a `region`; otherwise it answers from the
  question and `state`.
* `at` (for `read`, and for `decide` together with `screen = true` or a
  `window`): a point, such as one `find` returned. The call reads a 400 by 160
  box centered on it.
* `region` (for `decide`; `min_sdk` 3.22.0): a rectangle `{ x, y, w, h }` in
  screen coordinates. The call reads only that rectangle, so small detail such
  as a health bar stays legible. A rectangle is shrunk only when it is over
  1,024 pixels of the display on its long edge (512 points on a 2x display).
  Pass `at` or `region`, not both.

`decide` takes 2 to 32 distinct `choices` of 1 to 200 characters and a question
of up to 4,000; `find` and `read` take a description of up to 1,000 characters.

No call is retried. A call ends by itself after about 10 seconds with the
timeout error in the list below.

Errors:

* `rb.ai needs Rebind Pro: AI is part of Rebind Pro. Sign in to start your free 7-day trial.`
  when signed out, or
  `rb.ai needs Rebind Pro: AI is part of Rebind Pro, and this account's trial has been used.`
* `rb.ai: This month's AI allowance is used up. Add credits to keep going; it resets on the 1st.`
* `rb.ai: the model took too long; try again`
* `rb.ai.find: needs the 'ai' permission: add -- rebind: permission=ai`
* `rb.ai.find waits for an answer: call it inside rb.run` (with the call's
  name), for a call made outside a coroutine.

## Facts and choices

**Needs:** `min_sdk=3.22.0`.

`decide` takes what the script already knows apart from the question:

* `state`: the facts, as a string or a table, which is sent as JSON. An empty
  string or table raises. Keep the question for the rule and put the facts
  here. With `state` and no look, the call sends no screenshot and needs no
  `screen` permission. A value JSON cannot hold raises with its key, such as
  `rb.ai.decide: opts.state.target.on_hit cannot be sent: it is a function`.
* `descriptions`: a table from a choice to what it means, up to 300 characters
  each. Every key must be one of `choices`.

Each call sends the question or description, the choices, `state` and
`descriptions` when given, and the screenshot when the call looks. Nothing else
from the script is sent.

## The whole answer

**Needs:** `min_sdk=3.22.0`.

`decide` returns a third value, a table:

| Field | Holds |
| - | - |
| `probabilities` | each choice's probability, by choice; `nil` when the answer came without them |
| `usage` | `input_tokens`, `cost_usd` (what the call counted against the AI allowance), `request_id`, `upstream_ms` and `model`; `nil` when the provider reported no cost |
| `ms` | milliseconds the call took from this computer |

To act only on a sure answer, compare `detail.probabilities[choice]` with a
threshold you have checked against your own cases. `confidence` is not the
picked choice's probability.

This script asks which kind of text is on the clipboard, passing it as
`state`, and logs the choice with its probability and the call's cost:

```luau theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
-- rebind: min_sdk=3.22.0 name=example version=1.0.0
rb.bind("F7", rb.async(function()
  local kind, _, detail = rb.ai.decide("Which kind of text is this?", {
    choices = { "error message", "address", "other" },
    state = { text = rb.clipboard.get() or "" },
  })
  rb.log.info(kind, detail.probabilities and detail.probabilities[kind], detail.usage and detail.usage.cost_usd)
end))
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.