> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rebind.gg/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Recipes

> Scripts that find, read, and judge what is on screen with rb.ai, and report back with rb.phone.

`rb.ai` lets a script ask a model about the screen: where something is
(`rb.ai.find`), what some text says (`rb.ai.read`), or which of several answers
fits (`rb.ai.decide`). Use it for things you can describe in words but cannot
pin down as a pixel color or an image.

Before you start:

* **Rebind Pro.** `rb.ai` and `rb.phone` are part of [Rebind Pro](/account/pro).
  Calls go through the account signed in to Rebind, and the first call from an
  account without Pro starts its free trial. Each call's cost counts against
  the monthly AI allowance, so call on a hotkey or a slow interval, never on
  every tick.
* **A coroutine.** Each call waits for the model, so it must run inside
  `rb.run` or a function wrapped in `rb.async`. Outside one, it raises an error.
* **The `ai` permission.** A script that declares permissions lists `ai`, or
  only the calls it uses: `ai:find`, `ai:read`, `ai:decide`. That needs
  `min_sdk=3.9.0` or later. See [Permissions](/concepts/permissions).
* **The screen.** `find` and `read` always send a screenshot, and `decide` sends
  one when you pass `screen = true` or a `window`. Screen reads work on Windows
  and macOS; see [Platforms](/concepts/platforms).

Every option, return value, and error message is in the
[SDK reference](/reference/sdk-reference), under **Rebind Pro: rb.ai and
rb.phone**.

## Click something you describe

Press a hotkey and the script finds the button by its description, moves the
cursor onto it, and clicks. `rb.ai.find` returns the centre of what you
described in screen coordinates, or `nil` when it is not on screen.

```luau theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
--[[
  rebind: min_sdk=3.9.0
  rebind: name=Click Described Button
  rebind: version=1.0.0
  rebind: permissions=ai:find
]]
local cfg = rb.ui.schema({
  hotkey = rb.ui.keybind("F8", { label = "Find and click" }),
  target = rb.ui.text("the blue Continue button", { label = "What to click", max_length = 200 }),
})

rb.bind(cfg.hotkey, rb.async(function()
  local point = rb.ai.find(cfg.target)
  if not point then
    rb.ui.notify("Not on screen: " .. cfg.target, "warning")
    return
  end
  rb.hid.move_to(point.x, point.y)
  rb.hid.press("Mouse1", 30)
end))
```

Without a `window` option, `find` looks at the display under the cursor. Pass
`{ window = "Title" }` to look inside one window instead. Describe the target
the way you would to a person: its label, color, and where it sits.

## Read a value off the screen

`rb.ai.read` returns the text you describe, exactly as it appears. When the text
is not on screen it returns an empty string, so check for `""` before using
the result.

```luau theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
--[[
  rebind: min_sdk=3.9.0
  rebind: name=Read Order Total
  rebind: version=1.0.0
  rebind: permissions=ai:read
]]
local cfg = rb.ui.schema({
  hotkey = rb.ui.keybind("F9", { label = "Read and copy" }),
  window = rb.ui.text("Checkout", { label = "Window title", max_length = 100 }),
})

rb.bind(cfg.hotkey, rb.async(function()
  local total = rb.ai.read("the order total", { window = cfg.window })
  if total == "" then
    rb.ui.notify("No order total in that window", "warning")
    return
  end
  rb.clipboard.set(total)
  rb.ui.notify("Copied " .. total, "success")
end))
```

A `window` names the window by title (on macOS, by title or app name), and the
call reads the pixels inside that window's rectangle. If no window matches, the
call raises an error naming the title.

## Read next to something you found

`find` and `read` combine: find a label, then read the value printed beside it.
The `at` option reads a small box centred on a point.

```luau theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
--[[
  rebind: min_sdk=3.9.0
  rebind: name=Read Beside Label
  rebind: version=1.0.0
  rebind: permissions=ai:find,ai:read
]]

rb.bind("F10", rb.async(function()
  local label = rb.ai.find("the Balance label")
  if not label then
    rb.log.warn("Balance label is not on screen")
    return
  end
  local value = rb.ai.read("the amount next to Balance", { at = label })
  rb.log.info("balance: " .. (if value == "" then "not found" else value))
end))
```

## Wait for something to happen

`rb.ai.decide` picks one of the choices you give it and returns the model's
confidence from 0 to 1. This script checks the screen once a minute and stops
when it is confident a long job has finished, then tells you.

```luau theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
--[[
  rebind: min_sdk=3.9.0
  rebind: name=Job Watcher
  rebind: version=1.0.0
  rebind: permissions=ai:decide
]]
local cfg = rb.ui.schema({
  question = rb.ui.text("Has the export finished?", { label = "Question", max_length = 300 }),
  every_s = rb.ui.slider(60, { min = 15, max = 600, suffix = "s", label = "Check every" }),
  confidence = rb.ui.slider(80, { min = 50, max = 99, suffix = "%", label = "Confidence needed" }),
})

rb.on("start", function()
  rb.run(function()
    while true do
      local answer, confidence = rb.ai.decide(cfg.question, {
        choices = { "yes", "no" },
        screen = true,
      })
      if answer == "yes" and confidence >= cfg.confidence / 100 then
        local sent = rb.phone.notify("Done: " .. cfg.question, "Job Watcher")
        if not sent.delivered then
          rb.ui.notify("Done: " .. cfg.question, "success")
        end
        return
      end
      rb.sleep(cfg.every_s * 1000)
    end
  end)
end)
```

Without `screen = true` or a `window`, `decide` answers from the question alone
and sends no screenshot. `rb.phone.notify` returns `{ delivered, reason }`.
There is no phone app yet, so it always returns `delivered = false` with the
reason `"no phone connected"`; the script falls back to a notification in
Rebind.

## Keep looks inside one app

A script targeted at an application can promise to read only that
application's window. Add `screen:target` to its permissions along with a
`process=` or `window=` target. Then `find` and `read` without a `window` look
at the target window instead of the whole display, and Rebind's install screen
shows the narrower grant.

```luau theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
--[[
  rebind: min_sdk=3.9.0
  rebind: name=Editor Error Reader
  rebind: version=1.0.0
  rebind: process=Code.exe
  rebind: permissions=ai:read,screen:target
]]

rb.bind("F11", rb.async(function()
  local message = rb.ai.read("the first error message in the Problems panel")
  if message ~= "" then
    rb.clipboard.set(message)
    rb.ui.notify("Copied the error", "success")
  end
end))
```

`screen:none` blocks every look, so a script with it can still use `decide`
without a screenshot but not `find` or `read`. Scoped permissions are explained
in [Permissions](/concepts/permissions).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.