AI Recipes

rb.ai lets a script ask a model about the screen: where something is (rb.ai.find), what some text says (rb.ai.read), or which of several answers fits (rb.ai.decide). Use it for things you can describe in words but cannot pin down as a pixel color or an image. Before you start:

Rebind Pro. rb.ai and rb.phone are part of Rebind Pro. Calls go through the account signed in to Rebind, and the first call from an account without Pro starts its free trial. Each call’s cost counts against the monthly AI allowance, so call on a hotkey or a slow interval, never on every tick.
A coroutine. Each call waits for the model, so it must run inside rb.run or a function wrapped in rb.async. Outside one, it raises an error.
The ai permission. A script that declares permissions lists ai, or only the calls it uses: ai:find, ai:read, ai:decide. That needs min_sdk=3.9.0 or later. See Permissions.
The screen. find and read always send a screenshot, and decide sends one when you pass screen = true or a window. Screen reads work on Windows and macOS; see Platforms.

Every option, return value, and error message is in the SDK reference, under Rebind Pro: rb.ai and rb.phone.

Click something you describe

Press a hotkey and the script finds the button by its description, moves the cursor onto it, and clicks. rb.ai.find returns the centre of what you described in screen coordinates, or nil when it is not on screen.

--[[
  rebind: min_sdk=3.9.0
  rebind: name=Click Described Button
  rebind: version=1.0.0
  rebind: permissions=ai:find
]]
local cfg = rb.ui.schema({
  hotkey = rb.ui.keybind("F8", { label = "Find and click" }),
  target = rb.ui.text("the blue Continue button", { label = "What to click", max_length = 200 }),
})

rb.bind(cfg.hotkey, rb.async(function()
  local point = rb.ai.find(cfg.target)
  if not point then
    rb.ui.notify("Not on screen: " .. cfg.target, "warning")
    return
  end
  rb.hid.move_to(point.x, point.y)
  rb.hid.press("Mouse1", 30)
end))

Without a window option, find looks at the display under the cursor. Pass { window = "Title" } to look inside one window instead. Describe the target the way you would to a person: its label, color, and where it sits.

Read a value off the screen

rb.ai.read returns the text you describe, exactly as it appears. When the text is not on screen it returns an empty string, so check for "" before using the result.

--[[
  rebind: min_sdk=3.9.0
  rebind: name=Read Order Total
  rebind: version=1.0.0
  rebind: permissions=ai:read
]]
local cfg = rb.ui.schema({
  hotkey = rb.ui.keybind("F9", { label = "Read and copy" }),
  window = rb.ui.text("Checkout", { label = "Window title", max_length = 100 }),
})

rb.bind(cfg.hotkey, rb.async(function()
  local total = rb.ai.read("the order total", { window = cfg.window })
  if total == "" then
    rb.ui.notify("No order total in that window", "warning")
    return
  end
  rb.clipboard.set(total)
  rb.ui.notify("Copied " .. total, "success")
end))

A window names the window by title (on macOS, by title or app name), and the call reads the pixels inside that window’s rectangle. If no window matches, the call raises an error naming the title.

Wait for something to happen

rb.ai.decide picks one of the choices you give it and returns the model’s confidence from 0 to 1. This script checks the screen once a minute and stops when it is confident a long job has finished, then tells you.

--[[
  rebind: min_sdk=3.9.0
  rebind: name=Job Watcher
  rebind: version=1.0.0
  rebind: permissions=ai:decide
]]
local cfg = rb.ui.schema({
  question = rb.ui.text("Has the export finished?", { label = "Question", max_length = 300 }),
  every_s = rb.ui.slider(60, { min = 15, max = 600, suffix = "s", label = "Check every" }),
  confidence = rb.ui.slider(80, { min = 50, max = 99, suffix = "%", label = "Confidence needed" }),
})

rb.on("start", function()
  rb.run(function()
    while true do
      local answer, confidence = rb.ai.decide(cfg.question, {
        choices = { "yes", "no" },
        screen = true,
      })
      if answer == "yes" and confidence >= cfg.confidence / 100 then
        local sent = rb.phone.notify("Done: " .. cfg.question, "Job Watcher")
        if not sent.delivered then
          rb.ui.notify("Done: " .. cfg.question, "success")
        end
        return
      end
      rb.sleep(cfg.every_s * 1000)
    end
  end)
end)

Without screen = true or a window, decide answers from the question alone and sends no screenshot. rb.phone.notify returns { delivered, reason }. There is no phone app yet, so it always returns delivered = false with the reason "no phone connected"; the script falls back to a notification in Rebind.

Keep looks inside one app

A script targeted at an application can promise to read only that application’s window. Add screen:target to its permissions along with a process= or window= target. Then find and read without a window look at the target window instead of the whole display, and Rebind’s install screen shows the narrower grant.

--[[
  rebind: min_sdk=3.9.0
  rebind: name=Editor Error Reader
  rebind: version=1.0.0
  rebind: process=Code.exe
  rebind: permissions=ai:read,screen:target
]]

rb.bind("F11", rb.async(function()
  local message = rb.ai.read("the first error message in the Problems panel")
  if message ~= "" then
    rb.clipboard.set(message)
    rb.ui.notify("Copied the error", "success")
  end
end))

screen:none blocks every look, so a script with it can still use decide without a screenshot but not find or read. Scoped permissions are explained in Permissions.

Getting Started

Concepts

Cookbook

Account

Marketplace

Reference

Remote Control

Help

Click something you describe

Read a value off the screen

Read next to something you found

Wait for something to happen

Keep looks inside one app

​Click something you describe

​Read a value off the screen

​Read next to something you found

​Wait for something to happen

​Keep looks inside one app

Click something you describe

Read a value off the screen

Read next to something you found

Wait for something to happen

Keep looks inside one app