rb.screen

rb.screen reads what is on the screen: the color of one pixel, the first match for a color or an image in a region, or a display as a PNG. A script reaches for it to react to what a game or app shows, such as a health bar changing color or a button appearing. Needs: the screen or screen:target permission for pixel reads in a script that declares permissions; a coroutine for capture, find_image, and a search_color region over 200,000 physical pixels, so call them inside rb.run or rb.async (see Where code runs). This script prints the color under the cursor on F6, then looks for red in a 400 by 300 box at the top left of the primary display:

-- rebind: min_sdk=3.9.0 name=example version=1.0.0 permissions=screen
rb.bind("F6", rb.async(function()
  print("under the cursor: " .. rb.screen.pixel_hex())
  local hit = rb.screen.search_color({ 0, 0, 400, 300 }, "ff0000", 0.1)
  if hit then
    print("red at " .. hit.x .. ", " .. hit.y)
  else
    print("no red in the box")
  end
end))

Signature	Coroutine only
`rb.screen.pixel_hex(x: number?, y: number?) -> string`	no
`rb.screen.search_color(region: { number }, color: string, tolerance: number?) -> PixelMatch?`	over 200,000 physical pixels
`rb.screen.list() -> { MonitorEntry }`	no
`rb.screen.capture(options: CaptureOptions?) -> CaptureResult`	yes
`rb.screen.find_image(needle: string, options: FindImageOptions?) -> ImageMatch?`	yes
`rb.screen.work_area(x: number?, y: number?) -> WindowPos`	no

Coordinates are logical screen pixels on the virtual desktop; displays left of or above the primary have negative coordinates. Outside a coroutine, capture raises Screen.Capture: call inside rb.run() or an rb.async handler; it yields instead of blocking the tick, and find_image raises the same message starting Screen.FindImage:.

rb.screen.pixel_hex

Returns the color of the pixel at x, y, or under the cursor when they are omitted, as "rrggbb", lowercase.

rb.screen.search_color

region: { x1, y1, x2, y2 }; x2 must be greater than x1 and y2 greater than y1.
color: six hex digits, "rrggbb", any case, without #; anything else raises SearchForColor: invalid color '<color>' (expected 6-char hex like "ff0000"). For a value from rb.ui.color, pass cfg.color:sub(2).
tolerance: 0 (exact, default) to 1, per channel as a fraction of 255.
Returns { x, y } of the first match, scanning left to right and top to bottom, or nil. A failed read raises.

A region up to 200,000 physical pixels is scanned where it is called. A larger one raises outside a coroutine: Screen.SearchForColor: region of <n> physical pixels exceeds the 200000 inline limit; call inside rb.run().

rb.screen.list

One { index, x, y, width, height, primary } per display, index starting at 1. Returns an empty table when displays cannot be listed.

rb.screen.capture

Options:

display: index from rb.screen.list(). Default: the display under the cursor, else the primary. Out of range raises Screen.Capture: display <n> out of range (1..<count>).
region: { x, y, w, h } in that display’s own coordinates, clamped to the display. A zero width or height raises.
max_edge: downscale so the longer side is at most this many pixels; 0 or omitted keeps native resolution.

Returns { image, width, height, display, cursor }: image is a base64 PNG, width and height are its pixel size after any downscale, display is the display’s { index, x, y, width, height, primary }, and cursor is the cursor position read with the frame. On a high-density display the image has more pixels than the logical region.

rb.screen.find_image

Captures like capture and scans for a needle image.

needle: path to a PNG in the script directory or its data folder. Pixels with alpha below 128 match anything.
options: display and region as in capture; tolerance as in search_color; scales, a list of 1 to 8 ratios between 0.1 and 10 to resize the needle by, tried in order (default { 1 }).
Returns { x, y, width, height } of the first match in screen coordinates, or nil.

The needle is compared at native pixel scale. A needle cut from a 2x display finds its 1x rendering with scales = { 1, 0.5 }. Each ratio costs one more scan, and resizing softens edges, so raise tolerance with it. This bind looks for ok-button.png on F6 and clicks the center of the first match:

-- rebind: min_sdk=3.9.0 name=example version=1.0.0
rb.bind("F6", rb.async(function()
  local hit = rb.screen.find_image("ok-button.png", { tolerance = 0.05 })
  if hit then
    rb.hid.move_to(hit.x + hit.width / 2, hit.y + hit.height / 2)
    rb.hid.press("Mouse1")
  end
end))

rb.screen.work_area

The usable desktop of the display containing the point, or the cursor when omitted: the display minus the taskbar. Windows only; elsewhere it raises.

Overview

Script basics

Input and output

Timing and script control

Panels, overlays and dialogs

Screen, windows and apps

Operating system

Files and data

Network and pipes

Rebind Pro

Lookup