Docs Center

UIA Scripts

Complete syntax and command reference for Windows UI Automation scripts.

What is a UIA script

A UIA script is the desktop execution script of the digital employee. Instead of browser automation it calls Windows UI Automation (UIA) in-process and drives Chrome, Edge, Notepad, Office and other desktop apps in the order: window → element → action.

One command per line, executed top-down; the first failure stops the run and reports the line number and error code (except find optional).

  • Locating only uses window titles and element properties — no URLs, no browser drivers
  • Two channels: A = UIA patterns (works in background); B = SendInput injection. The default natural level prefers real mouse/keyboard and uses patterns only as a fallback
  • Scripts declare inputs / outputs and are callable from workflow nodes; empty caller values fall back to script defaults
  • Supports conditions, loops, iteration, regex text processing and screenshots for waiting, retries and cleanup
  • Plain-text scripts, reviewable and version-controllable

Inputs and outputs

The outermost level may declare capability parameters and products with @input / @output. Declarations are metadata: parameter values are interpolated into strings and selectors via {name}.

Inputs (with defaults)

@input url 目标网址 =https://example.com
@input count 重复次数 =3
@input index 编辑器序号 =1

Outputs

@output result 读取到的结果

Format: "@input name description =default". Everything between the name and the equals sign is the description; the text after it is the default (quotes allowed). It applies when the caller omits the value or passes an empty string.

  • Parameters can be interpolated into strings, name/id/class selectors, and index:{param} to make positions configurable
  • If a name: value looks like /regex/, it is treated as a regex at runtime — one parameter switches the match mode
@input / @output are only allowed at the top level, never inside if / repeat / foreach blocks. Every emit target must be declared with @output or parsing fails.

Execution model and boundaries

AspectNotes
Where it runsInside the desktop app process (Rust + Windows UIA COM); no Node runtime, no browser drivers
ScopeThe current Windows desktop session: foreground windows, tray apps, browsers and clients
Element sourceThe UIA element tree: Name, AutomationId, ClassName, ControlType and other properties
PrivilegesSame integrity level is enough; to drive an elevated app the app itself must run elevated
Not supportedLocked screens, signed-out sessions and disconnected RDP sessions
Scan capA single lookup examines at most 20,000 candidates to protect against pathological pages; if the cap is hit the log warns about possible omissions
Stability tipsPrefer exact name:; use name~: or regex for localized UIs; use next/prev/near for unstable layouts; add index: / timeout: when needed

Table of contents

1. Lexical rules

Basics

  • One command per line; blank lines are ignored
  • A hash sign starts a comment to end of line; hash signs inside strings are literal
  • Strings use double quotes, e.g. name:"Notepad"
  • Variables are written $name: find produces elements, get / collect / let / extract / matches / strip produce text, and read lines / json can produce lists
  • Parameter interpolation uses {name}, only for declared @input values (or params supplied by the caller)
  • Strings and logs may also contain $variable, replaced with its current text value at runtime

Comments

# 打开 Chrome 窗口
window name~:"Chrome"        # 行尾注释同样有效

Parameter and variable interpolation

@input kw 关键词 =数字员工
set name:"搜索框" "搜索 {kw}"
get name:"搜索框" -> $text
log "关键词:{kw},输入框里现在是 $text"
Three kinds of variables: $element (from find), $text (from get / collect / let) and $list (multiple values from read lines / collect / json). They are not interchangeable.

2. Selectors

A selector describes an element with key:value pairs separated by spaces — all keys must match (AND).

KeyMeaningExample
name:Exact name matchname:"发送"
name~:Name contains substringname~:"地址"
name:/…/Name regex matchname:/Chrome.*/
type:UIA control typetype:button
id:AutomationId for elements (browser top-level windows usually have none; window ignores id)id:"submit"
id~:AutomationId contains a substring (or /regex/)id~:"send"
class:Window / control ClassNameclass:"Chrome_WidgetWin_1"
class~:ClassName contains a substring (or /regex/)class~:"WidgetWin"
value:Filter by current value (ValuePattern → TextPattern → name)value:"已发布"
value~:Current value contains a substring (or /regex/)value~:"成功"
visible:Only visible elements; 0 negates, true/false acceptedvisible:1
enabled:Only enabled elements; 0 negatesenabled:1
childof:Only direct children of that elementchildof:$panel
limit:Take at most N results (collect / count only)limit:20
index:Nth match from 1; index:{param} allowedindex:2
min:Minimum side in px, filters out icons/avatarsmin:24
timeout:Locate timeout in ms, default 3000timeout:8000
under:Search only inside that element subtreeunder:$tip
next:Next sibling after the anchornext:$box
prev:Previous sibling before the anchorprev:$box
near:Element near the anchor, by side/distancenear:$box
side:Direction for near: right / left / above / below / anyside:right
gap:Minimum gap in px for neargap:12
depth:How many parent levels next/prev walks up, 1-3depth:2

Absolute lookup

find type:edit name~:"搜索" -> $box
find type:button name:/Chrome.*/ index:{btn_index} timeout:8000 -> $btn

Relative lookup

Relative lookup uses a located element as the anchor. next / prev first walk the control tree siblings (optionally up via depth) and fall back to geometry; near sorts candidates by bounding-box direction and distance.

find type:text name~:"标题" -> $title
find next:$title type:text -> $nextItem
find prev:$title type:text depth:2 -> $prevItem
find near:$title type:button side:below gap:8 -> $btn
side / gap work only with near:; a selector may have only one relative anchor; the anchor must be located by find first.

Scope

find type:tooltip optional timeout:3000 -> $tip
find under:$tip type:text name~:"候选" optional -> $option
name~: and name:/…/ cannot be expressed as UIA property conditions: the engine grabs candidates by type / id / class first, then filters by name locally (up to 20,000 candidates per scan). If a regex contains #, wrap the whole pattern in double quotes so it is not read as a comment.

3. Supported control types

type: maps to the UIA ControlType. Common values below; other types work with their lowercase names too.

ValueControl
buttonButton
editEdit
textText
checkboxCheck box
radiobuttonRadio button
comboboxCombo box
list / listitemList / list item
tab / tabitemTab / tab item
menu / menuitemMenu / menu item
tree / treeitemTree / tree item
table / dataitemTable / data item
hyperlinkHyperlink
imageImage
tooltipTooltip
group / paneGroup / pane (often used as scope)
documentDocument (browser page body)
windowWindow

4. Command summary

The engine currently ships 48 commands, ordered by purpose below and documented group by group.

CommandPurpose
windowLocate a top-level window and bring it forward; window back returns, window close closes the current one
gotoOpen a URL via the address bar, read it back and wait for the title expect to match
urlRead the current page address into $url
tabList, switch, open or close browser tabs
findLocate an element into $element; optional visual fallback with find image / find text
clickClick an element (real mouse first, patterns as fallback); click at clicks a coordinate
hoverMove the real mouse onto an element without clicking, for hover menus
dragPress at a start point, drag and release, for sliders, sorting and drag-and-drop
selectChoose a dropdown option (expand and match first, real click as fallback)
toggleSet a checkbox / switch to a state without redundant clicks
invokeTrigger a control through UIA patterns without moving the mouse
focusFocus an element without changing content
expandExpand virtual-scroll content (down button, Ctrl+End fallback)
setWrite text (real input first on natural; ValuePattern first on light/off)
letStore a literal or variable into $text
typeType into the focused control; ≥40 chars are pasted in chunks; type $var allowed
keySend hotkeys such as ctrl+l
scrollScroll the wheel at the window centre
clipboardRead from or write to the system clipboard
dialogAccept or dismiss a native system dialog (Enter / Esc)
downloadWait for a browser download to finish and return its path
getRead element text into $text
checkAssert content / state / count and fail otherwise; -> $ok makes it non-fatal
collectCollect all matching text into $text, optionally sliced between anchors
countCount matching elements into $text; timeout: polls until they appear
extractExtract the first capture group by regex
matchesCollect all regex matches
stripRemove all regex matches
firstTake the first list item into $var; empty lists yield ""
dropRemove the first matching item from a list into $rest
pushAppend one item to a list
lenCount list items into $var
incIncrement an integer by a delta (inc $var 2)
readRead a text file into $var; lines splits it into a list
appendAppend one line to a text file
writeOverwrite a text file with a list, one item per line
postSend a JSON POST into $resp; non-2xx fails, retry supported
jsonRead a JSON field by dot path into $var
newestGet the newest image path from the download folder; with optional, a timeout yields an empty value instead of aborting
waitSleep for milliseconds; wait visible / gone / stable wait for a condition
retryRetry a block when it fails, up to n times
logAppend a line to the run log
emitWrite a capability output
shotCapture the current window as PNG
ifConditional branch (== / != / contains / in and more)
repeatLoop by count (param allowed)
foreachIterate a list variable (quiet suppresses successful step logs)
breakBreak out of the nearest loop

5. window: locate a windowCore

Locates a top-level window by title and brings it forward; later find / click operate inside it. Every switch pushes the current window onto a stack. Locating runs a series of pre-checks: the window still exists, its title/class matches, its position is usable (only minimized or off-screen windows are restored and placed in the left third), and it is brought to the foreground when needed.

window name:"记事本"
window name~:"Chrome"
window name:/.*Excel.*/
window class:"Chrome_WidgetWin_1"
window name~:"Chrome" expect:"币安广场" timeout:8000

expect: window content assertion

expect:"text" makes window keep waiting after locating until the window title contains that text, confirming the page has loaded onto the target site instead of sitting on a blank or sign-in page; a timeout returns E_WINDOW_MISMATCH. The wait uses the window timeout (5000 ms by default).

window back: return to the previous window

After handling a file dialog or another temporary window, window back restores the previous one instead of repeating its title.

find type:button name~:"上传" -> $up
click $up
window class:"#32770"
type "C:\tmp\cover.png"
key enter
window back

window close: close the current window

Sends a close request to the located window, useful for cleaning up temporary windows after a run; a window with an unsaved-changes prompt is not force-closed, so handle the dialog in the script. The current window becomes invalid afterwards — run window again before further actions.

window needs one of name: / name~: / name:/…/ / class:; missing windows return E_WINDOW_NOT_FOUND. window back without a previous window fails the same way. id: is an element AutomationId and is not used for window lookup.

Browser window ownership

After the digital employee launches a browser it marks the main window it detects. From then on, a window selector that looks like a browser target (its name contains Chrome / Chromium / Edge / Firefox / Browser, or its class is Chrome_WidgetWin_* / MozillaWindowClass) only matches marked windows; a Chrome window opened manually by the user is never mistaken for the target.

When no marked browser window exists, the pre-run check opens the browser using the configured launch command and takes the new window over. For Chrome / Edge / Chromium / Firefox it appends a new-window flag (--new-window / -new-window) so the mark lands on a new window rather than the user's old one; if no new window appears, the current main window is taken over. The mark lives on the window object and survives app restarts until that window closes.

6. find: locate an elementCore

Searches descendants of the located window and stores the result in an $element variable for later actions.

find type:edit name:"地址栏" timeout:10000 -> $addr
find name~:"搜索" type:button -> $searchBtn
find near:$searchBtn type:text side:below gap:8 -> $hint

Requires "-> $var"; on a miss it retries every 200ms until timeout: expires, then returns E_ELEMENT_NOT_FOUND.

optional: tolerate a miss

find type:button name~:"上传" optional timeout:5000 -> $up
if $up == "" {
    log "没有上传按钮,跳过配图"
}

With optional a miss is not an error: the variable becomes an empty string and later if-checks can take a fallback path. The element also stores its name text for logs and checks; unnamed elements store 1 (meaning found) to avoid false empty results.

optional only suppresses the lookup error: clicking an empty variable still fails with E_ELEMENT_NOT_FOUND, so put the fallback inside an if.
When the element tree has no usable target (canvas, custom-drawn controls, remote desktop, image buttons), use the visual fallback in the next section: find image for template matching, find text for OCR, then click the produced point with click at.

7. Visual fallback: template matching & OCR

find image runs local template matching against the captured window (locate an anchor image) and find text uses Windows OCR to find on-screen text. Both store the matched physical screen point as "x,y" in a variable, which click at $pt can then click — ideal when the element tree cannot give coordinates.

find image:"compose.png" region:$panel min:0.88 scales:0.8~1.25 -> $pt
click at $pt
find text:"导出" region:$panel min:0.8 timeout:5000 -> $txtPt
click at $txtPt

Parameters

ParameterDescription
image:"anchor.png"Anchor path: absolute, ~/ (user home) or relative; relative paths resolve inside the run storage directory (take a shot first, then crop)
text:"导出"Text to find; matched against OCR output by similarity
region:$containerLocate a container element with find first to limit the search to its visible rect; omitted, the whole window is searched
min:0.88Similarity threshold: 0.88 for image, 0.8 for text
scales:0.8~1.25Anchor scale range for DPI differences; image only
timeout:msRetries every 200 ms until it matches, capped by the capability timeout
optionalA timeout is not an error: the variable becomes an empty string for an if fallback

Preparing an anchor image

An anchor is a PNG crop of the target area — keep only the button or icon, avoid large backgrounds, and stay under roughly 300×300. Take a shot of the window and crop it, or export from the design file; widen scales when system scaling differs.

window must run before visual lookup: the engine brings the window to the foreground and captures it. Occlusion, off-screen targets and system scaling all affect matches; OCR needs the matching language pack installed, otherwise nothing is recognised.

8. click: click an element

Target is an $element variable or an inline selector (selectors also accept under / next / prev / near). On the default natural level a real mouse moves along a track into the visible area and clicks; the hit is then verified with ElementFromPoint — a miss is retried at the new position or handled via UIA patterns (Invoke → Toggle → SelectionItem → LegacyIAccessible) instead of failing hard. light / off use patterns directly; light still moves in and hovers first.

click $searchBtn
click name:"发送" type:button
click name~:"确定" index:2

click at: click by coordinate

When an element cannot be located, click by coordinate in three forms: click at 120,340 for a physical screen point; click at $el 50%,30% for a relative position inside an element's visible rect (0–100, vertical centred when omitted); click at $pt for points produced by find image / find text. Coordinate clicks keep the foreground check.

click at 120,340
click at $canvas 50%,30%
find image:"play.png" -> $pt
click at $pt
The foreground pre-check applies only to the coordinate-click fallback: the owning window must still exist, it is activated when needed, and a refused activation fails with E_WINDOW_NOT_FOREGROUND instead of clicking another window. UIA pattern clicks need no foreground and never steal focus.

9. focus: focus an element

Only moves focus to the element (SetFocus) without touching content or caret position — useful before sending hotkeys.

focus $editor
key ctrl+home
type "开头的话"

10. set: write textCore

On the default natural level the control is focused and written: focus goes to the control first, and a real click inside the visible area is added only when SetFocus did not get keyboard focus; ≥40 chars are pasted in clipboard chunks, shorter text is typed character by character. The value is read back automatically — an empty or clearly short result is retried with the other input method, and a still-mismatched result fails with E_ACTION_FAILED so a silently empty write is never reported as success. light / off try ValuePattern.SetValue first, then fall back to real input when unsupported or ineffective. Line breaks are kept as-is with no extra blank lines (\r\n, a lone \r and \n all count as one break). Text is written line by line: each line is pasted or typed, and real newline keys are sent between lines (Shift+Enter by default, so chat editors never submit early) — Slate / Draft editors do not split pasted \n into paragraphs, so clipboard newlines alone collapse into one line. Editors that only accept a bare Enter can opt in with "newlineKey": "enter" in the capability config.

set $addr "https://example.com"
set name:"搜索框" "搜索 {kw}"
set $title $text
The value comes last: a string literal or a $variable (set $var $src). The target (variable or selector) comes before it.

11. type: type text

Injects text into the focused control via Unicode, so CJK and symbols work without an IME; ≥40 chars switch to chunked clipboard paste (12–25 chars per chunk), shorter text is typed with a human rhythm, and newlines send a real Enter. The text may come from a variable: type $var. Focus is verified first: if it still sits in the browser address bar the step fails with E_ACTION_FAILED instead of turning the text into a URL.

type "你好,世界"
type "{kw}"
type $text
Before type / key / scroll / shot runs, the target window must still exist and be in the foreground: it is brought forward first when needed, and if activating is refused the step fails with E_WINDOW_NOT_FOREGROUND instead of sending input to another window; a closed window reports E_WINDOW_CLOSED.

12. key: hotkeys

Keys separated by plus signs with the main key last; supports ctrl / shift / alt / enter / tab / esc / space / delete / backspace / home / end / arrows / pageup / pagedown and single letters/digits.

key ctrl+l
key enter
key ctrl+shift+tab

13. scroll: wheel

Scrolls the wheel at the window centre: scroll up/down with a step count (integer or {param}); each command is capped at 100 steps and defaults to 3.

scroll down 3
scroll up {max_scroll}
window must run before scroll; when precision matters, locate with near: and click instead of blind scrolling.

14. get: read textCore

Hyperlinks read Name first (their Value is the URL); otherwise ValuePattern → TextPattern → Name. The result goes into a $text variable for log / emit / if and text processing.

get $addr -> $url
get name~:"结果" -> $result
emit answer $result

Like find, "-> $var" is required.

15. collect: collect text

Collects the text of every matching element into one $text variable joined by semicolons; hyperlinks also use Name. Selectors accept under / next / prev / near.

collect type:hyperlink -> $links
collect type:text after:"提示词" until:"复制" timeout:15000 -> $answer

With after: / until: the result is the slice between the anchor and the stop marker, joined by newlines. They are text anchors (a /regex/ is allowed), not element variables; while a page streams, the engine waits for two identical samples before returning.

collect rescans until timeout: (default 3000ms, slower once candidates appear); if nothing is collected it returns E_ELEMENT_NOT_FOUND. Use find for a single element.

16. expand: reveal all content

For virtual-scrolling editors / threads: finds a small unnamed button above-right of the input (a down arrow) and clicks it so history renders fully; after 10 seconds it falls back to Ctrl+End.

find type:edit index:1 -> $box
expand $box
collect type:text after:"提示词" until:"复制" -> $body
expand needs the input located by find first; it makes a best effort and does not fail when the down button is missing.

17. hover / drag: hover and drag

hover moves the real mouse to the element centre and pauses without clicking, for hover menus, overlays and tooltips; drag $from to $to presses, moves in segments and releases, for sliders, list sorting and drag-and-drop uploads. Both accept $element variables and inline selectors.

hover $menu
find type:listitem name~:"导出报表" -> $item
click $item
drag $slider to $target
drag type:"listitem" name~:"卡片 1" to type:"list" name~:"回收站"
Hover menus animate for tens of milliseconds; add wait visible <target> or wait stable before clicking to avoid a miss. Very short drags may be treated as clicks — move sliders in several segments.

18. select / toggle: dropdowns and switches

select $el "option text" (or a $variable) expands the dropdown, matches the option text in its subtree, and falls back to a real click when needed; toggle $el on|off reads the current state first and does nothing when it already matches — true/false and 1/0 are accepted.

select $category "发布能力"
select $city $targetCity
toggle $remember on
toggle $switch false
Option text must match the UI exactly (spaces, full-width characters and ellipses all matter). Custom-drawn dropdowns that are not real ComboBoxes should be driven with click + find/click on the option.

19. clipboard: clipboard

clipboard get -> $var reads the clipboard text; clipboard set "text" or clipboard set $var writes it. Useful for controls whose text cannot be read directly, or for pasting long text into an editor in one go.

clipboard get -> $code
check $code matches:/^\d{6}$/
clipboard set "已处理:{name}"
clipboard set $text
clipboard set overwrites the user's clipboard; before reading, make sure the target app finished copying (wait stable or a fixed wait helps).

20. dialog: native dialogs

Handles native system dialogs on the window selected by window (bringing it to the foreground) without locating the dialog itself: dialog accept presses Enter, dialog dismiss presses Esc. Typical cases are download confirmations, certificate warnings and exit prompts.

click $download
wait visible class:"#32770" timeout:15000
dialog accept
wait gone class:"#32770" timeout:10000 -> $closed
In-page popups are not native dialogs and still need find / click; dialog only responds to real system dialogs and relies on window to locate the owning window first.

21. url: read the current address

url -> $var reads the current page address from the browser address bar through UIA without any browser API — useful for verifying navigation or persisting links.

window name~:"Chrome"
url -> $link
check $link contains "/publish"
append "~/published-urls.txt" $link
Locate a browser window with window first; non-browser windows or a missing address bar fail with E_ELEMENT_NOT_FOUND.

22. tab: browser tabs

tab list -> $var lists tab titles; tab switch 2 or tab switch "title" activates a tab; tab new ["URL"] opens one; tab close [index] closes the given tab (the current one when omitted). Relies on the browser exposing its tab strip to UIA.

tab list -> $tabs
tab new "https://example.com"
tab switch "工作台"
wait stable type:document stable:1000 timeout:60000 -> $body
tab close 1
Tab-strip structure varies between Chromium versions; when it is not exposed the command fails with E_ELEMENT_NOT_FOUND, so fall back to key ctrl+tab. A switched tab may still be loading — follow with wait visible / wait stable. Use indexes when titles repeat.

23. download wait: wait for a download

download wait timeout:60000 -> $var polls the browser download folder, skips temporary files such as .crdownload / .part, only accepts files written around this run, and returns the absolute path once the size settles.

click $export
wait gone type:progressbar timeout:120000
download wait timeout:180000 -> $file
log "已下载:$file"
Assumes the browser's default download folder; with concurrent downloads the first settled file is returned. Raise timeout for slow networks or large files.

24. read / append / write: text files

read "path" -> $var reads text, and with lines splits it into a list (empty lines skipped); append "path" "text" appends one line (newlines are folded into spaces); write "path" $list overwrites with one item per line, and an empty list truncates the file. Paths accept absolute, ~/ and %USERPROFILE%; relative paths resolve against the user home.

read "~/queue.txt" lines -> $queue
first $queue -> $item
append "~/done.txt" "{name}\t$item"
drop $queue $item -> $rest
write "~/queue.txt" $rest
read treats a missing file as empty so first runs never fail; append / write create parent folders. Build lists with read lines or collect — hand-joined strings are treated as a single item by first / drop.

25. post / json: HTTP requests and JSON fields

post "url" "{\"key\":$var}" -> $resp sends a JSON POST; non-2xx responses fail and retry 3 adds automatic retries. json $resp "data.items" -> $value reads by dot path, joining arrays into a list and yielding an empty string for a missing path — handy for enriching runs with your own API.

post "https://api.example.com/tasks" "{\"page\":$page}" retry 2 -> $resp
json $resp "data.items" -> $items
len $items -> $n
foreach $item in $items {
    log "任务:$item"
}
Responses must be JSON for json extraction; {params} and $vars are expanded inside the body. Never hard-code API keys in scripts — keep secrets in the capability config.

26. let: store a value

Stores a literal or another variable into a $text variable, handy for taking a copy before repeated processing.

let $raw "{body}"
let $copy $raw

27. extract: regex slice

Slices a variable with /regex/: the first capture group when present, otherwise the whole match; a total miss keeps the original value.

let $raw "{body}"
extract $raw /^([^;]+)/ -> $head
extract $raw /^[^;]+;?/ -> $rest

28. matches: all matches

Finds every /regex/ match (the whole match, not capture groups), joined by semicolons; no match yields an empty string.

let $raw "{body}"
matches $raw "/#[^#,。!?;:、]+/" -> $tags
foreach $t in $tags {
    log "标签:$t"
}

29. strip: remove matches

Removes every /regex/ match from a variable — commonly used to strip hashtags out of a body.

let $raw "{body}"
strip $raw "/#[^#,。!?;:、]+/" -> $plain
set $editor $plain

30. List helpers: first / drop / push / len / inc

Lists are structured variables produced by read lines / collect and friends: first $list -> $item takes the head, len $list -> $n counts items, drop $list $item -> $rest removes the first match, push $list value appends one, and inc $var 2 increments an integer (missing variables start at 0). Logs and outputs render lists joined by ";".

read "~/queue.txt" lines -> $queue
len $queue -> $n
first $queue -> $item
drop $queue $item -> $rest
write "~/queue.txt" $rest
push $done $item
inc $page {page_size}
Build lists only with commands like read lines / collect — hand-joined strings count as one item and break first / drop. first returns an empty string for an empty list, and drop removes only the first match, so loop for duplicates.

31. newest: newest image

Returns the newest image path in a folder. An empty folder argument means %USERPROFILE%\Downloads; it polls for 30 seconds and accepts only files written in the last 5 minutes (png / jpg / jpeg / webp / gif).

By default a 30-second timeout raises E_ELEMENT_NOT_FOUND and aborts the script. When the image is optional, append optional: on timeout the variable is set to an empty string and execution continues (the log states that no new image appeared), so the script can branch with if $img == "". Wrapping newest in an if block does not help: errors inside a block still abort.

# 必需:等不到就中止
newest "" -> $img

# 可选:等不到就置空并继续
newest "" -> $img optional
if $img == "" {
    log "本次没有配图"
}
newest only finds the path; uploading is up to the script (fill the file dialog and press Enter). Temporary downloads in progress are ignored.

32. wait: sleep

wait 800
wait {wait_ms}

Sleeps the given milliseconds (capped at 600000ms per call). Prefer find / collect timeout: for conditional waits over fixed sleeps.

33. log: run log

log "开始执行"
log "关键词:{kw},结果:$result"

Writes to the work log streamed in real time; supports {param} and $variable interpolation.

34. if: conditional

A condition is left operand, operator and right operand (each a $variable or literal); operators are ==, !=, contains. Blocks open with a trailing { and close with } on its own line.

get $addr -> $url
if $url contains "example.com" {
    log "已在目标站点"
}
if $up == "" {
    log "按钮不存在,走备用路径"
}

35. repeat: loop

The count is an integer or a {param}; a single run is capped at 1000 iterations. break inside the body ends it early.

repeat 3 {
    key tab
    wait 300
}

36. foreach: iterate a list

Splits the list variable on semicolons (trimming blanks and skipping empty items), stores the current item in $item each round, and scopes the item variable to the body.

let $raw "{body}"
matches $raw "/#[^#,。!?;:、]+/" -> $tags
foreach $t in $tags {
    log "标签:$t"
}

37. break: break a loop

Breaks the nearest repeat / foreach; inside an if it propagates to the enclosing loop.

repeat 10 {
    get name~:"状态" -> $s
    if $s contains "完成" {
        break
    }
    wait 500
}

38. emit: write outputs

Writes a $text variable or literal into capability outputs; the name must match an @output declaration and lands in the run result for later nodes.

emit result $result
emit site "示例站点"

39. shot: screenshot

Captures the located window as PNG into the run storage directory; the output name is the file name and the value is the absolute path.

shot "step1.png"
Window must be located before shot, otherwise E_WINDOW_NOT_FOUND.

40. goto: open a URL and confirm it loadedCore

Opens an address in the located browser window: it locates the address bar via UIA, clicks to focus it, types the URL and reads it back before pressing Enter. If the text did not land it presses Esc to dismiss popups such as "Restore pages?" and retries (up to 3 times); if it still fails it returns E_ACTION_FAILED without pressing Enter. The first goto also waits config.humanize.gotoSettleMs (5000 ms by default, 0 disables), letting welcome pages and prompts finish. A trailing space is appended to the typed URL so the address bar does not rewrite it with history/suggestion matches (which could open a different address on Enter); the read-back comparison ignores that space and navigation is unaffected.

@input url 目标网址 =https://example.com
@input expect 标题关键词 =示例
window name:/.../ timeout:8000
goto "{url}" expect:"{expect}" timeout:20000

expect accepts plain text or /regex/ and asserts the window title like window expect before continuing; a timeout returns E_NAVIGATE_TIMEOUT. The address may come from a variable: goto $url.

Drop the old key ctrl+l + wait 1500 pattern: fixed sleeps are slow and unreliable, while goto reads the address bar back and confirms the page via the title.

41. invoke: trigger via patterns

Triggers the control through UIA patterns (Invoke → Toggle → SelectionItem → LegacyIAccessible) without moving the mouse. It is the right choice for controls that real clicks cannot hit: Chrome's hidden <input type=file> spans several toolbar icons and is skipped by hit testing, so click is a gamble while invoke always opens the file dialog. On failure it re-locates the element once and retries.

find type:button name~:"上传" -> $up
invoke $up
invoke does not steal focus or move onto the control; use click when a hover or real click is required.

42. check: assert content and stateCore

check is a read-only assertion that never moves focus or changes content. Content comparisons support == (ignoring zero-width and whitespace differences), contains, ~= (edit-distance similarity, tolerance:0–1, default 0.9) and matches:/regex/; state checks support exists:1 / visible:1 / enabled:1 / focused:1 / checked:1; counts use check $n >= 2.

find type:edit index:1 -> $editor
get $editor -> $text
check $text contains "已发布"
check $text ~= "{expected}" tolerance:0.9
check $editor exists:1 visible:1
count type:listitem visible:1 -> $n
check $n >= 1
Without -> $ok it is a hard assertion that fails with E_CHECK_FAILED and stops the run; with -> $ok it is a soft assertion that only stores 1/0. Adding timeout: polls every 200 ms until satisfied — handy for waiting until a button becomes enabled or a "Saved" label appears. set / type do not guarantee the written value, so always verify with check.

43. count: count matching elements

Counts matching elements into a $text variable, typically to confirm a list really rendered its rows. With timeout: it polls every 200 ms until a match appears; without it the count is a single snapshot and zero is not an error. limit:n caps how many are counted.

count type:listitem visible:1 timeout:5000 -> $n
log "列表项数量:$n"
check $n >= 1

44. retry: retry a block on failure

retry n { … } retries the whole block: any failing statement restarts it from the top, up to n times (integer or {param}, capped at 1000). Each failure is logged separately and the last error is raised after all attempts. Unlike repeat, retry only restarts after a failure.

retry 3 {
    find name:"保存" type:button timeout:5000 -> $btn
    click $btn
    wait stable name:"已保存" timeout:5000 -> $state
}
if $state contains "已保存" {
    emit result "ok"
}
A failure automatically saves a screenshot whose path appears in the error and log; keep only repeatable actions inside the block to avoid double-submitting.

45. wait visible / gone / stable

wait visible <target> [timeout:ms] waits until the element exists and is visible (10000 ms default); wait gone <target> [timeout:ms] [-> $ok] waits until it disappears or the window closes (5000 ms default), and with -> $ok it becomes a check: 1 when gone, 0 on timeout without aborting; wait stable <target> [stable:ms] [timeout:ms] [-> $out] waits until the text stays unchanged and non-empty for stable ms (800 default, 20000 timeout) and can store the settled text.

click $up
window class:"#32770" timeout:8000
type "C:\tmp\cover.png"
key enter
window back
wait gone class:"#32770" timeout:20000 -> $closed
if $closed != "1" {
    log "文件对话框没有关闭,上传可能未提交"
}
wait stable type:document stable:1500 timeout:90000 -> $answer
log "回答已稳定:$answer"
Fixed wait never shortens and is capped by the run budget; when a condition can be expressed, prefer wait visible / gone / stable for both speed and stability.

46. Humanization (on by default)

Humanization lives in the engine and applies without touching scripts: the mouse follows a cubic Bézier path (slower at both ends, with jitter and overshoot), the landing point is never the centre; text is typed with a human rhythm and long text is pasted in chunks; the wheel scrolls in segments; window switches, navigations and page loads pause by level; explicit waits never shorten. Each run logs the active level on its first line.

"humanize": {
  "level": "natural",   # off | light | natural | careful(默认 natural)
  "speed": 1.0,         # 节奏倍率 0.3~3.0,越大越快
  "mouse": true,
  "typing": true,
  "dwell": true,
  "pasteOver": 40,      # ≥ 该字数走粘贴
  "gotoSettleMs": 5000, # 首次 goto 前等待浏览器稳定
  "seed": 0
}
A global default can be stored in settings (key=humanize, value a level name or JSON) and overridden per capability via config.humanize; "off" or false disables it entirely. Internal pauses are capped by config.maxMs. If a site is unstable on natural, drop to light or raise speed.

Keeping the screen awake and lock detection

Real input injected by channel B is discarded while the display is off or the machine sleeps (the script finishes but nothing is written). Each capability therefore blocks display-off and sleep by default (which also blocks idle auto-lock) and restores power behaviour on exit; a dark display is woken first. Both are capability config keys: keepAwake (default true) and allowLockedScreen (default false).

"keepAwake": true,        # 执行期阻止息屏/睡眠,结束恢复
"allowLockedScreen": false, # 锁屏时直接报 E_SESSION_LOCKED
A locked screen (Win+L / the lock screen) cannot be unlocked by a program: it is the Winlogon secure desktop, simulated input is dropped, and letting the app type the password itself is unsafe. All you can do is prevent idle locking and fail fast with E_SESSION_LOCKED.

47. Reference recipes: verify first

The snippets below are copy-paste ready. The rule: follow every action that "looks successful" with a check / wait so a click that did nothing is never reported as success.

Open → write → send → confirm

@input url 目标网址
@input body 正文
window name:/.../ timeout:8000
goto "{url}" expect:/.../ timeout:20000
find type:edit index:1 -> $editor
set $editor "{body}"
check $editor exists:1
find name:/发送|Send/ type:button timeout:5000 -> $send
click $send
wait gone type:edit index:1 timeout:15000 -> $closed
if $closed != "1" {
    log "编辑器没有收起,发送可能未生效"
}

Upload an image and confirm the dialog closed

newest "" -> $img
find type:button name~:"上传" -> $up
invoke $up
window class:"#32770" timeout:8000
type "$img"
key enter
window back
wait gone class:"#32770" timeout:20000 -> $closed
check $closed == "1"

Streaming answer: wait until settled, then read

wait stable type:document stable:1500 timeout:90000 -> $answer
collect type:text after:"提示词" until:"复制" timeout:15000 -> $body
extract $body /^([^;]+)/ -> $head
emit reply $body
Use hard assertions when continuing after a failure makes no sense; when a fallback is needed write check … -> $ok and branch on if $ok != "1".

48. Error code reference

CodeMeaningWhat to check
E_SCRIPT_PARSEScript failed to parseRead the line number in the log; check } pairing, selector keys, -> $var, variable definition order and @output declarations
E_PARAM_INVALIDInvalid or missing parameterCheck @input declarations and caller params; unsupported key names; numeric params (wait/repeat/scroll) must be non-negative integers
E_UIA_INITUIA initialisation failedCheck the UIA item in Settings → Environment check; restart the engine
E_WINDOW_NOT_FOUNDWindow not foundMake sure the app is open; use name~: or regex for fuzzy titles; window back needs a preceding window
E_WINDOW_MISMATCHWindow content mismatchThe expect: assertion did not match before the timeout: check that the page finished loading and the title is correct; raise timeout: or use a more stable keyword
E_WINDOW_CLOSEDWindow closedThe target window closed mid-run: run window again so the script re-locates or reopens the browser
E_WINDOW_NOT_FOREGROUNDWindow cannot be foregroundedA full-screen or elevated window holds the foreground: close it and retry; keyboard input is never sent blindly
E_ELEMENT_NOT_FOUNDElement not foundVerify name and type; raise timeout:; add index: or use next/prev/near; watch for the scan-cap warning
E_NAVIGATE_TIMEOUTgoto timed outThe expect title did not appear before the timeout: check the URL, that the page opens and the title keyword is stable; raise goto's timeout: if needed
E_WAIT_TIMEOUTConditional wait timed outwait visible / gone / stable did not satisfy its condition in time: raise timeout: or use a more stable target (gone / stable with -> $var do not abort)
E_CHECK_FAILEDAssertion failedA hard assertion failed: the error includes expected/actual and the first difference; confirm the previous write really landed, then relax to ~= or add timeout: polling
E_CHECK_UNSUPPORTEDControl cannot be checkedThe target has no ValuePattern / TextPattern (content unreadable) or no TogglePattern (checked unreadable); switch to get + text checks
E_TIMEOUTCapability timed outExceeded config.maxMs (30 min by default, 0 unlimited): split long runs or raise the budget; per-statement timeout: is also capped by the remaining budget
E_SESSION_LOCKEDSession lockedThe screen is locked (Win+L / lock screen). The lock screen is the Winlogon secure desktop, simulated input is discarded and user-mode apps cannot unlock it, so the run aborts with this hint; set config.allowLockedScreen=1 to skip the check
E_IMPL_RETIREDImplementation retiredA retired browser-automation capability was called: rewrite it as a UIA script
E_ACTION_FAILEDAction failedElement may be hidden or disabled; click or focus it before acting

49. Example: write into Notepad

@input text 写入内容 =数字员工 UIA 脚本
@output body 写入后的内容
window name~:"记事本"
set name~:"文本编辑器" "{text}"
get name~:"文本编辑器" -> $read
emit body $read

Notepad's edit area may be named "Text editor" or "Document" depending on version — name~: is safer. Interpolate the parameter with {text}.

50. Example: open a URL in Chrome

@input url 目标网址 =https://example.com
@output after-open.png 截图路径
window name~:"Chrome"
key ctrl+l
type "{url}"
key enter
wait 1200
shot "after-open.png"

The address bar can also use channel A: find type:edit name~:"Address bar" -> $addr then set $addr "{url}".

51. Example: relative lookup and optional element

@input anchor 起始文本 =提示词
@input stop 结束文本 =复制
@output body 读取到的正文
window name~:"Chrome"
find type:text name~:"{anchor}" -> $a
find near:$a type:button side:below gap:8 optional timeout:5000 -> $more
if $more != "" {
    click $more
}
collect type:text after:"{anchor}" until:"{stop}" timeout:15000 -> $body
emit body $body

Locate the anchor with find, then use near to find the button below it. The button may be absent, so optional plus an if-check handles it. collect slices the answer by text anchors without relying on fixed indices.

52. Example: poll a status and stop early

@input expect 期望文本 =完成
@output status 命中时的状态
@output done 是否命中
let $done "0"
repeat 10 {
    get name~:"状态" -> $s
    if $s contains "{expect}" {
        let $done "1"
        emit status $s
        break
    }
    wait 600
}
emit done $done

break ends repeat immediately; the done output tells the caller whether it matched, so no extra text check is needed.