What is a UIA script
A UIA script is the desktop execution script of the digital employee. Instead of browser automation it calls Windows UI Automation (UIA) in-process and drives Chrome, Edge, Notepad, Office and other desktop apps in the order: window → element → action.
One command per line, executed top-down; the first failure stops the run and reports the line number and error code (except find optional).
- Locating only uses window titles and element properties — no URLs, no browser drivers
- Two channels: A = UIA patterns (works in background); B = SendInput injection. The default natural level prefers real mouse/keyboard and uses patterns only as a fallback
- Scripts declare inputs / outputs and are callable from workflow nodes; empty caller values fall back to script defaults
- Supports conditions, loops, iteration, regex text processing and screenshots for waiting, retries and cleanup
- Plain-text scripts, reviewable and version-controllable
Inputs and outputs
The outermost level may declare capability parameters and products with @input / @output. Declarations are metadata: parameter values are interpolated into strings and selectors via {name}.
Inputs (with defaults)
@input url 目标网址 =https://example.com
@input count 重复次数 =3
@input index 编辑器序号 =1Outputs
@output result 读取到的结果Format: "@input name description =default". Everything between the name and the equals sign is the description; the text after it is the default (quotes allowed). It applies when the caller omits the value or passes an empty string.
- Parameters can be interpolated into strings, name/id/class selectors, and index:{param} to make positions configurable
- If a name: value looks like /regex/, it is treated as a regex at runtime — one parameter switches the match mode
Execution model and boundaries
| Aspect | Notes |
|---|---|
| Where it runs | Inside the desktop app process (Rust + Windows UIA COM); no Node runtime, no browser drivers |
| Scope | The current Windows desktop session: foreground windows, tray apps, browsers and clients |
| Element source | The UIA element tree: Name, AutomationId, ClassName, ControlType and other properties |
| Privileges | Same integrity level is enough; to drive an elevated app the app itself must run elevated |
| Not supported | Locked screens, signed-out sessions and disconnected RDP sessions |
| Scan cap | A single lookup examines at most 20,000 candidates to protect against pathological pages; if the cap is hit the log warns about possible omissions |
| Stability tips | Prefer exact name:; use name~: or regex for localized UIs; use next/prev/near for unstable layouts; add index: / timeout: when needed |
Table of contents
1. Lexical rules
Basics
- One command per line; blank lines are ignored
- A hash sign starts a comment to end of line; hash signs inside strings are literal
- Strings use double quotes, e.g. name:"Notepad"
- Variables are written $name: find produces elements, get / collect / let / extract / matches / strip produce text, and read lines / json can produce lists
- Parameter interpolation uses {name}, only for declared @input values (or params supplied by the caller)
- Strings and logs may also contain $variable, replaced with its current text value at runtime
Comments
# 打开 Chrome 窗口
window name~:"Chrome" # 行尾注释同样有效Parameter and variable interpolation
@input kw 关键词 =数字员工
set name:"搜索框" "搜索 {kw}"
get name:"搜索框" -> $text
log "关键词:{kw},输入框里现在是 $text"2. Selectors
A selector describes an element with key:value pairs separated by spaces — all keys must match (AND).
| Key | Meaning | Example |
|---|---|---|
| name: | Exact name match | name:"发送" |
| name~: | Name contains substring | name~:"地址" |
| name:/…/ | Name regex match | name:/Chrome.*/ |
| type: | UIA control type | type:button |
| id: | AutomationId for elements (browser top-level windows usually have none; window ignores id) | id:"submit" |
| id~: | AutomationId contains a substring (or /regex/) | id~:"send" |
| class: | Window / control ClassName | class:"Chrome_WidgetWin_1" |
| class~: | ClassName contains a substring (or /regex/) | class~:"WidgetWin" |
| value: | Filter by current value (ValuePattern → TextPattern → name) | value:"已发布" |
| value~: | Current value contains a substring (or /regex/) | value~:"成功" |
| visible: | Only visible elements; 0 negates, true/false accepted | visible:1 |
| enabled: | Only enabled elements; 0 negates | enabled:1 |
| childof: | Only direct children of that element | childof:$panel |
| limit: | Take at most N results (collect / count only) | limit:20 |
| index: | Nth match from 1; index:{param} allowed | index:2 |
| min: | Minimum side in px, filters out icons/avatars | min:24 |
| timeout: | Locate timeout in ms, default 3000 | timeout:8000 |
| under: | Search only inside that element subtree | under:$tip |
| next: | Next sibling after the anchor | next:$box |
| prev: | Previous sibling before the anchor | prev:$box |
| near: | Element near the anchor, by side/distance | near:$box |
| side: | Direction for near: right / left / above / below / any | side:right |
| gap: | Minimum gap in px for near | gap:12 |
| depth: | How many parent levels next/prev walks up, 1-3 | depth:2 |
Absolute lookup
find type:edit name~:"搜索" -> $box
find type:button name:/Chrome.*/ index:{btn_index} timeout:8000 -> $btnRelative lookup
Relative lookup uses a located element as the anchor. next / prev first walk the control tree siblings (optionally up via depth) and fall back to geometry; near sorts candidates by bounding-box direction and distance.
find type:text name~:"标题" -> $title
find next:$title type:text -> $nextItem
find prev:$title type:text depth:2 -> $prevItem
find near:$title type:button side:below gap:8 -> $btnScope
find type:tooltip optional timeout:3000 -> $tip
find under:$tip type:text name~:"候选" optional -> $option3. Supported control types
type: maps to the UIA ControlType. Common values below; other types work with their lowercase names too.
| Value | Control |
|---|---|
| button | Button |
| edit | Edit |
| text | Text |
| checkbox | Check box |
| radiobutton | Radio button |
| combobox | Combo box |
| list / listitem | List / list item |
| tab / tabitem | Tab / tab item |
| menu / menuitem | Menu / menu item |
| tree / treeitem | Tree / tree item |
| table / dataitem | Table / data item |
| hyperlink | Hyperlink |
| image | Image |
| tooltip | Tooltip |
| group / pane | Group / pane (often used as scope) |
| document | Document (browser page body) |
| window | Window |
4. Command summary
The engine currently ships 48 commands, ordered by purpose below and documented group by group.
| Command | Purpose |
|---|---|
| window | Locate a top-level window and bring it forward; window back returns, window close closes the current one |
| goto | Open a URL via the address bar, read it back and wait for the title expect to match |
| url | Read the current page address into $url |
| tab | List, switch, open or close browser tabs |
| find | Locate an element into $element; optional visual fallback with find image / find text |
| click | Click an element (real mouse first, patterns as fallback); click at clicks a coordinate |
| hover | Move the real mouse onto an element without clicking, for hover menus |
| drag | Press at a start point, drag and release, for sliders, sorting and drag-and-drop |
| select | Choose a dropdown option (expand and match first, real click as fallback) |
| toggle | Set a checkbox / switch to a state without redundant clicks |
| invoke | Trigger a control through UIA patterns without moving the mouse |
| focus | Focus an element without changing content |
| expand | Expand virtual-scroll content (down button, Ctrl+End fallback) |
| set | Write text (real input first on natural; ValuePattern first on light/off) |
| let | Store a literal or variable into $text |
| type | Type into the focused control; ≥40 chars are pasted in chunks; type $var allowed |
| key | Send hotkeys such as ctrl+l |
| scroll | Scroll the wheel at the window centre |
| clipboard | Read from or write to the system clipboard |
| dialog | Accept or dismiss a native system dialog (Enter / Esc) |
| download | Wait for a browser download to finish and return its path |
| get | Read element text into $text |
| check | Assert content / state / count and fail otherwise; -> $ok makes it non-fatal |
| collect | Collect all matching text into $text, optionally sliced between anchors |
| count | Count matching elements into $text; timeout: polls until they appear |
| extract | Extract the first capture group by regex |
| matches | Collect all regex matches |
| strip | Remove all regex matches |
| first | Take the first list item into $var; empty lists yield "" |
| drop | Remove the first matching item from a list into $rest |
| push | Append one item to a list |
| len | Count list items into $var |
| inc | Increment an integer by a delta (inc $var 2) |
| read | Read a text file into $var; lines splits it into a list |
| append | Append one line to a text file |
| write | Overwrite a text file with a list, one item per line |
| post | Send a JSON POST into $resp; non-2xx fails, retry supported |
| json | Read a JSON field by dot path into $var |
| newest | Get the newest image path from the download folder; with optional, a timeout yields an empty value instead of aborting |
| wait | Sleep for milliseconds; wait visible / gone / stable wait for a condition |
| retry | Retry a block when it fails, up to n times |
| log | Append a line to the run log |
| emit | Write a capability output |
| shot | Capture the current window as PNG |
| if | Conditional branch (== / != / contains / in and more) |
| repeat | Loop by count (param allowed) |
| foreach | Iterate a list variable (quiet suppresses successful step logs) |
| break | Break out of the nearest loop |
5. window: locate a windowCore
Locates a top-level window by title and brings it forward; later find / click operate inside it. Every switch pushes the current window onto a stack. Locating runs a series of pre-checks: the window still exists, its title/class matches, its position is usable (only minimized or off-screen windows are restored and placed in the left third), and it is brought to the foreground when needed.
window name:"记事本"
window name~:"Chrome"
window name:/.*Excel.*/
window class:"Chrome_WidgetWin_1"
window name~:"Chrome" expect:"币安广场" timeout:8000expect: window content assertion
expect:"text" makes window keep waiting after locating until the window title contains that text, confirming the page has loaded onto the target site instead of sitting on a blank or sign-in page; a timeout returns E_WINDOW_MISMATCH. The wait uses the window timeout (5000 ms by default).
window back: return to the previous window
After handling a file dialog or another temporary window, window back restores the previous one instead of repeating its title.
find type:button name~:"上传" -> $up
click $up
window class:"#32770"
type "C:\tmp\cover.png"
key enter
window backwindow close: close the current window
Sends a close request to the located window, useful for cleaning up temporary windows after a run; a window with an unsaved-changes prompt is not force-closed, so handle the dialog in the script. The current window becomes invalid afterwards — run window again before further actions.
Browser window ownership
After the digital employee launches a browser it marks the main window it detects. From then on, a window selector that looks like a browser target (its name contains Chrome / Chromium / Edge / Firefox / Browser, or its class is Chrome_WidgetWin_* / MozillaWindowClass) only matches marked windows; a Chrome window opened manually by the user is never mistaken for the target.
6. find: locate an elementCore
Searches descendants of the located window and stores the result in an $element variable for later actions.
find type:edit name:"地址栏" timeout:10000 -> $addr
find name~:"搜索" type:button -> $searchBtn
find near:$searchBtn type:text side:below gap:8 -> $hintRequires "-> $var"; on a miss it retries every 200ms until timeout: expires, then returns E_ELEMENT_NOT_FOUND.
optional: tolerate a miss
find type:button name~:"上传" optional timeout:5000 -> $up
if $up == "" {
log "没有上传按钮,跳过配图"
}With optional a miss is not an error: the variable becomes an empty string and later if-checks can take a fallback path. The element also stores its name text for logs and checks; unnamed elements store 1 (meaning found) to avoid false empty results.
7. Visual fallback: template matching & OCR
find image runs local template matching against the captured window (locate an anchor image) and find text uses Windows OCR to find on-screen text. Both store the matched physical screen point as "x,y" in a variable, which click at $pt can then click — ideal when the element tree cannot give coordinates.
find image:"compose.png" region:$panel min:0.88 scales:0.8~1.25 -> $pt
click at $pt
find text:"导出" region:$panel min:0.8 timeout:5000 -> $txtPt
click at $txtPtParameters
| Parameter | Description |
|---|---|
| image:"anchor.png" | Anchor path: absolute, ~/ (user home) or relative; relative paths resolve inside the run storage directory (take a shot first, then crop) |
| text:"导出" | Text to find; matched against OCR output by similarity |
| region:$container | Locate a container element with find first to limit the search to its visible rect; omitted, the whole window is searched |
| min:0.88 | Similarity threshold: 0.88 for image, 0.8 for text |
| scales:0.8~1.25 | Anchor scale range for DPI differences; image only |
| timeout:ms | Retries every 200 ms until it matches, capped by the capability timeout |
| optional | A timeout is not an error: the variable becomes an empty string for an if fallback |
Preparing an anchor image
An anchor is a PNG crop of the target area — keep only the button or icon, avoid large backgrounds, and stay under roughly 300×300. Take a shot of the window and crop it, or export from the design file; widen scales when system scaling differs.
8. click: click an element
Target is an $element variable or an inline selector (selectors also accept under / next / prev / near). On the default natural level a real mouse moves along a track into the visible area and clicks; the hit is then verified with ElementFromPoint — a miss is retried at the new position or handled via UIA patterns (Invoke → Toggle → SelectionItem → LegacyIAccessible) instead of failing hard. light / off use patterns directly; light still moves in and hovers first.
click $searchBtn
click name:"发送" type:button
click name~:"确定" index:2click at: click by coordinate
When an element cannot be located, click by coordinate in three forms: click at 120,340 for a physical screen point; click at $el 50%,30% for a relative position inside an element's visible rect (0–100, vertical centred when omitted); click at $pt for points produced by find image / find text. Coordinate clicks keep the foreground check.
click at 120,340
click at $canvas 50%,30%
find image:"play.png" -> $pt
click at $pt9. focus: focus an element
Only moves focus to the element (SetFocus) without touching content or caret position — useful before sending hotkeys.
focus $editor
key ctrl+home
type "开头的话"10. set: write textCore
On the default natural level the control is focused and written: focus goes to the control first, and a real click inside the visible area is added only when SetFocus did not get keyboard focus; ≥40 chars are pasted in clipboard chunks, shorter text is typed character by character. The value is read back automatically — an empty or clearly short result is retried with the other input method, and a still-mismatched result fails with E_ACTION_FAILED so a silently empty write is never reported as success. light / off try ValuePattern.SetValue first, then fall back to real input when unsupported or ineffective. Line breaks are kept as-is with no extra blank lines (\r\n, a lone \r and \n all count as one break). Text is written line by line: each line is pasted or typed, and real newline keys are sent between lines (Shift+Enter by default, so chat editors never submit early) — Slate / Draft editors do not split pasted \n into paragraphs, so clipboard newlines alone collapse into one line. Editors that only accept a bare Enter can opt in with "newlineKey": "enter" in the capability config.
set $addr "https://example.com"
set name:"搜索框" "搜索 {kw}"
set $title $text11. type: type text
Injects text into the focused control via Unicode, so CJK and symbols work without an IME; ≥40 chars switch to chunked clipboard paste (12–25 chars per chunk), shorter text is typed with a human rhythm, and newlines send a real Enter. The text may come from a variable: type $var. Focus is verified first: if it still sits in the browser address bar the step fails with E_ACTION_FAILED instead of turning the text into a URL.
type "你好,世界"
type "{kw}"
type $text12. key: hotkeys
Keys separated by plus signs with the main key last; supports ctrl / shift / alt / enter / tab / esc / space / delete / backspace / home / end / arrows / pageup / pagedown and single letters/digits.
key ctrl+l
key enter
key ctrl+shift+tab13. scroll: wheel
Scrolls the wheel at the window centre: scroll up/down with a step count (integer or {param}); each command is capped at 100 steps and defaults to 3.
scroll down 3
scroll up {max_scroll}14. get: read textCore
Hyperlinks read Name first (their Value is the URL); otherwise ValuePattern → TextPattern → Name. The result goes into a $text variable for log / emit / if and text processing.
get $addr -> $url
get name~:"结果" -> $result
emit answer $resultLike find, "-> $var" is required.
15. collect: collect text
Collects the text of every matching element into one $text variable joined by semicolons; hyperlinks also use Name. Selectors accept under / next / prev / near.
collect type:hyperlink -> $links
collect type:text after:"提示词" until:"复制" timeout:15000 -> $answerWith after: / until: the result is the slice between the anchor and the stop marker, joined by newlines. They are text anchors (a /regex/ is allowed), not element variables; while a page streams, the engine waits for two identical samples before returning.
16. expand: reveal all content
For virtual-scrolling editors / threads: finds a small unnamed button above-right of the input (a down arrow) and clicks it so history renders fully; after 10 seconds it falls back to Ctrl+End.
find type:edit index:1 -> $box
expand $box
collect type:text after:"提示词" until:"复制" -> $body17. hover / drag: hover and drag
hover moves the real mouse to the element centre and pauses without clicking, for hover menus, overlays and tooltips; drag $from to $to presses, moves in segments and releases, for sliders, list sorting and drag-and-drop uploads. Both accept $element variables and inline selectors.
hover $menu
find type:listitem name~:"导出报表" -> $item
click $item
drag $slider to $target
drag type:"listitem" name~:"卡片 1" to type:"list" name~:"回收站"18. select / toggle: dropdowns and switches
select $el "option text" (or a $variable) expands the dropdown, matches the option text in its subtree, and falls back to a real click when needed; toggle $el on|off reads the current state first and does nothing when it already matches — true/false and 1/0 are accepted.
select $category "发布能力"
select $city $targetCity
toggle $remember on
toggle $switch false19. clipboard: clipboard
clipboard get -> $var reads the clipboard text; clipboard set "text" or clipboard set $var writes it. Useful for controls whose text cannot be read directly, or for pasting long text into an editor in one go.
clipboard get -> $code
check $code matches:/^\d{6}$/
clipboard set "已处理:{name}"
clipboard set $text20. dialog: native dialogs
Handles native system dialogs on the window selected by window (bringing it to the foreground) without locating the dialog itself: dialog accept presses Enter, dialog dismiss presses Esc. Typical cases are download confirmations, certificate warnings and exit prompts.
click $download
wait visible class:"#32770" timeout:15000
dialog accept
wait gone class:"#32770" timeout:10000 -> $closed21. url: read the current address
url -> $var reads the current page address from the browser address bar through UIA without any browser API — useful for verifying navigation or persisting links.
window name~:"Chrome"
url -> $link
check $link contains "/publish"
append "~/published-urls.txt" $link22. tab: browser tabs
tab list -> $var lists tab titles; tab switch 2 or tab switch "title" activates a tab; tab new ["URL"] opens one; tab close [index] closes the given tab (the current one when omitted). Relies on the browser exposing its tab strip to UIA.
tab list -> $tabs
tab new "https://example.com"
tab switch "工作台"
wait stable type:document stable:1000 timeout:60000 -> $body
tab close 123. download wait: wait for a download
download wait timeout:60000 -> $var polls the browser download folder, skips temporary files such as .crdownload / .part, only accepts files written around this run, and returns the absolute path once the size settles.
click $export
wait gone type:progressbar timeout:120000
download wait timeout:180000 -> $file
log "已下载:$file"24. read / append / write: text files
read "path" -> $var reads text, and with lines splits it into a list (empty lines skipped); append "path" "text" appends one line (newlines are folded into spaces); write "path" $list overwrites with one item per line, and an empty list truncates the file. Paths accept absolute, ~/ and %USERPROFILE%; relative paths resolve against the user home.
read "~/queue.txt" lines -> $queue
first $queue -> $item
append "~/done.txt" "{name}\t$item"
drop $queue $item -> $rest
write "~/queue.txt" $rest25. post / json: HTTP requests and JSON fields
post "url" "{\"key\":$var}" -> $resp sends a JSON POST; non-2xx responses fail and retry 3 adds automatic retries. json $resp "data.items" -> $value reads by dot path, joining arrays into a list and yielding an empty string for a missing path — handy for enriching runs with your own API.
post "https://api.example.com/tasks" "{\"page\":$page}" retry 2 -> $resp
json $resp "data.items" -> $items
len $items -> $n
foreach $item in $items {
log "任务:$item"
}26. let: store a value
Stores a literal or another variable into a $text variable, handy for taking a copy before repeated processing.
let $raw "{body}"
let $copy $raw27. extract: regex slice
Slices a variable with /regex/: the first capture group when present, otherwise the whole match; a total miss keeps the original value.
let $raw "{body}"
extract $raw /^([^;]+)/ -> $head
extract $raw /^[^;]+;?/ -> $rest28. matches: all matches
Finds every /regex/ match (the whole match, not capture groups), joined by semicolons; no match yields an empty string.
let $raw "{body}"
matches $raw "/#[^#,。!?;:、]+/" -> $tags
foreach $t in $tags {
log "标签:$t"
}29. strip: remove matches
Removes every /regex/ match from a variable — commonly used to strip hashtags out of a body.
let $raw "{body}"
strip $raw "/#[^#,。!?;:、]+/" -> $plain
set $editor $plain30. List helpers: first / drop / push / len / inc
Lists are structured variables produced by read lines / collect and friends: first $list -> $item takes the head, len $list -> $n counts items, drop $list $item -> $rest removes the first match, push $list value appends one, and inc $var 2 increments an integer (missing variables start at 0). Logs and outputs render lists joined by ";".
read "~/queue.txt" lines -> $queue
len $queue -> $n
first $queue -> $item
drop $queue $item -> $rest
write "~/queue.txt" $rest
push $done $item
inc $page {page_size}31. newest: newest image
Returns the newest image path in a folder. An empty folder argument means %USERPROFILE%\Downloads; it polls for 30 seconds and accepts only files written in the last 5 minutes (png / jpg / jpeg / webp / gif).
By default a 30-second timeout raises E_ELEMENT_NOT_FOUND and aborts the script. When the image is optional, append optional: on timeout the variable is set to an empty string and execution continues (the log states that no new image appeared), so the script can branch with if $img == "". Wrapping newest in an if block does not help: errors inside a block still abort.
# 必需:等不到就中止
newest "" -> $img
# 可选:等不到就置空并继续
newest "" -> $img optional
if $img == "" {
log "本次没有配图"
}32. wait: sleep
wait 800
wait {wait_ms}Sleeps the given milliseconds (capped at 600000ms per call). Prefer find / collect timeout: for conditional waits over fixed sleeps.
33. log: run log
log "开始执行"
log "关键词:{kw},结果:$result"Writes to the work log streamed in real time; supports {param} and $variable interpolation.
34. if: conditional
A condition is left operand, operator and right operand (each a $variable or literal); operators are ==, !=, contains. Blocks open with a trailing { and close with } on its own line.
get $addr -> $url
if $url contains "example.com" {
log "已在目标站点"
}
if $up == "" {
log "按钮不存在,走备用路径"
}35. repeat: loop
The count is an integer or a {param}; a single run is capped at 1000 iterations. break inside the body ends it early.
repeat 3 {
key tab
wait 300
}36. foreach: iterate a list
Splits the list variable on semicolons (trimming blanks and skipping empty items), stores the current item in $item each round, and scopes the item variable to the body.
let $raw "{body}"
matches $raw "/#[^#,。!?;:、]+/" -> $tags
foreach $t in $tags {
log "标签:$t"
}37. break: break a loop
Breaks the nearest repeat / foreach; inside an if it propagates to the enclosing loop.
repeat 10 {
get name~:"状态" -> $s
if $s contains "完成" {
break
}
wait 500
}38. emit: write outputs
Writes a $text variable or literal into capability outputs; the name must match an @output declaration and lands in the run result for later nodes.
emit result $result
emit site "示例站点"39. shot: screenshot
Captures the located window as PNG into the run storage directory; the output name is the file name and the value is the absolute path.
shot "step1.png"40. goto: open a URL and confirm it loadedCore
Opens an address in the located browser window: it locates the address bar via UIA, clicks to focus it, types the URL and reads it back before pressing Enter. If the text did not land it presses Esc to dismiss popups such as "Restore pages?" and retries (up to 3 times); if it still fails it returns E_ACTION_FAILED without pressing Enter. The first goto also waits config.humanize.gotoSettleMs (5000 ms by default, 0 disables), letting welcome pages and prompts finish. A trailing space is appended to the typed URL so the address bar does not rewrite it with history/suggestion matches (which could open a different address on Enter); the read-back comparison ignores that space and navigation is unaffected.
@input url 目标网址 =https://example.com
@input expect 标题关键词 =示例
window name:/.../ timeout:8000
goto "{url}" expect:"{expect}" timeout:20000expect accepts plain text or /regex/ and asserts the window title like window expect before continuing; a timeout returns E_NAVIGATE_TIMEOUT. The address may come from a variable: goto $url.
41. invoke: trigger via patterns
Triggers the control through UIA patterns (Invoke → Toggle → SelectionItem → LegacyIAccessible) without moving the mouse. It is the right choice for controls that real clicks cannot hit: Chrome's hidden <input type=file> spans several toolbar icons and is skipped by hit testing, so click is a gamble while invoke always opens the file dialog. On failure it re-locates the element once and retries.
find type:button name~:"上传" -> $up
invoke $up42. check: assert content and stateCore
check is a read-only assertion that never moves focus or changes content. Content comparisons support == (ignoring zero-width and whitespace differences), contains, ~= (edit-distance similarity, tolerance:0–1, default 0.9) and matches:/regex/; state checks support exists:1 / visible:1 / enabled:1 / focused:1 / checked:1; counts use check $n >= 2.
find type:edit index:1 -> $editor
get $editor -> $text
check $text contains "已发布"
check $text ~= "{expected}" tolerance:0.9
check $editor exists:1 visible:1
count type:listitem visible:1 -> $n
check $n >= 143. count: count matching elements
Counts matching elements into a $text variable, typically to confirm a list really rendered its rows. With timeout: it polls every 200 ms until a match appears; without it the count is a single snapshot and zero is not an error. limit:n caps how many are counted.
count type:listitem visible:1 timeout:5000 -> $n
log "列表项数量:$n"
check $n >= 144. retry: retry a block on failure
retry n { … } retries the whole block: any failing statement restarts it from the top, up to n times (integer or {param}, capped at 1000). Each failure is logged separately and the last error is raised after all attempts. Unlike repeat, retry only restarts after a failure.
retry 3 {
find name:"保存" type:button timeout:5000 -> $btn
click $btn
wait stable name:"已保存" timeout:5000 -> $state
}
if $state contains "已保存" {
emit result "ok"
}45. wait visible / gone / stable
wait visible <target> [timeout:ms] waits until the element exists and is visible (10000 ms default); wait gone <target> [timeout:ms] [-> $ok] waits until it disappears or the window closes (5000 ms default), and with -> $ok it becomes a check: 1 when gone, 0 on timeout without aborting; wait stable <target> [stable:ms] [timeout:ms] [-> $out] waits until the text stays unchanged and non-empty for stable ms (800 default, 20000 timeout) and can store the settled text.
click $up
window class:"#32770" timeout:8000
type "C:\tmp\cover.png"
key enter
window back
wait gone class:"#32770" timeout:20000 -> $closed
if $closed != "1" {
log "文件对话框没有关闭,上传可能未提交"
}wait stable type:document stable:1500 timeout:90000 -> $answer
log "回答已稳定:$answer"46. Humanization (on by default)
Humanization lives in the engine and applies without touching scripts: the mouse follows a cubic Bézier path (slower at both ends, with jitter and overshoot), the landing point is never the centre; text is typed with a human rhythm and long text is pasted in chunks; the wheel scrolls in segments; window switches, navigations and page loads pause by level; explicit waits never shorten. Each run logs the active level on its first line.
"humanize": {
"level": "natural", # off | light | natural | careful(默认 natural)
"speed": 1.0, # 节奏倍率 0.3~3.0,越大越快
"mouse": true,
"typing": true,
"dwell": true,
"pasteOver": 40, # ≥ 该字数走粘贴
"gotoSettleMs": 5000, # 首次 goto 前等待浏览器稳定
"seed": 0
}Keeping the screen awake and lock detection
Real input injected by channel B is discarded while the display is off or the machine sleeps (the script finishes but nothing is written). Each capability therefore blocks display-off and sleep by default (which also blocks idle auto-lock) and restores power behaviour on exit; a dark display is woken first. Both are capability config keys: keepAwake (default true) and allowLockedScreen (default false).
"keepAwake": true, # 执行期阻止息屏/睡眠,结束恢复
"allowLockedScreen": false, # 锁屏时直接报 E_SESSION_LOCKED47. Reference recipes: verify first
The snippets below are copy-paste ready. The rule: follow every action that "looks successful" with a check / wait so a click that did nothing is never reported as success.
Open → write → send → confirm
@input url 目标网址
@input body 正文
window name:/.../ timeout:8000
goto "{url}" expect:/.../ timeout:20000
find type:edit index:1 -> $editor
set $editor "{body}"
check $editor exists:1
find name:/发送|Send/ type:button timeout:5000 -> $send
click $send
wait gone type:edit index:1 timeout:15000 -> $closed
if $closed != "1" {
log "编辑器没有收起,发送可能未生效"
}Upload an image and confirm the dialog closed
newest "" -> $img
find type:button name~:"上传" -> $up
invoke $up
window class:"#32770" timeout:8000
type "$img"
key enter
window back
wait gone class:"#32770" timeout:20000 -> $closed
check $closed == "1"Streaming answer: wait until settled, then read
wait stable type:document stable:1500 timeout:90000 -> $answer
collect type:text after:"提示词" until:"复制" timeout:15000 -> $body
extract $body /^([^;]+)/ -> $head
emit reply $body48. Error code reference
| Code | Meaning | What to check |
|---|---|---|
| E_SCRIPT_PARSE | Script failed to parse | Read the line number in the log; check } pairing, selector keys, -> $var, variable definition order and @output declarations |
| E_PARAM_INVALID | Invalid or missing parameter | Check @input declarations and caller params; unsupported key names; numeric params (wait/repeat/scroll) must be non-negative integers |
| E_UIA_INIT | UIA initialisation failed | Check the UIA item in Settings → Environment check; restart the engine |
| E_WINDOW_NOT_FOUND | Window not found | Make sure the app is open; use name~: or regex for fuzzy titles; window back needs a preceding window |
| E_WINDOW_MISMATCH | Window content mismatch | The expect: assertion did not match before the timeout: check that the page finished loading and the title is correct; raise timeout: or use a more stable keyword |
| E_WINDOW_CLOSED | Window closed | The target window closed mid-run: run window again so the script re-locates or reopens the browser |
| E_WINDOW_NOT_FOREGROUND | Window cannot be foregrounded | A full-screen or elevated window holds the foreground: close it and retry; keyboard input is never sent blindly |
| E_ELEMENT_NOT_FOUND | Element not found | Verify name and type; raise timeout:; add index: or use next/prev/near; watch for the scan-cap warning |
| E_NAVIGATE_TIMEOUT | goto timed out | The expect title did not appear before the timeout: check the URL, that the page opens and the title keyword is stable; raise goto's timeout: if needed |
| E_WAIT_TIMEOUT | Conditional wait timed out | wait visible / gone / stable did not satisfy its condition in time: raise timeout: or use a more stable target (gone / stable with -> $var do not abort) |
| E_CHECK_FAILED | Assertion failed | A hard assertion failed: the error includes expected/actual and the first difference; confirm the previous write really landed, then relax to ~= or add timeout: polling |
| E_CHECK_UNSUPPORTED | Control cannot be checked | The target has no ValuePattern / TextPattern (content unreadable) or no TogglePattern (checked unreadable); switch to get + text checks |
| E_TIMEOUT | Capability timed out | Exceeded config.maxMs (30 min by default, 0 unlimited): split long runs or raise the budget; per-statement timeout: is also capped by the remaining budget |
| E_SESSION_LOCKED | Session locked | The screen is locked (Win+L / lock screen). The lock screen is the Winlogon secure desktop, simulated input is discarded and user-mode apps cannot unlock it, so the run aborts with this hint; set config.allowLockedScreen=1 to skip the check |
| E_IMPL_RETIRED | Implementation retired | A retired browser-automation capability was called: rewrite it as a UIA script |
| E_ACTION_FAILED | Action failed | Element may be hidden or disabled; click or focus it before acting |
49. Example: write into Notepad
@input text 写入内容 =数字员工 UIA 脚本
@output body 写入后的内容
window name~:"记事本"
set name~:"文本编辑器" "{text}"
get name~:"文本编辑器" -> $read
emit body $readNotepad's edit area may be named "Text editor" or "Document" depending on version — name~: is safer. Interpolate the parameter with {text}.
50. Example: open a URL in Chrome
@input url 目标网址 =https://example.com
@output after-open.png 截图路径
window name~:"Chrome"
key ctrl+l
type "{url}"
key enter
wait 1200
shot "after-open.png"The address bar can also use channel A: find type:edit name~:"Address bar" -> $addr then set $addr "{url}".
51. Example: relative lookup and optional element
@input anchor 起始文本 =提示词
@input stop 结束文本 =复制
@output body 读取到的正文
window name~:"Chrome"
find type:text name~:"{anchor}" -> $a
find near:$a type:button side:below gap:8 optional timeout:5000 -> $more
if $more != "" {
click $more
}
collect type:text after:"{anchor}" until:"{stop}" timeout:15000 -> $body
emit body $bodyLocate the anchor with find, then use near to find the button below it. The button may be absent, so optional plus an if-check handles it. collect slices the answer by text anchors without relying on fixed indices.
52. Example: poll a status and stop early
@input expect 期望文本 =完成
@output status 命中时的状态
@output done 是否命中
let $done "0"
repeat 10 {
get name~:"状态" -> $s
if $s contains "{expect}" {
let $done "1"
emit status $s
break
}
wait 600
}
emit done $donebreak ends repeat immediately; the done output tells the caller whether it matched, so no extra text check is needed.
