commit a2171a828859bdf299014f87a82951c897958cbc Author: ye.yp <1436237912@qq.com> Date: Fri Mar 13 20:46:34 2026 +0800 init diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..00f2d38 --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +__pycache__/ +*.pyc +.venv/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..f91df4c --- /dev/null +++ b/README.md @@ -0,0 +1,119 @@ +# AgentScope Skills Demo + +这是一个最小可运行示例，演示如何： + +- 使用 llama.cpp 的 OpenAI 兼容接口连接 `OpenAIChatModel` +- 为 `ReActAgent` 注册多个本地 skill +- 让智能体在任务中优先使用 skill，并通过 Python 工具执行代码 +- 在 Windows 桌面上执行截图、点击、输入等 GUI 自动化操作 + +## 文件结构 + +```text +. +├── demo_with_skills.py +├── requirements.txt +└── skills/ + ├── python_hello_skill/ + │ └── SKILL.md + └── windows_gui_operator/ + └── SKILL.md +``` + +运行 GUI skill 后，截图会保存在 `artifacts/screenshots/` 目录。 + +## 安装依赖 + +```bash +pip install -r requirements.txt +``` + +## 运行示例 + +```bash +python demo_with_skills.py +``` + +默认会进入交互式对话模式，你可以在终端中持续输入消息与模型对话： + +```text +user> 你好 +user> 请用 Python 打印 Hello World +user> /exit +``` + +脚本默认会隐藏模型的 thinking 过程，只显示最终可见回复。如果你想调试 thinking 输出，可以运行： + +```bash +python demo_with_skills.py --show-thinking +``` + +默认模式下，工具调用过程中的 `tool_use` 和 `tool_result` JSON 也会被隐藏，只保留最终自然语言回复。 + +当前版本会以可读形式打印 ReAct 过程，例如 `Action` 和 `Observation`，但不会直接输出原始 thinking 块和原始 JSON。 + +如果你想检查模型每一轮 assistant 原始返回了什么 block，可以运行： + +```bash +python demo_with_skills.py --debug-model-output +``` + +这会额外打印每轮模型的原始 content blocks，便于确认截图返回后模型是否真的在消费图片相关内容。 + +## Windows GUI Skill + +新增的 `windows_gui_operator` skill 会要求智能体按 ReAct 方式执行 GUI 操作： + +- 先获取当前活动窗口并截图 +- 再执行鼠标或键盘动作 +- 关键步骤后再次截图确认 +- 对输入类操作，必要时通过剪贴板做结果确认 + +当前脚本已注册以下 GUI 工具： + +- `save_desktop_screenshot` +- `get_mouse_position` +- `left_click` +- `type_text` +- `press_keys` +- `hotkey` +- `wait_seconds` +- `get_active_window_title` +- `get_clipboard_text` + +如果当前模型支持多模态，`save_desktop_screenshot` 会把本地截图文件同时作为图片结果返回给模型，模型可以直接基于截图继续判断下一步动作。 + +GUI skill 默认采用“观察 -> 执行一个最小动作 -> 复查 -> 必要时重试”的单步闭环，而不是预先规划固定长步骤后一路执行。 + +当前版本对 Windows GUI 的约束是：桌面任务必须通过键盘和鼠标完成，不提供直接启动应用或直接打开网址的非 GUI 工具。 + +如果你只想发送一条消息并立即退出，可以运行： + +```bash +python demo_with_skills.py --message "请用 Python 打印 Hello World" +``` + +如果你只想确认 skill 注册是否成功，而不立即调用远程模型，可以运行： + +```bash +python demo_with_skills.py --skip-agent-run +``` + +如果你需要替换 llama.cpp 服务地址或模型名，可以通过环境变量覆盖： + +```bash +set LLAMA_CPP_BASE_URL=http://127.0.0.1:8000/v1 +set LLAMA_CPP_MODEL_NAME=your-model-name +set LLAMA_CPP_API_KEY=EMPTY +python demo_with_skills.py +``` + +## 说明 + +示例脚本默认使用以下 llama.cpp OpenAI 兼容接口配置，并支持通过环境变量覆盖： + +- Base URL: `http://yyplab.site:8033/v1` +- Model: `Qwen3.5-122B-A10B-Q4_K_M-00001-of-00003.gguf` +- API Key: `EMPTY` + +如果你本地的 AgentScope 版本中，文件读取工具名称与脚本里的候选名称不同，请在 `demo_with_skills.py` 中补充对应函数名。 diff --git a/artifacts/screenshots/20260313_153941_025978_before_opening_chrome.png b/artifacts/screenshots/20260313_153941_025978_before_opening_chrome.png new file mode 100644 index 0000000..26989bf Binary files /dev/null and b/artifacts/screenshots/20260313_153941_025978_before_opening_chrome.png differ diff --git a/artifacts/screenshots/20260313_154918_142890_before_opening_chrome.png b/artifacts/screenshots/20260313_154918_142890_before_opening_chrome.png new file mode 100644 index 0000000..20a3004 Binary files /dev/null and b/artifacts/screenshots/20260313_154918_142890_before_opening_chrome.png differ diff --git a/artifacts/screenshots/20260313_160434_436113_verification.png b/artifacts/screenshots/20260313_160434_436113_verification.png new file mode 100644 index 0000000..625556a Binary files /dev/null and b/artifacts/screenshots/20260313_160434_436113_verification.png differ diff --git a/artifacts/screenshots/20260313_161750_181053_bing_search_page.png b/artifacts/screenshots/20260313_161750_181053_bing_search_page.png new file mode 100644 index 0000000..47765f0 Binary files /dev/null and b/artifacts/screenshots/20260313_161750_181053_bing_search_page.png differ diff --git a/artifacts/screenshots/20260313_164459_060540_initial_state.png b/artifacts/screenshots/20260313_164459_060540_initial_state.png new file mode 100644 index 0000000..607b185 Binary files /dev/null and b/artifacts/screenshots/20260313_164459_060540_initial_state.png differ diff --git a/artifacts/screenshots/20260313_164635_045328_after_launch_chrome.png b/artifacts/screenshots/20260313_164635_045328_after_launch_chrome.png new file mode 100644 index 0000000..6fb802d Binary files /dev/null and b/artifacts/screenshots/20260313_164635_045328_after_launch_chrome.png differ diff --git a/artifacts/screenshots/20260313_170818_906226_initial_state.png b/artifacts/screenshots/20260313_170818_906226_initial_state.png new file mode 100644 index 0000000..d87ef2c Binary files /dev/null and b/artifacts/screenshots/20260313_170818_906226_initial_state.png differ diff --git a/artifacts/screenshots/20260313_170951_432428_after_launch_chrome.png b/artifacts/screenshots/20260313_170951_432428_after_launch_chrome.png new file mode 100644 index 0000000..fad0417 Binary files /dev/null and b/artifacts/screenshots/20260313_170951_432428_after_launch_chrome.png differ diff --git a/artifacts/screenshots/20260313_171248_625142_current_state_retry.png b/artifacts/screenshots/20260313_171248_625142_current_state_retry.png new file mode 100644 index 0000000..59bfeb4 Binary files /dev/null and b/artifacts/screenshots/20260313_171248_625142_current_state_retry.png differ diff --git a/artifacts/screenshots/20260313_171337_704962_after_close_powershell.png b/artifacts/screenshots/20260313_171337_704962_after_close_powershell.png new file mode 100644 index 0000000..f92a7e7 Binary files /dev/null and b/artifacts/screenshots/20260313_171337_704962_after_close_powershell.png differ diff --git a/artifacts/screenshots/20260313_172605_878486_initial_state.png b/artifacts/screenshots/20260313_172605_878486_initial_state.png new file mode 100644 index 0000000..e12ff54 Binary files /dev/null and b/artifacts/screenshots/20260313_172605_878486_initial_state.png differ diff --git a/artifacts/screenshots/20260313_172726_238483_after_launch_chrome.png b/artifacts/screenshots/20260313_172726_238483_after_launch_chrome.png new file mode 100644 index 0000000..1a862b5 Binary files /dev/null and b/artifacts/screenshots/20260313_172726_238483_after_launch_chrome.png differ diff --git a/artifacts/screenshots/20260313_173252_094341_after_second_chrome_attempt.png b/artifacts/screenshots/20260313_173252_094341_after_second_chrome_attempt.png new file mode 100644 index 0000000..c31026b Binary files /dev/null and b/artifacts/screenshots/20260313_173252_094341_after_second_chrome_attempt.png differ diff --git a/artifacts/screenshots/20260313_174750_800070_initial_state.png b/artifacts/screenshots/20260313_174750_800070_initial_state.png new file mode 100644 index 0000000..504075b Binary files /dev/null and b/artifacts/screenshots/20260313_174750_800070_initial_state.png differ diff --git a/artifacts/screenshots/20260313_174828_851736_after_win_key.png b/artifacts/screenshots/20260313_174828_851736_after_win_key.png new file mode 100644 index 0000000..36cf51c Binary files /dev/null and b/artifacts/screenshots/20260313_174828_851736_after_win_key.png differ diff --git a/artifacts/screenshots/20260313_174910_971401_after_type_chrome.png b/artifacts/screenshots/20260313_174910_971401_after_type_chrome.png new file mode 100644 index 0000000..68c679e Binary files /dev/null and b/artifacts/screenshots/20260313_174910_971401_after_type_chrome.png differ diff --git a/artifacts/screenshots/20260313_174942_808197_after_enter_chrome.png b/artifacts/screenshots/20260313_174942_808197_after_enter_chrome.png new file mode 100644 index 0000000..5afcdd3 Binary files /dev/null and b/artifacts/screenshots/20260313_174942_808197_after_enter_chrome.png differ diff --git a/artifacts/screenshots/20260313_175403_231915_check_current_state.png b/artifacts/screenshots/20260313_175403_231915_check_current_state.png new file mode 100644 index 0000000..3c7e9f3 Binary files /dev/null and b/artifacts/screenshots/20260313_175403_231915_check_current_state.png differ diff --git a/artifacts/screenshots/20260313_175451_933505_after_second_win_key.png b/artifacts/screenshots/20260313_175451_933505_after_second_win_key.png new file mode 100644 index 0000000..5eb916d Binary files /dev/null and b/artifacts/screenshots/20260313_175451_933505_after_second_win_key.png differ diff --git a/artifacts/screenshots/20260313_175533_294892_after_type_chrome_second.png b/artifacts/screenshots/20260313_175533_294892_after_type_chrome_second.png new file mode 100644 index 0000000..ea5dc37 Binary files /dev/null and b/artifacts/screenshots/20260313_175533_294892_after_type_chrome_second.png differ diff --git a/artifacts/screenshots/20260313_175559_502498_after_enter_chrome_second.png b/artifacts/screenshots/20260313_175559_502498_after_enter_chrome_second.png new file mode 100644 index 0000000..45f0279 Binary files /dev/null and b/artifacts/screenshots/20260313_175559_502498_after_enter_chrome_second.png differ diff --git a/artifacts/screenshots/20260313_184636_815085_observe_initial.png b/artifacts/screenshots/20260313_184636_815085_observe_initial.png new file mode 100644 index 0000000..f3bac53 Binary files /dev/null and b/artifacts/screenshots/20260313_184636_815085_observe_initial.png differ diff --git a/artifacts/screenshots/20260313_184736_174539_observe_after_win_key.png b/artifacts/screenshots/20260313_184736_174539_observe_after_win_key.png new file mode 100644 index 0000000..6d22289 Binary files /dev/null and b/artifacts/screenshots/20260313_184736_174539_observe_after_win_key.png differ diff --git a/artifacts/screenshots/20260313_184842_035874_observe_after_chrome_launch.png b/artifacts/screenshots/20260313_184842_035874_observe_after_chrome_launch.png new file mode 100644 index 0000000..8ba1770 Binary files /dev/null and b/artifacts/screenshots/20260313_184842_035874_observe_after_chrome_launch.png differ diff --git a/artifacts/screenshots/20260313_184942_218905_observe_after_alt_tab.png b/artifacts/screenshots/20260313_184942_218905_observe_after_alt_tab.png new file mode 100644 index 0000000..848179a Binary files /dev/null and b/artifacts/screenshots/20260313_184942_218905_observe_after_alt_tab.png differ diff --git a/artifacts/screenshots/20260313_185047_898776_observe_after_second_alt_tab.png b/artifacts/screenshots/20260313_185047_898776_observe_after_second_alt_tab.png new file mode 100644 index 0000000..7d63f43 Binary files /dev/null and b/artifacts/screenshots/20260313_185047_898776_observe_after_second_alt_tab.png differ diff --git a/artifacts/screenshots/20260313_185139_914658_observe_after_third_alt_tab.png b/artifacts/screenshots/20260313_185139_914658_observe_after_third_alt_tab.png new file mode 100644 index 0000000..8a596fa Binary files /dev/null and b/artifacts/screenshots/20260313_185139_914658_observe_after_third_alt_tab.png differ diff --git a/artifacts/screenshots/20260313_190457_515687_observe_after_fourth_alt_tab.png b/artifacts/screenshots/20260313_190457_515687_observe_after_fourth_alt_tab.png new file mode 100644 index 0000000..408969f Binary files /dev/null and b/artifacts/screenshots/20260313_190457_515687_observe_after_fourth_alt_tab.png differ diff --git a/artifacts/screenshots/20260313_190635_497926_observe_after_click_taskbar.png b/artifacts/screenshots/20260313_190635_497926_observe_after_click_taskbar.png new file mode 100644 index 0000000..ed72d1e Binary files /dev/null and b/artifacts/screenshots/20260313_190635_497926_observe_after_click_taskbar.png differ diff --git a/artifacts/screenshots/20260313_190703_803916_observe_after_click_taskbar_result.png b/artifacts/screenshots/20260313_190703_803916_observe_after_click_taskbar_result.png new file mode 100644 index 0000000..e4f0343 Binary files /dev/null and b/artifacts/screenshots/20260313_190703_803916_observe_after_click_taskbar_result.png differ diff --git a/artifacts/screenshots/20260313_191915_091004_observe_initial.png b/artifacts/screenshots/20260313_191915_091004_observe_initial.png new file mode 100644 index 0000000..cc633dc Binary files /dev/null and b/artifacts/screenshots/20260313_191915_091004_observe_initial.png differ diff --git a/artifacts/screenshots/20260313_192008_351790_after_win_key.png b/artifacts/screenshots/20260313_192008_351790_after_win_key.png new file mode 100644 index 0000000..e81fa81 Binary files /dev/null and b/artifacts/screenshots/20260313_192008_351790_after_win_key.png differ diff --git a/artifacts/screenshots/20260313_192116_385684_after_chrome_launch.png b/artifacts/screenshots/20260313_192116_385684_after_chrome_launch.png new file mode 100644 index 0000000..21d6bfe Binary files /dev/null and b/artifacts/screenshots/20260313_192116_385684_after_chrome_launch.png differ diff --git a/artifacts/screenshots/20260313_192157_888669_check_chrome_status.png b/artifacts/screenshots/20260313_192157_888669_check_chrome_status.png new file mode 100644 index 0000000..dbecd0b Binary files /dev/null and b/artifacts/screenshots/20260313_192157_888669_check_chrome_status.png differ diff --git a/artifacts/screenshots/20260313_192326_382264_check_chrome_open.png b/artifacts/screenshots/20260313_192326_382264_check_chrome_open.png new file mode 100644 index 0000000..9949f4e Binary files /dev/null and b/artifacts/screenshots/20260313_192326_382264_check_chrome_open.png differ diff --git a/artifacts/screenshots/20260313_192553_073176_after_ctrl_l.png b/artifacts/screenshots/20260313_192553_073176_after_ctrl_l.png new file mode 100644 index 0000000..8a0124a Binary files /dev/null and b/artifacts/screenshots/20260313_192553_073176_after_ctrl_l.png differ diff --git a/artifacts/screenshots/20260313_193517_133400_observe_initial.png b/artifacts/screenshots/20260313_193517_133400_observe_initial.png new file mode 100644 index 0000000..36686d9 Binary files /dev/null and b/artifacts/screenshots/20260313_193517_133400_observe_initial.png differ diff --git a/artifacts/screenshots/20260313_194217_277472_观察当前桌面状态.png b/artifacts/screenshots/20260313_194217_277472_观察当前桌面状态.png new file mode 100644 index 0000000..68d9a58 Binary files /dev/null and b/artifacts/screenshots/20260313_194217_277472_观察当前桌面状态.png differ diff --git a/artifacts/screenshots/20260313_194250_215147_观察_Chrome_浏览器是否已打开.png b/artifacts/screenshots/20260313_194250_215147_观察_Chrome_浏览器是否已打开.png new file mode 100644 index 0000000..2703166 Binary files /dev/null and b/artifacts/screenshots/20260313_194250_215147_观察_Chrome_浏览器是否已打开.png differ diff --git a/artifacts/screenshots/20260313_194317_262059_观察_Chrome_浏览器是否已打开.png b/artifacts/screenshots/20260313_194317_262059_观察_Chrome_浏览器是否已打开.png new file mode 100644 index 0000000..fa28439 Binary files /dev/null and b/artifacts/screenshots/20260313_194317_262059_观察_Chrome_浏览器是否已打开.png differ diff --git a/artifacts/screenshots/20260313_194350_239941_观察当前浏览器状态.png b/artifacts/screenshots/20260313_194350_239941_观察当前浏览器状态.png new file mode 100644 index 0000000..3f75e94 Binary files /dev/null and b/artifacts/screenshots/20260313_194350_239941_观察当前浏览器状态.png differ diff --git a/artifacts/screenshots/20260313_194507_115789_观察_Chrome_浏览器是否已打开.png b/artifacts/screenshots/20260313_194507_115789_观察_Chrome_浏览器是否已打开.png new file mode 100644 index 0000000..74b9384 Binary files /dev/null and b/artifacts/screenshots/20260313_194507_115789_观察_Chrome_浏览器是否已打开.png differ diff --git a/demo_with_skills.py b/demo_with_skills.py new file mode 100644 index 0000000..5b0e4da --- /dev/null +++ b/demo_with_skills.py @@ -0,0 +1,904 @@ +# -*- coding: utf-8 -*- +"""使用 llama.cpp 的 OpenAI 兼容接口创建带有 skills 的 ReActAgent 示例。""" + +from __future__ import annotations + +import argparse +import asyncio +from datetime import datetime +import json +import os +from pathlib import Path +import subprocess +import time +from typing import Any +from agentscope.agent import ReActAgent +from agentscope.formatter import OpenAIChatFormatter +from agentscope.memory import InMemoryMemory +from agentscope.message import ImageBlock, Msg, TextBlock, URLSource +from agentscope.model import OpenAIChatModel +from agentscope.tool import ToolResponse, Toolkit, execute_python_code + +LLAMA_CPP_BASE_URL = os.getenv("LLAMA_CPP_BASE_URL", "http://yyplab.site:8033/v1") +LLAMA_CPP_MODEL_NAME = os.getenv( + "LLAMA_CPP_MODEL_NAME", + "Qwen3.5-35B-A3B-UD-Q8_K_XL.gguf", +) +LLAMA_CPP_API_KEY = os.getenv("LLAMA_CPP_API_KEY", "EMPTY") +WORKSPACE_DIR = Path(__file__).parent.resolve() +SKILLS_DIR = Path(__file__).parent / "skills" +SCREENSHOT_DIR = Path(__file__).parent / "artifacts" / "screenshots" +DEFAULT_GUI_MAX_STEPS = 60 + + +def _load_pyautogui(): + """延迟导入 pyautogui，避免缺依赖时脚本启动失败。""" + try: + import pyautogui # type: ignore + except ImportError as exc: + raise RuntimeError( + "缺少 pyautogui，请先执行 `pip install -r requirements.txt`。", + ) from exc + + pyautogui.FAILSAFE = True + pyautogui.PAUSE = 0.15 + return pyautogui + + +def _ensure_screenshot_dir() -> Path: + """确保截图目录存在。""" + SCREENSHOT_DIR.mkdir(parents=True, exist_ok=True) + return SCREENSHOT_DIR + + +def normalize_text(value: Any, default: str = "") -> str: + """将模型传入的参数规范化为字符串。""" + if value is None: + return default + if isinstance(value, str): + stripped = value.strip() + return stripped if stripped else default + return str(value) + + +def normalize_float(value: Any, default: float) -> float: + """将模型传入的参数规范化为浮点数。""" + if value is None: + return default + if isinstance(value, str): + stripped = value.strip() + if not stripped: + return default + return float(stripped) + return float(value) + + +def normalize_int(value: Any, default: int) -> int: + """将模型传入的参数规范化为整数。""" + if value is None: + return default + if isinstance(value, str): + stripped = value.strip() + if not stripped: + return default + return int(float(stripped)) + return int(value) + + +def normalize_keys(value: Any) -> list[str]: + """将模型传入的按键参数规范化为字符串列表。""" + if value is None: + return [] + if isinstance(value, str): + stripped = value.strip() + return [stripped] if stripped else [] + if isinstance(value, (list, tuple)): + return [normalize_text(item) for item in value if normalize_text(item)] + normalized = normalize_text(value) + return [normalized] if normalized else [] + + +def make_tool_response(text: str, metadata: dict[str, Any] | None = None) -> ToolResponse: + """构造符合 AgentScope 要求的工具响应。""" + return ToolResponse( + content=[ + TextBlock( + type="text", + text=text, + ), + ], + metadata=metadata, + ) + + +def make_success_response(summary: str, **metadata: Any) -> ToolResponse: + """构造成功响应。""" + payload = "\n".join( + [summary, *(f"{key}={value}" for key, value in metadata.items())], + ) + return make_tool_response(payload, metadata=metadata or None) + + +def make_error_response(action: str, exc: Exception) -> ToolResponse: + """构造失败响应，避免工具异常直接打断会话。""" + return make_tool_response( + f"action={action}\nstatus=error\nerror={type(exc).__name__}: {exc}", + metadata={ + "action": action, + "status": "error", + "error": f"{type(exc).__name__}: {exc}", + }, + ) + + +def make_multimodal_response( + summary: str, + image_path: str, + metadata: dict[str, Any] | None = None, +) -> ToolResponse: + """构造包含截图图片块的工具响应，供多模态模型直接观察。""" + return ToolResponse( + content=[ + TextBlock( + type="text", + text=summary, + ), + ImageBlock( + type="image", + source=URLSource( + type="url", + url=image_path, + ), + ), + ], + metadata=metadata, + ) + + +def read_local_text_file(path: str) -> ToolResponse: + """读取工作区内的文本文件，供 agent 查看 skill 说明。""" + try: + requested_path = Path(normalize_text(path)).expanduser().resolve() + if WORKSPACE_DIR not in requested_path.parents and requested_path != WORKSPACE_DIR: + return make_tool_response( + "action=read_local_text_file\nstatus=error\nerror=Path is outside the workspace.", + metadata={ + "action": "read_local_text_file", + "status": "error", + "path": str(requested_path), + }, + ) + + if not requested_path.is_file(): + return make_tool_response( + "action=read_local_text_file\nstatus=error\nerror=File does not exist.", + metadata={ + "action": "read_local_text_file", + "status": "error", + "path": str(requested_path), + }, + ) + + content = requested_path.read_text(encoding="utf-8") + return make_tool_response( + "\n".join( + [ + "action=read_local_text_file", + "status=ok", + f"path={requested_path}", + content, + ], + ), + metadata={ + "action": "read_local_text_file", + "status": "ok", + "path": str(requested_path), + }, + ) + except Exception as exc: + return make_error_response("read_local_text_file", exc) + + +def save_desktop_screenshot(label: str = "observe") -> ToolResponse: + """截取当前桌面并保存到本地，用于 GUI 操作前后的观察与确认。""" + try: + pyautogui = _load_pyautogui() + screenshot_dir = _ensure_screenshot_dir() + label = normalize_text(label, default="observe") + safe_label = "".join( + char if char.isalnum() or char in "-_" else "_" for char in label + ) + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f") + screenshot_path = screenshot_dir / f"{timestamp}_{safe_label}.png" + pyautogui.screenshot(str(screenshot_path)) + return make_multimodal_response( + summary=( + "action=save_desktop_screenshot\n" + "status=ok\n" + f"path={screenshot_path}\n" + f"label={label}\n" + "note=The attached image is the latest desktop screenshot." + ), + image_path=str(screenshot_path), + metadata={ + "action": "save_desktop_screenshot", + "status": "ok", + "path": str(screenshot_path), + "label": label, + }, + ) + except Exception as exc: + return make_error_response("save_desktop_screenshot", exc) + + +def get_mouse_position() -> ToolResponse: + """获取当前鼠标坐标。""" + try: + pyautogui = _load_pyautogui() + x_pos, y_pos = pyautogui.position() + return make_success_response( + "action=get_mouse_position\nstatus=ok", + x=x_pos, + y=y_pos, + ) + except Exception as exc: + return make_error_response("get_mouse_position", exc) + + +def left_click(x: int, y: int, clicks: int = 1, interval: float = 0.2) -> ToolResponse: + """在指定坐标执行鼠标左键单击。""" + try: + x = normalize_int(x, default=0) + y = normalize_int(y, default=0) + clicks = normalize_int(clicks, default=1) + interval = normalize_float(interval, default=0.2) + pyautogui = _load_pyautogui() + pyautogui.click(x=x, y=y, clicks=clicks, interval=interval, button="left") + return make_success_response( + "action=left_click\nstatus=ok", + x=x, + y=y, + clicks=clicks, + interval=interval, + ) + except Exception as exc: + return make_error_response("left_click", exc) + + +def type_text(text: str, interval: float = 0.03) -> ToolResponse: + """向当前焦点控件输入文本。""" + try: + text = normalize_text(text, default="") + if not text: + return make_tool_response( + "action=type_text\nstatus=error\nerror=Empty text is not allowed.", + metadata={ + "action": "type_text", + "status": "error", + "error": "Empty text is not allowed.", + }, + ) + interval = normalize_float(interval, default=0.03) + pyautogui = _load_pyautogui() + pyautogui.write(text, interval=interval) + return make_success_response( + "action=type_text\nstatus=ok", + length=len(text), + interval=interval, + ) + except Exception as exc: + return make_error_response("type_text", exc) + + +def press_keys(keys: list[str], interval: float = 0.1) -> ToolResponse: + """依次按下多个按键。""" + try: + keys = normalize_keys(keys) + if not keys: + return make_tool_response( + "action=press_keys\nstatus=error\nerror=At least one key is required.", + metadata={ + "action": "press_keys", + "status": "error", + "error": "At least one key is required.", + }, + ) + interval = normalize_float(interval, default=0.1) + pyautogui = _load_pyautogui() + pyautogui.press(keys, interval=interval) + return make_success_response( + "action=press_keys\nstatus=ok", + keys=keys, + interval=interval, + ) + except Exception as exc: + return make_error_response("press_keys", exc) + + +def hotkey(*keys: str) -> ToolResponse: + """按组合键，例如 Ctrl+A、Ctrl+C。""" + try: + keys = tuple(normalize_keys(keys)) + if not keys: + return make_tool_response( + "action=hotkey\nstatus=error\nerror=At least one key is required.", + metadata={ + "action": "hotkey", + "status": "error", + "error": "At least one key is required.", + }, + ) + pyautogui = _load_pyautogui() + pyautogui.hotkey(*keys) + return make_success_response( + "action=hotkey\nstatus=ok", + keys=list(keys), + ) + except Exception as exc: + return make_error_response("hotkey", exc) + + +def wait_seconds(seconds: float = 1.0) -> ToolResponse: + """等待指定秒数，给界面渲染或响应留出时间。""" + try: + seconds = normalize_float(seconds, default=1.0) + time.sleep(seconds) + return make_success_response( + "action=wait_seconds\nstatus=ok", + seconds=seconds, + ) + except Exception as exc: + return make_error_response("wait_seconds", exc) + + +def get_active_window_title() -> ToolResponse: + """获取当前活动窗口标题。""" + try: + pyautogui = _load_pyautogui() + title = "" + + get_title = getattr(pyautogui, "getActiveWindowTitle", None) + if callable(get_title): + title = get_title() or "" + + return make_success_response( + "action=get_active_window_title\nstatus=ok", + title=title, + ) + except Exception as exc: + return make_error_response("get_active_window_title", exc) + + +def get_clipboard_text() -> ToolResponse: + """读取 Windows 剪贴板文本，用于确认输入结果。""" + try: + completed = subprocess.run( + ["powershell", "-NoProfile", "-Command", "Get-Clipboard"], + capture_output=True, + text=True, + check=False, + encoding="utf-8", + ) + status = "ok" if completed.returncode == 0 else "error" + return make_tool_response( + "\n".join( + [ + "action=get_clipboard_text", + f"status={status}", + f"text={completed.stdout.strip()}", + f"stderr={completed.stderr.strip()}", + ], + ), + metadata={ + "action": "get_clipboard_text", + "status": status, + "text": completed.stdout.strip(), + "stderr": completed.stderr.strip(), + }, + ) + except Exception as exc: + return make_error_response("get_clipboard_text", exc) + + +def paste_text(text: str) -> ToolResponse: + """通过剪贴板粘贴文本，降低输入法对 ASCII 输入的干扰。""" + try: + text = normalize_text(text, default="") + if not text: + return make_tool_response( + "action=paste_text\nstatus=error\nerror=Empty text is not allowed.", + metadata={ + "action": "paste_text", + "status": "error", + "error": "Empty text is not allowed.", + }, + ) + + set_clipboard = subprocess.run( + ["powershell", "-NoProfile", "-Command", "Set-Clipboard -Value @'\n" + text + "\n'@"], + capture_output=True, + text=True, + check=False, + encoding="utf-8", + ) + if set_clipboard.returncode != 0: + return make_tool_response( + "\n".join( + [ + "action=paste_text", + "status=error", + f"stderr={set_clipboard.stderr.strip()}", + ], + ), + metadata={ + "action": "paste_text", + "status": "error", + "stderr": set_clipboard.stderr.strip(), + }, + ) + + pyautogui = _load_pyautogui() + pyautogui.hotkey("ctrl", "v") + return make_success_response( + "action=paste_text\nstatus=ok", + length=len(text), + ) + except Exception as exc: + return make_error_response("paste_text", exc) + + +def register_gui_tools(toolkit: Toolkit) -> None: + """注册本地 Windows GUI 操作工具。""" + gui_tools = [ + save_desktop_screenshot, + get_mouse_position, + left_click, + type_text, + paste_text, + press_keys, + hotkey, + wait_seconds, + get_active_window_title, + get_clipboard_text, + ] + + for tool_func in gui_tools: + toolkit.register_tool_function(tool_func) + + +def register_local_skills(toolkit: Toolkit) -> None: + """自动注册 skills 目录下的全部本地 skills。""" + for skill_dir in sorted(SKILLS_DIR.iterdir()): + if skill_dir.is_dir() and (skill_dir / "SKILL.md").exists(): + toolkit.register_agent_skill(str(skill_dir)) + + +def strip_thinking_from_msg(msg: Msg) -> Msg: + """移除消息中的 thinking block。""" + all_blocks = msg.get_content_blocks() + visible_blocks = [ + block for block in all_blocks if block.get("type") != "thinking" + ] + + if len(visible_blocks) == len(all_blocks): + return msg + + filtered_msg_dict = msg.to_dict() + filtered_msg_dict["content"] = visible_blocks + return Msg.from_dict(filtered_msg_dict) + + +def keep_visible_text_blocks(msg: Msg) -> Msg | None: + """仅保留适合展示给用户的自然语言文本块。""" + visible_blocks = [] + + for block in msg.get_content_blocks(): + if block.get("type") != "text": + continue + + text = block.get("text", "") + if isinstance(text, str) and text.strip(): + visible_blocks.append(block) + + if not visible_blocks: + return None + + filtered_msg_dict = msg.to_dict() + filtered_msg_dict["content"] = visible_blocks + return Msg.from_dict(filtered_msg_dict) + + +class SilentThinkingFormatter(OpenAIChatFormatter): + """在发给 OpenAI 兼容接口前移除 thinking block。""" + + async def format(self, msgs: list[Msg], **kwargs: Any) -> list[dict[str, Any]]: + sanitized_msgs = [strip_thinking_from_msg(msg) for msg in msgs] + return await super().format(sanitized_msgs, **kwargs) + + +class SilentThinkingReActAgent(ReActAgent): + """仅打印可见文本，隐藏 ThinkingBlock 内容。""" + + def __init__(self, *args: Any, **kwargs: Any) -> None: + self.hide_thinking = kwargs.pop("hide_thinking", True) + self.debug_model_output = kwargs.pop("debug_model_output", False) + super().__init__(*args, **kwargs) + self._react_action_cache: dict[str, str] = {} + self._react_observation_cache: dict[str, str] = {} + self._suppress_max_iter_summary = False + + @staticmethod + def _summarize_text(text: str, limit: int = 240) -> str: + """压缩长文本，避免控制台输出过长。""" + normalized = " ".join(text.split()) + if len(normalized) <= limit: + return normalized + return f"{normalized[:limit]}..." + + def _format_tool_input(self, tool_input: Any) -> str: + """将工具输入格式化为简短可读文本。""" + try: + serialized = json.dumps(tool_input, ensure_ascii=False) + except TypeError: + serialized = str(tool_input) + return self._summarize_text(serialized) + + def _format_tool_output(self, output: Any) -> str: + """将工具输出格式化为简短可读文本。""" + if isinstance(output, list): + text_parts = [] + for item in output: + if isinstance(item, dict) and item.get("type") == "text": + text_parts.append(item.get("text", "")) + else: + text_parts.append(str(item)) + return self._summarize_text("\n".join(text_parts)) + + return self._summarize_text(str(output)) + + def _print_raw_model_output(self, msg: Msg) -> None: + """打印模型原始返回内容，便于调试多模态观察链路。""" + if msg.role != "assistant": + return + + try: + raw_content = json.dumps( + msg.get_content_blocks(), + ensure_ascii=False, + indent=2, + ) + except TypeError: + raw_content = str(msg.get_content_blocks()) + + print("===== Raw Model Output =====") + print(raw_content) + print("===== End Raw Model Output =====") + + async def _print_react_process(self, msg: Msg, last: bool) -> None: + """打印 ReAct 中的 Action / Observation 步骤。""" + if not last: + return + + for block in msg.get_content_blocks(): + block_type = block.get("type") + + if block_type == "tool_use": + tool_name = block.get("name", "unknown_tool") + tool_input = self._format_tool_input(block.get("input", {})) + cache_key = f"{msg.id}:{block.get('id', tool_name)}" + cache_value = f"{tool_name} {tool_input}" + if self._react_action_cache.get(cache_key) == cache_value: + continue + self._react_action_cache[cache_key] = cache_value + print(f"{msg.name}[Action]: {cache_value}") + + elif block_type == "tool_result": + tool_name = block.get("name", "unknown_tool") + tool_output = self._format_tool_output(block.get("output", "")) + cache_key = f"{msg.id}:{block.get('id', tool_name)}" + cache_value = f"{tool_name} -> {tool_output}" + if self._react_observation_cache.get(cache_key) == cache_value: + continue + self._react_observation_cache[cache_key] = cache_value + print(f"{msg.name}[Observation]: {cache_value}") + + async def print(self, msg: Msg, last: bool = True, speech=None) -> None: + """展示自然语言回复，并打印可读的 ReAct 过程。""" + await self._print_react_process(msg, last=last) + + if self.debug_model_output and last: + self._print_raw_model_output(msg) + + filtered_msg = msg + if self.hide_thinking: + filtered_msg = strip_thinking_from_msg(filtered_msg) + + filtered_msg = keep_visible_text_blocks(filtered_msg) + if filtered_msg is None: + return + + await super().print(filtered_msg, last=last, speech=speech) + + async def _summarizing(self) -> Msg: + """在 GUI 单步模式下，达到上限时不要生成误导性总结。""" + if self._suppress_max_iter_summary: + return Msg(self.name, "", "assistant") + return await super()._summarizing() + + +def looks_like_gui_task(text: str) -> bool: + """粗略判断用户请求是否属于 GUI 自动化任务。""" + gui_keywords = [ + "打开", + "浏览器", + "chrome", + "edge", + "bing", + "窗口", + "桌面", + "点击", + "输入", + "回车", + "搜索", + "gui", + ] + lowered = text.lower() + return any(keyword in text or keyword in lowered for keyword in gui_keywords) + + +def should_auto_continue_gui(reply: Msg | None) -> bool: + """判断 GUI 任务回复是否表现为中途暂停而非真正完成。""" + if reply is None: + return False + + text = (reply.get_text_content() or "").strip() + if not text: + return True + + stop_markers = [ + "是否需要我继续", + "是否需要我继续执行", + "下一步计划", + "当前任务状态总结", + "当前问题", + "我将", + "准备继续", + "让我尝试", + "继续执行", + "尚未成功", + "未成功", + "仍然是", + "仍为", + "当前活动窗口是", + "需要先打开", + "尝试打开", + "尝试关闭", + "回到桌面", + "", + ] + done_markers = [ + "任务已完成", + "已经完成", + "已成功", + "搜索结果", + "已打开 bing", + "已输入test", + "已回车搜索", + "明确无法继续", + "无法继续", + ] + blocked_markers = [ + "需要用户", + "请用户", + "权限不足", + "无法定位", + "无法识别", + "无法访问", + "明确受阻", + ] + + if any(marker in text for marker in done_markers): + return False + + if any(marker in text for marker in blocked_markers): + return False + + if any(marker in text for marker in stop_markers): + return True + + # For GUI tasks, default to continue unless the reply clearly indicates + # success or a real blocker. This prevents the agent from stopping after + # an intermediate summary. + return True + + +def register_optional_skill_access_tools(toolkit: Toolkit) -> None: + """注册受限的本地文本读取工具，避免 agent 使用 shell 绕开 GUI。""" + toolkit.register_tool_function(read_local_text_file) + print("已注册可选工具: read_local_text_file") + + +def build_agent(show_thinking: bool, debug_model_output: bool) -> ReActAgent: + """创建一个带有示例 skill 的 ReActAgent。""" + toolkit = Toolkit() + toolkit.register_tool_function(execute_python_code) + register_optional_skill_access_tools(toolkit) + register_gui_tools(toolkit) + register_local_skills(toolkit) + + agent_cls = SilentThinkingReActAgent + formatter = ( + OpenAIChatFormatter(promote_tool_result_images=True) + if show_thinking + else SilentThinkingFormatter(promote_tool_result_images=True) + ) + + agent = agent_cls( + name="Jarvis", + sys_prompt=( + "你是一个名为 Jarvis 的助手。请优先使用已注册的 skill 完成任务。" + "对于 GUI 自动化任务，不要在中途停下来询问用户是否继续；" + "你应当持续执行观察、操作、复查循环，直到任务完成或明确无法继续。" + "每一轮只做一个最小必要动作，并在动作后立刻重新观察。" + "不要预设固定的长操作顺序，不要在旧假设上连续执行多步。" + ), + model=OpenAIChatModel( + model_name=LLAMA_CPP_MODEL_NAME, + api_key=LLAMA_CPP_API_KEY, + stream=True, + client_kwargs={ + "base_url": LLAMA_CPP_BASE_URL, + "timeout": 60, + }, + generate_kwargs={ + "temperature": 0.7, + }, + ), + max_iters=8, + hide_thinking=not show_thinking, + debug_model_output=debug_model_output, + formatter=formatter, + toolkit=toolkit, + memory=InMemoryMemory(), + ) + + return agent + + +def parse_args() -> argparse.Namespace: + """解析命令行参数。""" + parser = argparse.ArgumentParser(description="AgentScope skills demo") + parser.add_argument( + "--skip-agent-run", + action="store_true", + help="只打印 skill 注册结果，不调用远程模型。", + ) + parser.add_argument( + "--message", + help="发送给 ReActAgent 的单轮消息；不传时进入交互式对话。", + ) + parser.add_argument( + "--show-thinking", + action="store_true", + help="在控制台显示模型 thinking 内容。默认隐藏。", + ) + parser.add_argument( + "--user-name", + default="user", + help="交互式对话中使用的用户名。", + ) + parser.add_argument( + "--debug-model-output", + action="store_true", + help="打印每轮 assistant 的原始模型返回内容，便于调试截图观察链路。", + ) + return parser.parse_args() + + +async def chat_once(agent: ReActAgent, user_name: str, user_input: str) -> None: + """发送一条用户消息。""" + msg = Msg( + name=user_name, + content=user_input, + role="user", + ) + + is_gui_task = looks_like_gui_task(user_input) + original_max_iters = agent.max_iters + original_suppress = getattr(agent, "_suppress_max_iter_summary", False) + + if is_gui_task: + agent.max_iters = 1 + if hasattr(agent, "_suppress_max_iter_summary"): + agent._suppress_max_iter_summary = True + + try: + reply = await agent(msg) + + if not is_gui_task: + return + + for _ in range(DEFAULT_GUI_MAX_STEPS): + if not should_auto_continue_gui(reply): + return + + follow_up = Msg( + name=user_name, + content=( + "继续执行当前 GUI 任务，不要停下来询问是否继续。" + "请继续按照观察、操作、复查的闭环执行，直到任务完成或明确无法继续。" + "如果你刚才输出的是计划、状态总结，或原样的 文本，请立即把它落实为真实工具调用并继续。" + "下一轮只做一个最小必要动作，并在动作后立刻截图复查。" + "这一轮最多只允许一次真实工具动作，不要连续做多步。" + ), + role="user", + ) + reply = await agent(follow_up) + finally: + agent.max_iters = original_max_iters + if hasattr(agent, "_suppress_max_iter_summary"): + agent._suppress_max_iter_summary = original_suppress + + +async def chat_loop(agent: ReActAgent, user_name: str) -> None: + """通过命令行输入与模型持续对话。""" + print("\n===== 交互式对话已启动 =====") + print("输入 /exit 结束对话，输入 /help 查看提示。") + + while True: + try: + user_input = await asyncio.to_thread(input, f"\n{user_name}> ") + except (EOFError, KeyboardInterrupt): + print("\n对话结束。") + return + + user_input = user_input.strip() + + if not user_input: + continue + + if user_input in {"/exit", "/quit"}: + print("对话结束。") + return + + if user_input == "/help": + print("直接输入消息即可和模型对话。使用 /exit 或 /quit 退出。") + continue + + try: + await chat_once(agent, user_name, user_input) + except Exception as exc: + print(f"本轮对话执行失败：{type(exc).__name__}: {exc}") + print("你可以继续输入下一条消息，或输入 /exit 退出。") + + +async def main() -> None: + args = parse_args() + agent = build_agent( + show_thinking=args.show_thinking, + debug_model_output=args.debug_model_output, + ) + + print("===== 已注册 skill 提示词 =====") + print(agent.toolkit.get_agent_skill_prompt()) + + print("\n===== 当前模型配置 =====") + print(f"base_url={LLAMA_CPP_BASE_URL}") + print(f"model_name={LLAMA_CPP_MODEL_NAME}") + print(f"show_thinking={args.show_thinking}") + print(f"debug_model_output={args.debug_model_output}") + + if args.skip_agent_run: + print("\n===== 已跳过远程模型调用 =====") + return + + if args.message: + print("\n===== 单轮对话模式 =====") + await chat_once(agent, args.user_name, args.message) + return + + await chat_loop(agent, args.user_name) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..e82dd11 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,4 @@ +agentscope +openai +pyautogui +pillow diff --git a/skills/python_hello_skill/SKILL.md b/skills/python_hello_skill/SKILL.md new file mode 100644 index 0000000..099009c --- /dev/null +++ b/skills/python_hello_skill/SKILL.md @@ -0,0 +1,31 @@ +--- +name: python_hello_skill +description: 当用户希望用 Python 执行一个简单示例、打印文本或快速验证代码时，优先使用这个技能。 +--- + +# Python Hello Skill + +当用户要求你用 Python 输出一段文本、演示最小代码示例或确认 Python 工具是否可用时，按下面步骤执行： + +1. 识别用户是否需要真正运行 Python，而不是只给出代码。 +2. 如果需要运行，优先调用 Python 执行工具，而不是只描述做法。 +3. 对于最小示例，默认使用下面这段代码： + +```python +print("Hello from skill demo!") +``` + +4. 运行完成后，简要总结执行结果。 +5. 如果用户没有明确指定输出内容，可以使用上面的默认文本。 + +## 适用场景 + +- 用户说“用 Python 打个招呼” +- 用户说“帮我运行一个 Hello World” +- 用户说“验证一下 Python 工具能不能用” + +## 注意事项 + +- 除非用户要求复杂逻辑，否则保持代码最小化。 +- 优先直接执行，不要只返回伪代码。 +- 如果工具执行失败，说明失败原因并给出可复现的 Python 代码。 diff --git a/skills/windows_gui_operator/SKILL.md b/skills/windows_gui_operator/SKILL.md new file mode 100644 index 0000000..4abc9c2 --- /dev/null +++ b/skills/windows_gui_operator/SKILL.md @@ -0,0 +1,115 @@ +--- +name: windows_gui_operator +description: 在 Windows 桌面环境中执行 GUI 自动化。优先把截图直接提供给多模态模型观察，并按“观察-操作-复查-必要时重试”的闭环执行。 +--- + +# Windows GUI Operator + +当用户要求你操作 Windows 图形界面时，必须严格遵循下面的 ReAct 工作流。 + +这个 skill 假设当前模型具备多模态图像理解能力，因此你应该把截图结果当作“观察输入”的核心证据，而不是只依赖文字描述。 + +## 核心原则 + +- 必须先观察，再操作，再复查；如果没有达到目标，继续下一轮观察和调整。 +- 每个关键步骤都要保留本地截图，截图既要保存到本地，也要作为图片结果交给模型判断。 +- 所有桌面操作都只能通过 GUI 完成，不允许通过 shell、命令行、直接启动 URL 或直接启动应用来绕过界面。 +- 不要盲点、盲输；在缺少坐标或上下文时，先获取更多观察信息。 +- 每次完成关键动作后，都要做确认，而不是假设动作已经生效。 +- 若当前动作未达到效果，必须继续循环，直到完成目标或明确受阻。 +- 不要在任务中途向用户提问“是否继续”。除非已经明确受阻，否则应自动进入下一轮观察和操作。 +- 如果你输出的只是计划、状态总结、解释文字，或者原样的 `` 文本，这都不算完成；你必须继续把它落实为真实工具调用。 +- 一次只做一个最小必要动作。不要预先规划一长串固定操作并连续执行。 +- 下一步动作必须基于“刚刚最新的一张截图和最新观察结果”，而不是基于上一个假设继续硬做。 +- 在决定下一步动作前，先明确描述最新截图里的关键界面状态，例如开始菜单是否打开、浏览器是否已经出现、地址栏是否聚焦。 + +## 单步闭环工作流 + +每一轮都按下面顺序进行： + +1. `Observe`： + - 调用 `get_active_window_title` + - 调用 `save_desktop_screenshot` + - 读取截图对应的图片内容，先简短描述你看到了什么，再判断当前界面状态和下一步动作 +2. `Act`： + - 只选择一个最小必要动作执行 + - 可用工具包括：`left_click`、`type_text`、`press_keys`、`hotkey`、`wait_seconds` +3. `Verify`： + - 再次调用 `save_desktop_screenshot` + - 结合新截图、活动窗口标题、工具返回文本进行判断 + - 对输入任务，优先额外用 `get_clipboard_text` 做校验 +4. `Decide`： + - 如果目标已达成，结束 + - 如果目标未达成但可继续，基于最新截图重新判断，再进入下一轮 Observe + - 如果无法继续，明确说明阻塞点和建议 + +不要只执行单轮。默认至少假设需要多轮迭代，直到确认成功。 +不要在中途仅输出状态总结后停止等待用户确认。 +如果当前轮没有真正完成目标，就继续下一轮，而不是把控制权交还给用户。 + +## 决策要求 + +- 每一轮只能提交一个动作决策，例如“按 Win 键”或“点击地址栏”或“输入文本”。 +- 每一轮最多只允许一次真实工具动作；动作完成后必须先复查，不能在同一轮继续第二个 GUI 动作。 +- 不要在同一轮里把“按 Win -> 输入 chrome -> 回车 -> 打开地址栏 -> 输入网址 -> 回车”当作一个固定脚本整体执行。 +- 如果某一步出现偏差，例如焦点不对、窗口没切换、输入框没获得焦点，就必须停下来重新观察，而不是继续执行原计划。 +- 如果你不确定按钮、输入框或窗口状态，就先截图再决定，不要赌测。 + +## 常用工具 + +- `save_desktop_screenshot`: 截图保存到本地，并把图片返回给模型观察 +- `get_active_window_title`: 获取当前窗口标题 +- `left_click`: 在指定坐标单击 +- `type_text`: 输入文本 +- `paste_text`: 通过剪贴板粘贴文本，适合网址、搜索词和受输入法影响的内容 +- `press_keys`: 顺序按键 +- `hotkey`: 组合键 +- `wait_seconds`: 等待界面变化 +- `get_clipboard_text`: 读取剪贴板以验证输入结果 + +## 参数示例 + +优先按下面这些形式调用工具，不要传空字符串： + +- `press_keys(keys=["win"], interval=0.1)` +- `type_text(text="chrome", interval=0.03)` +- `press_keys(keys=["enter"], interval=0.1)` +- `hotkey("ctrl", "l")` +- `paste_text(text="https://www.bing.com")` +- `wait_seconds(seconds=1.0)` +- `left_click(x=300, y=120, clicks=1, interval=0.2)` +- `paste_text(text="test")` +- `press_keys(keys=["enter"], interval=0.1)` +- `hotkey("ctrl", "l")` + +如果你暂时不确定具体参数，也要使用合理默认值，而不是传空字符串。 +对网址、英文搜索词和命令词，优先使用 `paste_text`，不要依赖当前输入法状态。 + +## 确认策略 + +当前模型具备多模态能力时，你应当把截图内容纳入判断依据。 +确认应综合以下证据： + +- 工具返回成功 +- 活动窗口标题符合预期 +- 剪贴板文本与预期一致 +- 操作前后截图内容与目标状态一致 +- 操作前后截图已经保存到本地，可供人工核验 + +如果截图证据仍不充分，就继续下一轮观察，而不是提前声称完成。 + +## 失败与重试策略 + +- 每次动作失败后，先截图复查，再决定下一步。 +- 不要连续盲目重复同一个点击动作超过 3 次。 +- 如果无法定位目标控件，先通过截图重新判断布局，再进行更小步的操作。 +- 如果应用未启动成功，优先通过开始菜单、任务栏或桌面图标继续用 GUI 方式尝试。 +- 如果经过多轮仍不能完成，给出清晰的阻塞原因。 +- 如果某个动作连续两轮都没有带来预期变化，优先改变策略，而不是机械重复原动作链。 + +## 注意事项 + +- 坐标点击前，优先让用户提供窗口状态、分辨率或目标位置描述。 +- 涉及敏感操作时，先复述将要执行的动作，再执行。 +- 如果动作可能产生副作用，拆成更小步骤，并在每一步后确认。 +- 若界面未响应，先 `wait_seconds`，再重新截图，不要连续重复点击。