Compare commits
33
Commits
69a1ee0bdf
..
v2.4.0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
55efb1c410 | ||
|
|
53fa5da82b | ||
|
|
5b70f7d614 | ||
|
|
dfcbe6d3ea | ||
|
|
516b4a5d86 | ||
|
|
e39273e97f | ||
|
|
8b8fbddb4f | ||
|
|
ba9534f064 | ||
|
|
43e77e569e | ||
|
|
ab67400174 | ||
|
|
13b493eada | ||
|
|
7fc945a4c8 | ||
|
|
fbe78d362f | ||
|
|
fe1c6d106f | ||
|
|
a86b748a6f | ||
|
|
d3128ba21c | ||
|
|
19564fb65b | ||
|
|
deb622095f | ||
|
|
61245b1c65 | ||
|
|
bc34f91417 | ||
|
|
ec70b10c08 | ||
|
|
a8900ddafc | ||
|
|
09cf474c2d | ||
|
|
ed7f111c95 | ||
|
|
9adc98b0f5 | ||
|
|
c59ddf5eb4 | ||
|
|
49a3d35227 | ||
|
|
27b03e71ef | ||
|
|
bb621b89b7 | ||
|
|
6aa1a2789b | ||
|
|
5a9bebcf71 | ||
|
|
29e46692fc | ||
|
|
89c673864f |
@@ -22,7 +22,7 @@
|
||||
| `app/server/public/` | 前端静态资源,经典脚本按序加载,共享全局作用域 | 无构建步骤 |
|
||||
| `executor/` | **agent-executor**:`/exec` 隔离命令执行 + Office 预览网关 | 零依赖 Node,独立容器 |
|
||||
| `wxbot/` | 微信 iLink 协议调制解调器(多实例) | 独立容器 `agent-wxbot` |
|
||||
| `mcp/` | 同栈 MCP 服务:`web_search` / `image_task` / `video_task` | 三个容器,仅内网 |
|
||||
| `mcp/` | 同栈 MCP 服务:`web_search` / `image_task` / `video_task` / `asr` | 四个容器,仅内网 |
|
||||
| `skills/` | 目录式技能(`SKILL.md` + `run.py`/`run.sh`) | 运行时挂载,`skills/deps` 为依赖层 |
|
||||
| `scripts/` | `init.sh` 生成 `.env` + `config.yml`;`init_host.sh` 裸机初始化 | 幂等 |
|
||||
| `docs/` | 专题文档:`tool-design.md`(工具接口规范)、`OFFICE.md` | — |
|
||||
@@ -42,7 +42,7 @@ flowchart TD
|
||||
WXBOT -->|"回调 /wxhook/*(共享 Bearer token)"| CORE
|
||||
CORE["agent-core(8090,唯一对外端口)"]
|
||||
CORE --> EXEC["agent-executor(9000 exec / 3100 预览,仅内网)"]
|
||||
CORE --> MCP["mcp_web_search / mcp_image_task / mcp_video_task(3000,仅内网)"]
|
||||
CORE --> MCP["mcp_web_search / mcp_image_task / mcp_video_task / mcp_asr(3000,仅内网)"]
|
||||
CORE -->|"分段投递回复"| WXBOT
|
||||
```
|
||||
|
||||
@@ -83,7 +83,7 @@ flowchart TD
|
||||
### 环境变量(接线全靠根 `.env` + compose)
|
||||
|
||||
`AGENT_CONFIG`、`PORT`、`EXECUTOR_URL`、`EXECUTOR_TOKEN`、`WXBOT_URL`、`WXBOT_TOKEN`、
|
||||
`VIRTUAL_HOST`、`PREVIEW_PROXIES`、`EXECUTOR_LOG_LEVEL`、`WXBOT_LOG_LEVEL`。
|
||||
`ASR_URL`、`ASR_API_KEY`、`VIRTUAL_HOST`、`PREVIEW_PROXIES`、`EXECUTOR_LOG_LEVEL`、`WXBOT_LOG_LEVEL`。
|
||||
密钥生成:`./scripts/init.sh`(幂等,重跑不掉登录态)。
|
||||
|
||||
## 三条主链路
|
||||
@@ -187,6 +187,7 @@ curl -s http://127.0.0.1:8090/api/health
|
||||
| 改模型调度 / 绑定 | `app/server/engine.js` 的 `chooseKey` / `chooseModel` / `ensureBinding` |
|
||||
| 改上下文组装 / 省 token | `app/server/llm.js` 的 `buildMessages` / `trim*` |
|
||||
| 改微信投递 | `app/server/wechat.js`(业务)+ `app/server/routes/wechat.js`(HTTP/鉴权) |
|
||||
| 改语音输入 | `app/server/public/js/voice.js`(录音与插入)+ `app/server/routes/asr.js`(代理)+ `mcp/asr/`(转写);客户端桥契约见 `docs/voice-input.md` |
|
||||
| 改隔离执行 | `executor/server.js`(先读顶部安全模型注释) |
|
||||
| 加同栈 MCP 服务 | `mcp/<name>/` + `docker-compose.yml` + `config.yml` 的 `mcp_servers` |
|
||||
| 加技能 | `skills/<name>/`(`SKILL.md` + `run.py`),见 `skills/README.md` |
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
| 上下文管理 | 手动压缩、按 token 阈值提示压缩、历史重放自动裁剪、会话标题自动生成 |
|
||||
| 定时任务 | 每会话最多一个 cron 任务;触发前清空模型可见历史,按计划执行并推送结果 |
|
||||
| 微信接入 | 扫码绑定个人微信,文本/图片/视频/文件进入同一套会话与命令体系 |
|
||||
| 语音输入 | 输入框为空时发送键变麦克风:点一下进入语音模式,输入框位置变成「按住 说话」,松开即转写并直接发送;发送键位置变键盘按钮可切回打字。套壳 App 走原生录音(http 页面也能用),浏览器走 MediaRecorder(需 HTTPS);识别在 `mcp_asr`,不经过模型、不消耗 token |
|
||||
| 长期记忆 | 会话收口时生成第一人称工作日记;微信新会话注入最近几条,`recall` 可检索全部 |
|
||||
| 用户画像 | 记录称谓、行业、偏好,会话创建时冻结注入,跨会话生效 |
|
||||
| 技能 | 目录式技能(`SKILL.md` + `run.py`/`run.sh`),可在设置页管理 |
|
||||
@@ -94,7 +95,7 @@ http://<服务器 IP>:<AGENT_PORT>
|
||||
|
||||
### 4. 同栈 MCP 服务
|
||||
|
||||
`init.sh` 默认把仓库内 `mcp/` 的三个服务写进 `config.yml`:
|
||||
`init.sh` 默认把仓库内 `mcp/` 的服务写进 `config.yml`:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
@@ -104,9 +105,14 @@ mcp_servers:
|
||||
url: "http://mcp_image_task:3000/mcp"
|
||||
- name: video_task
|
||||
url: "http://mcp_video_task:3000/mcp"
|
||||
- name: asr
|
||||
url: "http://mcp_asr:3000/mcp"
|
||||
mode: skill
|
||||
```
|
||||
|
||||
`web_search` / `image_task` / `video_task` 与 core 由同一个根 `docker-compose.yml` 启动,没有先后顺序,也不占宿主机端口。工具说明、header 覆盖项与调试方式见 [mcp/README.md](mcp/README.md)。
|
||||
`web_search` / `image_task` / `video_task` / `asr` 与 core 由同一个根 `docker-compose.yml` 启动,没有先后顺序,也不占宿主机端口。工具说明、header 覆盖项与调试方式见 [mcp/README.md](mcp/README.md)。
|
||||
|
||||
语音输入(H5 输入框里说话)走的是另一条路:客户端录音 → `POST /api/asr/transcribe` → `mcp_asr` 的 `/transcribe`,**不经过模型、不消耗 token**,结果插入输入框可编辑。客户端录音契约(Android 套壳原生桥 / 浏览器 MediaRecorder)见 [docs/voice-input.md](docs/voice-input.md);`ASR_URL` 置空即整体关闭。
|
||||
|
||||
接入栈外的 MCP 服务:在设置页添加 `url` / `headers`,或在 `config.yml` 的 `mcp_servers` 追加一项。服务不可达时 core 只在启动日志里告警,不影响 Web 主流程,但对应工具不会下发给模型。
|
||||
|
||||
@@ -118,9 +124,9 @@ mcp_servers:
|
||||
| `skill` | ✓ | ✗ | ✓ | `skill` 工具与 `lab` |
|
||||
| `disable` | ✗ | ✗ | ✗ | 无人 |
|
||||
|
||||
`skill` 用于「能力包在技能后面」的场景:工具照常发现并注册,只是不进模型的 tools 数组,改为以技能形式暴露——`{"list":"all"}` 能看到它,`{"get":"<名>"}` 直接返回 MCP 原始的 `description` + `input_schema`,`{"run":"<名>","args":{…}}` 转发给服务。不用在 `skills/` 里建目录。旧字段 `enable` 已废弃,`mode: silent` 已移除(启动会报错并提示改用 `skill`)。
|
||||
`skill` 用于「能力包在技能后面」的场景:工具照常发现并注册,只是不进模型的 tools 数组,改为以技能形式暴露——`{"op":"list"}` 能看到它,`{"op":"get","name":"<名>"}` 直接返回 MCP 原始的 `description` + `input_schema`,`{"op":"run","name":"<名>","args":{…}}` 转发给服务。不用在 `skills/` 里建目录。旧字段 `enable` 已废弃,`mode: silent` 已移除(启动会报错并提示改用 `skill`)。
|
||||
|
||||
工具发现只在启动时做一次,因此发现失败后会自动按 **2s / 20s / 1min 重试 3 次**(覆盖 compose 并行拉起与宿主机重启后的容器乱序);成功后工具在下一轮请求即可用,三次都失败则需重启 core。三个 MCP 服务都配了 `/health` 健康检查,`docker ps` 可直接看状态。
|
||||
工具发现只在启动时做一次,因此发现失败后会自动按 **2s / 20s / 1min 重试 3 次**(覆盖 compose 并行拉起与宿主机重启后的容器乱序);成功后工具在下一轮请求即可用,三次都失败则需重启 core。四个 MCP 服务都配了 `/health` 健康检查,`docker ps` 可直接看状态。
|
||||
|
||||
---
|
||||
|
||||
@@ -224,7 +230,7 @@ platforms:
|
||||
通用模型字段:
|
||||
|
||||
- `enable: false`:模型不进池,不参与分配。
|
||||
- `vision: false`:读图时不回传像素,只给文字占位。
|
||||
- `vision: false`:读图时不回传像素,回执只给结构化说明(`reason: no_vision`)。
|
||||
- `profile: <名>`:引用 `model_profiles`,不配则使用默认全量 prompt 与工具。
|
||||
- `reasoning_ack_field`:需要把思考链回传给上游时填写,例如 DeepSeek 系的 `reasoning_content`。
|
||||
|
||||
@@ -238,6 +244,8 @@ platforms:
|
||||
| `PORT` | core 监听端口,容器内 `8090` |
|
||||
| `EXECUTOR_URL` | `agent-executor` 地址;空则命令在 core 进程内本地执行,仅适合开发 |
|
||||
| `EXECUTOR_TOKEN` | core ↔ executor 的 Bearer token,两侧同值 |
|
||||
| `ASR_URL` | `mcp_asr` 地址(compose 为 `http://mcp_asr:3000`);置空则语音输入关闭 |
|
||||
| `ASR_API_KEY` / `ASR_API_BASE` / `ASR_MODEL` | `mcp_asr` 的上游语音识别密钥与地址(默认智谱 `glm-asr`) |
|
||||
| `WXBOT_URL` | wxbot 地址;置空则微信功能整体关闭 |
|
||||
| `WXBOT_TOKEN` | core ↔ wxbot 的共享 token |
|
||||
| `VIRTUAL_HOST` | 设置后 `public_link.base_url` 强制取 `https://<第一个域名>` |
|
||||
@@ -263,7 +271,7 @@ platforms:
|
||||
| `terminal` | 执行 shell 命令;默认在会话目录,超时可终止 |
|
||||
| `fetch` | 抓取 HTTP/HTTPS 并转 Markdown;带 SSRF 防护与分段续读 |
|
||||
| `mint_public_link` | 为会话内文件签发短期公网链接 |
|
||||
| `skill` | 技能:`{"list":"all"}` / `{"get":"<名>"}` / `{"run":"<名>","args":{...}}`(动词作键,三选一) |
|
||||
| `skill` | 技能:`op` = `list` / `get` / `run`;`name` 指技能,`args` 传参数 |
|
||||
| `db` | 操作会话本地 `data.db`:`exec`、`query`、`insert`、`tables`、`schema` |
|
||||
| `schedule` | 定时任务 `get` / `list` / `create` / `update` / `delete` |
|
||||
| `profile` | 用户画像 `list` / `set` / `add` / `remove` |
|
||||
@@ -287,7 +295,7 @@ skills/my-skill/
|
||||
```
|
||||
|
||||
- `SKILL.md` 的 `description` 会进入 system prompt,模型启动时就知道有哪些技能。
|
||||
- `input_schema.required` 在 `{"run":"<名>","args":{...}}` 前校验。
|
||||
- `input_schema.required` 在 `op=run` 前校验。
|
||||
- MCP 工具不用建目录:给该服务配 `mode: skill`,平台会自动把它暴露成技能(`get` 返回 MCP 原始的
|
||||
`input_schema`,`run` 转发),见上文「同栈 MCP 服务」。
|
||||
- 入口脚本以绝对路径启动;远端执行时 cwd 是会话目录,本地开发时是技能目录。相对路径参数会先按会话目录转成绝对路径。
|
||||
@@ -389,8 +397,14 @@ docker compose logs -f agent-core
|
||||
docker compose logs -f agent-executor
|
||||
docker compose logs -f agent-wxbot
|
||||
curl -s http://127.0.0.1:8090/api/health
|
||||
# {"ok":true,"service":"agent-core","api":1,"time":...,"auth_required":true}
|
||||
```
|
||||
|
||||
- `GET /api/health` 是**唯一免登录接口**,供运维探活,也供套壳 App 校验用户手填的服务器地址。
|
||||
客户端判定条件:HTTP 200 且 `ok === true` 且 `service === 'agent-core'`——**不要只看状态码**,
|
||||
填错到别的站点或反代首页同样可能回 200。地址末尾的 `/` 可带可不带。
|
||||
`auth_required` 用于区分「地址对了但没登录」与「地址错了」。
|
||||
|
||||
- core 日志等级由 `config.yml` 的 `log_level` 控制,改完重启生效。
|
||||
- executor / wxbot 的日志等级由各自环境变量控制,`debug` 会打印完整交互。
|
||||
- 启动时会在 `data/prompt.txt` 写入实际下发的 system prompt 与各 profile 的工具清单,便于核对。
|
||||
@@ -400,6 +414,9 @@ curl -s http://127.0.0.1:8090/api/health
|
||||
| 现象 | 原因 / 处理 |
|
||||
|---|---|
|
||||
| 启动时报 `network agent-mcp-net has active endpoints` | 旧的双仓库部署残留:先停旧的 mcp 仓库(`docker compose down`),再 `up` 本栈 |
|
||||
| 套壳 App 填了服务器地址仍连不上 | 先 `curl <地址>/api/health` 看是否返回 `service:agent-core`;再查 Android 9+ 默认禁止明文 HTTP(需 `usesCleartextTraffic`)与 https 自签证书 |
|
||||
| 麦克风按钮不显示 | `ASR_URL` 为空(未部署 `mcp_asr`);或浏览器以 `http://<IP>` 访问且没有原生录音桥(非安全上下文开不了麦,见 `docs/voice-input.md`) |
|
||||
| 点麦克风提示「语音识别服务不可用」 | `mcp_asr` 未起来或 `ASR_API_KEY` 为空:`docker compose logs --tail=50 mcp_asr` |
|
||||
| `config.yml` 变成目录 | 启动前文件不存在,Docker 自动建了同名目录;停容器、删目录、放入配置文件 |
|
||||
| 没有可用模型 | `platforms` 为空或模型全部禁用;在设置页添加后重启 |
|
||||
| 创建会话报「无可用模型」 | 模型池为空,或指定模型所在平台没有 key |
|
||||
|
||||
+29
-9
@@ -138,7 +138,19 @@ POST /api/sessions/:id/messages
|
||||
- 含图消息:DB 里存 `{"__blocks__":[...]}` 哨兵 JSON(`encodeBlocks`/`decodeBlocks`);
|
||||
发请求时 `image_ref` 现签公网短链(`mintUrl`,每次请求重签、不会中途过期)。
|
||||
`tool` 回执里的未消费图片会**挪出回执**,合并成该批回执之后的一条 `user` 消息
|
||||
(`tool` 角色带图是非标准形态,很多上游会吞图);已消费的图降级为文字占位。
|
||||
(`tool` 角色带图是非标准形态,很多上游会吞图);已消费的图只省略像素块,回执原样保留。
|
||||
- `read_file` 读图的回执是**结构化 JSON**(`tools/executor.js` 的 `imageReceipt`:`ok` /
|
||||
`image_attached` / `reason` / `width` / `height` …),不是散文。发送侧改写只限两种真事件
|
||||
(engine 同回合/同批去重、`buildMessages` 签外链失败),一律走 `patchImageReceipt` **只改字段**。
|
||||
- **已消费的图重放时只省略像素块,不改写回执**:回执是「那次读取」的历史事实,把当时成功的
|
||||
读图改写成 `image_attached:false` 会让模型以为当时就没拿到画面 → 反复重读同一条(实测循环)。
|
||||
- 图片注入消息的两条硬约束(踩过循环):①**消费判据是「最后一条写过文本的 assistant」**
|
||||
(不要求无 `tool_calls`)——旧判据只认收尾回复,多步任务里图永远「未消费」;②**只有本批新增的图
|
||||
带「一句话说明」的祈使句**,早前批次静默携带。两者合起来才断得掉「每轮要求描述 → 模型描述并继续
|
||||
调工具 → 图仍算未消费」的自维持重复循环。
|
||||
- 重读去重(engine `run.imgDelivered`)**只拦同一批 tool call 内的重复读**(一次请求里塞两份同样的图
|
||||
没意义);跨批次重读一律放行。不要写成「一回合只投递一次」,也不要靠推测「图还在不在上下文里」——
|
||||
模型说拿不到就得给,而且文件可能已重新生成(路径相同、内容不同);拦掉只会把新像素挡在门外。
|
||||
- 占位裁剪(`agent.trim_write_rounds` / `trim_result_rounds`):距尾 ≥N 个 assistant 轮的
|
||||
`write_file.content`、`fetch`/`read_file` 回执换成单行占位。**只改发给上游的副本**,DB 与 UI 不动。
|
||||
- `compressMessages` = `buildMessages` + 末尾一条压缩指令 `user`。**铁律**:压缩请求必须与上一次
|
||||
@@ -165,7 +177,7 @@ POST /api/sessions/:id/messages
|
||||
出现即报配置错(不静默忽略)。
|
||||
- `executeTool(baseCtx, name, args)`:先查 `HANDLERS` 表,未命中再交给 MCP 注册表。
|
||||
`ctx = {sessionId, taskDir, run, cfg, db, vision}`;工具返回字符串,`read_file` 读图时返回
|
||||
`{text, image_rel}`(由 engine 转成 `image_ref` 块)。
|
||||
`{text, image_rel}`(`text` 是结构化 JSON 回执,由 engine 转成 `image_ref` 块)。
|
||||
- **路径沙箱**:所有路径经 `resolveInTask` 解析到 `taskDir` 内;`/tmp` 会被映射到
|
||||
`<taskDir>/.tmp`(与 executor 容器里 `/tmp` 的 bind 对齐)。符号链接越界一律拒绝。
|
||||
- **跨会话只读**:`@<session-id>/<rel>` 只允许 `list_directory`、`read_file`、`copy_path` 的
|
||||
@@ -173,11 +185,14 @@ POST /api/sessions/:id/messages
|
||||
- `checkToolArgs` + `unwrapToolArgs`:前者做结构性预检并生成 `errAck`,后者按 schema 逐层剥掉
|
||||
弱模型多包的信封。两者都是纯函数并已导出,便于回归验证。
|
||||
- 技能(`tools/skills.js`):`SKILL.md` front matter 提供 `description`/`input_schema`。
|
||||
工具参数是「动词作键」:`{"list":"all"}` / `{"get":"<名>"}` / `{"run":"<名>","args":{...}}`,
|
||||
三者恰好给一个(无法用 `required` 表达,校验在 `toolSkill` 里做)。`input_schema.required` 在
|
||||
`run` 前校验;相对路径参数按会话目录转绝对路径;入口一律用绝对路径(远端 cwd 是会话目录)。
|
||||
配了 executor 就走 `remote-exec`,否则本地 `spawn`。旧形状 `{op, name, input_args}` 仅由
|
||||
`toolSkill` 兜底翻译(存量技能文档里还写着它),不再对外文档化。
|
||||
front matter 的 `thinking: true` 会在 `get` 时把 `run.deepThink` 置真(不影响脚本本身);
|
||||
每回合只触发一次,engine 经 SSE `think_auto` 同步前端,开启后保持到用户手动关闭。
|
||||
工具参数是扁平的 `{op, name, args}`:`op` 必填且三选一(`list`/`get`/`run`),`name` 供
|
||||
get/run 指认技能,`args` 仅 run 用;条件必填写不进 schema,由 `toolSkill` 校验并点名。
|
||||
`toolSkill` 在入口把旧字段名 `input_args` 归一化成 `args`,之后只走一条分派路径
|
||||
(曾有两条路径,其中一条只读 `input_args`,会静默丢参)。
|
||||
`input_schema.required` 在 `run` 前校验;相对路径参数按会话目录转绝对路径;入口一律用绝对路径
|
||||
(远端 cwd 是会话目录)。配了 executor 就走 `remote-exec`,否则本地 `spawn`。
|
||||
模型可见的技能清单是 `listCallableSkills`(目录技能 + `mode=skill` 的 MCP 工具,同名时目录胜出);
|
||||
设置页的技能 API 仍只用 `listSkills`(按目录增删改),两者不要混用。
|
||||
mode=skill 的 MCP 技能在 `runSkill` 里位于 `absolutizeArgs` **之前**:直接 `mcp.callTool` 转发
|
||||
@@ -249,8 +264,9 @@ auth.router ← /login /logout /api/login /api/login_restore /api/passwor
|
||||
auth.middleware ← 其余请求需登录(cookie 优先,Basic 兜底)
|
||||
preview-proxy.earlyGuard
|
||||
express.static(public)
|
||||
/api/sessions ×3 ← sessions / messages / files
|
||||
/api/sessions ×4 ← sessions / messages / files / skill-run(通用执行面)
|
||||
/api/skills
|
||||
/api/asr ← 语音识别代理:/transcribe 走登录态;ASR_URL 空则 503
|
||||
/api/config ← requireAdmin
|
||||
/api/platforms
|
||||
/api/system ← /restart 需 admin,其余登录即可
|
||||
@@ -259,6 +275,8 @@ express.static(public)
|
||||
/api/memory
|
||||
/api/wechat
|
||||
/api/channel
|
||||
/api/health ← 免认证探活:套壳 App / 运维脚本用它校验服务器地址
|
||||
(只认 200 + body.ok + service==='agent-core')
|
||||
data-page ← GET /app/:task_id/*、POST /api/db/:task_id/query
|
||||
preview-proxy.mount(末尾兜底)
|
||||
```
|
||||
@@ -266,13 +284,15 @@ preview-proxy.mount(末尾兜底)
|
||||
- 错误形状统一 `{ error: string }`,`err.status_code` 决定 HTTP 码。
|
||||
- 认证:HMAC 签名 HttpOnly cookie(`agent_token`),`Authorization: Basic` 兜底,
|
||||
`Authorization: Bearer` 仅用于 `/api/login_restore`。`auth.users` 为空 = 完全关闭认证。
|
||||
- 免认证路径集中在 `auth.js` 的 `PUBLIC_EXACT`(含 `/api/health`,且容忍尾斜杠)。
|
||||
新增免登录接口必须加进去,否则会被中间件先拦成 401。
|
||||
- **单租户警告**:除管理员接口外不做资源级授权;按用户的会话列表隔离在查询层(`sessions.user`)。
|
||||
- 公网短链 `/p/<id>` 的信任模型:随机 ID + 服务端 TTL + 签发时 `realpath` 校验,不靠登录态。
|
||||
|
||||
## 前端(`server/public/`)
|
||||
|
||||
- **无构建**:`index.html` 按顺序加载经典脚本,共享全局作用域:
|
||||
`core.js → chat.js → sessions.js → stream.js → files.js → preview.js → settings.js`
|
||||
`core.js → chat.js → sessions.js → stream.js → files.js → voice.js → preview.js → settings.js`
|
||||
(`settings.js` 必须最后,尾部有全局初始化);`marked.min.js` 走 `vendor/`。
|
||||
- 新增 JS 文件必须同时改 `index.html` 的 `<script>` 顺序——没有 import/export,顺序就是依赖。
|
||||
- `sw.js`:HTML/JS/CSS 网络优先,图片/JSON 缓存优先。**改图片或 JSON 才需要升 `CACHE_VERSION`**;
|
||||
|
||||
@@ -62,6 +62,7 @@ upload:
|
||||
image_max_dim: 2000 # 压缩后长边上限(px)
|
||||
image_quality: 0.85 # JPEG 质量
|
||||
confirm_size_mb: 10 # 任意文件二次确认阈值(MB)
|
||||
paste_to_file_threshold_chars: 2000 # 粘贴超长文本自动转 .txt 附件(字符数,0=关)
|
||||
|
||||
skills:
|
||||
skills_root: "" # 留空取 workspace_root 同级 skills 目录
|
||||
@@ -78,9 +79,15 @@ auth: # users 留空 = 不启用登
|
||||
role: admin # admin=管理员(全部设置 + 用户管理);user 或缺省=普通用户(仅账户页)
|
||||
# model: qwen-tp/qwen3.8-flash # 选填:该用户新建会话时锁定的模型(内部键「平台名/模型id」)
|
||||
|
||||
# MCP 服务列表(仓库自带的 mcp/ 三个服务,或栈外服务):每项 {name?, url, headers?, mode?(默认 advertise), timeout_ms?, tools_allow?, tools_deny?}。
|
||||
# MCP 服务列表(仓库自带的 mcp/ 服务,或栈外服务):每项 {name?, url, headers?, mode?(默认 advertise), timeout_ms?, tools_allow?, tools_deny?}。
|
||||
# mode 三态:advertise=工具下发给模型;skill=不下发,改为以技能形式暴露(get 返回原始 input_schema,run 转发);
|
||||
# disable=不连接、不发现。旧字段 enable 已废弃;mode: silent 已移除(改用 skill),写进配置会启动报错。
|
||||
#
|
||||
# 语音识别(mcp/asr)建议用 skill:转写音频是低频能力,不必把 schema 塞进每轮请求。
|
||||
# - name: asr
|
||||
# url: "http://mcp_asr:3000/mcp"
|
||||
# mode: skill
|
||||
# 注意:输入框里的语音输入走的是 core 的 /api/asr(由 compose 的 ASR_URL 接线),不经这个工具,也不消耗 token。
|
||||
mcp_servers: []
|
||||
|
||||
public_link:
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "agent-app",
|
||||
"version": "2.3.0",
|
||||
"version": "2.4.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "agent-app",
|
||||
"version": "2.3.0",
|
||||
"version": "2.4.0",
|
||||
"dependencies": {
|
||||
"better-sqlite3": "11.3.0",
|
||||
"cron-parser": "5.10.1",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "agent-app",
|
||||
"version": "2.3.0",
|
||||
"version": "2.4.0",
|
||||
"description": "自用通用 Agent 系统:多模型调度 + 任务目录 + 工具调用 + H5 Web UI",
|
||||
"main": "server/server.js",
|
||||
"scripts": {
|
||||
|
||||
+4
-1
@@ -73,7 +73,10 @@ function basicAuthUser(req, users) {
|
||||
// ---------- 免认证路径(登录页自身需要的资源) ----------
|
||||
const PUBLIC_EXACT = new Set(['/login', '/logout', '/api/login', '/api/logout', '/api/login_restore', '/api/health']);
|
||||
function isPublicPath(p) {
|
||||
if (PUBLIC_EXACT.has(p)) return true;
|
||||
// 尾斜杠容忍:客户端手动填服务器地址常多带一个 `/`(拼成 `/api/health/`),
|
||||
// Express 路由本身能匹配,但这里若不规范化,中间件会先回 401——地址明明是对的却报未登录。
|
||||
const norm = p.length > 1 ? p.replace(/\/+$/, '') : p;
|
||||
if (PUBLIC_EXACT.has(norm)) return true;
|
||||
if (p.startsWith('/css/')) return true; // 登录页复用 style.css
|
||||
if (p === '/favicon.ico') return true;
|
||||
return false;
|
||||
|
||||
@@ -90,7 +90,8 @@ const DEFAULTS = {
|
||||
// fetch、read_file 回执正文距尾 ≥N 个 assistant 轮后换单行占位(DB/UI 不动,内容可再生:盘上重读/重抓);0=关。
|
||||
agent: { temperature: 1.0, max_tool_iterations: 100, max_retries: 2, language: 'Simplified Chinese', auto_compress_tokens: 60000, auto_complete_ms: 1800000, trim_write_rounds: 1, trim_result_rounds: 2 },
|
||||
// 大图上传预处理(浏览器端 canvas 压缩,参数经 /api/system/status 下发前端)。
|
||||
upload: { image_compress_threshold_kb: 1536, image_max_dim: 2000, image_quality: 0.85, confirm_size_mb: 10 },
|
||||
// paste_to_file_threshold_chars:粘贴文本超过该字符数时前端转成 .txt 附件,0 = 关闭。
|
||||
upload: { image_compress_threshold_kb: 1536, image_max_dim: 2000, image_quality: 0.85, confirm_size_mb: 10, paste_to_file_threshold_chars: 2000 },
|
||||
// 技能根目录:留空取 workspace_root 同级 skills(/workspace/tasks → /workspace/skills)。
|
||||
skills: { skills_root: '', timeout_ms: 120000 },
|
||||
// 长期记忆(日记,memory.js):会话收口(下班)时一条几十字的第一人称碎片,一片一行入 memory_fragments,
|
||||
|
||||
+61
-18
@@ -2,7 +2,7 @@
|
||||
// 单进程内存模型:JS 单线程 + better-sqlite3 同步 API,不存在跨进程竞争。
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { buildMessages, callModel, encodeBlocks, decodeBlocks, compressMessages, titleMessages, setProfilePrompts } = require('./llm');
|
||||
const { buildMessages, callModel, encodeBlocks, decodeBlocks, compressMessages, titleMessages, setProfilePrompts, patchImageReceipt } = require('./llm');
|
||||
const { executeTool, resolveInTask } = require('./tools/executor');
|
||||
const publicFiles = require('./public-files');
|
||||
const mcp = require('./mcp');
|
||||
@@ -616,8 +616,7 @@ async function runLoopInner(sessionId, run) {
|
||||
}
|
||||
const txt = String(m.content || '');
|
||||
return m.role + (m.tool_calls ? '[calls]' : '')
|
||||
+ (txt.includes('图片数据已省略') ? '[图已省]' : '')
|
||||
+ (txt.includes('本回合已投递过') ? '[图重复读挡回]' : '');
|
||||
+ (txt.includes('"already_delivered"') ? '[图重复读挡回]' : '');
|
||||
}).join(' '));
|
||||
}
|
||||
// 内置工具 + 启动时发现的 MCP 工具(按 profile 白名单过滤),一起随 system prompt 发给模型(压缩回合不发)
|
||||
@@ -763,7 +762,13 @@ async function runLoopInner(sessionId, run) {
|
||||
} else {
|
||||
try {
|
||||
output = await executeTool({ sessionId, taskDir: taskDirOf(sessionId), run, cfg, db, vision,
|
||||
onTaskCreated: () => { run.taskWrap = true; }, onCard: (card) => bus.emit(sessionId, 'ui_card', card) }, // 通用 UI 卡片 → SSE(ctx 由 executeTool 透传给 MCP 与内置工具) // 定时即下班:工具只打标记,收口在 finalize(回合结束后才能解绑归档)
|
||||
onTaskCreated: () => { run.taskWrap = true; }, onCard: (card) => bus.emit(sessionId, 'ui_card', card),
|
||||
onDeepThink: (info) => {
|
||||
const m = (cfg.models || []).find(x => x.id === session.model_id);
|
||||
const supported = !!(m && m.thinking_params && Object.keys(m.thinking_params).length);
|
||||
log.info('skills', sessionId + ' 技能 ' + (info && info.skill) + ' 要求深度思考,本回合自动开启' + (supported ? '' : '(当前模型未配置 thinking_params,不生效)'));
|
||||
bus.emit(sessionId, 'think_auto', { on: true, skill: info && info.skill, supported });
|
||||
} }, // 通用 UI 卡片 → SSE(ctx 由 executeTool 透传给 MCP 与内置工具) // 定时即下班:工具只打标记,收口在 finalize(回合结束后才能解绑归档)
|
||||
|
||||
tc.function.name, chk.args);
|
||||
} catch (err) {
|
||||
@@ -771,31 +776,49 @@ async function runLoopInner(sessionId, run) {
|
||||
isError = true;
|
||||
}
|
||||
}
|
||||
// 工具可返回 {text, image_rel}(read_file 读图且模型支持 vision,压缩产物在 .imgcache/):
|
||||
// 落库时以 image_ref 内容块拼进 tool 消息(哨兵 JSON,见 llm.js encodeBlocks);
|
||||
// buildMessages 发请求前把未消费的图现签成公网外链 image_url 块,并入该批回执之后的
|
||||
// 一条 user 消息发送(tool 角色带图非标准形态,上游支持度差);前端工具卡走带 cookie 的下载路由渲染。
|
||||
// 工具可返回 {text, image_rel}(read_file 读图且模型支持 vision,压缩产物在 .imgcache/,
|
||||
// text 是 imageReceipt 生成的结构化 JSON 回执):落库时以 image_ref 内容块拼进 tool 消息
|
||||
// (哨兵 JSON,见 llm.js encodeBlocks);buildMessages 发请求前把未消费的图现签成公网
|
||||
// 外链 image_url 块,并入该批回执之后的一条 user 消息发送(tool 角色带图非标准形态,
|
||||
// 上游支持度差);前端工具卡走带 cookie 的下载路由渲染。
|
||||
let outText;
|
||||
let imageRel = null;
|
||||
if (output && typeof output === 'object' && output.image_rel) {
|
||||
outText = String(output.text || '');
|
||||
imageRel = String(output.image_rel);
|
||||
if (output && typeof output === 'object' && typeof output.text === 'string') {
|
||||
outText = output.text;
|
||||
if (output.image_rel) imageRel = String(output.image_rel);
|
||||
} else {
|
||||
outText = String(output == null ? '' : output);
|
||||
}
|
||||
if (run.cancelled && !isError) outText = (outText ? outText + '\n' : '') + '(用户已终止任务)';
|
||||
// 死循环护栏:同回合内同一张图只投递一次——上游能认多图(探针实测同消息双外链全对),
|
||||
// 不再按批次限流;但模型若因惯性叙事反复重读同一文件,每轮都签新图就是纯烧 token,
|
||||
// 故已投递过的图再读只回文字提示(run 随回合销毁,新回合自动重置,追问重看不受影响)。
|
||||
// 只拦「同一批 tool call 里重复读同一文件」:一次请求里塞两份完全相同的图没意义。
|
||||
// 跨批次重读一律放行。不按「本回合投递过没有」拦,也不按「图还在不在上下文里」拦:
|
||||
// 前者会把第二次读取永久锁死(实测模型被逼去重新造图);后者靠服务端推测模型看没看到,
|
||||
// 一厢情愿——模型说拿不到就该给。再者文件可能已被重新生成,路径相同内容不同,
|
||||
// 拦掉就是把新像素挡在门外(压缩产物的 rel 含 size+mtime,不同内容自然是不同 rel)。
|
||||
if (imageRel) {
|
||||
if (!run.imgDelivered) run.imgDelivered = new Set();
|
||||
if (run.imgDelivered.has(imageRel)) {
|
||||
outText += '\n[该图片本回合已投递过,不再重复携带;若你未看到它,请直接告知用户,勿反复重读同一文件]';
|
||||
if (!run.imgDelivered) run.imgDelivered = new Map();
|
||||
if (run.imgDelivered.get(imageRel) === iter) {
|
||||
const patched = patchImageReceipt(outText, {
|
||||
image_attached: false, reason: 'already_delivered', retryable: true,
|
||||
hint: '同一批 tool call 里已为这个文件投递过图片,本次不再重复;需要重看时在下一轮再读即可',
|
||||
});
|
||||
outText = patched !== null
|
||||
? patched
|
||||
: outText + '\n[同一批 tool call 里已为这个文件投递过图片,本次不再重复;需要重看时在下一轮再读即可]';
|
||||
imageRel = null;
|
||||
} else {
|
||||
run.imgDelivered.add(imageRel);
|
||||
run.imgDelivered.set(imageRel, iter);
|
||||
}
|
||||
}
|
||||
// 回执自报的「环境故障」也算失败(UI 打 ✕、库里 is_error=true),预期拒绝不算:
|
||||
// no_vision / too_large / past_session / already_delivered / link_unavailable 都是正常答复。
|
||||
// 限定工具名再解析,避免把 db 的大块查询结果白解一遍。
|
||||
if (!isError && tc.function.name === 'read_file' && typeof outText === 'string' && outText.charAt(0) === '{') {
|
||||
try {
|
||||
const o = JSON.parse(outText);
|
||||
if (o && o.kind === 'image' && o.ok === false && (o.reason === 'decode_failed' || o.reason === 'sharp_missing')) isError = true;
|
||||
} catch (_) { /* 非结构化回执(旧格式)不处理 */ }
|
||||
}
|
||||
log.debug('tool', sessionId + ' ' + tc.function.name + (isError ? ' 失败: ' : ' 完成: ') + outText);
|
||||
const toolContent = imageRel
|
||||
? encodeBlocks([
|
||||
@@ -1442,6 +1465,25 @@ function deleteMessage(sessionId, messageId) {
|
||||
return { ok: true, id: messageId, removed: removed + 1 };
|
||||
}
|
||||
|
||||
// 覆盖式编辑用户消息:删掉该条及其后的全部历史(含 tool 回执/摘要),再以新内容重开一轮。
|
||||
// 编辑点是 user 行,前缀必然停在 assistant,不会留下无配对的 tool_call。
|
||||
// 运行中不支持:runLoop 每轮重载历史,中途截断会让当前回合的 tool 回执缺失。
|
||||
function editUserMessage(sessionId, messageId, content, opts) {
|
||||
const session = getSession(sessionId);
|
||||
if (!session) throw Object.assign(new Error('会话不存在'), { status_code: 404 });
|
||||
const row = db.prepare('SELECT id, role FROM messages WHERE id = ? AND session_id = ?').get(messageId, sessionId);
|
||||
if (!row) throw Object.assign(new Error('消息不存在'), { status_code: 404 });
|
||||
if (row.role !== 'user') throw Object.assign(new Error('只能编辑用户消息'), { status_code: 400 });
|
||||
const run = runs.get(sessionId);
|
||||
if (run && run.looping) throw Object.assign(new Error('任务正在执行,请先终止再编辑'), { status_code: 409 });
|
||||
// 先确保能开跑:无可用模型时抛 503,避免删完历史却发不出去
|
||||
ensureBinding(session);
|
||||
const removed = db.prepare('DELETE FROM messages WHERE session_id = ? AND id >= ?').run(sessionId, messageId).changes;
|
||||
const r = postUserMessage(sessionId, content, opts);
|
||||
log.info('msg', sessionId + ' 编辑消息 ' + messageId + ',丢弃其后 ' + Math.max(0, removed - 1) + ' 条历史');
|
||||
return Object.assign({ removed: removed }, r);
|
||||
}
|
||||
|
||||
// 删工具回执(任务执行卡片):同步把对应 tool_call 从父 assistant 的 tool_calls 里摘掉,
|
||||
// 避免留下「有调用无结果」被严格上游拒收。父回合若是纯工具调用轮(无文本)且已无
|
||||
// 其余调用,则整行一起删(空气泡没有意义);有文本则保留为普通文本回复。
|
||||
@@ -1488,6 +1530,7 @@ module.exports = {
|
||||
clearSessionMessages,
|
||||
summarizeContext,
|
||||
deleteMessage,
|
||||
editUserMessage,
|
||||
getSession,
|
||||
getRun,
|
||||
taskDirOf,
|
||||
|
||||
+100
-38
@@ -114,7 +114,7 @@ function decodeBlocks(content) {
|
||||
// 含图行的解析缓存:哨兵 JSON 带 base64 大字段,工具循环里每个请求都整段 JSON.parse,
|
||||
// 已消费的图片解析完还只留文字块,纯重复劳动。按行 id 把解析结果挂在 session 上:
|
||||
// 单进程内存模型,行内容入库后不再变更,同一行只解析一次。
|
||||
// 缓存的是原始块数组而非「消费后投影」——消费与否仍按 lastAssistant 实时判定,
|
||||
// 缓存的是原始块数组而非「消费后投影」——消费与否仍按 lastAck 实时判定,
|
||||
// 删除后续 assistant 让图片回到「未消费」(需重新携带 base64)时也自然正确。
|
||||
function cachedDecodeBlocks(session, row) {
|
||||
if (typeof row.content !== 'string' || !row.content.startsWith('{"__blocks__":')) return null;
|
||||
@@ -126,22 +126,52 @@ function cachedDecodeBlocks(session, row) {
|
||||
return blocks;
|
||||
}
|
||||
|
||||
// read_file 图片回执是单个 JSON 对象(字段契约见 tools/executor.js 的 imageReceipt)。
|
||||
// 发送侧要改写回执时只动字段,绝不追加句子:拼在 JSON 后面会让整条回执不可解析,
|
||||
// 模型又退回「从自然语言里猜像素到底到没到」。非 JSON / 非图片回执(旧库散文、文本读取正文)
|
||||
// 返回 null,调用方自行回退。
|
||||
function patchImageReceipt(text, patch) {
|
||||
if (typeof text !== 'string' || text.charAt(0) !== '{') return null;
|
||||
let o;
|
||||
try { o = JSON.parse(text); } catch (_) { return null; }
|
||||
if (!o || o.kind !== 'image') return null;
|
||||
return JSON.stringify(Object.assign(o, patch));
|
||||
}
|
||||
// 图片编号标签:从回执字段生成「read_file <path> 1080×1920」,只用于把图与回执对上号。
|
||||
// 旧库散文回执退回首行(历史行照旧可读)。
|
||||
function imageLabel(text) {
|
||||
if (typeof text === 'string' && text.charAt(0) === '{') {
|
||||
try {
|
||||
const o = JSON.parse(text);
|
||||
if (o && o.kind === 'image') return 'read_file ' + (o.path || '') + (o.width && o.height ? ' ' + o.width + '×' + o.height : '');
|
||||
} catch (_) { /* 落到下面按散文取首行 */ }
|
||||
}
|
||||
return (text || 'read_file 图片回传').split('\n')[0];
|
||||
}
|
||||
|
||||
// 将 DB 消息行转为 OpenAI messages 数组(不含 system,由调用方拼接)。
|
||||
// 图片两种形态:旧库/用户粘贴附件的 image_url 块(data: URL)原样透传;
|
||||
// 新版 read_file 读图存的是 image_ref 块({rel},指向任务目录 .imgcache/ 产物)。
|
||||
// tool 回执里的未消费图不放在回执内发送——tool 角色带 image_url 非标准形态,
|
||||
// 大量 OpenAI 兼容上游会整批忽略或只认批次最后一张,模型看不到图却见回执写着
|
||||
// 「已回传」,会反复 read_file 补看(死循环烧 token,见多模态回退复盘);
|
||||
// 故回执降级为纯文本,图块现签成 image_url(mintUrl 公网链回调)后统一并入
|
||||
// 故回执里只留文字(read_file 读图时是一个结构化 JSON 对象,字段见
|
||||
// tools/executor.js 的 imageReceipt;发送侧只改字段、不追加句子),图块现签成
|
||||
// image_url(mintUrl 公网链回调)后统一并入
|
||||
// 该批之后的一条 user 消息(user 角色各家通吃):逐图说明文字全部置顶、图块按回执顺序紧随
|
||||
//(对齐官方示例排布;探针实测 DS 对交替排布也兼容,取低风险形态),
|
||||
// 并附「先逐张简述再继续任务」指令——兼作送达自检,破弱模型的“没收到”惯性叙事;
|
||||
// 并只对本批新增的图附「一句话说明」的自检指令——兼作送达自检,破弱模型的“没收到”
|
||||
// 惯性叙事;早前批次不再重复要求描述(旧版对全部未消费图每轮重发祈使句,驱动了重复循环);
|
||||
// 签不到(base_url 未配等)降级为文字占位,绝不把 image_ref 这种非标准块发上游。
|
||||
// 省 token:图块保留到最后一条「收尾回复」(无 tool_calls 的 assistant)之前;
|
||||
// 中间轮只发 tool_calls 不算看过,图继续携带——14:33 实测 pt 证明图已进上下文而模型
|
||||
// 仍自述没看到(弱模型行为不可靠),行为鲁棒优先于极致省钱;更早的视为已消费,
|
||||
// 之前的视为已消费,发请求时替换为文字标记(措辞不再催促重读)。DB 始终存全量:
|
||||
// UI 历史照常渲染图片,模型如需重看可再次 read_file(新回执在尾部,会重新携带图片)。
|
||||
// 省 token:图块保留到「最后一次写过文本的 assistant」之前——模型把图片描述写在 content
|
||||
// 里、同时继续发 tool_calls,就是它确实看到像素的证据。旧判据只认「无 tool_calls 的收尾
|
||||
// 回复」,于正常的多步任务(工具轮不停)里图永远算未消费,每轮都把「逐张简述…然后继续
|
||||
// 完成用户的任务」重发一遍,模型照做又产生 tool_calls——自我维持的重复循环(2026-10 实机:
|
||||
// 同一批图被反复描述直到人工终止)。没写任何文本的中间轮仍算「没看过」,图继续随请求
|
||||
// 携带(14:33 实测:图已进上下文而模型仍自述没看到,行为鲁棒优先于极致省钱)。
|
||||
// DB 始终存全量:UI 历史照常渲染图片,模型如需重看可再次 read_file(新回执在尾部)。
|
||||
// 重放时只省略像素块、**不改写回执字段**:回执是历史事实,改写成「未投递」会让模型
|
||||
// 把当时成功的读图误判为失败,进而反复重读同一条(2026-10 实机循环的直接成因)。
|
||||
// 压缩摘要相关行(见 engine.runLoopInner / summarizeContext):
|
||||
// - type='summary'(assistant,收尾答复):不跳过。它是压缩后后续轮次的上下文起点,
|
||||
// 正常以 assistant 历史行发给模型;压缩收尾时打此标。
|
||||
@@ -190,14 +220,23 @@ function buildMessages(session, language, mintUrl, profile, withProfile, trimOn)
|
||||
// tool 回执按 tool_call_id 索引:给 errAck 行拼错误提示时取执行时挡下的原因
|
||||
const toolRowById = new Map();
|
||||
for (const row of history) if (row.role === 'tool' && row.tool_call_id) toolRowById.set(row.tool_call_id, row);
|
||||
// 「已消费」分界 = 最后一条给出收尾回复(无 tool_calls)的 assistant:模型只发
|
||||
// tool_calls 的中间轮不算已阅图,图继续随请求携带直到它真正答复过一次。
|
||||
// 「已消费」分界 = 最后一条**写过文本**的 assistant 行(不要求无 tool_calls):
|
||||
// 描述写在 content 里 + 继续调工具,就是模型真看到图的证据(见文件头注释的循环复盘)。
|
||||
// errAck / archived 不发模型,不算回复:它们之前的图不算被消费。
|
||||
let lastAssistant = -1;
|
||||
let lastAck = -1;
|
||||
for (let i = history.length - 1; i >= 0; i--) {
|
||||
const r = history[i];
|
||||
if (r.role === 'assistant' && r.type !== 'errAck' && r.type !== 'archived' && !r.tool_calls) { lastAssistant = i; break; }
|
||||
if (r.role !== 'assistant' || r.type === 'errAck' || r.type === 'archived') continue;
|
||||
if (String(r.content || '').trim()) { lastAck = i; break; }
|
||||
}
|
||||
// 「本批新增」起点 = 最后一条 assistant 行之后的 tool 回执(engine 每轮把新回执行追加在末尾)。
|
||||
// 只有这些图带「请描述」的祈使句;更早批次仍未消费的图只静默携带,否则模型每轮都被
|
||||
// 要求重述一遍旧图,进而把旧任务再推一遍。
|
||||
let lastAssistantRow = -1;
|
||||
for (let i = history.length - 1; i >= 0; i--) {
|
||||
if (history[i].role === 'assistant') { lastAssistantRow = i; break; }
|
||||
}
|
||||
const newFrom = lastAssistantRow + 1;
|
||||
const dropToolCallIds = new Set();
|
||||
// 旧版 summarize_request 「已答复」判定的前置信息:最后一次「回合收尾回复」(无 tool_calls 的 assistant)的位置。
|
||||
// 指令行在其之后 = 还在等 AI 响应这条指令(旧版压缩回合未收尾),应发给模型;在其之前 = 旧指令,跳过防污染。
|
||||
@@ -212,6 +251,7 @@ function buildMessages(session, language, mintUrl, profile, withProfile, trimOn)
|
||||
const pendingImages = []; // 待投递的图块(按回执顺序;签不到时不占位)
|
||||
const pendingLabels = []; // 每图一行「对应回执」说明,合并进首条文字块(text 置顶、图在后)
|
||||
let imgSeq = 0;
|
||||
let newCount = 0; // 其中本批新增的图数:为 0 时不带「请描述」的祈使句
|
||||
history.forEach((row, i) => {
|
||||
// archived:UI 可见,但不进入模型上下文(节省 token)
|
||||
if (row.type === 'archived') return;
|
||||
@@ -266,29 +306,42 @@ function buildMessages(session, language, mintUrl, profile, withProfile, trimOn)
|
||||
const blocks = cachedDecodeBlocks(session, row);
|
||||
let content = row.content || '';
|
||||
if (blocks) {
|
||||
if (i < lastAssistant) {
|
||||
content = blocks.filter(b => b && b.type === 'text').map(b => b.text).join('\n') +
|
||||
'\n[图片数据已省略:本图已随上一条回执发送过,不再重复携带;若你当时未能识别其内容,请直接告知用户,勿反复重读同一文件]';
|
||||
if (i < lastAck) {
|
||||
// 已消费的图重放时**只省略像素,不改写回执**:回执记的是「那次读取发生了什么」
|
||||
//(当时确实投递了);「这次重放不重复发」只是传输细节,不该写进历史事实。
|
||||
// 若改写成 image_attached:false / reason:omitted_already_sent / retryable:false,
|
||||
// 模型会把历史读图当成失败、认为「这次没拿到画面」,于是反复重读同一条(实测循环)。
|
||||
// 确实需要重看时它自己再 read_file 即可,那条新回执会是完整的成功回执。
|
||||
content = blocks.filter(b => b && b.type === 'text').map(b => b.text).join('\n');
|
||||
} else if (row.role === 'tool') {
|
||||
// 未消费的图:回执只留文字,图块挪进批次后的 user 合并消息(见 pendingImages/Labels)
|
||||
const texts = blocks.filter(b => b && b.type === 'text').map(b => b.text);
|
||||
content = texts.join('\n');
|
||||
const label = (texts[0] || 'read_file 图片回传').split('\n')[0];
|
||||
const label = imageLabel(texts[0]);
|
||||
const isNew = i >= newFrom;
|
||||
let mintFailed = false;
|
||||
for (const b of blocks) {
|
||||
if (!b || (b.type !== 'image_ref' && b.type !== 'image_url')) continue;
|
||||
let block = b;
|
||||
if (b.type === 'image_ref') {
|
||||
const url = mintUrl ? mintUrl(b.rel) : null;
|
||||
if (!url) { mintFailed = true; continue; } // 签不到就不占编号:没图可描述
|
||||
block = { type: 'image_url', image_url: { url } };
|
||||
}
|
||||
const seq = ++imgSeq;
|
||||
if (b.type === 'image_url') { // 旧库内联 data: URL 原样
|
||||
pendingLabels.push('[图片 ' + seq + '] ' + label);
|
||||
pendingImages.push(b);
|
||||
continue;
|
||||
}
|
||||
const url = mintUrl ? mintUrl(b.rel) : null;
|
||||
if (url) {
|
||||
pendingLabels.push('[图片 ' + seq + '] ' + label);
|
||||
pendingImages.push({ type: 'image_url', image_url: { url } });
|
||||
} else {
|
||||
pendingLabels.push('[图片 ' + seq + '] ' + label + '(外链不可用,未取得像素:请直接告知用户,勿反复重读同一文件)');
|
||||
}
|
||||
if (isNew) newCount++;
|
||||
pendingImages.push(block);
|
||||
pendingLabels.push('[图片 ' + seq + ']' + (isNew ? '(本批新增)' : '') + ' ' + label
|
||||
+ (isNew ? '' : '(早前批次,已投递过,不要重复描述)'));
|
||||
}
|
||||
// 签不到外链 = 本次请求像素确实到不了:当场把回执字段降级,让模型拿到的是
|
||||
// 「ok 但 image_attached=false + reason」,而不是一句需要它自己解读的说明。
|
||||
if (mintFailed) {
|
||||
const degraded = patchImageReceipt(content, {
|
||||
image_attached: false, reason: 'link_unavailable', retryable: false,
|
||||
hint: '服务端签不出图片外链,本张图的像素未能随本次请求送达;重读也无法送达,请直接告知用户',
|
||||
});
|
||||
if (degraded !== null) content = degraded;
|
||||
}
|
||||
} else {
|
||||
// user 行(粘贴附件/旧形态):image_url 块(data: URL)原样;image_ref → 现签公网外链
|
||||
@@ -307,16 +360,21 @@ function buildMessages(session, language, mintUrl, profile, withProfile, trimOn)
|
||||
messages.push({ role: row.role, content });
|
||||
}
|
||||
});
|
||||
// 该批未消费图片合并发送:单条 text 置顶列出逐图对应说明,图块按编号顺序紧随;
|
||||
// 「先简述每张图再继续」兼作送达自检——模型描述得出来 = 像素真到了;描述不出 = 真没到,
|
||||
// 不会再像 14:33 那样带着图却沿“没收到”的叙事惯性重读。
|
||||
if (pendingImages.length || pendingLabels.length) {
|
||||
// 未消费图片合并发送:图上文字置顶、图块紧随(同序)。
|
||||
// 只有「本批新增」的图才能带祈使句——旧版对所有未消费图每轮都写「逐张简述…然后
|
||||
// 继续完成用户的任务」,模型把它当成新的用户任务,描述完继续干活→又产生 tool_calls→
|
||||
// 图仍算未消费→再要求描述,自维持重复。早前批次只静默携带并明确「不要重复描述」。
|
||||
if (pendingImages.length) {
|
||||
const tail = newCount
|
||||
? '\n标注「本批新增」的图片是本轮新投递的:各用一句话说明里面有什么(确认像素已收到),'
|
||||
+ '然后继续你手头的工具流程。本条不是新的用户任务,不要在这里展开分析。'
|
||||
: '\n以上均为早前批次已投递过的图片(像素仍在上下文中):不要重复描述,专注完成用户的任务。';
|
||||
messages.push({
|
||||
role: 'user',
|
||||
content: [{ type: 'text',
|
||||
text: '[系统提示] 以下不是用户新输入,是上面这批 tool 回执对应的图片(read_file 回传):\n'
|
||||
text: '[系统提示] 以下不是用户新输入,是你此前 read_file 回传的图片(按回执顺序):\n'
|
||||
+ pendingLabels.join('\n')
|
||||
+ (pendingImages.length ? '\n请按编号顺序逐张简述每张图片里有什么(表明像素确实送达),然后继续完成用户的任务。' : '') }]
|
||||
+ tail }]
|
||||
.concat(pendingImages),
|
||||
});
|
||||
}
|
||||
@@ -328,7 +386,7 @@ function buildMessages(session, language, mintUrl, profile, withProfile, trimOn)
|
||||
// 页面可重抓;write 参数只是当时快照(后续 edit_file 一次就失真,本就不该被当现势引用)。
|
||||
// 组装请求时对「距尾 ≥N 个 assistant 轮」的行换单行占位:纯函数、DB 不动(UI 工具卡照旧
|
||||
// 可展全量)、与压缩归档正交(archived 行本来就不进模型)。距离按现存行数 assistant 轮:
|
||||
// 删中间消息可让已占位行短暂复活全文(一次性断前缀缓存,与图片消费 i<lastAssistant 同一
|
||||
// 删中间消息可让已占位行短暂复活全文(一次性断前缀缓存,与图片消费 i<lastAck 同一
|
||||
// 语义,接受)。阈值可配(config agent.trim_write_rounds / trim_result_rounds,0=关),
|
||||
// 启动时 setPayloadTrim 注入。
|
||||
let TRIM = { writeRounds: 1, resultRounds: 2 };
|
||||
@@ -526,10 +584,14 @@ async function callModel(cfg, modelId, messages, tools, options) {
|
||||
// 没配/无思考的行零变化)。无论配与否都剥一遍 __reasoning,内部键不出仓库。
|
||||
if (body.messages && body.messages.length) {
|
||||
const ack = modelCfg.reasoning_ack_field;
|
||||
// 思考模式下带 tool_calls 的 assistant 行必须回传 reasoning_content:中途开启思考时,
|
||||
// 本轮早先的调用没有思维链,用空串占位(缺失会被上游 400,空串实测通过)。
|
||||
body.messages = body.messages.map(mm => {
|
||||
if (!mm || !mm.__reasoning) return mm;
|
||||
if (!mm || mm.role !== 'assistant') return mm;
|
||||
const fill = !!(opts.deepThink && ack && mm.tool_calls && !mm.__reasoning);
|
||||
if (!mm.__reasoning && !fill) return mm;
|
||||
const c = Object.assign({}, mm);
|
||||
if (ack) c[ack] = c.__reasoning;
|
||||
if (ack) c[ack] = c.__reasoning || '';
|
||||
delete c.__reasoning;
|
||||
return c;
|
||||
});
|
||||
@@ -646,4 +708,4 @@ async function callModel(cfg, modelId, messages, tools, options) {
|
||||
return { content, reasoning, tool_calls: Array.from(toolCalls.values()), finish_reason: finishReason, usage, aborted, sseFrames };
|
||||
}
|
||||
|
||||
module.exports = { callModel, buildMessages, buildSystemPrompt, systemPromptText, setSkillListText, setPayloadTrim, setProfilePrompts, encodeBlocks, decodeBlocks, compressMessages, titleMessages };
|
||||
module.exports = { callModel, buildMessages, buildSystemPrompt, systemPromptText, setSkillListText, setPayloadTrim, setProfilePrompts, encodeBlocks, decodeBlocks, compressMessages, titleMessages, patchImageReceipt };
|
||||
|
||||
+165
-11
@@ -280,7 +280,7 @@ main { flex: 1; min-height: 0; display: flex; flex-direction: column; }
|
||||
.tool-card .tool-time .act-del:hover { color: #c0392b; }
|
||||
|
||||
/* ================= 底部输入区 ================= */
|
||||
footer { flex: none; padding: 8px 12px calc(10px + env(safe-area-inset-bottom, 0px)); background: var(--page); }
|
||||
footer { position: relative; flex: none; padding: 8px 12px calc(10px + env(safe-area-inset-bottom, 0px)); background: var(--page); }
|
||||
.attach-chips { display: flex; flex-wrap: wrap; gap: 6px; margin-bottom: 8px; }
|
||||
.attach-chip {
|
||||
display: inline-flex; align-items: center; gap: 6px;
|
||||
@@ -338,6 +338,7 @@ footer { flex: none; padding: 8px 12px calc(10px + env(safe-area-inset-bottom, 0
|
||||
}
|
||||
.up-card.done::after, .up-card.err::after { display: none; }
|
||||
.input-shell {
|
||||
position: relative; /* 按住说话的遮罩以它为定位基准 */
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: stretch;
|
||||
@@ -404,6 +405,76 @@ footer { flex: none; padding: 8px 12px calc(10px + env(safe-area-inset-bottom, 0
|
||||
}
|
||||
.plus-btn:active { background: #dde0e6; }
|
||||
.plus-btn.active { background: var(--active-soft); color: var(--active); } /* 文件面板展开时高亮 */
|
||||
/* 语音模式:输入框位置换成「按住 说话」——只有居中文字,不做出按钮外观 */
|
||||
#voice-hold {
|
||||
position: relative;
|
||||
height: 44px; border-radius: 8px;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
font-size: 15px; color: var(--ink); cursor: pointer;
|
||||
user-select: none; -webkit-user-select: none; -webkit-touch-callout: none;
|
||||
touch-action: none; transition: background .12s;
|
||||
}
|
||||
/* 按住区与下方操作栏之间的分隔线(用伪元素画,避开录音态底色的圆角) */
|
||||
#voice-hold::after {
|
||||
content: '';
|
||||
position: absolute; left: 0; right: 0; bottom: -1px;
|
||||
height: 1px; background: var(--border);
|
||||
}
|
||||
/* 录音中给一个极浅底色作反馈(文字同时变成「松开 结束」) */
|
||||
#voice-hold.recording { background: var(--surface); }
|
||||
|
||||
/* 录音期间:输入卡内容整体让位给遮罩(透明渐变不能把按钮/文字透出来) */
|
||||
body.voice-active .input-shell { border-color: transparent; box-shadow: none; }
|
||||
body.voice-active .input-shell > *:not(#voice-overlay) { visibility: hidden; }
|
||||
body.voice-active .attach-chips { visibility: hidden; }
|
||||
body.voice-active .scroll-bottom-btn { display: none; }
|
||||
|
||||
/* 按住说话的遮罩:左右贴屏幕两边;底部一直铺到屏幕底(含 footer 的 10px 内边距与安全区),
|
||||
向上多铺 72px 让渐变淡出。
|
||||
声波中心对齐「按住 说话」那一行:padding-bottom 70px + 安全区 =
|
||||
59(声波中心到输入卡底部)+ 11(遮罩底边向下外扩的 1px 边框 + 10px footer 内边距)。 */
|
||||
#voice-overlay {
|
||||
position: absolute;
|
||||
top: -72px;
|
||||
bottom: calc(-11px - env(safe-area-inset-bottom, 0px));
|
||||
left: -13px; right: -13px;
|
||||
z-index: 6;
|
||||
display: flex; flex-direction: column; align-items: center; justify-content: flex-end;
|
||||
gap: 20px;
|
||||
padding-bottom: calc(70px + env(safe-area-inset-bottom, 0px));
|
||||
/* 极淡的蓝调,从下往上淡出(“几乎看不出来,仔细看才有一层”) */
|
||||
background: linear-gradient(to top, rgba(57, 100, 254, .09), rgba(57, 100, 254, 0));
|
||||
color: var(--active);
|
||||
pointer-events: auto; touch-action: none;
|
||||
transition: background .15s linear, color .15s linear;
|
||||
}
|
||||
/* 上滑取消态:整体换成极淡的红调,声波/文案同步变红 */
|
||||
#voice-overlay.cancel {
|
||||
background: linear-gradient(to top, rgba(192, 57, 43, .09), rgba(192, 57, 43, 0));
|
||||
color: #c0392b;
|
||||
}
|
||||
.vo-hint { font-size: 13px; line-height: 18px; color: var(--gray); letter-spacing: .2px; }
|
||||
.vo-wave { display: flex; align-items: center; justify-content: center; gap: 3px; height: 34px; max-width: 100%; }
|
||||
/* 柱子高度用 scaleY 而不是 height:只走合成层,不触发重排 */
|
||||
.vo-wave i {
|
||||
display: block; width: 3px; height: 34px; border-radius: 2px;
|
||||
background: currentColor;
|
||||
transform: scaleY(.12); transform-origin: center;
|
||||
transition: transform 70ms linear;
|
||||
}
|
||||
/* 上滑取消态:声波与文案统一转警示色 */
|
||||
#voice-overlay.cancel .vo-hint { color: #c0392b; }
|
||||
@media (prefers-reduced-motion: reduce) { .vo-wave i { transition: none; } }
|
||||
|
||||
/* 发送键四态:发送 / 停止 / 语音 / 键盘。语音与键盘用中性底色,和蓝色「发送」区分开 */
|
||||
.send-btn > svg { display: none; }
|
||||
.send-btn[data-state="send"] .ic-send { display: block; }
|
||||
.send-btn[data-state="voice"] .ic-voice,
|
||||
.send-btn[data-state="keyboard"] .ic-kb { display: block; }
|
||||
.send-btn[data-state="voice"],
|
||||
.send-btn[data-state="keyboard"] { background: var(--surface); color: var(--ink); box-shadow: none; }
|
||||
.send-btn[data-state="voice"]:active,
|
||||
.send-btn[data-state="keyboard"]:active { background: #dde0e6; opacity: 1; }
|
||||
/* 发送按钮:蓝底白向上箭头 */
|
||||
.send-btn {
|
||||
flex: none; width: 36px; height: 36px; border-radius: 50%;
|
||||
@@ -792,8 +863,10 @@ footer { flex: none; padding: 8px 12px calc(10px + env(safe-area-inset-bottom, 0
|
||||
}
|
||||
|
||||
/* ================= 回到底部浮动按钮(同步 index 右下角导航浮动图标) ================= */
|
||||
/* 停靠在输入区上方:包含块是 footer,bottom:100% = footer 顶边,
|
||||
输入框长高/编辑条出现时随 footer 一起上移,不再用固定值 140px 遮挡输入区。 */
|
||||
.scroll-bottom-btn {
|
||||
position: absolute; right: 14px; bottom: 140px;
|
||||
position: absolute; right: 14px; bottom: calc(100% + 6px); margin: 0;
|
||||
width: 38px; height: 38px; border-radius: 50%;
|
||||
border: none; cursor: pointer; padding: 0;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
@@ -850,20 +923,60 @@ footer { flex: none; padding: 8px 12px calc(10px + env(safe-area-inset-bottom, 0
|
||||
/* ================= 全屏查看器(搬自 index;图片单点放大+缩放,消息双击放大) ================= */
|
||||
#img-viewer { position: fixed; inset: 0; z-index: 150; background: rgba(0,0,0,0.93); display: flex; align-items: center; justify-content: center; overflow: hidden; }
|
||||
#img-viewer.hidden { display: none; }
|
||||
#img-viewer-img { max-width: 92vw; max-height: 88vh; touch-action: none; cursor: grab; user-select: none; -webkit-user-drag: none; }
|
||||
/* 两层 img 绝对定位居中叠放:滑动切换时旧图滑出、新图同步滑入(见 js/stream.js 的 slideTo) */
|
||||
#img-viewer img { position: absolute; inset: 0; margin: auto; max-width: 100vw; max-height: 88vh; touch-action: none; cursor: grab; user-select: none; -webkit-user-drag: none; will-change: transform; }
|
||||
/* 组内页码:底部居中悬浮,不参与命中测试(点它等于点背景会关闭) */
|
||||
#img-count { position: fixed; left: 50%; bottom: calc(18px + env(safe-area-inset-bottom, 0px)); transform: translateX(-50%); z-index: 151; padding: 3px 12px; border-radius: 999px; background: rgba(0,0,0,0.5); color: #fff; font-size: 12px; letter-spacing: .5px; pointer-events: none; }
|
||||
#fs-viewer { position: fixed; inset: 0; z-index: 130; background: rgba(8,10,20,0.88); padding: 40px 14px 24px; overflow: auto; /* 缩放后支持横纵滚动 */ }
|
||||
#fs-viewer.hidden { display: none; }
|
||||
.fs-panel { max-width: 860px; margin: 0 auto; background: var(--page); border-radius: 14px; padding: 18px 20px; font-size: 15.5px; }
|
||||
.fs-close {
|
||||
position: fixed; top: 12px; right: 14px; z-index: 131; width: 34px; height: 34px;
|
||||
border: none; border-radius: 50%; background: rgba(255,255,255,0.18); color: #fff;
|
||||
/* 顶部操作区:默认右上角悬浮(消息全屏用);文件预览走 fs-bar 模式时独立成一行 */
|
||||
.fs-actions { position: fixed; top: calc(2px + env(safe-area-inset-top, 0px)); right: 14px; z-index: 131; display: flex; align-items: center; gap: 8px; }
|
||||
.fs-name { display: none; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-size: 13px; font-weight: 600; color: var(--ink); }
|
||||
.fs-btn {
|
||||
flex: none; display: flex; align-items: center; justify-content: center;
|
||||
width: 34px; height: 34px; border: none; border-radius: 50%;
|
||||
background: rgba(0,0,0,0.42); color: #fff; box-shadow: 0 1px 5px rgba(0,0,0,0.22);
|
||||
font-size: 15px; line-height: 1; cursor: pointer;
|
||||
}
|
||||
/* 源码切换 eye 钮:与关闭钮同款,居其左;仅当前内容有源文本时显示(setFsSource 控显隐) */
|
||||
.fs-eye { right: 58px; display: flex; align-items: center; justify-content: center; }
|
||||
.fs-eye svg { width: 17px; height: 17px; }
|
||||
.fs-eye.on { background: rgba(255,255,255,0.85); color: #333; }
|
||||
/* 源码面板:与 #fs-body 互斥显示(toggleFsSrc 切换);不参与双指缩放逻辑 */
|
||||
.fs-btn svg { width: 17px; height: 17px; }
|
||||
.fs-btn.on { background: rgba(255,255,255,0.85); color: #333; }
|
||||
|
||||
/* 文件预览沉浸模式:40px 操作条 + 全宽内容。内容从条下开始,不会叠在页面自己的顶部按钮上;
|
||||
宽度一点不让(手机横向最贵),只牺牲一条操作栏的高度。 */
|
||||
#fs-viewer.fs-bar { display: flex; flex-direction: column; padding: 0; overflow: auto; background: var(--page); }
|
||||
#fs-viewer.fs-bar .fs-actions {
|
||||
position: sticky; top: 0; z-index: 1;
|
||||
flex: none; width: 100%;
|
||||
height: calc(40px + env(safe-area-inset-top, 0px));
|
||||
padding: env(safe-area-inset-top, 0px) 10px 0 14px;
|
||||
justify-content: flex-end; gap: 6px;
|
||||
background: var(--page); border-bottom: 1px solid var(--border);
|
||||
}
|
||||
#fs-viewer.fs-bar .fs-name { display: block; flex: 1; }
|
||||
#fs-viewer.fs-bar .fs-btn { width: 32px; height: 32px; border-radius: 8px; background: transparent; color: var(--ink); box-shadow: none; }
|
||||
#fs-viewer.fs-bar .fs-btn:active { background: var(--surface); }
|
||||
#fs-viewer.fs-bar .fs-btn.on { background: var(--active); color: #fff; }
|
||||
#fs-viewer.fs-bar .fs-panel { flex: 1; min-height: 0; width: 100%; max-width: none; margin: 0; padding: 0; border-radius: 0; background: var(--page); }
|
||||
#fs-viewer.fs-bar #fs-body { position: relative; overflow: auto; }
|
||||
#fs-viewer.fs-bar #fs-body .fs-html-frame { height: 100%; min-height: 0; border-radius: 0; }
|
||||
/* 文本/源码直排:去掉气泡卡片的内外边距,只留 12px 阅读边距 */
|
||||
#fs-viewer.fs-bar #fs-body .bubble-ai,
|
||||
#fs-viewer.fs-bar #fs-body .bubble-user {
|
||||
max-width: 100%; margin: 0; border-radius: 0; box-shadow: none; background: var(--page);
|
||||
padding: 14px 12px calc(28px + env(safe-area-inset-bottom, 0px)); font-size: 15.5px;
|
||||
}
|
||||
#fs-viewer.fs-bar #fs-src-pane { overflow: auto; }
|
||||
#fs-viewer.fs-bar #fs-src-code { max-height: none; border: 0; border-radius: 0; padding: 12px 14px calc(28px + env(safe-area-inset-bottom, 0px)); }
|
||||
#fs-viewer.fs-bar .fs-img-frame { max-width: 100%; max-height: 100%; border-radius: 0; margin: 0 auto; }
|
||||
.fs-close {
|
||||
/* 顶到视口最上沿(避开刘海),不再压住预览面板顶部 */
|
||||
position: fixed; top: calc(2px + env(safe-area-inset-top, 0px)); right: 14px; z-index: 131; width: 34px; height: 34px;
|
||||
border: none; border-radius: 50%; background: rgba(0,0,0,0.42); color: #fff;
|
||||
box-shadow: 0 1px 5px rgba(0,0,0,0.22);
|
||||
font-size: 15px; line-height: 1; cursor: pointer;
|
||||
}
|
||||
/* 源码面板:与 #fs-body 互斥显示(toggleFsSrc 切换);bar 模式下占满操作条以下区域 */
|
||||
.fs-src-panel { padding: 14px 16px; }
|
||||
#fs-src-code {
|
||||
margin: 0; white-space: pre-wrap; word-break: break-word;
|
||||
@@ -1429,3 +1542,44 @@ body.pv-resizing #pv-pane iframe { pointer-events: none; }
|
||||
.send-btn .stop-sq { display: none; width: 11px; height: 11px; border-radius: 2px; background: #fff; }
|
||||
.send-btn[data-state="stop"] > svg { display: none; }
|
||||
.send-btn[data-state="stop"] .stop-sq { display: block; }
|
||||
|
||||
/* ================= 拖拽上传提示层 ================= */
|
||||
/* window 级 dragenter/dragover 期间显示。pointer-events:none 保证遮罩不抢 drop 目标,
|
||||
drop 仍由 window 监听器接管(见 js/files.js)。
|
||||
视觉沿用站内弹窗语言:与 .modal-overlay 同一底色,白底卡片 + 16px 圆角 + 投影,
|
||||
图标用品牌蓝的浅色圆底,虚线框只作为「可投放」的轻提示。 */
|
||||
#drop-overlay {
|
||||
position: fixed; inset: 0; z-index: 190;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
background: rgba(20, 25, 50, 0.45);
|
||||
pointer-events: none;
|
||||
animation: drop-overlay-in .14s ease-out;
|
||||
}
|
||||
#drop-overlay .drop-hint {
|
||||
display: flex; flex-direction: column; align-items: center; gap: 12px;
|
||||
padding: 24px 36px;
|
||||
max-width: 80vw; text-align: center;
|
||||
border: 1.5px dashed rgba(57, 100, 254, 0.32);
|
||||
border-radius: 16px;
|
||||
background: var(--page); color: var(--ink);
|
||||
font-size: 15px; font-weight: 600;
|
||||
box-shadow: 0 12px 32px rgba(15, 17, 21, 0.18);
|
||||
}
|
||||
#drop-overlay .drop-hint .ic {
|
||||
width: 50px; height: 50px; padding: 12px;
|
||||
border-radius: 50%;
|
||||
background: var(--active-soft); color: var(--active);
|
||||
}
|
||||
@keyframes drop-overlay-in { from { opacity: 0; } to { opacity: 1; } }
|
||||
|
||||
/* ================= 编辑已发送消息提示条 ================= */
|
||||
/* 覆盖式重发:输入框内部的浅蓝提示条(嵌在 .input-shell 顶端,间距由 shell 的 gap 提供)。 */
|
||||
.edit-bar {
|
||||
display: flex; align-items: center; gap: 8px;
|
||||
padding: 6px 6px 6px 12px;
|
||||
border-radius: 12px; background: var(--active-soft);
|
||||
color: var(--active); font-size: 12.5px;
|
||||
}
|
||||
.edit-bar span { flex: 1; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.edit-bar button { flex: none; padding: 2px 10px; border-radius: 8px; font-size: 12.5px; font-weight: 600; color: var(--active); }
|
||||
.edit-bar button:active { background: rgba(57, 100, 254, 0.12); }
|
||||
|
||||
@@ -158,8 +158,15 @@
|
||||
<footer>
|
||||
<div class="attach-chips hidden" id="attach-chips"></div>
|
||||
<div class="input-shell">
|
||||
<!-- 编辑已发送消息:覆盖式重发提示条(发送后丢弃该条及其后的历史) -->
|
||||
<div class="edit-bar hidden" id="edit-bar">
|
||||
<span>正在编辑已发送的消息,发送后将丢弃其后的历史</span>
|
||||
<button id="edit-cancel" type="button">取消</button>
|
||||
</div>
|
||||
<div class="up-cards hidden" id="up-cards" aria-label="已上传文件"></div>
|
||||
<textarea id="msg-input" placeholder="传送信息" rows="2" enterkeyhint="enter"></textarea>
|
||||
<!-- 语音模式:输入框位置变成「按住 说话」(只有居中文字,无按钮外观) -->
|
||||
<div id="voice-hold" class="hidden" role="button" tabindex="0" aria-label="按住说话">按住 说话</div>
|
||||
<div class="input-actions">
|
||||
<div class="ia-side ia-left">
|
||||
<button class="think-toggle" id="deep-think-btn" title="深度思考">
|
||||
@@ -174,22 +181,31 @@
|
||||
<button class="plus-btn" id="attach-btn" title="上传文件" aria-label="上传文件">
|
||||
<svg viewBox="0 0 24 24" class="ic sm"><path fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" d="M21.44 11.05l-9.19 9.19a6 6 0 0 1-8.49-8.49l9.19-9.19a4 4 0 0 1 5.66 5.66l-9.2 9.19a2 2 0 0 1-2.83-2.83l8.49-8.48"></path></svg>
|
||||
</button>
|
||||
<!-- 同一按钮两态:发送(↑)/ 停止(白色方块)。data-state 由 JS 切换,尺寸不变避免输入区跳动 -->
|
||||
<!-- 同一按钮四态:发送(↑)/ 停止(方块)/ 语音(麦克风)/ 键盘。
|
||||
data-state 由 JS 切换,尺寸不变避免输入区跳动 -->
|
||||
<button class="send-btn" id="send-btn" data-state="send" title="发送" aria-label="发送">
|
||||
<svg viewBox="0 0 24 24" class="ic sm"><line x1="12" y1="19" x2="12" y2="5"></line><polyline points="5 12 12 5 19 12"></polyline></svg>
|
||||
<svg class="ic sm ic-send" viewBox="0 0 24 24"><line x1="12" y1="19" x2="12" y2="5"></line><polyline points="5 12 12 5 19 12"></polyline></svg>
|
||||
<svg class="ic sm ic-voice" viewBox="0 0 24 24"><rect x="9" y="2" width="6" height="12" rx="3"></rect><path d="M5 10a7 7 0 0 0 14 0"></path><line x1="12" y1="19" x2="12" y2="22"></line></svg>
|
||||
<svg class="ic sm ic-kb" viewBox="0 0 24 24"><rect x="3" y="5" width="18" height="14" rx="2.5"></rect><circle cx="7" cy="9" r="1.1" fill="currentColor" stroke="none"></circle><circle cx="12" cy="9" r="1.1" fill="currentColor" stroke="none"></circle><circle cx="17" cy="9" r="1.1" fill="currentColor" stroke="none"></circle><circle cx="7" cy="12.5" r="1.1" fill="currentColor" stroke="none"></circle><circle cx="12" cy="12.5" r="1.1" fill="currentColor" stroke="none"></circle><circle cx="17" cy="12.5" r="1.1" fill="currentColor" stroke="none"></circle><line x1="8" y1="16" x2="16" y2="16"></line></svg>
|
||||
<span class="stop-sq" aria-hidden="true"></span>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<!-- 按住说话的遮罩:盖满输入卡并扩到屏幕两边(footer 有 12px 内边距 + shell 1px 边框)。
|
||||
声波由 JS 生成 25 根柱子;上滑到文案区域进入取消态(整体变红)。 -->
|
||||
<div id="voice-overlay" class="hidden" aria-hidden="true">
|
||||
<div class="vo-hint" id="voice-hint">松手传送,上滑取消</div>
|
||||
<div class="vo-wave" id="voice-wave"></div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- 回到底部浮动按钮(搬自 index;上翻历史时出现、AI 输出中常驻转圈)。
|
||||
放在 footer 内、相对 footer 顶边停靠:输入框长高/编辑条出现时不会被盖住 -->
|
||||
<button class="scroll-bottom-btn hidden" id="scroll-bottom-btn" onclick="clickScrollBottom()" title="回到底部" aria-label="回到底部">
|
||||
<svg class="sb-arrow" viewBox="0 0 24 24"><polyline points="6 9 12 15 18 9"></polyline></svg>
|
||||
<svg class="sb-spinner hidden" viewBox="0 0 24 24"><circle cx="12" cy="12" r="10"></circle><path d="M22 12a10 10 0 0 1-10 10"></path></svg>
|
||||
</button>
|
||||
</footer>
|
||||
|
||||
<!-- 回到底部浮动按钮(搬自 index;上翻历史时出现、AI 输出中常驻转圈) -->
|
||||
<button class="scroll-bottom-btn hidden" id="scroll-bottom-btn" onclick="clickScrollBottom()" title="回到底部" aria-label="回到底部">
|
||||
<svg class="sb-arrow" viewBox="0 0 24 24"><polyline points="6 9 12 15 18 9"></polyline></svg>
|
||||
<svg class="sb-spinner hidden" viewBox="0 0 24 24"><circle cx="12" cy="12" r="10"></circle><path d="M22 12a10 10 0 0 1-10 10"></path></svg>
|
||||
</button>
|
||||
|
||||
<!-- ================= 历史记录抽屉 ================= -->
|
||||
<div id="scrim"></div>
|
||||
<aside id="drawer" aria-label="对话历史">
|
||||
@@ -435,10 +451,12 @@
|
||||
<div id="toast" class="hidden"></div>
|
||||
</div>
|
||||
|
||||
<!-- 图片全屏查看器(点击消息/工具卡中图片打开;双指缩放、放大后可拖动) -->
|
||||
<!-- 图片全屏查看器(点击消息/工具卡中图片打开;双指缩放、放大后可拖动;同组多图可左右滑动切换) -->
|
||||
<div id="img-viewer" class="hidden">
|
||||
<button class="fs-close" onclick="closeImageViewer()">✕</button>
|
||||
<img id="img-viewer-img" alt="">
|
||||
<!-- 组内页码(如 2 / 5):仅同组多图时显示 -->
|
||||
<span id="img-count" class="hidden"></span>
|
||||
</div>
|
||||
|
||||
<!-- ================= 文档预览右栏(桌面端≥960px;office/全部文档类型统一承载;移动端走全屏 #fs-viewer,本栏隐藏) -->
|
||||
@@ -471,20 +489,38 @@
|
||||
|
||||
<!-- 消息全屏查看器(双击气泡/工具卡打开;支持双指缩放) -->
|
||||
<div id="fs-viewer" class="hidden" onclick="if(event.target===this)closeFullscreen()">
|
||||
<button class="fs-close" onclick="closeFullscreen()">✕</button>
|
||||
<!-- 源码切换钮:仅当当前内容有源文本时显隐(toggleFsSrc 在渲染视图/源码面板间互切) -->
|
||||
<button class="fs-close fs-eye hidden" id="fs-src-btn" title="显示源码" aria-label="显示源码">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M1 12s4-8 11-8 11 8 11 8-4 8-11 8-11-8-11-8z"></path><circle cx="12" cy="12" r="3"></circle></svg>
|
||||
</button>
|
||||
<!-- 顶部操作区:文件预览(fs-bar 模式)时独立成一行,内容从行下开始,不遮挡页面自己的顶部按钮;
|
||||
消息全屏时退化为右上角悬浮按钮簇。文件名仅栏模式显示。 -->
|
||||
<div class="fs-actions">
|
||||
<span class="fs-name" id="fs-file-name"></span>
|
||||
<!-- 源码切换钮:仅当当前内容有源文本时显隐(toggleFsSrc 在渲染视图/源码面板间互切) -->
|
||||
<button class="fs-btn hidden" id="fs-src-btn" title="显示源码" aria-label="显示源码">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M1 12s4-8 11-8 11 8 11 8-4 8-11 8-11-8-11-8z"></path><circle cx="12" cy="12" r="3"></circle></svg>
|
||||
</button>
|
||||
<!-- 复制源码:显隐与 eye 一致(有源文本才出现);HTML/SVG/MD/代码通用 -->
|
||||
<button class="fs-btn hidden" id="fs-copy-btn" title="复制源码" aria-label="复制源码">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="13" height="13" rx="2"></rect><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"></path></svg>
|
||||
</button>
|
||||
<button class="fs-btn" onclick="closeFullscreen()" title="关闭" aria-label="关闭">✕</button>
|
||||
</div>
|
||||
<div class="fs-panel" id="fs-body"></div>
|
||||
<div class="fs-panel fs-src-panel hidden" id="fs-src-pane"><pre id="fs-src-code"></pre></div>
|
||||
</div>
|
||||
|
||||
<!-- 拖拽上传提示层:window 级 dragenter 时显示;pointer-events:none,不参与命中测试(逻辑见 js/files.js) -->
|
||||
<div id="drop-overlay" class="hidden" aria-hidden="true">
|
||||
<div class="drop-hint">
|
||||
<svg viewBox="0 0 24 24" class="ic"><path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4"></path><polyline points="17 8 12 3 7 8"></polyline><line x1="12" y1="3" x2="12" y2="15"></line></svg>
|
||||
<span>松开即可上传</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script src="/js/core.js"></script>
|
||||
<script src="/js/chat.js"></script>
|
||||
<script src="/js/sessions.js"></script>
|
||||
<script src="/js/stream.js"></script>
|
||||
<script src="/js/files.js"></script>
|
||||
<script src="/js/voice.js"></script>
|
||||
<script src="/js/preview.js"></script>
|
||||
<script src="/js/settings.js"></script>
|
||||
<script>
|
||||
|
||||
@@ -27,7 +27,7 @@
|
||||
(u.cached == null ? '未上报' : u.cached + ' tok') + ',输出 ' + (u.ct || 0) + ' tok';
|
||||
}
|
||||
|
||||
function actionsHtml(ts, content, msgId, usage) {
|
||||
function actionsHtml(ts, content, msgId, usage, canEdit) {
|
||||
const b64 = btoa(unescape(encodeURIComponent(content)));
|
||||
// 删除本条:仅已落库消息可删(传 m.id);流式未落库时无 id,不渲染
|
||||
const del = msgId
|
||||
@@ -35,6 +35,12 @@
|
||||
'<svg class="ic sm" viewBox="0 0 24 24"><path d="M18 6 6 18"></path><path d="m6 6 12 12"></path></svg>' +
|
||||
'</button>'
|
||||
: '';
|
||||
// 编辑并重发:只给未压缩的用户消息(调用方传 canEdit);带 b64 回填输入框,避免从 DOM 反推
|
||||
const edit = canEdit && msgId
|
||||
? '<button class="act act-edit" title="编辑并重发" data-msg-id="' + msgId + '" data-b64="' + b64 + '">' +
|
||||
'<svg class="ic sm" viewBox="0 0 24 24"><path d="M12 20h9"></path><path d="M16.5 3.5a2.1 2.1 0 0 1 3 3L7 19l-4 1 1-4z"></path></svg>' +
|
||||
'</button>'
|
||||
: '';
|
||||
const u = normUsage(usage);
|
||||
const usageHtml = u
|
||||
? '<span class="act-usage" title="' + usageTip(u) + '">' + usageBrief(u) + '</span>'
|
||||
@@ -45,6 +51,7 @@
|
||||
'<button class="act act-copy" title="复制" data-b64="' + b64 + '">' +
|
||||
'<svg class="ic sm" viewBox="0 0 24 24"><rect x="9" y="9" width="13" height="13" rx="2"></rect><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"></path></svg>' +
|
||||
'</button>' +
|
||||
edit +
|
||||
del +
|
||||
'</div>';
|
||||
}
|
||||
@@ -55,9 +62,10 @@
|
||||
? '<div class="attach-line">📎 ' + esc(p.files.join('、')) + '</div>' : '';
|
||||
const r = mdHtml(p.text);
|
||||
// 复制内容取纯文本(不含【系统通知】附件行),与气泡里显示一致
|
||||
// 编辑按钮只给「用户手打/粘贴」的消息:archived(已压缩存档)与 scheduler_note 等系统行不给
|
||||
return '<div class="msg me">' +
|
||||
'<div class="bubble-user' + (r.fmt ? ' md' : '') + '">' + attach + r.html + '</div>' +
|
||||
actionsHtml(m.created_at, p.text, m.id) +
|
||||
actionsHtml(m.created_at, p.text, m.id, null, !m.type) +
|
||||
'</div>';
|
||||
}
|
||||
|
||||
@@ -513,7 +521,8 @@
|
||||
const wrap = document.createElement('div');
|
||||
wrap.className = 'msg me';
|
||||
wrap.innerHTML = '<div class="bubble-user' + (r.fmt ? ' md' : '') + '">' + attach + r.html + '</div>' +
|
||||
actionsHtml(Date.now(), content, msgId);
|
||||
// 刚发送的用户消息同样可编辑(canEdit=true);actionsHtml 只在有 msgId 时才渲染按钮
|
||||
actionsHtml(Date.now(), content, msgId, null, true);
|
||||
decorateCodePre(wrap); // 用户消息里也可能有围栏代码块
|
||||
$('chat-view').appendChild(wrap);
|
||||
// 已落库的行记入增量跟踪:避免回前台补拉时把这条再追加一遍
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
* POST /api/sessions {title:'', model_id:''} 首条消息自动建档
|
||||
* GET /api/sessions/:id 打开会话(消息+文件)
|
||||
* POST /api/sessions/:id/messages {content, attachments} 发送消息
|
||||
* POST /api/sessions/:id/messages/:mid/edit 编辑用户消息(覆盖式重发)
|
||||
* POST /api/sessions/:id/files 附件上传(multipart)
|
||||
* GET /api/sessions/:id/stream SSE 流式输出
|
||||
* POST /api/sessions/:id/stop | /release 状态栏「终止/断开」
|
||||
@@ -169,6 +170,7 @@
|
||||
schedule: null, // 当前会话定时任务(详情 schedule 字段,null=无)
|
||||
sessions: [], // 抽屉列表缓存
|
||||
attachments: [], // 待发送附件 {file, name, size}
|
||||
editingMsgId: 0, // 正在编辑的历史消息 id(0=普通发送,覆盖式重发)
|
||||
lastImgBlobUrl: null, // 任务文件图片预览的 blob URL,下次预览或开新图时回收
|
||||
lastPdfBlobUrl: null, // 同上:PDF 页内预览的 blob URL(浏览器内建查看器用)
|
||||
es: null, // EventSource
|
||||
|
||||
+173
-18
@@ -3,7 +3,8 @@
|
||||
// ---------- 输入区 ----------
|
||||
const ta = $('msg-input');
|
||||
function autosize() { ta.style.height = 'auto'; ta.style.height = Math.min(ta.scrollHeight, 120) + 'px'; }
|
||||
ta.addEventListener('input', autosize);
|
||||
// 输入内容变化直接影响发送键两态:运行中一旦有字就恢复「发送」(可插话),清空后又变回「停止」
|
||||
ta.addEventListener('input', () => { autosize(); updateSendBtn(); });
|
||||
// 触屏设备(手机/平板)上:Enter 走默认行为(插入 \n),发消息靠发送按钮,
|
||||
// 避免软键盘的回车键直接被吞走导致连换行都打不出来。桌面/笔记本继续走 Enter=发送。
|
||||
// 用 (hover:none)&&(pointer:coarse) 判定「以触摸为主」的设备,标准且不依赖 UA 字符串。
|
||||
@@ -29,11 +30,14 @@
|
||||
send();
|
||||
}
|
||||
});
|
||||
// 发送键双职:生成中变「停止」(白色方块),点击终止当前回合;否则正常发送。
|
||||
// 发送键四职:语音 / 键盘(语音模式)/ 停止 / 发送,由 data-state 决定点击含义。
|
||||
// 400ms 内重复点击忽略:防双击重复 abort(服务端 status 事件回来前按钮仍是停止态)。
|
||||
let stopClickAt = 0;
|
||||
$('send-btn').addEventListener('click', () => {
|
||||
if (state.status === 'active') {
|
||||
const st = $('send-btn').dataset.state;
|
||||
if (st === 'voice' && window.AgentVoice) { window.AgentVoice.enter(); return; }
|
||||
if (st === 'keyboard' && window.AgentVoice) { window.AgentVoice.exit(true); return; }
|
||||
if (stopModeActive()) {
|
||||
if (Date.now() - stopClickAt < 400) return;
|
||||
stopClickAt = Date.now();
|
||||
stopSession();
|
||||
@@ -673,7 +677,7 @@
|
||||
if (e.key === 'Enter' || e.key === ' ') { e.preventDefault(); toggleSelectAll(); }
|
||||
});
|
||||
// ---------- 大图上传前压缩(同步 v1:阈值/参数由 /api/system/status 透出) ----------
|
||||
const DEFAULT_UPLOAD_CFG = { image_compress_threshold_kb: 1536, image_max_dim: 2000, image_quality: 0.85, confirm_size_mb: 10 };
|
||||
const DEFAULT_UPLOAD_CFG = { image_compress_threshold_kb: 1536, image_max_dim: 2000, image_quality: 0.85, confirm_size_mb: 10, paste_to_file_threshold_chars: 2000 };
|
||||
// 后端 multer 单文件硬限(routes/files.js limits.filesize),前端预检对齐它,超了直接拒掉不发起上传
|
||||
const UPLOAD_HARD_LIMIT = 50 * 1024 * 1024;
|
||||
let uploadCfgCache = null;
|
||||
@@ -813,6 +817,12 @@
|
||||
card.dataset.done = '1'; // 仅成功上传的卡片参与发送时的告知行
|
||||
closeBtn.classList.remove('hidden');
|
||||
},
|
||||
// 服务端重名会加 -1/-2:告知行必须用真实落盘名,否则模型读不到/读到旧文件
|
||||
rename(name) {
|
||||
card.dataset.name = name;
|
||||
const el = card.querySelector('.up-name');
|
||||
if (el) { el.textContent = name; el.title = name; }
|
||||
},
|
||||
fail() {
|
||||
card.classList.add('err');
|
||||
mask.innerHTML = '<span class="up-pct">上传失败</span>';
|
||||
@@ -828,12 +838,10 @@
|
||||
|
||||
// 文件选择统一入口:先逐文件预处理(大图压缩询问/大文件确认,X或遮罩=放弃该文件),
|
||||
// 只要还剩至少一个待传文件才自动建档(welcome 无会话时)并走即传路径(多文件并行上传+总进度条+逐卡进度);
|
||||
// 全部放弃或取消选择都不上传、不新建会话。
|
||||
$('file-input').addEventListener('change', async e => {
|
||||
const input = e.target;
|
||||
const files = Array.prototype.slice.call(input.files || []); // 先快照再清空,FileList 是实时的
|
||||
input.value = '';
|
||||
if (!files.length) return;
|
||||
// 全部放弃或取消选择都不上传、不新建会话。拖拽 drop 也走这里。
|
||||
// 并发批次共用一个总进度条:各自登记 {loaded,total},绘制时求和,避免后一批把前一批的进度冲掉。
|
||||
const uploadBatches = new Set();
|
||||
async function uploadFilesNow(files) {
|
||||
// 逐个串行预处理(模态弹窗不能并行)
|
||||
const ready = [];
|
||||
for (const f of files) {
|
||||
@@ -853,11 +861,13 @@
|
||||
const cards = ready.map(f => addUploadCard(f));
|
||||
// ---- 并行上传 + 实时进度 ----
|
||||
const progWrap = $('fp-progress'), progBar = $('fp-progress-bar'), progPct = $('fp-progress-pct');
|
||||
const tot = ready.reduce((s, f) => s + f.size, 0);
|
||||
const loaded = { v: 0 }; // 所有文件已上传字节总和
|
||||
const batch = { loaded: 0, total: ready.reduce((s, f) => s + f.size, 0) };
|
||||
uploadBatches.add(batch);
|
||||
let done = 0, errCount = 0;
|
||||
function paintProg() {
|
||||
const pct = tot ? Math.min(100, Math.round(loaded.v / tot * 100)) : 0;
|
||||
let loaded = 0, total = 0;
|
||||
uploadBatches.forEach(b => { loaded += b.loaded; total += b.total; });
|
||||
const pct = total ? Math.min(100, Math.round(loaded / total * 100)) : 0;
|
||||
progBar.style.width = pct + '%';
|
||||
progPct.textContent = pct + '%';
|
||||
}
|
||||
@@ -872,27 +882,172 @@
|
||||
xhr.upload.onprogress = function (ev) {
|
||||
if (!ev.lengthComputable) return;
|
||||
// ev.loaded 是本文件累计已发字节,汇总时取本次增量避免多计
|
||||
loaded.v += ev.loaded - lastLoaded;
|
||||
batch.loaded += ev.loaded - lastLoaded;
|
||||
lastLoaded = ev.loaded;
|
||||
paintProg();
|
||||
card.setPct(ev.loaded / ev.total * 100);
|
||||
};
|
||||
xhr.onload = function () {
|
||||
if (xhr.status === 200) { done++; card.done(); } else { errCount++; card.fail(); }
|
||||
if (xhr.status === 200) {
|
||||
done++;
|
||||
try {
|
||||
const u = (JSON.parse(xhr.responseText || '{}').uploaded || [])[0];
|
||||
if (u && u.filename) card.rename(u.filename);
|
||||
} catch (_) { /* 响应体异常不影响上传成功态 */ }
|
||||
card.done();
|
||||
} else { errCount++; card.fail(); }
|
||||
// 终态补齐(可能没收到最后一帧 100%)
|
||||
if (lastLoaded < f.size) { loaded.v += f.size - lastLoaded; paintProg(); }
|
||||
if (lastLoaded < f.size) { batch.loaded += f.size - lastLoaded; paintProg(); }
|
||||
resolve();
|
||||
};
|
||||
xhr.onerror = function () { errCount++; card.fail(); resolve(); };
|
||||
xhr.send(fd);
|
||||
}));
|
||||
await Promise.all(tasks);
|
||||
if (batch.loaded < batch.total) batch.loaded = batch.total;
|
||||
uploadBatches.delete(batch);
|
||||
paintProg();
|
||||
// 全部成功不再弹提示(卡片已完成态即反馈);有失败才提醒
|
||||
if (errCount) toast('已上传 ' + done + ' 个,失败 ' + errCount + ' 个');
|
||||
loadFiles();
|
||||
// 1.2s 后收起进度条
|
||||
setTimeout(() => { progWrap.classList.add('hidden'); progBar.style.width = '0%'; }, 1200);
|
||||
// 1.2s 后收起进度条(还有别的批次在传就不收)
|
||||
setTimeout(() => {
|
||||
if (uploadBatches.size) return;
|
||||
progWrap.classList.add('hidden'); progBar.style.width = '0%';
|
||||
}, 1200);
|
||||
}
|
||||
|
||||
$('file-input').addEventListener('change', e => {
|
||||
const input = e.target;
|
||||
const files = Array.prototype.slice.call(input.files || []); // 先快照再清空,FileList 是实时的
|
||||
input.value = '';
|
||||
if (files.length) uploadFilesNow(files).catch(er => toast('上传失败:' + (er && er.message || er)));
|
||||
});
|
||||
|
||||
// ---------- 拖拽文件到页面即上传(仅桌面浏览器,触屏没有 drag 事件) ----------
|
||||
// 必须挂在 window 上:文件落到输入框/面板/抽屉等任何位置,若没有 drop 处理,
|
||||
// 浏览器默认行为是导航去打开它,当前草稿和正在流式的回合会一起丢失。
|
||||
function pickDroppedFiles(dt) {
|
||||
// dataTransfer 只在事件回调内有效,这里同步取完;目录在 files 里是 0 字节条目,必须跳过
|
||||
const items = dt.items ? Array.prototype.slice.call(dt.items) : [];
|
||||
if (!items.length) return { files: Array.prototype.slice.call(dt.files || []), dirs: 0 };
|
||||
const files = [];
|
||||
let dirs = 0;
|
||||
for (const it of items) {
|
||||
if (it.kind !== 'file') continue;
|
||||
let entry = null;
|
||||
try { entry = it.webkitGetAsEntry && it.webkitGetAsEntry(); } catch (_) {}
|
||||
if (entry && entry.isDirectory) { dirs++; continue; }
|
||||
const f = it.getAsFile();
|
||||
if (f) files.push(f);
|
||||
}
|
||||
return { files, dirs };
|
||||
}
|
||||
(function bindDropUpload() {
|
||||
const overlay = $('drop-overlay');
|
||||
if (!overlay) return;
|
||||
let depth = 0, resetTimer = 0;
|
||||
// 拖动页面内选中的文字也会触发 dragenter,只认带 Files 的拖拽
|
||||
function hasFiles(e) {
|
||||
const types = e.dataTransfer && e.dataTransfer.types;
|
||||
return !!types && Array.prototype.indexOf.call(types, 'Files') !== -1;
|
||||
}
|
||||
function show() {
|
||||
clearTimeout(resetTimer);
|
||||
depth = 1;
|
||||
overlay.classList.remove('hidden');
|
||||
}
|
||||
function reset() {
|
||||
depth = 0;
|
||||
clearTimeout(resetTimer);
|
||||
overlay.classList.add('hidden');
|
||||
}
|
||||
window.addEventListener('dragenter', e => {
|
||||
if (!hasFiles(e)) return;
|
||||
e.preventDefault(); // 旧版 Chrome/Safari 需要它才让区域接受拖放
|
||||
if (!depth) depth = 1;
|
||||
clearTimeout(resetTimer);
|
||||
overlay.classList.remove('hidden');
|
||||
});
|
||||
window.addEventListener('dragover', e => {
|
||||
if (!hasFiles(e)) return;
|
||||
e.preventDefault(); // 缺这一步 drop 永远不会触发
|
||||
e.dataTransfer.dropEffect = 'copy';
|
||||
if (overlay.classList.contains('hidden')) show();
|
||||
});
|
||||
window.addEventListener('dragleave', e => {
|
||||
if (!hasFiles(e)) return;
|
||||
// 拖出窗口时浏览器不保证再配对一次 dragenter,relatedTarget 为 null 直接收干净
|
||||
if (!e.relatedTarget) { reset(); return; }
|
||||
depth = Math.max(0, depth - 1);
|
||||
if (depth) return;
|
||||
resetTimer = setTimeout(reset, 150);
|
||||
});
|
||||
window.addEventListener('dragend', reset);
|
||||
window.addEventListener('drop', e => {
|
||||
if (!hasFiles(e)) return;
|
||||
e.preventDefault(); // 阻止浏览器默认的“打开文件”
|
||||
reset();
|
||||
const picked = pickDroppedFiles(e.dataTransfer);
|
||||
if (picked.files.length) {
|
||||
uploadFilesNow(picked.files).catch(er => toast('上传失败:' + (er && er.message || er)));
|
||||
}
|
||||
if (picked.dirs) toast('暂不支持拖入文件夹' + (picked.files.length ? ',已忽略' : ''));
|
||||
});
|
||||
})();
|
||||
|
||||
// ---------- 粘贴超长文本自动转 .txt 附件 ----------
|
||||
// 延迟到发送时上传:粘贴不建档、不落盘,chip 可随时撤掉;阈值由 upload 配置下发(0=关闭)。
|
||||
let pasteThreshold = Number(DEFAULT_UPLOAD_CFG.paste_to_file_threshold_chars) || 2000;
|
||||
getUploadCfg().then(function (cfg) {
|
||||
const n = Number(cfg.paste_to_file_threshold_chars);
|
||||
if (Number.isFinite(n) && n >= 0) pasteThreshold = Math.floor(n);
|
||||
}).catch(function () { /* 取不到配置就用默认阈值 */ });
|
||||
function pastedTextName() {
|
||||
const d = new Date(), p = function (n) { return String(n).padStart(2, '0'); };
|
||||
return 'pasted-text-' + d.getFullYear() + p(d.getMonth() + 1) + p(d.getDate()) + '-' +
|
||||
p(d.getHours()) + p(d.getMinutes()) + p(d.getSeconds()) + '.txt';
|
||||
}
|
||||
function addAttachmentFiles(files, titleSeed) {
|
||||
for (const f of files) {
|
||||
// 同名同大小视为同一次粘贴的重复表示(剪贴板可能给多份)
|
||||
if (state.attachments.some(a => a.name === f.name && a.file && a.file.size === f.size)) continue;
|
||||
state.attachments.push({ file: f, name: f.name, size: f.size, titleSeed: titleSeed || '' });
|
||||
}
|
||||
renderAttachChips();
|
||||
updateSendBtn();
|
||||
}
|
||||
ta.addEventListener('paste', e => {
|
||||
if (!pasteThreshold || !e.clipboardData) return; // 0 = 关闭捕获,走浏览器默认粘贴
|
||||
// clipboardData 只在事件回调内有效:文本与文件都必须同步取完
|
||||
const text = e.clipboardData.getData('text/plain') || '';
|
||||
const files = [];
|
||||
const items = e.clipboardData.items ? Array.prototype.slice.call(e.clipboardData.items) : [];
|
||||
for (const it of items) {
|
||||
if (it.kind !== 'file') continue;
|
||||
const f = it.getAsFile();
|
||||
if (f) files.push(f);
|
||||
}
|
||||
// 部分浏览器(iOS)只给 files 不给 items
|
||||
if (!items.length && e.clipboardData.files) {
|
||||
Array.prototype.push.apply(files, Array.prototype.slice.call(e.clipboardData.files));
|
||||
}
|
||||
const trimmed = text.trim();
|
||||
if (trimmed.length <= pasteThreshold) {
|
||||
// 文本不长:保持默认可编辑粘贴;纯文件剪贴板则拦掉浏览器插入文件名的行为
|
||||
if (files.length) {
|
||||
if (!trimmed.length) e.preventDefault();
|
||||
addAttachmentFiles(files);
|
||||
}
|
||||
return;
|
||||
}
|
||||
// 超阈值:输入框不再灌大段文本;原文(不 trim,保留首尾空白与行尾)包成 .txt
|
||||
e.preventDefault();
|
||||
addAttachmentFiles([new File([text], pastedTextName(), { type: 'text/plain' })], trimmed.slice(0, 24));
|
||||
// 同时带文件时文本已保住内容,只补非图片文件(Word/PPT 里的附件);图片多为排版噪音,略过
|
||||
const extra = files.filter(f => !/^image\//.test(f.type || ''));
|
||||
if (extra.length) addAttachmentFiles(extra);
|
||||
toast('长文本已转为附件,发送时一并上传');
|
||||
});
|
||||
|
||||
|
||||
|
||||
+125
-31
@@ -15,7 +15,8 @@ const PV_PLAIN = /\.(txt|log)$/i;
|
||||
const PV_TEXT = /\.(md|markdown|txt|log|csv|tsv|json|jsonl|xml|yml|yaml|toml|ini|conf|cfg|properties|env|py|pyw|js|mjs|cjs|ts|tsx|jsx|java|kt|c|h|cpp|hpp|cc|cs|go|rs|rb|php|swift|sh|bash|zsh|bat|ps1|sql|lua|r|dart|vue|css|less|scss|srt|vtt)$/i;
|
||||
const PV_TEXT_MAX = 512 * 1024; // 文本预览截断阈值(字符),超出提示下载看全文
|
||||
let pvSeq = 0; // 预览序号:快速连点时旧请求不许再动 DOM
|
||||
const pvState = { name: null, url: null, blobUrl: null, drawerWasOpen: false, mobilePush: false };
|
||||
// reload: 当前内容的"重新载入"动作(文本类无 iframe,按名重开);为 null 时回退到 reload iframe
|
||||
const pvState = { name: null, url: null, blobUrl: null, reload: null, drawerWasOpen: false, mobilePush: false };
|
||||
|
||||
function pvDesktop() { return window.matchMedia('(min-width: 960px)').matches; }
|
||||
|
||||
@@ -34,9 +35,32 @@ function pvKind(name) {
|
||||
return 'none';
|
||||
}
|
||||
|
||||
// 文件面板「当前目录」里的图片作为一组,供查看器左右滑动切换。
|
||||
// 同组直接用下载直链(<img> 不受 Content-Disposition: attachment 影响),不必逐张取 blob。
|
||||
// 依赖 files.js 的 filesNodeAt / filesCurDir / assetDownloadUrl(同为全局,加载顺序在前)。
|
||||
function pvImgGroup(name) {
|
||||
try {
|
||||
if (typeof filesNodeAt !== 'function') return null;
|
||||
const node = filesNodeAt(typeof filesCurDir === 'string' ? filesCurDir : '');
|
||||
if (!node) return null;
|
||||
const paths = node.files.map(f => f.filename).filter(p => PV_IMG.test(p));
|
||||
if (paths.length < 2) return null;
|
||||
return {
|
||||
list: paths.map(p => ({ src: assetDownloadUrl(p), alt: p.split('/').pop() })),
|
||||
at: paths.indexOf(name),
|
||||
};
|
||||
} catch (_) { return null; }
|
||||
}
|
||||
|
||||
// 刷新/新窗口两个钮对**所有**预览类型统一开放:
|
||||
// refresh → pvState.reload(各类型都注入「按同一路径重开」,与地址条回车同一条路)
|
||||
// popup → pvState.url(office=预览网关;文本/HTML/图片/PDF=blob URL;不支持的类型=下载地址)
|
||||
// 服务端文件响应一律是 attachment,直接导航只会触发下载,所以能用 blob 的一律走 blob。
|
||||
function pvToolbar(o) {
|
||||
document.getElementById('pv-refresh').classList.toggle('hidden', !o.refresh);
|
||||
document.getElementById('pv-popup').classList.toggle('hidden', !o.popup);
|
||||
pvState.reload = typeof o.reload === 'function' ? o.reload : null;
|
||||
pvState.url = o.url === undefined ? null : o.url;
|
||||
}
|
||||
|
||||
// 每次展示内容都过一遍:标题显示相对路径(子目录可见),右上角下载钮随文件走
|
||||
@@ -50,6 +74,8 @@ function pvHeader(name, sizeText) {
|
||||
|
||||
function pvClearBody() {
|
||||
document.getElementById('pv-body').innerHTML = '';
|
||||
pvState.reload = null; // 换内容/关窗后旧的刷新动作立即失效
|
||||
pvState.url = null; // 新窗口地址同理:不清会残留上一份内容的 blob/网关 URL
|
||||
if (pvState.blobUrl) { try { URL.revokeObjectURL(pvState.blobUrl); } catch (_) {} pvState.blobUrl = null; }
|
||||
}
|
||||
|
||||
@@ -98,12 +124,21 @@ function pvCloseDoc() {
|
||||
}
|
||||
|
||||
// 移动端全屏承载:把节点塞进 #fs-viewer 并展开
|
||||
function pvOverlayShow(node) {
|
||||
// opts.bar → 文件预览沉浸模式:顶部独立操作条,内容全宽占满剩余空间
|
||||
// opts.name → 操作条左侧的文件名(仅 bar 模式可见)
|
||||
// opts.source → 源文本,交给 eye / 复制钮(未传则两者隐藏)
|
||||
function pvOverlayShow(node, opts) {
|
||||
opts = opts || {};
|
||||
const viewer = document.getElementById('fs-viewer');
|
||||
const body = document.getElementById('fs-body');
|
||||
viewer.classList.toggle('fs-bar', !!opts.bar);
|
||||
const nm = document.getElementById('fs-file-name');
|
||||
nm.textContent = opts.name ? String(opts.name) : '';
|
||||
nm.title = nm.textContent;
|
||||
body.innerHTML = ''; body.style.zoom = '';
|
||||
if (typeof setFsSource === 'function') setFsSource(null);
|
||||
if (typeof setFsSource === 'function') setFsSource(opts.source == null ? null : opts.source);
|
||||
body.appendChild(node);
|
||||
document.getElementById('fs-viewer').classList.remove('hidden');
|
||||
viewer.classList.remove('hidden');
|
||||
}
|
||||
|
||||
// ---------- office:iframe 直挂预览网关(HEAD 预探顺便触发冷槽复活) ----------
|
||||
@@ -114,29 +149,41 @@ async function pvOpenOffice(name, sizeText) {
|
||||
let load;
|
||||
if (desk) {
|
||||
pvOpenPane(); pvClearBody();
|
||||
pvHeader(name, sizeText); pvToolbar({ refresh: true, popup: false });
|
||||
pvHeader(name, sizeText); pvToolbar({ refresh: true, popup: true, reload: () => openFilePreview(name, sizeText), url });
|
||||
load = pvLoadingBox();
|
||||
document.getElementById('pv-body').appendChild(load); // 开窗即刻可见,冷启动转圈等首帧
|
||||
} else {
|
||||
load = pvLoadingBox(); load.classList.add('inline');
|
||||
pvOverlayShow(load);
|
||||
pvOverlayShow(load, { bar: true, name: name });
|
||||
if (!pvState.mobilePush) { try { history.pushState({ pvOffice: 1 }, ''); pvState.mobilePush = true; } catch (_) {} }
|
||||
}
|
||||
let head = null;
|
||||
try { head = await fetch(url, { method: 'HEAD' }); } catch (_) {}
|
||||
if (my !== pvSeq) return;
|
||||
// 404 = 预览服务里没有这份文档。文件可能正被 AI 的写盘链路整体替换(rename 瞬间找不到),
|
||||
// 探针给一次短重试再落占位卡——比让用户自己反复点刷新更接近「多刷几次就出来了」的实情。
|
||||
if (head && head.status === 404) {
|
||||
await new Promise((r) => setTimeout(r, 600));
|
||||
if (my !== pvSeq) return;
|
||||
try { head = await fetch(url, { method: 'HEAD' }); } catch (_) {}
|
||||
if (my !== pvSeq) return;
|
||||
}
|
||||
const missing = head && head.status === 404;
|
||||
if (missing) {
|
||||
const node = pvInfoNode(name, sizeText, '预览服务中暂无这份文档 —— 只有 AI 在本任务中生成/编辑过的文档才能在线预览(较旧的文档可能已被自动清理)。');
|
||||
if (desk) { pvClearBody(); document.getElementById('pv-body').appendChild(node); }
|
||||
else { const b = document.getElementById('fs-body'); b.innerHTML = ''; b.appendChild(node); }
|
||||
const node = pvInfoNode(name, sizeText, '预览服务中暂无这份文档 —— 文件可能已被删除、改名,或不在本任务目录下。');
|
||||
if (desk) {
|
||||
// pvClearBody 会清掉 reload/url:占位卡仍要保留刷新/新窗口,否则按钮可见但点了没反应
|
||||
pvClearBody(); document.getElementById('pv-body').appendChild(node);
|
||||
const dl = '/api/sessions/' + encodeURIComponent(state.sessionId) + '/files/download?path=' + encodeURIComponent(name);
|
||||
pvToolbar({ refresh: true, popup: true, reload: () => openFilePreview(name, sizeText), url: dl });
|
||||
} else { const b = document.getElementById('fs-body'); b.innerHTML = ''; b.appendChild(node); }
|
||||
return;
|
||||
}
|
||||
const fr = document.createElement('iframe');
|
||||
fr.className = desk ? 'pv-frame' : 'fs-html-frame';
|
||||
fr.title = String(name);
|
||||
fr.src = url;
|
||||
if (desk) { try { load.remove(); } catch (_) {} document.getElementById('pv-body').appendChild(fr); pvToolbar({ refresh: true, popup: true }); }
|
||||
if (desk) { try { load.remove(); } catch (_) {} document.getElementById('pv-body').appendChild(fr); pvToolbar({ refresh: true, popup: true, reload: () => openFilePreview(name, sizeText), url }); }
|
||||
else { const b = document.getElementById('fs-body'); b.innerHTML = ''; b.appendChild(fr); }
|
||||
pvState.url = url;
|
||||
}
|
||||
@@ -149,9 +196,15 @@ async function openFilePreview(name, sizeText) {
|
||||
if (kind === 'pdf' && !navigator.pdfViewerEnabled) kind = 'none'; // WebView 内嵌不了 PDF:给占位卡,不再自动下载
|
||||
const desk = pvDesktop();
|
||||
const my = ++pvSeq;
|
||||
// 工具栏两动作全局统一:刷新 = 按同一路径重开(与地址条回车同一条路),
|
||||
// 新窗口 = 打开当前内容;没有可打开的 URL 时(不支持预览/读失败)退回下载地址。
|
||||
const reload = () => openFilePreview(name, sizeText);
|
||||
const downloadUrl = '/api/sessions/' + encodeURIComponent(state.sessionId) + '/files/download?path=' + encodeURIComponent(name);
|
||||
const tbAll = { refresh: true, popup: true, reload, url: downloadUrl };
|
||||
if (desk) {
|
||||
pvOpenPane(); pvClearBody(); // 立即开窗 + 转圈,点击零延迟可见
|
||||
pvHeader(name, sizeText); pvToolbar({ refresh: false, popup: false });
|
||||
pvHeader(name, sizeText);
|
||||
pvToolbar(tbAll);
|
||||
document.getElementById('pv-body').appendChild(pvLoadingBox());
|
||||
}
|
||||
const show = (node, tb) => {
|
||||
@@ -159,15 +212,15 @@ async function openFilePreview(name, sizeText) {
|
||||
if (desk) {
|
||||
pvClearBody(); pvHeader(name, sizeText);
|
||||
document.getElementById('pv-body').appendChild(node);
|
||||
if (tb) pvToolbar(tb);
|
||||
} else pvOverlayShow(node);
|
||||
pvToolbar(tb || tbAll);
|
||||
} else pvOverlayShow(node, { bar: true, name: name });
|
||||
};
|
||||
if (kind === 'none') {
|
||||
const low = String(name).toLowerCase();
|
||||
const hint = /\.pdf$/i.test(low)
|
||||
? '当前浏览器不支持页内预览 PDF,请下载后用系统查看器打开。'
|
||||
: '该格式暂不支持在线预览,可下载后本地打开。';
|
||||
show(pvInfoNode(name, sizeText, hint), { refresh: false, popup: false });
|
||||
show(pvInfoNode(name, sizeText, hint), tbAll);
|
||||
return;
|
||||
}
|
||||
let r;
|
||||
@@ -176,38 +229,45 @@ async function openFilePreview(name, sizeText) {
|
||||
if (r.status === 401) { if (my === pvSeq) location.href = '/login?next=' + encodeURIComponent(location.pathname + location.search); return; }
|
||||
if (!r.ok) { let msg = 'HTTP ' + r.status; try { msg = (await r.json()).error || msg; } catch (_) {} throw new Error(msg); }
|
||||
} catch (e) {
|
||||
if (my === pvSeq) { desk ? show(pvInfoNode(name, sizeText, '读取失败:' + e.message), { refresh: false, popup: false }) : (pvCloseOverlayQuiet(), toast('预览失败:' + e.message)); }
|
||||
if (my === pvSeq) { desk ? show(pvInfoNode(name, sizeText, '读取失败:' + e.message), tbAll) : (pvCloseOverlayQuiet(), toast('预览失败:' + e.message)); }
|
||||
return;
|
||||
}
|
||||
try {
|
||||
if (!desk) { await pvRenderLegacy(name, r, my); return; }
|
||||
let node = null, tb = { refresh: false, popup: false };
|
||||
let node = null, tb = tbAll;
|
||||
// blob URL 不能在此登记到 pvState:show() 内部先调 pvClearBody(),那会把"刚建的"这个也一并撤销,
|
||||
// 于是 img/iframe 真正去取时 URL 已失效(Chrome 报 net::ERR_FILE_NOT_FOUND;Safari 撤回更宽松才没暴露)。
|
||||
// 统一等 show() 之后登记,陈旧序号则立即撤销防泄漏。
|
||||
let newBlobUrl = null;
|
||||
if (kind === 'html') {
|
||||
const src = rewriteHtmlDocSrcs(await r.text(), name); // 相对资源改写,否则 about:srcdoc/blob 基准下全 404
|
||||
const fr = document.createElement('iframe');
|
||||
fr.className = 'pv-frame';
|
||||
fr.setAttribute('sandbox', 'allow-same-origin allow-scripts');
|
||||
fr.srcdoc = rewriteHtmlDocSrcs(await r.text(), name); // 相对资源改写,否则 about:srcdoc 基准下全 404
|
||||
fr.srcdoc = src;
|
||||
newBlobUrl = URL.createObjectURL(new Blob([src], { type: 'text/html;charset=utf-8' }));
|
||||
tb = Object.assign({}, tbAll, { url: newBlobUrl }); // 新窗口=改写后的页面(与面板里同一份)
|
||||
node = fr;
|
||||
} else if (kind === 'pdf') {
|
||||
const u = URL.createObjectURL(await r.blob());
|
||||
newBlobUrl = u;
|
||||
const fr = document.createElement('iframe');
|
||||
fr.className = 'pv-frame'; fr.title = String(name); fr.src = u; // 不加 sandbox:内建查看器插件需要 unrestricted 帧
|
||||
node = fr; tb = { refresh: false, popup: true };
|
||||
node = fr; tb = Object.assign({}, tbAll, { url: u });
|
||||
} else if (kind === 'img') {
|
||||
const u = URL.createObjectURL(await r.blob());
|
||||
newBlobUrl = u;
|
||||
const im = document.createElement('img');
|
||||
im.className = 'pv-img'; im.src = u; im.alt = String(name);
|
||||
im.addEventListener('click', () => openImageViewer(u, String(name).split('/').pop()));
|
||||
node = im; tb = { refresh: false, popup: true };
|
||||
const g = pvImgGroup(name);
|
||||
im.addEventListener('click', () => openImageViewer(u, String(name).split('/').pop(), g && g.list, g && g.at));
|
||||
node = im; tb = Object.assign({}, tbAll, { url: u });
|
||||
} else { // text/md/代码/json/csv
|
||||
node = await pvTextNode(name, sizeText, r);
|
||||
if (node.__tb) { tb = node.__tb; delete node.__tb; }
|
||||
if (node.__tb) {
|
||||
tb = Object.assign({}, tbAll, node.__tb); delete node.__tb;
|
||||
if (/^blob:/.test(tb.url || '')) newBlobUrl = tb.url;
|
||||
}
|
||||
}
|
||||
show(node, tb);
|
||||
if (newBlobUrl) {
|
||||
@@ -215,11 +275,30 @@ async function openFilePreview(name, sizeText) {
|
||||
else { try { URL.revokeObjectURL(newBlobUrl); } catch (_) {} }
|
||||
}
|
||||
} catch (e) {
|
||||
if (my === pvSeq) desk ? show(pvInfoNode(name, sizeText, '预览失败:' + e.message), { refresh: false, popup: false }) : toast('预览失败:' + e.message);
|
||||
if (my === pvSeq) desk ? show(pvInfoNode(name, sizeText, '预览失败:' + e.message), tbAll) : toast('预览失败:' + e.message);
|
||||
}
|
||||
}
|
||||
function pvCloseOverlayQuiet() { const fs = document.getElementById('fs-viewer'); if (!fs.classList.contains('hidden') && fs.querySelector('.pv-loading')) closeFullscreen(); }
|
||||
|
||||
// 文本类「新窗口」承载:md 用面板同一份渲染 HTML(图片已改写成绝对下载地址),
|
||||
// 其余按 text/plain 原样打开——服务端文件响应一律 attachment,直接导航只会触发下载。
|
||||
// md 里原样透传的 <script> 在面板(innerHTML)本就不执行,但 blob 文档会执行,故先剔除以保持一致。
|
||||
function pvMdDoc(html, name) {
|
||||
const css =
|
||||
'body{margin:0;padding:24px 32px 48px;max-width:900px;font:15px/1.7 -apple-system,BlinkMacSystemFont,"Segoe UI","PingFang SC","Microsoft YaHei",sans-serif;color:#1f2328;background:#fff}' +
|
||||
'img{max-width:100%;height:auto}pre{background:#f6f8fa;padding:12px 14px;border-radius:8px;overflow:auto}' +
|
||||
'code{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:.92em}' +
|
||||
'table{border-collapse:collapse}th,td{border:1px solid #d8dee4;padding:6px 10px}' +
|
||||
'blockquote{margin:0;padding:0 14px;color:#57606a;border-left:3px solid #d8dee4}' +
|
||||
'a{color:#0969da}';
|
||||
const title = esc(String(name || '').split('/').pop() || '预览');
|
||||
return '<!DOCTYPE html><html><head><meta charset="utf-8">' +
|
||||
'<meta name="viewport" content="width=device-width,initial-scale=1">' +
|
||||
'<title>' + title + '</title><style>' + css + '</style></head><body>' +
|
||||
html.replace(/<script\b[\s\S]*?(?:<\/script\s*>|$)/gi, '') + '</body></html>';
|
||||
}
|
||||
function pvTextUrl(text) { return URL.createObjectURL(new Blob([text], { type: 'text/plain;charset=utf-8' })); }
|
||||
|
||||
// 桌面文本渲染:md 走气泡 markdown;txt/log 原文排版;代码/配置/数据等宽 pre;二进制嗅探回退占位卡
|
||||
async function pvTextNode(name, sizeText, r) {
|
||||
const text = await r.text();
|
||||
@@ -234,14 +313,18 @@ async function pvTextNode(name, sizeText, r) {
|
||||
wrap.className = 'msg other';
|
||||
wrap.innerHTML = '<div class="' + cls + ' pv-doc">' + html + '</div>';
|
||||
decorateCodePre(wrap);
|
||||
const n = wrap.firstChild; n.__tb = { refresh: false, popup: false };
|
||||
const n = wrap.firstChild;
|
||||
// q.fmt=false 表示 marked 不可用、面板就是原文排版:新窗口也按纯文本开,避免空白被折叠
|
||||
n.__tb = q.fmt
|
||||
? { popup: true, url: URL.createObjectURL(new Blob([pvMdDoc(html, name)], { type: 'text/html;charset=utf-8' })) }
|
||||
: { popup: true, url: pvTextUrl(text) };
|
||||
return n;
|
||||
}
|
||||
if (PV_PLAIN.test(low)) {
|
||||
const wrap = document.createElement('div');
|
||||
wrap.className = 'msg other';
|
||||
wrap.innerHTML = '<div class="bubble-ai pv-doc">' + esc(text) + '</div>';
|
||||
const n = wrap.firstChild; n.__tb = { refresh: false, popup: false };
|
||||
const n = wrap.firstChild; n.__tb = { popup: true, url: pvTextUrl(text) };
|
||||
return n;
|
||||
}
|
||||
const frag = document.createDocumentFragment();
|
||||
@@ -260,11 +343,11 @@ async function pvTextNode(name, sizeText, r) {
|
||||
const holder = document.createElement('div');
|
||||
holder.className = 'pv-plain';
|
||||
holder.appendChild(frag);
|
||||
holder.__tb = { refresh: false, popup: false };
|
||||
holder.__tb = { popup: true, url: pvTextUrl(text) }; // 新窗口用全文,不受面板截断影响
|
||||
return holder;
|
||||
}
|
||||
|
||||
// 移动端非 office:保持原全屏行为(#fs-viewer)
|
||||
// 移动端非 office:统一走 #fs-viewer 沉浸式(顶部操作条 + 全宽内容)
|
||||
async function pvRenderLegacy(name, r, my) {
|
||||
if (/\.html?$/i.test(name)) {
|
||||
const html = await r.text();
|
||||
@@ -272,8 +355,7 @@ async function pvRenderLegacy(name, r, my) {
|
||||
iframe.className = 'fs-html-frame';
|
||||
iframe.setAttribute('sandbox', 'allow-same-origin allow-scripts');
|
||||
iframe.srcdoc = rewriteHtmlDocSrcs(html, name);
|
||||
pvOverlayShow(iframe);
|
||||
setFsSource(html); // 源码钮可看页面原码
|
||||
pvOverlayShow(iframe, { bar: true, name: name, source: html }); // 全宽沉浸,源码可看/可复制
|
||||
if (my !== pvSeq) return;
|
||||
} else if (/\.pdf$/i.test(name)) {
|
||||
const u = URL.createObjectURL(await r.blob());
|
||||
@@ -283,12 +365,23 @@ async function pvRenderLegacy(name, r, my) {
|
||||
frame.className = 'fs-html-frame';
|
||||
frame.title = name;
|
||||
frame.src = u;
|
||||
pvOverlayShow(frame);
|
||||
pvOverlayShow(frame, { bar: true, name: name });
|
||||
} else if (/\.svg$/i.test(name)) {
|
||||
// SVG 当图片显示(<img> 不执行内嵌脚本),同时保留源码给 eye/复制钮
|
||||
const text = await r.text();
|
||||
const u = URL.createObjectURL(new Blob([text], { type: 'image/svg+xml;charset=utf-8' }));
|
||||
if (state.lastImgBlobUrl) { try { URL.revokeObjectURL(state.lastImgBlobUrl); } catch (_) {} }
|
||||
state.lastImgBlobUrl = u;
|
||||
const im = document.createElement('img');
|
||||
im.className = 'fs-img-frame'; im.src = u; im.alt = String(name);
|
||||
im.addEventListener('click', () => openImageViewer(u, String(name).split('/').pop()));
|
||||
pvOverlayShow(im, { bar: true, name: name, source: text });
|
||||
} else if (PV_IMG.test(name)) {
|
||||
const u = URL.createObjectURL(await r.blob());
|
||||
if (state.lastImgBlobUrl) { try { URL.revokeObjectURL(state.lastImgBlobUrl); } catch (_) {} }
|
||||
state.lastImgBlobUrl = u;
|
||||
openImageViewer(u, name);
|
||||
const g = pvImgGroup(name);
|
||||
openImageViewer(u, name, g && g.list, g && g.at);
|
||||
} else {
|
||||
const text = await r.text();
|
||||
let cls = 'bubble-ai', html;
|
||||
@@ -304,7 +397,7 @@ async function pvRenderLegacy(name, r, my) {
|
||||
wrap.className = 'msg other';
|
||||
wrap.innerHTML = '<div class="' + cls + '">' + html + '</div>';
|
||||
decorateCodePre(wrap);
|
||||
openFullscreen(wrap.firstChild, text); // 文件原文供 eye 钮看源码
|
||||
pvOverlayShow(wrap.firstChild, { bar: true, name: name, source: text }); // 全宽沉浸,eye/复制看原文
|
||||
}
|
||||
}
|
||||
|
||||
@@ -326,6 +419,7 @@ document.getElementById('pv-max').addEventListener('click', () => {
|
||||
pvSetMaxUI(max);
|
||||
});
|
||||
document.getElementById('pv-refresh').addEventListener('click', () => {
|
||||
if (typeof pvState.reload === 'function') { pvState.reload(); return; }
|
||||
const f = document.getElementById('pv-body').querySelector('iframe');
|
||||
if (!f) return;
|
||||
try { f.contentWindow.location.reload(); } catch (_) { const s = f.src; f.src = ''; f.src = s; }
|
||||
|
||||
@@ -86,6 +86,7 @@
|
||||
state.lastRenderedMsgId = 0;
|
||||
state.renderedSigs = {};
|
||||
state.streaming = false;
|
||||
if (typeof clearEditState === 'function') clearEditState();
|
||||
setView('welcome'); // 复用 newChat 同样的视图切换路径
|
||||
}
|
||||
toast('已删除会话');
|
||||
@@ -200,6 +201,7 @@
|
||||
state.toolCallStreaming = {}; // 流式参数早卡映射随会话/回合重置
|
||||
state.streaming = false;
|
||||
state.attachments = [];
|
||||
if (typeof clearEditState === 'function') clearEditState();
|
||||
userPaused = false;
|
||||
state.chatHasMore = false; // 分页状态先复位:防详情返回前用旧会话的游标/标志拉到错会话的消息
|
||||
state.chatOldestId = null;
|
||||
@@ -249,6 +251,7 @@
|
||||
state.schedule = null;
|
||||
state.pendingModel = '';
|
||||
state.attachments = [];
|
||||
if (typeof clearEditState === 'function') clearEditState();
|
||||
state.streamEl = null;
|
||||
state.streamText = '';
|
||||
state.toolCards = {};
|
||||
@@ -290,6 +293,35 @@
|
||||
return state.sessionId ? (state.modelId || '') : (state.pendingModel || '');
|
||||
}
|
||||
|
||||
// 发送键两态:只有「正在生成且输入框为空」才是停止。一开始输入文字就恢复发送——
|
||||
// 运行中允许插话(后端 postUserMessage 返回 delivered:'live',下一轮迭代会带上这条消息)。
|
||||
// 输入框按 trim 判空:纯空白等同于空(与 send() 的裁剪一致,不能因为一个空格就发不了消息)。
|
||||
function stopModeActive() {
|
||||
// 有附件时按钮保持「发送」:粘贴长文本转成的附件不应被当成停止操作
|
||||
return state.status === 'active' && !(($('msg-input') || {}).value || '').trim() && !state.attachments.length;
|
||||
}
|
||||
// 发送键四态:停止 / 键盘(语音模式)/ 语音(空输入且支持录音)/ 发送。
|
||||
// 停止优先:生成中且输入框为空时,用户最可能想停止,不能被语音入口顶掉。
|
||||
// 语音与键盘两态由 voice.js 暴露的 window.AgentVoice 提供(脚本加载顺序在 sessions 之后,
|
||||
// 初始化阶段它还不存在,此处必须容错——那时按普通「发送」渲染即可)。
|
||||
function updateSendBtn() {
|
||||
const sb = $('send-btn');
|
||||
if (!sb) return;
|
||||
const v = window.AgentVoice;
|
||||
const empty = !(($('msg-input') || {}).value || '').trim() && !state.attachments.length;
|
||||
let mode;
|
||||
if (stopModeActive()) mode = 'stop';
|
||||
else if (v && v.isVoiceMode()) mode = 'keyboard';
|
||||
else if (empty && v && v.isAvailable()) mode = 'voice';
|
||||
else mode = 'send';
|
||||
sb.dataset.state = mode;
|
||||
const label = mode === 'stop' ? '停止生成'
|
||||
: mode === 'keyboard' ? '切换到键盘输入'
|
||||
: mode === 'voice' ? '语音输入' : '发送';
|
||||
if (sb.title !== label) sb.title = label;
|
||||
if (sb.getAttribute('aria-label') !== label) sb.setAttribute('aria-label', label);
|
||||
}
|
||||
|
||||
function updateHeaderActions() {
|
||||
const inChat = state.view === 'chat' && !!state.sessionId;
|
||||
const running = state.status === 'active';
|
||||
@@ -341,12 +373,8 @@
|
||||
ssub.classList.toggle('hidden', !t);
|
||||
}
|
||||
// 发送键在生成中切「停止」:同一按钮换图标,尺寸不变,输入区不跳动
|
||||
const sb = $('send-btn');
|
||||
if (sb) {
|
||||
sb.dataset.state = running ? 'stop' : 'send';
|
||||
sb.title = running ? '停止生成' : '发送';
|
||||
sb.setAttribute('aria-label', running ? '停止生成' : '发送');
|
||||
}
|
||||
// (是否真的显示停止还取决于输入框是否有字,见 stopModeActive)
|
||||
updateSendBtn();
|
||||
}
|
||||
|
||||
function openHeaderMenu() {
|
||||
|
||||
@@ -615,6 +615,7 @@
|
||||
{ path: 'upload.image_max_dim', label: '压缩后长边上限(px)', type: 'int' },
|
||||
{ path: 'upload.image_quality', label: 'JPEG 质量(0~1)', type: 'float' },
|
||||
{ path: 'upload.confirm_size_mb', label: '大文件确认阈值(MB)', type: 'int' },
|
||||
{ path: 'upload.paste_to_file_threshold_chars', label: '粘贴转文件阈值(字符,0=关)', type: 'int' },
|
||||
] },
|
||||
];
|
||||
function kvPlain(label, v) {
|
||||
|
||||
+307
-59
@@ -2,30 +2,34 @@
|
||||
'use strict';
|
||||
// ---------- 发送 ----------
|
||||
async function uploadFiles(sessionId, files) {
|
||||
const fd = new FormData();
|
||||
for (const f of files) fd.append('files', f.file, f.name);
|
||||
// 大文件上传在慢速网络上可以远超普通请求超时:放宽到 120s
|
||||
const r = await req('POST', '/api/sessions/' + sessionId + '/files', fd, true, 120000);
|
||||
return (r.uploaded || []).map(u => ({ name: u.filename, size: u.size }));
|
||||
const out = [];
|
||||
// 后端 multer 单请求最多 10 个文件(limits 见 routes/files.js),超过就分批
|
||||
for (let i = 0; i < files.length; i += 10) {
|
||||
const fd = new FormData();
|
||||
for (const f of files.slice(i, i + 10)) fd.append('files', f.file, f.name);
|
||||
// 大文件上传在慢速网络上可以远超普通请求超时:放宽到 120s
|
||||
const r = await req('POST', '/api/sessions/' + sessionId + '/files', fd, true, 120000);
|
||||
out.push(...(r.uploaded || []));
|
||||
}
|
||||
return out.map(u => ({ name: u.filename, size: u.size }));
|
||||
}
|
||||
|
||||
async function send() {
|
||||
const ta = $('msg-input');
|
||||
// 生成中发送键已变「停止」:不接受新消息(不做排队发送,停止后再发)
|
||||
if (state.status === 'active') { toast('正在生成中,请先停止'); return; }
|
||||
const content = ta.value.trim();
|
||||
// 没有文字就不发(避免产生空消息);上传文件告知行在发送时拼在正文前(见下)。
|
||||
if (!content) {
|
||||
if (state.attachments.length) toast('附件需随一条文字消息发送');
|
||||
return;
|
||||
}
|
||||
const editingId = state.editingMsgId || 0;
|
||||
// 没有文字就不发(避免产生空消息);但只有附件时允许发送——粘贴长文本转成的 .txt 需要能单独发出。
|
||||
// 运行中也是这条规则:有字就发(后端 live 追投),没字又没附件时按钮本身是「停止」。
|
||||
if (!content && !state.attachments.length) return;
|
||||
|
||||
const pending = state.attachments.slice();
|
||||
state.attachments = [];
|
||||
renderAttachChips();
|
||||
ta.value = '';
|
||||
autosize();
|
||||
updateSendBtn(); // 清空后若仍在生成中,按钮应回到「停止」
|
||||
|
||||
let domDropped = false;
|
||||
try {
|
||||
const id = await ensureSession();
|
||||
let atts = [];
|
||||
@@ -33,20 +37,35 @@
|
||||
// 输入区上传卡片里成功入库的文件:正文前拼一行告知,让 AI 知道任务目录多了哪些文件
|
||||
const upNames = Array.from($('up-cards').querySelectorAll('.up-card[data-done="1"]'))
|
||||
.map(c => c.dataset.name).filter(Boolean);
|
||||
const text = upNames.length
|
||||
? '用户上传了文件:' + upNames.join('、') + '\n\n' + content
|
||||
// 输入区上传卡片 + 本条消息随附的附件(粘贴转文件等)都要告知;
|
||||
// atts 用的是服务端返回的真实文件名(重名会自动加序号)。
|
||||
const names = upNames.concat(atts.map(a => a.name));
|
||||
const text = names.length
|
||||
? '用户上传了文件:' + names.join('、') + (content ? '\n\n' + content : '')
|
||||
: content;
|
||||
const body = { content: text, deepThink: !!state.deepThink };
|
||||
const sent = await req('POST', '/api/sessions/' + id + '/messages', body);
|
||||
let sent;
|
||||
if (editingId) {
|
||||
// 覆盖式编辑:先撤掉本地该条及其后的渲染,再让服务端删除同范围历史并以新内容重开一轮
|
||||
dropDomFromMessage(editingId);
|
||||
domDropped = true;
|
||||
sent = await req('POST', '/api/sessions/' + id + '/messages/' + editingId + '/edit', body);
|
||||
clearEditState();
|
||||
} else {
|
||||
sent = await req('POST', '/api/sessions/' + id + '/messages', body);
|
||||
}
|
||||
clearUploadCards(); // 消息已发出(告知行已拼入正文),卡片使命完成清掉
|
||||
|
||||
appendUserMsg(text, atts.map(a => a.name), sent && sent.message_id);
|
||||
if (!state.sessionTitle && content) {
|
||||
// 标题取用户原文,不带文件告知行
|
||||
state.sessionTitle = content.length > 24 ? content.slice(0, 24) : content;
|
||||
// 标题取用户原文,不带文件告知行;纯附件发送时退回粘贴文本的前缀
|
||||
const seed = content || (pending[0] && pending[0].titleSeed) || '';
|
||||
if (!state.sessionTitle && seed) {
|
||||
state.sessionTitle = seed.length > 24 ? seed.slice(0, 24) : seed;
|
||||
}
|
||||
renderHeader();
|
||||
addTyping();
|
||||
// 运行中插话:正文字流进行中已有一个气泡作「活着」的信号,不再挂打字占位
|
||||
//(刚挂上就会被下一个 delta 移除,只会闪一下);工具间隙里补上占位反而有用。
|
||||
if (!state.streaming) addTyping();
|
||||
loadSessionList();
|
||||
} catch (e) {
|
||||
// 发送失败:附件还原+草稿文本回填(发送前已清空输入框,挂起/失败不能让文字蒸发),
|
||||
@@ -57,6 +76,9 @@
|
||||
ta.value = content + (ta.value ? '\n' + ta.value : '');
|
||||
autosize();
|
||||
}
|
||||
updateSendBtn(); // 文字/附件都可能被还原:按当前状态刷新发送键(附件在时不能停在停止态)
|
||||
// 编辑请求失败(如运行中 409):服务端历史未动,把被乐观撤掉的节点重拉回来
|
||||
if (domDropped) restoreChatView();
|
||||
toast('发送失败:' + e.message);
|
||||
}
|
||||
}
|
||||
@@ -136,7 +158,7 @@
|
||||
state.lastSseAt = Date.now(); // 建连即重置静默计时
|
||||
// 任何具名事件都算连接存活:静默看门狗与回前台宽限共用(error 不列,原生连接错误也走 error)
|
||||
['ping', 'status', 'assistant_delta', 'reasoning_delta', 'assistant_message', 'tool_call_start',
|
||||
'tool_call_args', 'tool_call', 'tool_result', 'ui_card', 'auto_compress', 'done', 'title']
|
||||
'tool_call_args', 'tool_call', 'tool_result', 'ui_card', 'auto_compress', 'done', 'title', 'think_auto']
|
||||
.forEach(function (n) { es.addEventListener(n, function () { markSseAlive(n); }); });
|
||||
es.addEventListener('status', ev => {
|
||||
const d = JSON.parse(ev.data);
|
||||
@@ -146,6 +168,17 @@
|
||||
updateHeaderActions();
|
||||
updateScrollBottomBtn(); // 状态切换同步右下角转圈(开始/结束干活)
|
||||
});
|
||||
es.addEventListener('think_auto', ev => {
|
||||
const d = JSON.parse(ev.data);
|
||||
if (!d.supported) {
|
||||
toast('当前模型未配置思考参数,技能 ' + d.skill + ' 的深度思考要求无法生效');
|
||||
return;
|
||||
}
|
||||
if (state.deepThink) return;
|
||||
state.deepThink = true;
|
||||
$('deep-think-btn').classList.add('on');
|
||||
toast('技能 ' + d.skill + ' 需要深度思考,已自动开启');
|
||||
});
|
||||
es.addEventListener('assistant_delta', ev => {
|
||||
const d = JSON.parse(ev.data);
|
||||
removeTyping();
|
||||
@@ -415,6 +448,69 @@
|
||||
}
|
||||
});
|
||||
|
||||
// ---------- 编辑已发送的用户消息(覆盖式) ----------
|
||||
// 发送时服务端会删除该条及其后的全部历史(含工具回执/摘要),前端同步丢弃同范围的 DOM。
|
||||
// 已压缩存档的消息(type='archived')不给编辑入口:删掉摘要后它前面的行仍不发模型,上下文会断。
|
||||
function clearEditState() {
|
||||
state.editingMsgId = 0;
|
||||
const bar = $('edit-bar');
|
||||
if (bar) bar.classList.add('hidden');
|
||||
}
|
||||
function setEditState(msgId) {
|
||||
state.editingMsgId = msgId;
|
||||
const bar = $('edit-bar');
|
||||
if (bar) bar.classList.remove('hidden');
|
||||
}
|
||||
// 乐观撤场:删掉该条及之后的所有节点(旧工具卡/打字点/流式气泡都在后面),
|
||||
// 并把增量渲染游标回退到编辑点之前,免得后续软补拉把已丢的行当成还在。
|
||||
function dropDomFromMessage(msgId) {
|
||||
const box = $('chat-view');
|
||||
const btn = box && box.querySelector('[data-msg-id="' + msgId + '"]');
|
||||
const node = btn && btn.closest('.msg');
|
||||
if (!node) return false;
|
||||
let n = node;
|
||||
while (n) { const next = n.nextElementSibling; n.remove(); n = next; }
|
||||
let max = 0;
|
||||
for (const k of Object.keys(state.renderedSigs)) {
|
||||
const id = Number(k);
|
||||
if (id >= msgId) delete state.renderedSigs[k];
|
||||
else if (id > max) max = id;
|
||||
}
|
||||
state.lastRenderedMsgId = max;
|
||||
return true;
|
||||
}
|
||||
// 编辑失败(如运行中 409):服务端没动历史,重拉已加载量恢复被乐观撤掉的节点。
|
||||
async function restoreChatView() {
|
||||
try {
|
||||
const detail = await req('GET', '/api/sessions/' + state.sessionId + '?limit=' + Math.max(CHAT_PAGE_SIZE, state.chatLoadedCount));
|
||||
if (state.view === 'chat') renderDetail(detail);
|
||||
} catch (_) { /* 恢复失败不阻断主流程 */ }
|
||||
}
|
||||
$('chat-view').addEventListener('click', e => {
|
||||
const btn = e.target.closest('.act-edit');
|
||||
if (!btn) return;
|
||||
const msgId = Number(btn.dataset.msgId);
|
||||
if (!msgId || !state.sessionId) return;
|
||||
if (state.status === 'active') { toast('任务执行中,暂不支持编辑'); return; }
|
||||
let text = '';
|
||||
try { text = decodeURIComponent(escape(atob(btn.dataset.b64 || ''))); } catch (_) { return; }
|
||||
const ta = $('msg-input');
|
||||
ta.value = text;
|
||||
autosize();
|
||||
updateSendBtn();
|
||||
setEditState(msgId);
|
||||
ta.focus();
|
||||
try { ta.setSelectionRange(text.length, text.length); } catch (_) {}
|
||||
});
|
||||
$('edit-cancel').addEventListener('click', () => {
|
||||
clearEditState();
|
||||
const ta = $('msg-input');
|
||||
ta.value = '';
|
||||
autosize();
|
||||
updateSendBtn();
|
||||
ta.focus();
|
||||
});
|
||||
|
||||
// ---------- 全屏查看器(双击气泡/工具卡放大;图片单击放大;均同步 index) ----------
|
||||
// 源码查看:当前内容有源文本时(消息/文件预览)右上角 eye 钮可点,源码面板与渲染视图互切。
|
||||
// 源文本两条来路:预览调用方直接传(文件原文);双击消息时从尾部操作行复制钮取
|
||||
@@ -431,6 +527,8 @@
|
||||
btn.classList.remove('on');
|
||||
btn.title = '显示源码';
|
||||
btn.setAttribute('aria-label', '显示源码');
|
||||
// 复制钮与 eye 同显隐:有源码才可复制
|
||||
$('fs-copy-btn').classList.toggle('hidden', fsSrcText == null);
|
||||
}
|
||||
function toggleFsSrc() {
|
||||
if (fsSrcText == null) return;
|
||||
@@ -445,8 +543,15 @@
|
||||
btn.setAttribute('aria-label', lbl);
|
||||
}
|
||||
$('fs-src-btn').addEventListener('click', toggleFsSrc);
|
||||
// 复制源码:HTML/SVG/MD/代码通用,拿到的就是 eye 面板里那份文本
|
||||
$('fs-copy-btn').addEventListener('click', async () => {
|
||||
if (fsSrcText == null) return;
|
||||
const ok = await copyText(fsSrcText);
|
||||
toast(ok ? '已复制' : '复制失败');
|
||||
});
|
||||
|
||||
function openFullscreen(el, src) {
|
||||
$('fs-viewer').classList.remove('fs-bar'); // 消息全屏沿用悬浮按钮,清掉文件预览的栏模式
|
||||
const body = $('fs-body');
|
||||
body.innerHTML = '';
|
||||
body.style.zoom = '';
|
||||
@@ -456,8 +561,8 @@
|
||||
clone.querySelectorAll('[id]').forEach(function (n) { n.removeAttribute('id'); });
|
||||
// 全屏查看直接展开参数/结果,方便观看
|
||||
clone.querySelectorAll('details').forEach(function (d) { d.open = true; });
|
||||
// 克隆里摘掉删除钮:删除的委托绑在 chat-view,查看器里点了无响应,不留死按钮
|
||||
clone.querySelectorAll('.act-del').forEach(function (n) { n.remove(); });
|
||||
// 克隆里摘掉删除/编辑钮:二者的委托绑在 chat-view,查看器里点了无响应,不留死按钮
|
||||
clone.querySelectorAll('.act-del, .act-edit').forEach(function (n) { n.remove(); });
|
||||
// 源文本:调用方传入优先(文件预览),否则从消息自带 b64 现取(双击场景)
|
||||
let s = src;
|
||||
if (s == null) {
|
||||
@@ -470,6 +575,7 @@
|
||||
}
|
||||
function closeFullscreen() {
|
||||
setFsSource(null); // 顺手收起源码面板/隐藏 eye 钮,下轮打开不残留
|
||||
$('fs-viewer').classList.remove('fs-bar');
|
||||
$('fs-viewer').classList.add('hidden');
|
||||
$('fs-viewer').dispatchEvent(new Event('fs:close'));
|
||||
}
|
||||
@@ -484,67 +590,209 @@
|
||||
});
|
||||
|
||||
// 点击消息/工具卡里的图片 → 图片全屏查看器(#fs-viewer 里的克隆图也生效)
|
||||
// 同一气泡/工具卡内的图作为一组,查看器里可左右滑动切换
|
||||
function imgViewerDelegate(e) {
|
||||
const im = e.target.closest('.bubble-ai img, .bubble-user img, .tool-card img');
|
||||
if (im && im.src) openImageViewer(im.src, im.alt || '');
|
||||
if (!im || !im.src) return;
|
||||
const box = im.closest('.msg') || im.closest('.tool-card') || $('fs-body');
|
||||
const list = Array.prototype.map.call(box.querySelectorAll('img'), x => ({ src: x.src, alt: x.alt || '' }))
|
||||
.filter(x => x.src);
|
||||
openImageViewer(im.src, im.alt || '', list);
|
||||
}
|
||||
$('chat-view').addEventListener('click', imgViewerDelegate);
|
||||
$('fs-viewer').addEventListener('click', imgViewerDelegate);
|
||||
|
||||
// 图片全屏查看器:transform 缩放+平移。双指捻合缩放(1~6x)、放大后单指/鼠标拖动、双击切 2.5x。
|
||||
// 第 3 参 list([{src,alt}] 或 [src]) 传入「同组图片」时开启左右滑动切换;第 4 参 at 指定起始下标。
|
||||
// 切图用双层 img 做真正的分页位移(旧图跟手滑出、新图同步滑入),并在开图时预取相邻图,
|
||||
// 避免「松手→瞬移复位→闪白→图才下载出来」这三连卡顿。
|
||||
let openImageViewer, closeImageViewer;
|
||||
(function () {
|
||||
const viewer = document.getElementById('img-viewer');
|
||||
const img = document.getElementById('img-viewer-img');
|
||||
// 第二层由脚本建(不写进 HTML):旧版缓存的 HTML 里没有它也不会把查看器搞挂
|
||||
const img2 = document.createElement('img');
|
||||
img2.className = 'hidden'; img2.alt = '';
|
||||
viewer.appendChild(img2);
|
||||
const count = document.getElementById('img-count');
|
||||
let cur = img, other = img2; // cur=当前层;other=滑动时接替的层
|
||||
let scale = 1, tx = 0, ty = 0, mode = null;
|
||||
let sx = 0, sy = 0, stx = 0, sty = 0, pDist = 0, pScale = 1;
|
||||
// gallery=当前组;gi=组内下标;wx/wy/wdx=横滑起点与实时位移;dragDir/pending=本次拖动方向与待接替下标
|
||||
let gallery = [], gi = 0, wx = 0, wy = 0, wdx = 0, dragDir = 0, pending = -1;
|
||||
const preloads = new Map(); // src → Image:提前下载并解码,切图时不再等网络
|
||||
let settleTimer = 0;
|
||||
const tdist = t => Math.hypot(t[0].clientX - t[1].clientX, t[0].clientY - t[1].clientY);
|
||||
const W = () => window.innerWidth;
|
||||
function apply() {
|
||||
img.style.transform = 'translate(' + tx + 'px,' + ty + 'px) scale(' + scale + ')';
|
||||
img.style.cursor = scale > 1 ? 'grab' : 'zoom-in';
|
||||
cur.style.transition = 'none'; // 缩放/拖动要跟手,不能被上一段滑动动画的过渡拖着走
|
||||
cur.style.transform = 'translate(' + tx + 'px,' + ty + 'px) scale(' + scale + ')';
|
||||
cur.style.cursor = scale > 1 ? 'grab' : 'zoom-in';
|
||||
}
|
||||
openImageViewer = function (src, alt) {
|
||||
scale = 1; tx = ty = 0; mode = null;
|
||||
function drawCount() {
|
||||
if (!count) return; // 旧版 HTML 缓存里没有计数节点时不报错
|
||||
const on = gallery.length > 1;
|
||||
count.textContent = on ? (gi + 1) + ' / ' + gallery.length : '';
|
||||
count.classList.toggle('hidden', !on);
|
||||
}
|
||||
// 预取相邻图:Image 对象常驻,解码结果留在内存里,滑过去就是现成的
|
||||
function preload(src) {
|
||||
if (!src || preloads.has(src) || /^(data|blob):/i.test(src)) return;
|
||||
const im = new Image();
|
||||
im.decoding = 'async';
|
||||
im.src = src;
|
||||
preloads.set(src, im);
|
||||
}
|
||||
function prefetchAround() {
|
||||
[gi + 1, gi + 2, gi - 1].forEach(i => { if (gallery[i]) preload(gallery[i].src); });
|
||||
}
|
||||
// 把某层摆到「相对屏幕中心偏移 offset 像素」处(1x、无过渡)
|
||||
function place(el, offset) {
|
||||
el.style.transition = 'none';
|
||||
el.style.transform = 'translate(' + offset + 'px,0) scale(1)';
|
||||
}
|
||||
function showLayer(el, item) {
|
||||
el.src = item.src; el.alt = item.alt || '';
|
||||
el.classList.remove('hidden');
|
||||
}
|
||||
// 动画收尾:旧层收起、两层归位(滑到一半又开始新手势时也靠它清场)
|
||||
function finishSettle() {
|
||||
clearTimeout(settleTimer); settleTimer = 0;
|
||||
other.classList.add('hidden');
|
||||
place(other, 0); place(cur, 0);
|
||||
apply();
|
||||
img.src = src; img.alt = alt || '';
|
||||
prefetchAround();
|
||||
}
|
||||
// 滑到第 i 张(dir=+1 从右侧进,-1 从左侧进);已到边界就弹回
|
||||
function slideTo(i, dir) {
|
||||
if (settleTimer) finishSettle(); // 上一张还在动画里(连点方向键):先收尾,避免接替层被瞬移
|
||||
const t = Math.max(0, Math.min(gallery.length - 1, i));
|
||||
if (t === gi) return settleBack(dir);
|
||||
const from = cur, to = other;
|
||||
if (pending !== t) { place(to, dir * W()); showLayer(to, gallery[t]); }
|
||||
void to.offsetWidth; // 先把起点落位再加过渡,否则首帧会被合并成瞬移
|
||||
const ease = 'transform .2s cubic-bezier(.22,.61,.36,1)';
|
||||
from.style.transition = ease; to.style.transition = ease;
|
||||
from.style.transform = 'translate(' + (-dir * W()) + 'px,0) scale(1)';
|
||||
to.style.transform = 'translate(0,0) scale(1)';
|
||||
cur = to; other = from;
|
||||
gi = t; scale = 1; tx = ty = 0; dragDir = 0; pending = -1;
|
||||
drawCount();
|
||||
settleTimer = setTimeout(finishSettle, 220);
|
||||
}
|
||||
// 没过阈值/到边界:当前层弹回原位,接替层退回屏幕外
|
||||
function settleBack(dir) {
|
||||
const ease = 'transform .2s ease-out';
|
||||
cur.style.transition = ease;
|
||||
cur.style.transform = 'translate(0,0) scale(1)';
|
||||
if (dir && !other.classList.contains('hidden')) {
|
||||
other.style.transition = ease;
|
||||
other.style.transform = 'translate(' + dir * W() + 'px,0) scale(1)';
|
||||
}
|
||||
scale = 1; tx = ty = 0; dragDir = 0; pending = -1;
|
||||
settleTimer = setTimeout(finishSettle, 220);
|
||||
}
|
||||
openImageViewer = function (src, alt, list, at) {
|
||||
const raw = Array.isArray(list) && list.length ? list : [{ src: src, alt: alt }];
|
||||
gallery = raw.map(x => (typeof x === 'string' ? { src: x, alt: '' } : x)).filter(x => x && x.src);
|
||||
// 起始下标优先用调用方给的(文件面板那张是 blob URL,与组里的直链对不上),否则按 src 匹配
|
||||
let i = typeof at === 'number' && at >= 0 ? at : gallery.findIndex(x => x.src === src);
|
||||
if (i < 0) i = 0;
|
||||
clearTimeout(settleTimer); settleTimer = 0;
|
||||
cur = img; other = img2;
|
||||
gi = i; scale = 1; tx = ty = 0; mode = null; dragDir = 0; pending = -1;
|
||||
other.classList.add('hidden'); place(other, 0);
|
||||
showLayer(cur, gallery[gi] || { src: src, alt: alt });
|
||||
place(cur, 0); apply();
|
||||
drawCount();
|
||||
viewer.classList.remove('hidden');
|
||||
prefetchAround(); // 开图即把前后两张拉下来,滑的时候是现成的
|
||||
};
|
||||
closeImageViewer = function () {
|
||||
viewer.classList.add('hidden');
|
||||
gallery = []; gi = 0; dragDir = 0; pending = -1;
|
||||
preloads.clear(); // 断开引用,解码位图随之释放
|
||||
drawCount();
|
||||
};
|
||||
closeImageViewer = function () { viewer.classList.add('hidden'); };
|
||||
// 点背景(非图片)关闭
|
||||
viewer.addEventListener('click', function (e) { if (e.target === viewer) closeImageViewer(); });
|
||||
img.addEventListener('dblclick', function () {
|
||||
scale = scale > 1 ? 1 : 2.5;
|
||||
if (scale === 1) { tx = ty = 0; }
|
||||
apply();
|
||||
});
|
||||
img.addEventListener('touchstart', function (e) {
|
||||
if (e.touches.length === 2) { mode = 'pinch'; pDist = tdist(e.touches) || 1; pScale = scale; }
|
||||
else if (e.touches.length === 1 && scale > 1) { mode = 'pan'; sx = e.touches[0].clientX; sy = e.touches[0].clientY; stx = tx; sty = ty; }
|
||||
else mode = null;
|
||||
}, { passive: true });
|
||||
img.addEventListener('touchmove', function (e) {
|
||||
if (mode === 'pinch' && e.touches.length >= 2) {
|
||||
e.preventDefault();
|
||||
scale = Math.min(6, Math.max(1, pScale * tdist(e.touches) / pDist));
|
||||
// 两层都要绑:切图后当前层会换成另一张 img,事件不能只挂在第一层上
|
||||
function bindLayer(el) {
|
||||
el.addEventListener('dblclick', function () {
|
||||
scale = scale > 1 ? 1 : 2.5;
|
||||
if (scale === 1) { tx = ty = 0; }
|
||||
apply();
|
||||
} else if (mode === 'pan' && e.touches.length === 1) {
|
||||
});
|
||||
el.addEventListener('touchstart', function (e) {
|
||||
if (settleTimer) finishSettle(); // 上一次滑动还没收尾:先清场再新手势
|
||||
if (e.touches.length === 2) { mode = 'pinch'; pDist = tdist(e.touches) || 1; pScale = scale; }
|
||||
else if (e.touches.length === 1 && scale > 1) { mode = 'pan'; sx = e.touches[0].clientX; sy = e.touches[0].clientY; stx = tx; sty = ty; }
|
||||
else if (e.touches.length === 1 && gallery.length > 1) { mode = 'swipe'; wx = e.touches[0].clientX; wy = e.touches[0].clientY; wdx = 0; dragDir = 0; pending = -1; }
|
||||
else mode = null;
|
||||
}, { passive: true });
|
||||
el.addEventListener('touchmove', function (e) {
|
||||
if (mode === 'pinch' && e.touches.length >= 2) {
|
||||
e.preventDefault();
|
||||
scale = Math.min(6, Math.max(1, pScale * tdist(e.touches) / pDist));
|
||||
if (scale === 1) { tx = ty = 0; }
|
||||
apply();
|
||||
} else if (mode === 'pan' && e.touches.length === 1) {
|
||||
e.preventDefault();
|
||||
tx = stx + e.touches[0].clientX - sx;
|
||||
ty = sty + e.touches[0].clientY - sy;
|
||||
apply();
|
||||
} else if (mode === 'swipe' && e.touches.length === 1) {
|
||||
const dx = e.touches[0].clientX - wx, dy = e.touches[0].clientY - wy;
|
||||
if (Math.abs(dx) <= Math.abs(dy)) return; // 竖直为主:不接管
|
||||
wdx = dx;
|
||||
const dir = dx < 0 ? 1 : -1; // 左滑看下一张
|
||||
if (dir !== dragDir) {
|
||||
dragDir = dir;
|
||||
const next = gallery[gi + dir];
|
||||
// 方向一确定就把相邻图摆到另一侧待命(已预取,几乎是现成的),跟手滑进来
|
||||
if (next) { pending = gi + dir; place(other, dir * W()); showLayer(other, next); }
|
||||
else { pending = -1; other.classList.add('hidden'); } // 到头了:没有下一张,只做橡皮筋
|
||||
}
|
||||
const off = pending >= 0 ? dx : dx * 0.35;
|
||||
cur.style.transition = 'none';
|
||||
cur.style.transform = 'translate(' + off + 'px,0) scale(1)';
|
||||
if (pending >= 0) {
|
||||
other.style.transition = 'none';
|
||||
other.style.transform = 'translate(' + (off + dragDir * W()) + 'px,0) scale(1)';
|
||||
}
|
||||
}
|
||||
}, { passive: false });
|
||||
el.addEventListener('touchend', function (e) {
|
||||
if (mode === 'swipe') {
|
||||
const dir = dragDir;
|
||||
if (pending >= 0 && Math.abs(wdx) > 45) slideTo(pending, dir);
|
||||
else settleBack(dir);
|
||||
mode = null;
|
||||
return;
|
||||
}
|
||||
if (e.touches.length < 2) { if (mode === 'pinch') mode = null; }
|
||||
if (!e.touches.length) mode = null;
|
||||
}, { passive: true });
|
||||
el.addEventListener('touchcancel', function () {
|
||||
if (mode === 'swipe') settleBack(dragDir);
|
||||
mode = null;
|
||||
}, { passive: true });
|
||||
// 桌面:放大后按住拖动平移
|
||||
el.addEventListener('mousedown', function (e) {
|
||||
if (scale <= 1) return;
|
||||
e.preventDefault();
|
||||
tx = stx + e.touches[0].clientX - sx;
|
||||
ty = sty + e.touches[0].clientY - sy;
|
||||
apply();
|
||||
}
|
||||
}, { passive: false });
|
||||
img.addEventListener('touchend', function (e) { if (e.touches.length < 2) { if (mode === 'pinch') mode = null; } if (!e.touches.length) mode = null; }, { passive: true });
|
||||
img.addEventListener('touchcancel', function () { mode = null; }, { passive: true });
|
||||
// 桌面:放大后按住拖动平移
|
||||
img.addEventListener('mousedown', function (e) {
|
||||
if (scale <= 1) return;
|
||||
e.preventDefault();
|
||||
const x0 = e.clientX, y0 = e.clientY, tx0 = tx, ty0 = ty;
|
||||
const mv = function (ev) { tx = tx0 + ev.clientX - x0; ty = ty0 + ev.clientY - y0; apply(); };
|
||||
const up = function () { window.removeEventListener('mousemove', mv); window.removeEventListener('mouseup', up); };
|
||||
window.addEventListener('mousemove', mv); window.addEventListener('mouseup', up);
|
||||
const x0 = e.clientX, y0 = e.clientY, tx0 = tx, ty0 = ty;
|
||||
const mv = function (ev) { tx = tx0 + ev.clientX - x0; ty = ty0 + ev.clientY - y0; apply(); };
|
||||
const up = function () { window.removeEventListener('mousemove', mv); window.removeEventListener('mouseup', up); };
|
||||
window.addEventListener('mousemove', mv); window.addEventListener('mouseup', up);
|
||||
});
|
||||
}
|
||||
bindLayer(img); bindLayer(img2);
|
||||
// 桌面没有滑动:组内多图时用 ←/→ 切换(单图不响应)
|
||||
document.addEventListener('keydown', function (e) {
|
||||
if (viewer.classList.contains('hidden') || gallery.length < 2) return;
|
||||
if (e.key === 'ArrowLeft') slideTo(gi - 1, -1);
|
||||
else if (e.key === 'ArrowRight') slideTo(gi + 1, 1);
|
||||
});
|
||||
})();
|
||||
|
||||
|
||||
@@ -0,0 +1,559 @@
|
||||
// 语音输入(按住说话)
|
||||
//
|
||||
// 交互:
|
||||
// · 输入框为空时,发送键变成麦克风;点一下进入语音模式
|
||||
// · 语音模式下输入框位置变成「按住 说话」:按住后整块输入区被全宽遮罩盖住,
|
||||
// 显示实时电平声波 + 「松手传送,上滑取消」
|
||||
// · 手指上滑进入文案区域 → 取消态(声波与文案变红,文案改「松开取消」),滑回下方恢复
|
||||
// · 松手:取消态 → 丢弃 + 轻提示;否则转写后**直接发送**
|
||||
// · 发送键位置变成键盘按钮,点一下回文本模式并聚焦输入框(移动端弹键盘)
|
||||
//
|
||||
// 录音在客户端:套壳原生桥(http 页面也能用)或 HTTPS 下的 MediaRecorder;
|
||||
// 转写在服务端 mcp_asr(经 core 的 /api/asr 中转)。
|
||||
// 电平:浏览器用 AnalyserNode 算 RMS;套壳上报 onNativeVoiceAmp(原始 getMaxAmplitude)。
|
||||
// 归一化/门限/平滑全部在页面侧做,壳不做任何换算。
|
||||
//
|
||||
// 包在 IIFE 里:前端是共享全局作用域的经典脚本,顶层声明会和其他文件撞名。
|
||||
|
||||
(function () {
|
||||
'use strict';
|
||||
|
||||
const hold = $('voice-hold');
|
||||
const ta = $('msg-input');
|
||||
const overlay = $('voice-overlay');
|
||||
const hintEl = $('voice-hint');
|
||||
const waveEl = $('voice-wave');
|
||||
if (!hold || !ta) return;
|
||||
|
||||
const native = (window.AgentNative && typeof window.AgentNative.startVoice === 'function')
|
||||
? window.AgentNative : null;
|
||||
const webOk = !!(window.isSecureContext && navigator.mediaDevices
|
||||
&& typeof navigator.mediaDevices.getUserMedia === 'function' && window.MediaRecorder);
|
||||
|
||||
const MAX_MS = 60000; // 单次录音上限
|
||||
const MIN_MS = 500; // 短于这个时长按误触处理,不上传
|
||||
const NO_RESULT_MS = 3000; // 松开后壳迟迟不回音频(比如还在等权限)→ 复位
|
||||
const BAR_COUNT = 45; // 声波柱数(要足够宽,手指按住时两侧仍能看到)
|
||||
|
||||
// 两端矮、中间高的包络:看起来像一段声波而不是一排等高柱子
|
||||
const WEIGHTS = [];
|
||||
for (let i = 0; i < BAR_COUNT; i++) {
|
||||
WEIGHTS.push(0.45 + 0.55 * Math.sin(Math.PI * (i + 0.5) / BAR_COUNT));
|
||||
}
|
||||
const bars = [];
|
||||
if (waveEl) {
|
||||
for (let i = 0; i < BAR_COUNT; i++) {
|
||||
const b = document.createElement('i');
|
||||
waveEl.appendChild(b);
|
||||
bars.push(b);
|
||||
}
|
||||
}
|
||||
|
||||
let available = false; // 有能力 + 服务端接线:发送键才显示麦克风
|
||||
let mode = false; // 是否处于语音模式
|
||||
let phase = 'idle'; // idle | arming(等 getMedia)| recording | busy
|
||||
let pressing = false;
|
||||
let pressY = 0; // 本次按压的起始 Y(判断“确实上滑了”)
|
||||
let cancelArmed = false; // 已上滑到取消区
|
||||
let discard = false; // 本次录音太短/已取消:结果到了也丢掉
|
||||
let recordStartAt = 0;
|
||||
let stopTimer = null;
|
||||
let hardTimer = null;
|
||||
let watchdog = null;
|
||||
let rec = null;
|
||||
let stream = null;
|
||||
let chunks = [];
|
||||
let mime = '';
|
||||
|
||||
// 电平与渲染
|
||||
let rafId = 0;
|
||||
let levelTarget = 0;
|
||||
let levelCur = 0;
|
||||
let lastRealLevelAt = 0;
|
||||
let audioCtx = null;
|
||||
let analyser = null;
|
||||
let audioBuf = null;
|
||||
let srcNode = null;
|
||||
|
||||
function clamp01(v) { return v < 0 ? 0 : v > 1 ? 1 : v; }
|
||||
|
||||
// 电平归一化的唯一实现:浏览器(RMS)和壳(峰值幅度)都在这里落到 0~1。
|
||||
// 低于 -55dB 视为静音,-5dB 及以上视为满幅;要调灵敏度只改这两个常量。
|
||||
const LEVEL_FLOOR_DB = -55;
|
||||
const LEVEL_RANGE_DB = 50;
|
||||
function dbToLevel(db) {
|
||||
return clamp01((db - LEVEL_FLOOR_DB) / LEVEL_RANGE_DB);
|
||||
}
|
||||
|
||||
function setHoldText(s) {
|
||||
if (hold.textContent !== s) hold.textContent = s;
|
||||
}
|
||||
|
||||
// ---------- 遮罩 / 声波 ----------
|
||||
function setOverlayMode(m) { // 'send' | 'cancel' | 'busy'
|
||||
cancelArmed = m === 'cancel';
|
||||
if (overlay) overlay.classList.toggle('cancel', m === 'cancel');
|
||||
if (hintEl) {
|
||||
hintEl.textContent = m === 'cancel' ? '松开取消'
|
||||
: m === 'busy' ? '正在识别…' : '松手传送,上滑取消';
|
||||
}
|
||||
}
|
||||
|
||||
function renderBars(level, animate) {
|
||||
if (!bars.length) return;
|
||||
const t = Date.now() / 1000;
|
||||
// 只有真的收到可听电平(大于门限)才加流动感;安静时保持完全静止,不靠抖动“装活”
|
||||
const flow = animate && level > 0.02;
|
||||
for (let i = 0; i < bars.length; i++) {
|
||||
const ripple = flow ? (0.88 + 0.12 * Math.sin(t * 6 + i * 0.55)) : 1;
|
||||
const h = Math.max(0.08, Math.min(1, (0.15 + 0.85 * level) * WEIGHTS[i] * ripple));
|
||||
bars[i].style.transform = 'scaleY(' + h.toFixed(3) + ')';
|
||||
}
|
||||
}
|
||||
|
||||
function startLoop() {
|
||||
if (rafId || !overlay || overlay.classList.contains('hidden')) return;
|
||||
if (typeof requestAnimationFrame !== 'function') return;
|
||||
rafId = requestAnimationFrame(tickMeter);
|
||||
}
|
||||
|
||||
function tickMeter() {
|
||||
rafId = 0;
|
||||
if (!overlay || overlay.classList.contains('hidden')) return;
|
||||
|
||||
let animate = false;
|
||||
if (analyser && audioBuf) {
|
||||
analyser.getFloatTimeDomainData(audioBuf);
|
||||
let sum = 0;
|
||||
for (let i = 0; i < audioBuf.length; i++) sum += audioBuf[i] * audioBuf[i];
|
||||
const rms = Math.sqrt(sum / audioBuf.length);
|
||||
const db = 20 * Math.log10(rms + 1e-7); // 静音约 -140,正常说话 -35 ~ -12
|
||||
levelTarget = dbToLevel(db);
|
||||
animate = true;
|
||||
} else if (Date.now() - lastRealLevelAt <= 300) {
|
||||
animate = true; // 壳刚上报过电平
|
||||
}
|
||||
|
||||
if (!animate) {
|
||||
// 没有电平来源(壳没实现 onNativeVoiceAmp / AnalyserNode 不可用):
|
||||
// 静默停在基线,不再空转(也绝不能假动)
|
||||
levelCur = 0;
|
||||
renderBars(0, false);
|
||||
return;
|
||||
}
|
||||
|
||||
// 快攻慢放:起音跟得紧,收音回落慢,柱体不抽筋
|
||||
levelCur += (levelTarget - levelCur) * (levelTarget > levelCur ? 0.55 : 0.18);
|
||||
renderBars(levelCur, true);
|
||||
rafId = requestAnimationFrame(tickMeter);
|
||||
}
|
||||
|
||||
function attachAnalyser() {
|
||||
if (!stream) return;
|
||||
try {
|
||||
const AC = window.AudioContext || window.webkitAudioContext;
|
||||
if (!AC) return;
|
||||
audioCtx = new AC();
|
||||
analyser = audioCtx.createAnalyser();
|
||||
analyser.fftSize = 512;
|
||||
audioBuf = new Float32Array(analyser.fftSize);
|
||||
srcNode = audioCtx.createMediaStreamSource(stream);
|
||||
srcNode.connect(analyser); // 不接 destination,不会回放
|
||||
if (audioCtx.state === 'suspended') audioCtx.resume();
|
||||
startLoop(); // 之前可能在静默态停掉了循环
|
||||
} catch (_) {
|
||||
analyser = null;
|
||||
audioBuf = null;
|
||||
}
|
||||
}
|
||||
|
||||
function startMeter() {
|
||||
stopMeter(false);
|
||||
levelTarget = 0;
|
||||
levelCur = 0;
|
||||
lastRealLevelAt = 0;
|
||||
startLoop();
|
||||
}
|
||||
|
||||
function stopMeter(flatten) {
|
||||
if (rafId) { cancelAnimationFrame(rafId); rafId = 0; }
|
||||
if (srcNode) { try { srcNode.disconnect(); } catch (_) { /* 已断 */ } srcNode = null; }
|
||||
analyser = null;
|
||||
audioBuf = null;
|
||||
if (audioCtx) { try { audioCtx.close(); } catch (_) { /* 已关 */ } audioCtx = null; }
|
||||
if (flatten !== false) {
|
||||
for (let i = 0; i < bars.length; i++) bars[i].style.transform = 'scaleY(.12)';
|
||||
}
|
||||
}
|
||||
|
||||
function showOverlay() {
|
||||
if (overlay) overlay.classList.remove('hidden');
|
||||
// 录音期间隐藏输入卡内容(否则透明渐变会把按钮/文字透出来)
|
||||
if (document.body) document.body.classList.add('voice-active');
|
||||
setOverlayMode('send');
|
||||
startMeter();
|
||||
}
|
||||
|
||||
function hideOverlay() {
|
||||
if (overlay) overlay.classList.add('hidden');
|
||||
if (document.body) document.body.classList.remove('voice-active');
|
||||
stopMeter();
|
||||
cancelArmed = false;
|
||||
}
|
||||
|
||||
function resetUI() {
|
||||
phase = 'idle';
|
||||
pressing = false;
|
||||
pressY = 0;
|
||||
discard = false;
|
||||
if (stopTimer) { clearTimeout(stopTimer); stopTimer = null; }
|
||||
if (hardTimer) { clearTimeout(hardTimer); hardTimer = null; }
|
||||
if (watchdog) { clearTimeout(watchdog); watchdog = null; }
|
||||
hold.classList.remove('recording');
|
||||
setHoldText('按住 说话');
|
||||
hideOverlay();
|
||||
updateSendBtn();
|
||||
}
|
||||
|
||||
function releaseStream() {
|
||||
if (!stream) return;
|
||||
try { stream.getTracks().forEach(function (t) { t.stop(); }); } catch (_) { /* 已关 */ }
|
||||
stream = null;
|
||||
}
|
||||
|
||||
function pickMime() {
|
||||
const list = ['audio/webm;codecs=opus', 'audio/webm', 'audio/mp4', 'audio/ogg;codecs=opus'];
|
||||
for (let i = 0; i < list.length; i++) {
|
||||
try { if (MediaRecorder.isTypeSupported(list[i])) return list[i]; } catch (_) { /* 忽略 */ }
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
// ---------- 模式切换 ----------
|
||||
function enter() {
|
||||
if (!available || mode) return;
|
||||
mode = true;
|
||||
ta.classList.add('hidden');
|
||||
hold.classList.remove('hidden');
|
||||
try { ta.blur(); } catch (_) { /* 忽略 */ }
|
||||
resetUI();
|
||||
updateSendBtn();
|
||||
}
|
||||
|
||||
function exit(focus) {
|
||||
if (!mode) return;
|
||||
cancelHold();
|
||||
mode = false;
|
||||
hold.classList.add('hidden');
|
||||
ta.classList.remove('hidden');
|
||||
autosize();
|
||||
updateSendBtn();
|
||||
// 必须在点击手势里同步 focus,移动端才会弹键盘
|
||||
if (focus) { try { ta.focus(); } catch (_) { /* 忽略 */ } }
|
||||
}
|
||||
|
||||
// ---------- 录音 ----------
|
||||
function startHold() {
|
||||
if (phase !== 'idle') return;
|
||||
pressing = true;
|
||||
discard = false;
|
||||
hold.classList.add('recording');
|
||||
setHoldText('松开 结束');
|
||||
showOverlay();
|
||||
|
||||
if (native) {
|
||||
// 原生侧即使要弹权限也先当「已开始」:松开时 stopVoice(),壳若还在等权限会忽略,
|
||||
// 由 watchdog 兜底复位(只有首次授权那一次会遇到)。
|
||||
phase = 'recording';
|
||||
recordStartAt = Date.now();
|
||||
stopTimer = setTimeout(finishHold, MAX_MS);
|
||||
hardTimer = setTimeout(function () {
|
||||
try { if (native.cancelVoice) native.cancelVoice(); } catch (_) { /* 忽略 */ }
|
||||
resetUI();
|
||||
toast('录音超时');
|
||||
}, MAX_MS + NO_RESULT_MS);
|
||||
try {
|
||||
native.startVoice();
|
||||
} catch (e) {
|
||||
resetUI();
|
||||
toast('无法开始录音:' + String((e && e.message) || e));
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
phase = 'arming'; // 等 getUserMedia/MediaRecorder 起来
|
||||
startWeb();
|
||||
}
|
||||
|
||||
async function startWeb() {
|
||||
try {
|
||||
stream = await navigator.mediaDevices.getUserMedia({ audio: true });
|
||||
} catch (e) {
|
||||
resetUI();
|
||||
const denied = e && (e.name === 'NotAllowedError' || e.name === 'SecurityError');
|
||||
toast(denied ? '麦克风权限被拒绝' : '无法打开麦克风:' + String((e && e.message) || e));
|
||||
return;
|
||||
}
|
||||
if (!pressing) { releaseStream(); resetUI(); return; } // 松开得比授权快
|
||||
attachAnalyser(); // 声波接同一路 MediaStream
|
||||
|
||||
mime = pickMime();
|
||||
try {
|
||||
rec = mime ? new MediaRecorder(stream, { mimeType: mime }) : new MediaRecorder(stream);
|
||||
} catch (_) {
|
||||
rec = new MediaRecorder(stream);
|
||||
mime = rec.mimeType || '';
|
||||
}
|
||||
chunks = [];
|
||||
rec.ondataavailable = function (ev) {
|
||||
if (ev.data && ev.data.size) chunks.push(ev.data);
|
||||
};
|
||||
rec.onstop = function () {
|
||||
const type = mime || 'audio/webm';
|
||||
const blob = new Blob(chunks, { type: type });
|
||||
chunks = [];
|
||||
rec = null;
|
||||
releaseStream();
|
||||
if (discard) { discard = false; resetUI(); toast('说话时间太短'); return; }
|
||||
upload(blob);
|
||||
};
|
||||
rec.onerror = function () {
|
||||
releaseStream();
|
||||
resetUI();
|
||||
toast('录音出错');
|
||||
};
|
||||
try {
|
||||
rec.start();
|
||||
} catch (e) {
|
||||
releaseStream();
|
||||
resetUI();
|
||||
toast('无法开始录音:' + String((e && e.message) || e));
|
||||
return;
|
||||
}
|
||||
|
||||
phase = 'recording';
|
||||
recordStartAt = Date.now();
|
||||
stopTimer = setTimeout(finishHold, MAX_MS);
|
||||
hardTimer = setTimeout(resetUI, MAX_MS + NO_RESULT_MS);
|
||||
}
|
||||
|
||||
// 松手:结束录音(太短则丢弃)
|
||||
function finishHold() {
|
||||
if (phase !== 'recording') return;
|
||||
if (stopTimer) { clearTimeout(stopTimer); stopTimer = null; }
|
||||
|
||||
const held = Date.now() - recordStartAt;
|
||||
if (held < MIN_MS) discard = true;
|
||||
phase = 'busy';
|
||||
hold.classList.remove('recording');
|
||||
setHoldText(discard ? '按住 说话' : '正在识别…');
|
||||
// 遮罩留着,文案换成「正在识别…」;柱子停在低位,等结果回来再收
|
||||
setOverlayMode(discard ? 'send' : 'busy');
|
||||
stopMeter();
|
||||
|
||||
if (native) {
|
||||
try {
|
||||
native.stopVoice();
|
||||
} catch (e) {
|
||||
resetUI();
|
||||
toast('结束录音失败:' + String((e && e.message) || e));
|
||||
return;
|
||||
}
|
||||
// 壳若因为权限弹窗没有真正开录,stopVoice() 会被忽略、结果不会回来
|
||||
watchdog = setTimeout(function () {
|
||||
if (phase !== 'busy') return;
|
||||
try { if (native.cancelVoice) native.cancelVoice(); } catch (_) { /* 忽略 */ }
|
||||
resetUI();
|
||||
}, NO_RESULT_MS);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
if (rec && rec.state !== 'inactive') rec.stop();
|
||||
else { releaseStream(); resetUI(); }
|
||||
} catch (_) {
|
||||
releaseStream();
|
||||
resetUI();
|
||||
}
|
||||
}
|
||||
|
||||
// 放弃本次录音(切模式 / 离开页面 / 上滑取消 / 壳回调 cancelled)
|
||||
function cancelHold() {
|
||||
if (phase === 'idle') return;
|
||||
const r = rec;
|
||||
rec = null;
|
||||
chunks = [];
|
||||
if (r) { r.onstop = null; r.onerror = null; }
|
||||
releaseStream();
|
||||
resetUI();
|
||||
try {
|
||||
if (r && r.state !== 'inactive') r.stop();
|
||||
else if (native && native.cancelVoice) native.cancelVoice();
|
||||
} catch (_) { /* 已停 */ }
|
||||
}
|
||||
|
||||
// ---------- 上传 + 自动发送 ----------
|
||||
function upload(blob) {
|
||||
if (!blob || !blob.size) { resetUI(); toast('没有录到声音'); return; }
|
||||
phase = 'busy';
|
||||
setOverlayMode('busy');
|
||||
req('POST', '/api/asr/transcribe', blob, true, 120000)
|
||||
.then(function (r) {
|
||||
const text = String((r && r.text) || '').replace(/\s+/g, ' ').trim();
|
||||
// 先收遮罩再做后续:toast 固定在 bottom:96px,遮罩还在时弹出来会压住声波
|
||||
resetUI();
|
||||
if (!text) { toast('没有识别到内容'); return; }
|
||||
ta.value = text;
|
||||
send();
|
||||
})
|
||||
.catch(function (e) {
|
||||
resetUI();
|
||||
toast('语音识别失败:' + String((e && e.message) || e));
|
||||
});
|
||||
}
|
||||
|
||||
// ---------- 按住 / 滑动 / 松开 ----------
|
||||
hold.addEventListener('pointerdown', function (e) {
|
||||
if (e && e.button != null && e.button !== 0) return; // 只响应左键/触摸
|
||||
e.preventDefault();
|
||||
if (phase !== 'idle') return;
|
||||
pressY = e.clientY || 0;
|
||||
try { hold.setPointerCapture(e.pointerId); } catch (_) { /* 忽略 */ }
|
||||
startHold();
|
||||
});
|
||||
|
||||
function onMove(e) {
|
||||
if (!pressing) return;
|
||||
if (phase !== 'recording' && phase !== 'arming') return;
|
||||
if (!overlay || overlay.classList.contains('hidden') || !hintEl) return;
|
||||
// 阈值取文案元素的实际下沿;同时要求相对按下点确实上滑了一段,
|
||||
// 否则首次 pointermove 的坐标噪声或极端布局会把状态直接判成取消(一进来就红)。
|
||||
const line = hintEl.getBoundingClientRect().bottom;
|
||||
const limit = pressY ? Math.min(line - 4, pressY - 24) : (line - 4);
|
||||
const y = e.clientY;
|
||||
if (!cancelArmed && y < limit) setOverlayMode('cancel');
|
||||
else if (cancelArmed && y > line + 4) setOverlayMode('send');
|
||||
}
|
||||
|
||||
function onRelease(e) {
|
||||
// 必须先进 pressing 判断:这个监听器也挂在 window 上,
|
||||
// 对无关的 pointerup 调 preventDefault 会吞掉全局 click 事件。
|
||||
if (!pressing) return;
|
||||
if (e) e.preventDefault();
|
||||
pressing = false;
|
||||
|
||||
if (cancelArmed) {
|
||||
// 不弹提示:toast 会盖住声波(取消本身已经有颜色/文案反馈)
|
||||
cancelHold();
|
||||
return;
|
||||
}
|
||||
if (phase === 'recording') finishHold();
|
||||
else if (phase === 'arming') cancelHold(); // 还没真正开录:当误触
|
||||
}
|
||||
|
||||
hold.addEventListener('pointerup', onRelease);
|
||||
hold.addEventListener('pointercancel', onRelease);
|
||||
hold.addEventListener('pointermove', onMove);
|
||||
// 兜底:个别环境下 setPointerCapture 不生效,手指移出按钮后也要跟手
|
||||
window.addEventListener('pointerup', onRelease);
|
||||
window.addEventListener('pointercancel', onRelease);
|
||||
window.addEventListener('pointermove', onMove);
|
||||
|
||||
// 长按不弹系统菜单(移动端)
|
||||
hold.addEventListener('contextmenu', function (e) { e.preventDefault(); });
|
||||
|
||||
// 键盘可达:空格/回车按下开始,松开结束
|
||||
hold.addEventListener('keydown', function (e) {
|
||||
if (e.repeat) return;
|
||||
if (e.key !== ' ' && e.key !== 'Enter') return;
|
||||
e.preventDefault();
|
||||
if (!pressing) startHold();
|
||||
});
|
||||
hold.addEventListener('keyup', function (e) {
|
||||
if (e.key !== ' ' && e.key !== 'Enter') return;
|
||||
e.preventDefault();
|
||||
onRelease(null);
|
||||
});
|
||||
|
||||
// 录音中离开页面:释放麦克风
|
||||
window.addEventListener('pagehide', cancelHold);
|
||||
|
||||
// ---------- 原生桥回调(契约见 docs/voice-input.md) ----------
|
||||
// 实时幅度:套壳每 ~100ms 上报一次 MediaRecorder.getMaxAmplitude() 的**原始值**(0~32767)。
|
||||
// 换算、门限、平滑全部在页面侧做——规则只存在这一份,壳不要做任何数学。
|
||||
function onNativeAmp(raw) {
|
||||
if (!native) return;
|
||||
const amp = Number(raw);
|
||||
if (!isFinite(amp)) return;
|
||||
lastRealLevelAt = Date.now();
|
||||
if (amp <= 0) {
|
||||
levelTarget = 0;
|
||||
} else {
|
||||
const db = 20 * Math.log10(Math.min(amp, 32767) / 32767);
|
||||
levelTarget = dbToLevel(db);
|
||||
}
|
||||
startLoop(); // 静默态下循环已停,下一帧数据到了再启
|
||||
}
|
||||
window.onNativeVoiceAmp = onNativeAmp;
|
||||
window.onNativeVoiceLevel = onNativeAmp; // 旧名兼容:同样按原始幅度解释
|
||||
|
||||
window.onNativeVoiceResult = function (b64, mimeType) {
|
||||
if (phase !== 'busy' && phase !== 'recording') return;
|
||||
if (watchdog) { clearTimeout(watchdog); watchdog = null; }
|
||||
if (discard) { discard = false; resetUI(); toast('说话时间太短'); return; }
|
||||
try {
|
||||
const bin = atob(String(b64 || ''));
|
||||
const arr = new Uint8Array(bin.length);
|
||||
for (let i = 0; i < bin.length; i++) arr[i] = bin.charCodeAt(i);
|
||||
upload(new Blob([arr], { type: mimeType || 'audio/mp4' }));
|
||||
} catch (_) {
|
||||
resetUI();
|
||||
toast('语音数据解析失败');
|
||||
}
|
||||
};
|
||||
|
||||
window.onNativeVoiceState = function (s) {
|
||||
if (!native || s !== 'recording' || phase !== 'recording') return;
|
||||
// 真正开录:60s 与「太短」判定从这一刻算(首次授权弹窗的时间不算)
|
||||
recordStartAt = Date.now();
|
||||
if (stopTimer) clearTimeout(stopTimer);
|
||||
stopTimer = setTimeout(finishHold, MAX_MS);
|
||||
if (!pressing) finishHold(); // 弹窗期间已经松手:立刻收口(held≈0 → 丢弃)
|
||||
};
|
||||
|
||||
window.onNativeVoiceError = function (code, message) {
|
||||
if (phase === 'idle') return;
|
||||
resetUI();
|
||||
// 生命周期打断(切后台/换页)属预期,不弹提示
|
||||
if (code === 'cancelled' || code === 'canceled') return;
|
||||
toast(code === 'too_short'
|
||||
? '说话时间太短,请按住再说'
|
||||
: '录音失败:' + String(message || code || '未知错误'));
|
||||
};
|
||||
|
||||
// ---------- 对外暴露给发送键 / sessions.js 的状态查询 ----------
|
||||
window.AgentVoice = {
|
||||
isAvailable: function () { return available; },
|
||||
isVoiceMode: function () { return mode; },
|
||||
enter: enter,
|
||||
exit: exit
|
||||
};
|
||||
|
||||
// 能力判定:桥优先;没有桥就得是安全上下文(浏览器不允许 http 页面开麦)
|
||||
if (!native && !webOk) {
|
||||
console.info('[voice] 语音入口未启用:没有 window.AgentNative,且当前不是安全上下文(需 HTTPS,或由套壳 App 注入原生桥)');
|
||||
return;
|
||||
}
|
||||
// 服务端没接线(core 的 ASR_URL 为空)就不给入口;探测失败则照常启用
|
||||
req('GET', '/api/asr/status').then(function (s) {
|
||||
if (s && s.enabled === false) {
|
||||
console.info('[voice] 语音入口未启用:服务端未配置语音识别(agent-core 的 ASR_URL 为空)');
|
||||
return;
|
||||
}
|
||||
available = true;
|
||||
updateSendBtn();
|
||||
}).catch(function () {
|
||||
available = true;
|
||||
updateSendBtn();
|
||||
});
|
||||
})();
|
||||
@@ -0,0 +1,71 @@
|
||||
// 语音识别代理:浏览器 / 套壳 App 录好的音频 → agent-core → mcp_asr → 文字。
|
||||
//
|
||||
// 为什么要 core 中转、而不是前端直连 ASR 容器:mcp_asr 只挂在 agent-mcp-net 内网、
|
||||
// 一个宿主端口都不占(它自身没有鉴权,暴露即被白嫖)。对外唯一的入口仍是 core,
|
||||
// 鉴权沿用全局登录态(auth.middleware 已挂在 /api 之前)。
|
||||
//
|
||||
// 开关:core 的 ASR_URL 为空 = 未部署语音识别,本路由一律 503,前端据此不显示麦克风。
|
||||
const express = require('express');
|
||||
const log = require('../log');
|
||||
|
||||
const router = express.Router();
|
||||
|
||||
const ASR_URL = String(process.env.ASR_URL || '').trim().replace(/\/+$/, '');
|
||||
// 与 mcp_asr 的上限保持一致(60s 的 AAC 约 240KB,这里给长音频留余量)
|
||||
const MAX_BYTES = 20 * 1024 * 1024;
|
||||
const TIMEOUT_MS = 120000;
|
||||
|
||||
// 只接音频二进制。全局 express.json(2mb) 只解析 application/json,两者不冲突。
|
||||
// 注意 WebView 可能发 audio/mp4、audio/webm;codecs=opus,统一按 audio/* 接。
|
||||
const rawAudio = express.raw({
|
||||
type: ['audio/*', 'application/octet-stream', 'application/mp4'],
|
||||
limit: '20mb',
|
||||
});
|
||||
|
||||
// 前端可用性探测(按钮是否显示)。不碰网络,纯看接线。
|
||||
router.get('/status', function (req, res) {
|
||||
res.json({ enabled: !!ASR_URL, max_bytes: MAX_BYTES });
|
||||
});
|
||||
|
||||
router.post('/transcribe', rawAudio, async function (req, res) {
|
||||
if (!ASR_URL) {
|
||||
return res.status(503).json({ error: '语音识别未配置(core 未设置 ASR_URL)' });
|
||||
}
|
||||
const buf = req.body;
|
||||
if (!Buffer.isBuffer(buf) || !buf.length) {
|
||||
return res.status(400).json({ error: '未收到音频数据(Content-Type 需为 audio/*)' });
|
||||
}
|
||||
try {
|
||||
const r = await fetch(ASR_URL + '/transcribe', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': req.headers['content-type'] || 'application/octet-stream' },
|
||||
body: buf,
|
||||
signal: AbortSignal.timeout(TIMEOUT_MS),
|
||||
});
|
||||
const text = await r.text();
|
||||
if (!r.ok) {
|
||||
// ASR 服务回的是 {error:string}:解出来再转,不然前端会看到两层套嵌的 JSON
|
||||
let msg = text.slice(0, 300);
|
||||
try { msg = JSON.parse(text).error || msg; } catch (_) { /* 非 JSON 保留原文 */ }
|
||||
log.warn('asr', '识别失败 HTTP ' + r.status + ':' + msg);
|
||||
// 4xx(静音 / 坏音频 / 过载)原样透传;5xx 统一当上游故障
|
||||
const code = r.status >= 400 && r.status < 500 ? r.status : 502;
|
||||
return res.status(code).json({ error: msg });
|
||||
}
|
||||
// 原样透传 ASR 的 {text, ms, backend}
|
||||
res.type('application/json').send(text);
|
||||
} catch (e) {
|
||||
log.warn('asr', '调用 ASR 服务失败:' + String(e && e.message || e));
|
||||
res.status(502).json({ error: '语音识别服务不可用:' + String(e && e.message || e) });
|
||||
}
|
||||
});
|
||||
|
||||
// express.raw 超限时抛 entity.too.large,默认会回 HTML;这里翻成前端能读的 JSON。
|
||||
router.use(function (err, req, res, next) {
|
||||
if (err && (err.type === 'entity.too.large' || err.status === 413)) {
|
||||
return res.status(413).json({ error: '音频过大(上限 ' + Math.round(MAX_BYTES / 1048576) + 'MB)' });
|
||||
}
|
||||
next(err);
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
@@ -1,7 +1,9 @@
|
||||
// data-page 技能的服务端承载,两条通道(配套 skills/data-page/SKILL.md 的契约):
|
||||
// 1. GET /app/{task_id}/... —— 静态托管任务目录的 .app/ 子树(index.html 兑底,不给列目录);
|
||||
// 目录名刻意带点:任务文件面板默认隐藏(?hidden=1 才列);
|
||||
// 只暴露 .app/ 而非整个任务目录,其余任务文件不从 URL 泄漏。
|
||||
// 只暴露 .app/ 而非整个任务目录,其余任务文件不从 URL 泄漏;
|
||||
// 允许符号链接(技能页面链回技能目录,不必每会话复制),
|
||||
// 但 realpath 必须落在会话目录或技能根目录内。
|
||||
// 2. POST /api/db/{task_id}/query —— 任务 data.db 的 SQL API(页面前端 fetch 调):任意单语句,
|
||||
// 执行核心与 db 工具共用 runTaskSql(executor.js),语义单一事实源。
|
||||
// 认证:挂在 server.js 的 auth.middleware 之后——信任模型与 files 路由一致(登录 ≈ 运维,可跨会话)。
|
||||
@@ -13,6 +15,7 @@ const fs = require('fs');
|
||||
const path = require('path');
|
||||
const engine = require('../engine');
|
||||
const { resolveInTask, runTaskSql } = require('../tools/executor');
|
||||
const { skillsRoot } = require('../tools/skills');
|
||||
|
||||
// 页面忘写 LIMIT 的服务端兜底:iterate 到量即停(不全量物化),超限报错引导分页。
|
||||
const API_MAX_ROWS = 2000;
|
||||
@@ -23,6 +26,19 @@ function notFound(res, msg) {
|
||||
res.status(404).json({ error: msg });
|
||||
}
|
||||
|
||||
// 符号链接白名单:技能页面不必每个会话复制一份,发布时链回技能目录(见各技能的 run.py)。
|
||||
// 链接只是字符串:executor 里看不到的目标也能被链上,而解析发生在 core 的命名空间
|
||||
// (例如 core 把 ./config.yml 挂在 /app/config.yml),所以 realpath 后必须落在
|
||||
// 「本会话目录」或「技能根目录」内,其余一律 404。
|
||||
function realInside(abs, roots) {
|
||||
let real;
|
||||
try { real = fs.realpathSync(abs); } catch (_) { return null; }
|
||||
for (const root of roots) {
|
||||
if (real === root || real.startsWith(root + path.sep)) return real;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// ---------- /app/{task_id}/...:任务 .app/ 子树静态托管(URL 路径与目录名解耦)----------
|
||||
router.use('/app', (req, res) => {
|
||||
try {
|
||||
@@ -44,7 +60,15 @@ router.use('/app', (req, res) => {
|
||||
try { st = fs.statSync(abs); } catch (_) { return notFound(res, '页面不存在'); }
|
||||
}
|
||||
if (!st.isFile()) return notFound(res, '页面不存在');
|
||||
res.sendFile(abs, err => {
|
||||
const candidates = [engine.taskDirOf(sessionId)];
|
||||
try { candidates.push(skillsRoot({ cfg: engine.getConfig() })); } catch (_) {}
|
||||
const roots = [];
|
||||
for (const dir of candidates) {
|
||||
try { roots.push(fs.realpathSync(dir)); } catch (_) {}
|
||||
}
|
||||
const real = realInside(abs, roots);
|
||||
if (!real) return notFound(res, '页面不存在');
|
||||
res.sendFile(real, err => {
|
||||
if (err && !res.headersSent) notFound(res, '页面不存在');
|
||||
});
|
||||
} catch (err) {
|
||||
|
||||
@@ -28,6 +28,17 @@ router.post('/:id/messages', wrap((req, res) => {
|
||||
}));
|
||||
|
||||
|
||||
// 覆盖式编辑:删除该条及其后的历史,再以新内容重开一轮(运行中 409)。
|
||||
router.post('/:id/messages/:mid/edit', wrap((req, res) => {
|
||||
const content = String((req.body && req.body.content) || '').trim();
|
||||
if (!content) return res.status(400).json({ error: 'content 不能为空' });
|
||||
const deepThink = !!(req.body && req.body.deepThink);
|
||||
const r = engine.editUserMessage(req.params.id, Number(req.params.mid), content, { deepThink });
|
||||
log.info('msg', (req.user || '-') + ' ' + req.params.id + ' 编辑消息 ' + req.params.mid +
|
||||
'(丢弃其后 ' + Math.max(0, (r.removed || 1) - 1) + ' 条)');
|
||||
res.json(r);
|
||||
}));
|
||||
|
||||
router.post('/:id/stop', wrap((req, res) => {
|
||||
const r = engine.stop(req.params.id);
|
||||
log.info('msg', (req.user || '-') + ' ' + req.params.id + ' 用户终止');
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
// 通用技能执行面:POST /api/sessions/:id/skills/:name/run
|
||||
//
|
||||
// 与 /api/db 的定位对称:data-page 给了页面「通用数据面」,这里补上「通用执行面」。
|
||||
// 页面类技能(如文档审核页)需要用户点一下就在服务端跑一次技能,而不经过模型:
|
||||
// - 不新开技能专属路由,core 不感知任何技能名、参数语义或产物路径;
|
||||
// - 参数校验、相对路径规整、远端 executor、超时/取消全部复用 tools/skills.runSkill,
|
||||
// 与模型侧 skill(op=run) 是同一事实源,不存在第二套执行逻辑;
|
||||
// - 本文件只做会话存在性、并发闸门、HTTP 错误映射。
|
||||
//
|
||||
// 认证:挂在 server.js 的 auth.middleware 之后,信任模型与 data-page/files 一致
|
||||
// (登录 ≈ 运维;能登录的人本来也能通过对话让模型跑任意技能,这里不新增权限面)。
|
||||
// 刻意不加 CORS:跨站 application/json POST 会被预检挡下。
|
||||
//
|
||||
// 并发:同一 (session, skill) 同时只允许一个请求;页面重复点按钮得到 409 而不是双跑。
|
||||
// 单实例内存 Set 即可(仓库整体就是单进程内存模型,重启即清空)。
|
||||
//
|
||||
// 响应契约:
|
||||
// - 进程成功退出 → HTTP 200 {ok:true, result}
|
||||
// stdout 能解析成 JSON 就返回对象,否则原样返回字符串;
|
||||
// - 技能自身业务失败(脚本以 exit 0 + {"ok":false} 表达)仍算进程成功,
|
||||
// 调用方查 result.ok(与 skill 工具的约定一致);
|
||||
// - 进程失败/超时 → HTTP 200 {ok:false, error}(与 data-page 的「HTTP 200 + error」一致);
|
||||
// - 会话不存在 / 参数形状错误 → 4xx {ok:false, error}。
|
||||
'use strict';
|
||||
const express = require('express');
|
||||
const engine = require('../engine');
|
||||
const skills = require('../tools/skills');
|
||||
const log = require('../log');
|
||||
|
||||
const router = express.Router();
|
||||
const inflight = new Set();
|
||||
|
||||
function sendErr(res, code, msg) {
|
||||
res.status(code).json({ ok: false, error: msg });
|
||||
}
|
||||
|
||||
router.post('/:id/skills/:name/run', async (req, res) => {
|
||||
const sessionId = req.params.id;
|
||||
const name = String(req.params.name || '').trim();
|
||||
if (!engine.getSession(sessionId)) return sendErr(res, 404, '会话不存在');
|
||||
if (!name) return sendErr(res, 400, '技能名不能为空');
|
||||
|
||||
const body = req.body || {};
|
||||
const args = body.args === undefined ? {} : body.args;
|
||||
if (args === null || typeof args !== 'object' || Array.isArray(args)) {
|
||||
return sendErr(res, 400, 'args 必须是 JSON 对象');
|
||||
}
|
||||
|
||||
const key = sessionId + '\n' + name;
|
||||
if (inflight.has(key)) return sendErr(res, 409, '该技能正在执行,请稍候');
|
||||
inflight.add(key);
|
||||
const t0 = Date.now();
|
||||
try {
|
||||
const ctx = {
|
||||
sessionId,
|
||||
taskDir: engine.taskDirOf(sessionId),
|
||||
cfg: engine.getConfig(),
|
||||
};
|
||||
const out = await skills.runSkill(ctx, name, args);
|
||||
let result = out;
|
||||
try { result = JSON.parse(out); } catch (_) { /* 非 JSON 技能:原样返回字符串 */ }
|
||||
log.info('skills', sessionId + ' HTTP 运行 ' + name + ' 完成(' + (Date.now() - t0) + 'ms)');
|
||||
res.json({ ok: true, result });
|
||||
} catch (err) {
|
||||
const msg = String((err && err.message) || err || '技能执行失败');
|
||||
if (err && err.name === 'SkillNotFoundError') return sendErr(res, 404, msg);
|
||||
// 脚本失败/超时不冒充 5xx:调用方统一查 ok/error(与 data-page 契约一致)
|
||||
log.warn('skills', sessionId + ' HTTP 运行 ' + name + ' 失败(' + (Date.now() - t0) + 'ms):' + msg.split('\n')[0]);
|
||||
res.json({ ok: false, error: msg });
|
||||
} finally {
|
||||
inflight.delete(key);
|
||||
}
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
@@ -77,7 +77,7 @@ function absInSkill(dir, rel) {
|
||||
// 技能发生任何变更(创建/重命名/删除/文件增删改都经过本模块)后,
|
||||
// 重建 system prompt 的 “可用技能” 段落(llm.setSkillListText):
|
||||
// 模型立即感知新技能或改名,无需重启。fire-and-forget:扫描失败不影响 API 响应,
|
||||
// skill {"list":"all"} 仍是实时兜底。
|
||||
// skill op=list 仍是实时兜底。
|
||||
function refreshSkillPrompt() {
|
||||
listCallableSkills({ cfg: engine.getConfig() })
|
||||
.then(list => llm.setSkillListText(formatSkillListForPrompt(list)))
|
||||
|
||||
@@ -15,7 +15,7 @@ const router = express.Router();
|
||||
const STARTED_AT = Date.now();
|
||||
// 版本以本文件字面量为准:容器部署只挂载 app/server,镜像里的 package.json 可能是旧的,
|
||||
// 读它会一直显示发版时的版本号(之前 2.1.0 改完仍显 2.0.0 即此因)。发版时记得同步这里。
|
||||
const VERSION = '2.3.0';
|
||||
const VERSION = '2.4.0';
|
||||
// git 提交号单独返回(提交时 post-commit 钩子自动生成),前端只显示简洁版本号
|
||||
const VERSION_COMMIT = versionInfo.commit || '';
|
||||
|
||||
|
||||
+24
-3
@@ -67,7 +67,13 @@ app.use(express.static(path.join(__dirname, 'public')));
|
||||
app.use('/api/sessions', require('./routes/sessions'));
|
||||
app.use('/api/sessions', require('./routes/messages'));
|
||||
app.use('/api/sessions', require('./routes/files'));
|
||||
// 通用技能执行面(页面类技能用,不经模型):POST /api/sessions/:id/skills/:name/run。
|
||||
// 与 /api/db 对称;执行核心仍复用 tools/skills.runSkill,core 不感知具体技能。
|
||||
app.use('/api/sessions', require('./routes/skill-run'));
|
||||
app.use('/api/skills', require('./routes/skills'));
|
||||
// 语音识别(可选):录音在客户端(套壳 App 原生桥或浏览器 MediaRecorder),转写在 mcp_asr。
|
||||
// ASR_URL 为空时接口回 503,前端据此不显示麦克风按钮(见 routes/asr.js)。
|
||||
app.use('/api/asr', require('./routes/asr'));
|
||||
// 系统配置含全部用户(脱敏密码)与平台密钥,且可整份改写 YAML:仅管理员可访问。
|
||||
app.use('/api/config', auth.requireAdmin, require('./routes/config'));
|
||||
app.use('/api/platforms', require('./routes/platforms'));
|
||||
@@ -142,7 +148,22 @@ app.get('/api/models', (req, res) => {
|
||||
res.json(rows);
|
||||
});
|
||||
|
||||
app.get('/api/health', (req, res) => res.json({ ok: true, time: Date.now() }));
|
||||
// 探活 + 服务器地址校验(免认证,见 auth.js 的 PUBLIC_EXACT)。
|
||||
// 套壳 App / 运维脚本会手动填服务器地址,必须能判断“这个地址到底是不是本服务”:
|
||||
// 只认 HTTP 200 + body.ok===true + service==='agent-core',不能只看状态码——
|
||||
// 填到别的站点、反代首页或劫持页同样可能回 200。
|
||||
// no-store:避免 WebView / 反向代理 / Service Worker 把旧响应当成“服务在线”。
|
||||
// auth_required 让客户端区分“连上了但没登录”和“地址填错了”。
|
||||
app.get('/api/health', (req, res) => {
|
||||
res.setHeader('Cache-Control', 'no-store');
|
||||
res.json({
|
||||
ok: true,
|
||||
service: 'agent-core',
|
||||
api: 1, // 接口代际:未来出现不兼容改动时升,客户端据此提示升级
|
||||
time: Date.now(),
|
||||
auth_required: !!(cfg.auth && cfg.auth.users && cfg.auth.users.length),
|
||||
});
|
||||
});
|
||||
|
||||
const PORT = process.env.PORT || cfg.port;
|
||||
|
||||
@@ -178,8 +199,8 @@ function dumpPromptBundle() {
|
||||
// 启动时先发现各 MCP 服务的工具(单个服务失败只警告,不阻止启动),
|
||||
// 并对技能列表做快照(注入 system prompt,见 llm.setSkillListText),完成后开始监听。
|
||||
mcp.init(cfg).then(async () => {
|
||||
// 技能列表快照:模型直接感知已有技能,免先 {"list":"all"} 探索;
|
||||
// 此处失败不阻塞启动,{"list":"all"} 仍可做实时兜底。
|
||||
// 技能列表快照:模型直接感知已有技能,免先 op=list 探索;
|
||||
// 此处失败不阻塞启动,op=list 仍可做实时兜底。
|
||||
try {
|
||||
const list = await skills.listCallableSkills({ cfg });
|
||||
llm.setSkillListText(skills.formatSkillListForPrompt(list));
|
||||
|
||||
@@ -1,104 +1,74 @@
|
||||
You are a general-purpose task assistant running on a private server via a web
|
||||
interface. You help with file generation, data processing, web fetching,
|
||||
You are a general-purpose task assistant on a private server, reached through a
|
||||
web interface. You help with file generation, data processing, web fetching,
|
||||
running commands, writing scripts, and similar tasks.
|
||||
|
||||
You have a dedicated working directory for each conversation. All file paths in
|
||||
tool calls are relative to that directory unless otherwise specified.
|
||||
Each conversation has its own working directory; tool paths are relative to it
|
||||
unless stated otherwise. Uploaded files appear there.
|
||||
|
||||
## Scope & Capability Boundaries
|
||||
## Capabilities
|
||||
|
||||
You are a file / data / information processing assistant. The tools available
|
||||
to you vary by session — always check the live tool list before answering.
|
||||
Your capabilities are exactly what the live tool list and the skills list below
|
||||
expose — they vary by session, so check them before answering. Do not assume a
|
||||
capability that isn't there.
|
||||
|
||||
In scope: understanding images (describe, OCR, read charts); deterministic image
|
||||
operations (format conversion, resize, crop, watermark); charts and diagrams
|
||||
from data; document processing and conversion (PDF / Markdown / Word / Excel /
|
||||
PowerPoint); data processing; running scripts and commands; fetching and
|
||||
summarizing web information; audio/video transcoding and basic editing;
|
||||
generative media when a tool for it is present in the current session (read that
|
||||
tool's own description for its capabilities and limits); SQLite operations.
|
||||
Never fake results: no placeholder files, no claiming a file was created, no
|
||||
crude substitutes (ASCII art, hand-drawn SVG) for what the user actually wants.
|
||||
|
||||
Out of scope: capabilities no available tool exposes in this session. Do not
|
||||
fake results — no placeholder files, no claiming a file was created, no crude
|
||||
substitutes (ASCII art, hand-drawn SVG) for what the user actually wants.
|
||||
|
||||
When the user asks for something that seems out of scope:
|
||||
|
||||
- Before concluding you can't do it, check for combinations: chaining tools, or
|
||||
combining tools with the user's own environment (their browser or device).
|
||||
Some requests are impossible from the server alone but become possible when
|
||||
the work is split between the server and the user's side.
|
||||
- If an available skill might handle it, check the skill before declaring it out
|
||||
of scope.
|
||||
- If it truly can't be done, say so in one or two sentences and stop — do not
|
||||
call tools just to reject a request, do not over-apologize. If a third-party
|
||||
tool covers it, mention one briefly. If part of the request is doable, offer
|
||||
that part.
|
||||
|
||||
## Communication
|
||||
|
||||
- Be concise and direct. Use plain language; format with Markdown when useful.
|
||||
- Be truthful. If unsure, say so.
|
||||
- Do not over-apologize.
|
||||
If something seems impossible, check relevant skills first, then whether tools
|
||||
can be combined, or whether the work can be split with the user's own
|
||||
environment (their browser or device). If it truly can't be done, say so in one
|
||||
or two sentences and offer whatever part is doable; do not call tools just to
|
||||
reject a request.
|
||||
|
||||
## Work Principles
|
||||
|
||||
- **Choose the simplest reliable path.** Prefer standard library or built-in
|
||||
tools. If a required package is missing, install it into a session-local venv
|
||||
when possible; otherwise use a fallback or report the limitation.
|
||||
- **Stop after repeated failures.** If the same approach fails more than once or
|
||||
twice, switch to a different approach or report failure. Do not keep retrying
|
||||
the same method.
|
||||
- **Report progress.** If a task will take many steps, state your plan briefly
|
||||
first and give periodic updates. Do not run silently for a long stretch.
|
||||
- **Match effort to the task.** Do not write complex scripts where a few
|
||||
commands suffice, and do not verify more deeply than the task requires. One
|
||||
basic check that the output is usable is enough.
|
||||
- Choose the simplest reliable path; do not write a script where a few commands
|
||||
suffice.
|
||||
- If the same approach fails twice, switch approaches or report the failure —
|
||||
do not keep retrying.
|
||||
- For multi-step work, state the plan briefly and give periodic updates.
|
||||
- Match effort to the task: one basic check that the output is usable is enough.
|
||||
|
||||
## Tool Use
|
||||
## Tools
|
||||
|
||||
- Use the most appropriate tool for each step.
|
||||
- Independent tool calls can be made in parallel.
|
||||
- Follow the parameter schemas exactly; parameters are passed as top-level
|
||||
fields of the tool call arguments.
|
||||
- Skills marked **required** in the skill list below must be used for their
|
||||
scope — do not hand-write scripts to accomplish what a required skill already
|
||||
does. For other skills, prefer them when they match the task.
|
||||
- `mint_public_link` creates a public, unauthenticated URL — never use it for
|
||||
files containing secrets.
|
||||
- Independent calls can run in parallel. Parameters are top-level fields of the
|
||||
tool call arguments — follow the schema exactly.
|
||||
- `mint_public_link` makes a file publicly reachable without authentication —
|
||||
never use it for files containing secrets.
|
||||
|
||||
## Skill Use
|
||||
|
||||
Before writing a script or installing a package, check the skill list below.
|
||||
If a skill covers the task, use it instead of reimplementing the same job in
|
||||
ad-hoc code — skills already handle the server-specific pitfalls. Read its
|
||||
interface with the `skill` tool (`{"op":"get","name":"<skill>"}`) before the
|
||||
first run, once per skill per conversation. `required` marks the skills whose
|
||||
domain must never be hand-rolled.
|
||||
|
||||
{APP_SKILLS_SECTION}
|
||||
|
||||
## File Management
|
||||
## Communication
|
||||
|
||||
- Work inside the working directory. Use clear file names.
|
||||
- Uploaded files appear in the working directory; check with `terminal` (e.g.
|
||||
`ls`) if needed.
|
||||
- Be concise and direct; Markdown when useful. If unsure, say so. Don't
|
||||
over-apologize.
|
||||
- When done, briefly state what was done and list created/modified files. Offer
|
||||
follow-ups only when useful.
|
||||
|
||||
## Completion
|
||||
## Environment
|
||||
|
||||
- After finishing, briefly state what was done and list created/modified files.
|
||||
- Offer follow-up options only if useful; do not do extra work automatically.
|
||||
|
||||
## System Environment
|
||||
|
||||
- OS: linux. Shell commands run under POSIX sh (`/bin/sh` is dash); use `bash`
|
||||
for scripts (`run.sh`). Keep `terminal` one-liners POSIX-compatible.
|
||||
- Isolated container: a command can access the session working directory, a
|
||||
session-scoped `/tmp`, read-only `/workspace/skills`, and read-only system
|
||||
paths. It cannot see other sessions.
|
||||
- `/tmp` is usable for scratch across `terminal` calls in the same conversation.
|
||||
`/dev/shm` and `/var/tmp` reset on every `terminal` call — do not use them to
|
||||
carry state across calls.
|
||||
- Python 3.11 is available; the system Python and its packages are read-only.
|
||||
To add packages, create a virtualenv inside the working directory
|
||||
(`python3 -m venv .venv`, then `.venv/bin/pip` and `.venv/bin/python`). Keep
|
||||
it in the working directory, not `/tmp`.
|
||||
- System packages cannot be installed at runtime (`apt-get install` fails on the
|
||||
read-only root filesystem). If a required CLI tool is missing, tell the user
|
||||
it must be added on the server.
|
||||
- No C/C++ compiler or `make` — `pip install` works only for packages that ship
|
||||
a prebuilt wheel or are pure Python.
|
||||
- For Chinese text in matplotlib: set `font.sans-serif` to `Noto Sans CJK SC`.
|
||||
- For Chinese ZIP files with GBK filenames, prefer `unar` or `7z x` to avoid
|
||||
garbled names; in Python, use automatic encoding detection.
|
||||
- Linux container, isolated per conversation: you see the working directory, a
|
||||
session-scoped `/tmp`, read-only `/workspace/skills`, and system paths — never
|
||||
other sessions.
|
||||
- `terminal` runs POSIX sh (`/bin/sh` is dash); use `bash` for scripts.
|
||||
- `/tmp` persists across calls in a conversation; `/dev/shm` and `/var/tmp` are
|
||||
wiped on every call.
|
||||
- Python 3.11; system packages are read-only. Add packages in a venv in the
|
||||
working directory: `python3 -m venv --system-site-packages .venv`, then
|
||||
`.venv/bin/pip` / `.venv/bin/python`. Don't reinstall packages that are
|
||||
already importable.
|
||||
- System packages can't be installed at runtime (`apt-get` fails). A missing CLI
|
||||
tool must be added on the server — tell the user.
|
||||
- No C/C++ compiler or `make`: `pip install` only works for pure-Python or
|
||||
prebuilt-wheel packages.
|
||||
- Chinese text in matplotlib: set `font.sans-serif` to `Noto Sans CJK SC`. For
|
||||
ZIPs with GBK filenames, use `unar` or `7z x` to avoid garbled names.
|
||||
|
||||
@@ -219,7 +219,7 @@
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "read_file",
|
||||
"description": "Reads the content of a file.\n\nText files: each returned line is prefixed with a line number right-aligned in a 6-character field, followed by a tab, then the line content.\n\nCross-session peek: `path` accepts '@<session-id>/<rel>' to read a text file from your OWN past session, read-only (find the id with recall op=search, browse with list_directory). It is a look, not an import: nothing lands in your session directory, so `terminal`, grep and `edit_file` cannot see it, and past images are refused. When you need to work with a file or view a past image, `copy_path` it first (its `source_path` takes the same '@' form). '@' is never accepted by `write_file` or `edit_file`.\n\nLarge text files (roughly over 16 KB) are not returned in full: you get an outline listing top-level symbols (or Markdown headings) with line numbers instead. The outline IS a successful response — do NOT retry the same call without a line range; use the outline line numbers with `start_line` / `end_line` to read specific sections. If no outline can be extracted, a first-1KB excerpt is returned instead. Supplying `start_line` or `end_line` disables outline mode.\n\nImage files (PNG, JPEG, WebP, GIF, BMP, TIFF) are returned as visual content you can analyze directly, but only if the current model supports vision; otherwise a text placeholder is returned — that placeholder is not an error. The tool result itself carries a text notice only; the actual image pixels arrive in a separate user message right after the batch of tool results: that message lists per-image labels matching each image to its read_file receipt, then the image blocks in receipt order, and asks you to briefly describe each image before continuing. Never re-read an image already delivered in the current turn; if pixels never arrive, tell the user instead of retrying. Very large images (roughly over 50 MB) are skipped, and images may be downscaled (long edge ~2000 px, re-encoded as JPEG) before being returned.\n\nBinary files are not returned as text; a short notice is returned instead. Text files over 32 MB cannot be read by this tool — use `terminal` (grep -n / head / sed -n).",
|
||||
"description": "Reads the content of a file.\n\nText files: each returned line is prefixed with a line number right-aligned in a 6-character field, followed by a tab, then the line content.\n\nCross-session peek: `path` accepts '@<session-id>/<rel>' to read a text file from your OWN past session, read-only (find the id with recall op=search, browse with list_directory). It is a look, not an import: nothing lands in your session directory, so `terminal`, grep and `edit_file` cannot see it, and past images are refused. When you need to work with a file or view a past image, `copy_path` it first (its `source_path` takes the same '@' form). '@' is never accepted by `write_file` or `edit_file`.\n\nLarge text files (roughly over 16 KB) are not returned in full: you get an outline listing top-level symbols (or Markdown headings) with line numbers instead. The outline IS a successful response — do NOT retry the same call without a line range; use the outline line numbers with `start_line` / `end_line` to read specific sections. If no outline can be extracted, a first-1KB excerpt is returned instead. Supplying `start_line` or `end_line` disables outline mode.\n\nImage files (PNG, JPEG, WebP, GIF, BMP, TIFF): the receipt is a single JSON object, not prose — read its fields instead of interpreting the text. Success shape: {\"ok\":true,\"tool\":\"read_file\",\"kind\":\"image\",\"path\":\"...\",\"image_attached\":true,\"width\":1080,\"height\":1920,\"format\":\"jpeg\",\"bytes_in\":3550000,\"bytes_out\":210000,\"resized\":true,\"source_width\":4000,\"source_height\":7111}. `width`/`height` are the delivered pixel size; `source_width`/`source_height` appear only when the image was downscaled. `image_attached:true` means the server produced the image and it will arrive as an image block in the separate user message that immediately follows the batch of tool results; that message labels each image `[图片 N] read_file <path> <w>×<h>` and asks you to describe it before continuing. The tool result itself never carries pixels. Refusal or failure shape: {\"ok\":false,\"kind\":\"image\",\"path\":\"...\",\"image_attached\":false,\"reason\":\"...\",\"retryable\":false,\"hint\":\"...\"} where `reason` is one of no_vision (this model cannot see images), too_large (roughly over 50 MB), past_session (import it with copy_path first), decode_failed, sharp_missing, already_delivered (this same image was already delivered by an earlier call in the SAME batch of tool calls, so this duplicate was not attached again — reading it again in a later batch always delivers the current pixels, which may have changed since), link_unavailable (no public URL could be minted, so the pixels did not arrive for that request). `ok:true` with `image_attached:false` is still a successful call, e.g. a deduplicated re-read. When `retryable` is false do not retry — tell the user instead. Images may be downscaled (long edge ~2000 px, re-encoded as JPEG) before delivery.\n\nBinary files are not returned as text; a short notice is returned instead. Text files over 32 MB cannot be read by this tool — use `terminal` (grep -n / head / sed -n).",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
@@ -337,25 +337,23 @@
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "skill",
|
||||
"description": "Run or inspect the skills installed on this server. A skill is a directory under the skill root whose SKILL.md defines its purpose and input_schema.\n\nPass EXACTLY ONE verb key:\n{\"list\":\"all\"} List every installed skill with a one-line description, read live from the skill root; an empty list is not an error.\n{\"get\":\"<name>\"} Return the skill's full raw SKILL.md (front matter included): purpose, input_schema and usage notes. Call it before run for a skill you have not used in this conversation.\n{\"run\":\"<name>\",\"args\":{...}} Execute the skill; args are key/value pairs matching its input_schema, and may be omitted for skills that take none.\n\nrun returns the skill's combined stdout+stderr as plain text after it exits (exit 0 with no output yields '(无输出)'); a non-zero exit raises an error containing the output and [exit code: N]. Relative paths in args are resolved against the session working directory, so pass output paths explicitly and never rely on the skill's cwd. Default timeout 120 seconds: the process group is killed and the error includes partial output collected so far (a timeout is not a skill failure).",
|
||||
"description": "Run or inspect the skills installed on this server. Each skill describes its purpose and input_schema.\n\n{\"op\":\"list\"} list every skill with a one-line description\n{\"op\":\"get\",\"name\":\"<skill>\"} return the skill's full details; call it before the first run\n{\"op\":\"run\",\"name\":\"<skill>\",\"args\":{...}} execute it; args follow its input_schema\n\nop is required. op=get and op=run require name. Put skill arguments inside args, never at the top level.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": ["op"],
|
||||
"properties": {
|
||||
"list": {
|
||||
"op": {
|
||||
"type": "string",
|
||||
"description": "Scope of the listing. The only scope today is \"all\" (every installed skill)."
|
||||
"enum": ["list", "get", "run"],
|
||||
"description": "Operation to perform on skills."
|
||||
},
|
||||
"get": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Skill name (subdirectory name under the skill root): return its SKILL.md."
|
||||
},
|
||||
"run": {
|
||||
"type": "string",
|
||||
"description": "Skill name: execute it with args."
|
||||
"description": "Skill name. Required with op=get and op=run."
|
||||
},
|
||||
"args": {
|
||||
"type": "object",
|
||||
"description": "Only with run: the skill's input arguments, e.g. {\"input\": \"file.md\", \"output\": \"out.pdf\"}. Omit it for skills that take no arguments."
|
||||
"description": "Only with op=run: the skill's input arguments, e.g. {\"input\":\"a.md\",\"output\":\"b.pdf\"}. Omit for skills that take no arguments."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+122
-80
@@ -129,6 +129,9 @@ const IMAGE_EXTS = new Set(['.png', '.jpg', '.jpeg', '.webp', '.gif', '.bmp', '.
|
||||
// read_file 读图(仅当会话绑定的模型 vision: true):sharp 压缩后落盘到任务目录 .imgcache/,
|
||||
// 返回 {text, image_rel};engine 拼成 image_ref 内容块落库,llm.buildMessages 发请求前
|
||||
// 现签成公网外链(GET /p/<id>)进 image_url 块,base64 不再进 DB/请求体。
|
||||
// text 一律是 imageReceipt 生成的 JSON 对象(字段契约见下),不是散文:
|
||||
// 这份回执会被三层改写(executor 产出 / engine 同回合去重 / llm 签外链失败降级),
|
||||
// 散文回执让模型只能靠语义猜「到底有没有图」,历史事故是反复 read_file 补看烧 token。
|
||||
const IMG_MAX_INPUT_BYTES = 50 * 1024 * 1024; // 与上传上限一致
|
||||
const IMG_MAX_SIDE = 2000; // 长边压缩阈值(px)
|
||||
const IMG_JPEG_QUALITY = 80;
|
||||
@@ -178,14 +181,51 @@ function outlineWrapper(outlineText) {
|
||||
'For example, to read a function shown as [L100-150], use start_line: 100 and end_line: 150.';
|
||||
}
|
||||
|
||||
// read_file 图片回执的统一信封。字段是模型唯一可判定的事实,回执不再写中文句子:
|
||||
// ok 调用本身是否成功(ok=true + image_attached=false 是合法形态:同回合重复读被去重)
|
||||
// path 模型请求的路径(@ 历史形原样保留);投递产物路径只在服务端内部用,不回执
|
||||
// image_attached 服务端是否产出了图块、会随本批回执之后的那条 user 消息投递(≠上游已收到)
|
||||
// reason 仅 image_attached=false 时有值,稳定枚举(no_vision / too_large / past_session /
|
||||
// decode_failed / sharp_missing / already_delivered / link_unavailable)——
|
||||
// 模型按它分支,不要读 hint 猜。注意:重放省略像素不属于回执事件,不写 reason。
|
||||
// retryable 重试同一调用是否有意义(图片这条路上恒 false:同样的字节只会得到同样的结果)
|
||||
// width/height 实际投递的像素尺寸;source_width/height + resized 只在确实缩放过时出现
|
||||
// hint 一句人话补充,仅供解释,不承载判定
|
||||
function imageReceipt(o) {
|
||||
const out = { ok: o.ok === true, tool: 'read_file', kind: 'image', path: o.path, image_attached: o.image_attached === true };
|
||||
for (const k of ['reason', 'retryable', 'width', 'height', 'format', 'bytes_in', 'bytes_out',
|
||||
'resized', 'source_width', 'source_height', 'hint']) {
|
||||
if (o[k] !== undefined) out[k] = o[k];
|
||||
}
|
||||
return JSON.stringify(out);
|
||||
}
|
||||
|
||||
// 用 sharp 压缩(按 EXIF 摆正、长边 ≤2000px、JPEG 质量 80)并落盘 .imgcache/。
|
||||
// 成功返回 {text, image_rel};产物文件名带源文件指纹(mtime+size),同图重复读复用同一份不增殖。
|
||||
// 成功返回 {text, image_rel},失败返回 {text}(text 都是 imageReceipt 的 JSON)。
|
||||
// 产物文件名带源文件指纹(mtime+size),同图重复读复用同一份不增殖。
|
||||
// 无收益守卫(与浏览器端压缩同款):重编码 JPEG 反而 ≥ 原图时不落盘,image_rel 直指原文件。
|
||||
// sharp 缺失或解码失败时退回纯文本提示,不让整个工具回合报错。
|
||||
// sharp 缺失或解码失败时只回执原因,不让整个工具回合报错。
|
||||
async function readImageAsBlocks(ctx, abs, rel, size) {
|
||||
let sharp;
|
||||
try { sharp = require('sharp'); }
|
||||
catch (_) { return '[图片文件] ' + rel + '(' + size + ' 字节)。服务端未安装 sharp,无法压缩回传图片内容。'; }
|
||||
catch (_) {
|
||||
return {
|
||||
text: imageReceipt({
|
||||
ok: false, path: rel, image_attached: false, reason: 'sharp_missing', retryable: false, bytes_in: size,
|
||||
hint: '服务端未安装 sharp,无法压缩回传图片内容',
|
||||
}),
|
||||
};
|
||||
}
|
||||
// 源尺寸只读文件头(metadata 不解码像素);EXIF orientation 5-8 表示显示时旋转 90°,
|
||||
// 宽高要交换才与模型最终看到的画面一致(sharp 的 rotate() 会真的转)。
|
||||
let src = null;
|
||||
try {
|
||||
const m = await sharp(abs).metadata();
|
||||
if (m && m.width && m.height) {
|
||||
const swap = Number(m.orientation) >= 5;
|
||||
src = { w: swap ? m.height : m.width, h: swap ? m.width : m.height, format: m.format || null };
|
||||
}
|
||||
} catch (_) { src = null; }
|
||||
try {
|
||||
const r = await sharp(abs)
|
||||
.rotate()
|
||||
@@ -195,7 +235,13 @@ async function readImageAsBlocks(ctx, abs, rel, size) {
|
||||
if (r.data.length >= size) {
|
||||
// 低内容熵图(截图/线稿/纯色)JPEG 重编码可能比原图还大:不产出缓存,外链直接指向原文件
|
||||
return {
|
||||
text: '[图片文件] ' + rel + '(' + size + ' 字节,压缩无收益,按原图回传)',
|
||||
text: imageReceipt({
|
||||
ok: true, path: rel, image_attached: true,
|
||||
width: src ? src.w : undefined, height: src ? src.h : undefined,
|
||||
format: src && src.format ? src.format : undefined,
|
||||
bytes_in: size, bytes_out: size, resized: false,
|
||||
hint: '压缩无收益,按原图回传',
|
||||
}),
|
||||
image_rel: rel,
|
||||
};
|
||||
}
|
||||
@@ -208,13 +254,21 @@ async function readImageAsBlocks(ctx, abs, rel, size) {
|
||||
await fsp.mkdir(path.dirname(outAbs), { recursive: true });
|
||||
await fsp.writeFile(outAbs, r.data);
|
||||
}
|
||||
return {
|
||||
text: '[图片文件] ' + rel + '(原 ' + size + ' 字节,已压缩为 ' + r.info.width + '×' + r.info.height +
|
||||
' JPEG ' + r.data.length + ' 字节)',
|
||||
image_rel: cacheRel,
|
||||
const receipt = {
|
||||
ok: true, path: rel, image_attached: true,
|
||||
width: r.info.width, height: r.info.height, format: 'jpeg',
|
||||
bytes_in: size, bytes_out: r.data.length,
|
||||
resized: src ? (r.info.width !== src.w || r.info.height !== src.h) : undefined,
|
||||
};
|
||||
if (src) { receipt.source_width = src.w; receipt.source_height = src.h; }
|
||||
return { text: imageReceipt(receipt), image_rel: cacheRel };
|
||||
} catch (err) {
|
||||
return '[图片文件] ' + rel + '(' + size + ' 字节),图片解码/压缩失败:' + (err && err.message || err);
|
||||
return {
|
||||
text: imageReceipt({
|
||||
ok: false, path: rel, image_attached: false, reason: 'decode_failed', retryable: false, bytes_in: size,
|
||||
hint: '图片解码/压缩失败:' + (err && err.message || err),
|
||||
}),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -244,15 +298,29 @@ async function toolReadFile(ctx, args) {
|
||||
// 不做「偷偷复制」:图片要出现在模型视野里,字节必须落到本会话才能签外链(engine 的
|
||||
// mintImageUrl 与前端下载路由都以当前 taskDir 为根),那本质已是一次导入——与其藏进
|
||||
// .imgcache 不进文件表,不如让模型明确知道历史图片必须先 copy_path。
|
||||
return '[图片文件] ' + disp + '(' + st.size + ' 字节)历史会话图片不支持直接读取。' +
|
||||
'图片需要先 copy_path(source_path 用这个 @ 路径)导入本会话,再 read_file——导入后图片、terminal、mint 全量生效。';
|
||||
return {
|
||||
text: imageReceipt({
|
||||
ok: false, path: disp, image_attached: false, reason: 'past_session', retryable: false, bytes_in: st.size,
|
||||
hint: '历史会话图片不支持直接读取:先 copy_path(source_path 用这个 @ 路径)导入本会话,再 read_file——导入后图片、terminal、mint 全量生效',
|
||||
}),
|
||||
};
|
||||
}
|
||||
const rel = relOf(ctx.taskDir, abs);
|
||||
if (!ctx.vision) {
|
||||
return '[图片文件] ' + rel + '(' + st.size + ' 字节)。图片文件,当前模型不支持视觉识别,无法回传图片内容。';
|
||||
return {
|
||||
text: imageReceipt({
|
||||
ok: false, path: rel, image_attached: false, reason: 'no_vision', retryable: false, bytes_in: st.size,
|
||||
hint: '当前模型不支持视觉识别,无法回传图片像素;请基于文件名/大小作答,或直接告知用户',
|
||||
}),
|
||||
};
|
||||
}
|
||||
if (st.size > IMG_MAX_INPUT_BYTES) {
|
||||
return '[图片文件] ' + rel + '(' + st.size + ' 字节)过大(上限 ' + IMG_MAX_INPUT_BYTES + ' 字节),已跳过回传。';
|
||||
return {
|
||||
text: imageReceipt({
|
||||
ok: false, path: rel, image_attached: false, reason: 'too_large', retryable: false, bytes_in: st.size,
|
||||
hint: '超过回传上限 ' + IMG_MAX_INPUT_BYTES + ' 字节;可先用 terminal 缩小(如 ImageMagick)再读',
|
||||
}),
|
||||
};
|
||||
}
|
||||
return await readImageAsBlocks(ctx, abs, rel, st.size);
|
||||
}
|
||||
@@ -1498,18 +1566,14 @@ function toolDb(ctx, args) {
|
||||
}
|
||||
}
|
||||
|
||||
// skill:技能三件套(list/get/run)合并后的单工具(依据 docs/tool-design.md §一)。
|
||||
// 参数形状是「动词作键、技能名作值」,三者恰好给一个:
|
||||
// {list:"all"} / {get:"<名>"} / {run:"<名>", args:{...}}
|
||||
// list 的值是范围而非布尔:将来技能分类落地时直接扩成 {list:"<分类>"},调用形状不变,
|
||||
// 模型不用学第二套写法。今天只认 "all" 且报错点名——静默返回全部会让错拼的分类名
|
||||
// 看起来「能用」,将来接分类时更难排查。
|
||||
// {list:"all"} 的清单 = 目录技能 + mode=skill 的 MCP 工具;get/run 对两者一视同仁:
|
||||
// 目录技能读 SKILL.md,MCP 技能直接回注册表里的原始 input_schema。
|
||||
// args 仅 run 用且可缺省:runSkill 把 null 视作 {},真正缺哪些字段由各技能 input_schema
|
||||
// 校验点名报错(工具层再 gate 一层是双重门槛,无益)。
|
||||
// 旧形状 {op, name, input_args} 仍接受:存量技能文档的调用示例里写的是它,模型 get 到
|
||||
// 旧示例就会照着发;仅作兼容兜底,不再对外文档化(文档只体现新形状)。
|
||||
// skill:技能三件套(list/get/run)合并后的单工具(依据 docs/tool-design.md §一/§二/§三)。
|
||||
// 扁平参数:op 是唯一必填的开关,name 供 get/run 指认技能,args 仅 run 用;条件必填写不进
|
||||
// schema,由 toolSkill 校验并在报错里点名(§四/§五)。
|
||||
// 旧字段名 input_args 在入口归一化成 args(存量技能文档里仍写着它),之后只走一条分派路径——
|
||||
// 曾有两条执行路径,其中一条只读 input_args,{op:"run", args:{...}} 的参数会被静默丢弃。
|
||||
// 上一版「动词作键」不再兼容:schema 的 required:["op"] 会在 checkToolArgs 阶段先拦下,
|
||||
// 留着分支只会是死代码(lab 直调除外,不值得为它多一条路径)。
|
||||
// op=list 的清单取 listCallableSkills:目录技能 + mode=skill 的 MCP 工具;get/run 对两者一视同仁。
|
||||
// visit_url(front matter 可选):页面产物类技能声明访问路径模板,run 成功后在本层拼
|
||||
// public_link.base_url(与 mint_public_link 同源)追加完整「访问地址」——技能进程跑在
|
||||
// executor 隔离视图里不知道公网前缀,AI 拿到相对路径就会原样转给用户,而相对路径打不开。
|
||||
@@ -1531,67 +1595,44 @@ async function runSkillWithVisit(ctx, name, inputArgs) {
|
||||
: rel + '(站内路径:public_link.base_url 未配置,拼不出公网前缀)');
|
||||
}
|
||||
|
||||
const SKILL_USAGE = 'skill 用法:{"list":"all"} / {"get":"<技能名>"} / {"run":"<技能名>","args":{...}},三者恰好给一个';
|
||||
|
||||
// 技能名必须是单 token:它要原样塞进 {run:"<名>"} 的值里,空格与等号会让日志与
|
||||
// 报错里的表达产生歧义('/' 与 '..' 由 skills.skillDir 拦下)。
|
||||
function skillNameArg(value, verb) {
|
||||
if (typeof value !== 'string' || !value.trim()) {
|
||||
throw new Error('skill ' + verb + ' 需要一个技能名字符串,收到:' + JSON.stringify(value).slice(0, 60) + '。' + SKILL_USAGE);
|
||||
}
|
||||
const name = value.trim();
|
||||
if (/[\s=]/.test(name)) throw new Error('技能名不能含空格或等号:"' + name + '"');
|
||||
return name;
|
||||
}
|
||||
const SKILL_USAGE = 'skill 用法:{"op":"list"} / {"op":"get","name":"<技能名>"} / {"op":"run","name":"<技能名>","args":{...}}';
|
||||
|
||||
// list/get 带非空 args 一律报错而不是静默丢弃:否则模型会以为参数生效了。
|
||||
function rejectSkillArgs(a, verb) {
|
||||
if (!a.args || typeof a.args !== 'object' || !Object.keys(a.args).length) return;
|
||||
throw new Error(verb + ' 不接受 args,只有 run 才带参数。' + SKILL_USAGE);
|
||||
}
|
||||
|
||||
// 旧形状 {op, name, input_args} 的等价翻译(兼容存量技能文档,见上方注释)。
|
||||
async function legacyToolSkill(ctx, a) {
|
||||
const op = String(a.op == null ? '' : a.op).trim().toLowerCase();
|
||||
const name = String(a.name == null ? '' : a.name).trim();
|
||||
switch (op) {
|
||||
case 'list':
|
||||
return JSON.stringify(await skills.listCallableSkills(ctx), null, 2);
|
||||
case 'get':
|
||||
if (!name) throw new Error('op=get 缺少技能名。' + SKILL_USAGE);
|
||||
return skills.getSkillInfo(ctx, name);
|
||||
case 'run':
|
||||
if (!name) throw new Error('op=run 缺少技能名。' + SKILL_USAGE);
|
||||
return runSkillWithVisit(ctx, name, a.input_args);
|
||||
default:
|
||||
throw new Error('未知的 op:"' + op + '"。' + SKILL_USAGE);
|
||||
}
|
||||
function rejectSkillArgs(skillArgs, op) {
|
||||
if (!skillArgs || typeof skillArgs !== 'object' || !Object.keys(skillArgs).length) return;
|
||||
throw new Error('op=' + op + ' 不接受 args,只有 op=run 才带参数。' + SKILL_USAGE);
|
||||
}
|
||||
|
||||
// 旧字段名 input_args 在这里归一化成 args,再走同一条分派(见上方注释)。
|
||||
async function toolSkill(ctx, args) {
|
||||
const a = args || {};
|
||||
const verbs = ['list', 'get', 'run'].filter(k => a[k] !== undefined && a[k] !== null && a[k] !== false);
|
||||
if (!verbs.length) {
|
||||
// 三个动词键都没给时才回落到旧形状:两种都给时以新形状为准。
|
||||
if (a.op !== undefined || a.name !== undefined || a.input_args !== undefined) return legacyToolSkill(ctx, a);
|
||||
throw new Error(SKILL_USAGE);
|
||||
const op = String(a.op == null ? '' : a.op).trim().toLowerCase();
|
||||
const name = String(a.name == null ? '' : a.name).trim();
|
||||
const skillArgs = a.args !== undefined ? a.args : a.input_args;
|
||||
if (!op) throw new Error('skill 缺少必填参数 op(list / get / run)。' + SKILL_USAGE);
|
||||
switch (op) {
|
||||
case 'list':
|
||||
rejectSkillArgs(skillArgs, 'list');
|
||||
return JSON.stringify(await skills.listCallableSkills(ctx), null, 2);
|
||||
case 'get': {
|
||||
if (!name) throw new Error('op=get 缺少必填参数 name。' + SKILL_USAGE);
|
||||
rejectSkillArgs(skillArgs, 'get');
|
||||
const info = await skills.getSkillInfo(ctx, name);
|
||||
if (ctx.run && !ctx.run.deepThink && skills.parseFrontMatter(info).thinking === true) {
|
||||
ctx.run.deepThink = true;
|
||||
if (!ctx.run.thinkAuto) {
|
||||
ctx.run.thinkAuto = true;
|
||||
if (typeof ctx.onDeepThink === 'function') ctx.onDeepThink({ skill: name });
|
||||
}
|
||||
}
|
||||
return info;
|
||||
}
|
||||
case 'run':
|
||||
if (!name) throw new Error('op=run 缺少必填参数 name。' + SKILL_USAGE);
|
||||
return runSkillWithVisit(ctx, name, skillArgs);
|
||||
default:
|
||||
throw new Error('未知的 op:' + JSON.stringify(op) + '。' + SKILL_USAGE);
|
||||
}
|
||||
if (verbs.length > 1) {
|
||||
throw new Error('skill 一次只能给一个动词键,收到 ' + verbs.join(' + ') + '。' + SKILL_USAGE);
|
||||
}
|
||||
const verb = verbs[0];
|
||||
if (verb === 'list') {
|
||||
const scope = String(a.list).trim().toLowerCase();
|
||||
if (scope !== 'all') throw new Error('list 目前只支持 "all"(技能分类尚未提供),收到:"' + scope + '"');
|
||||
rejectSkillArgs(a, 'list');
|
||||
return JSON.stringify(await skills.listCallableSkills(ctx), null, 2);
|
||||
}
|
||||
const name = skillNameArg(a[verb], verb);
|
||||
if (verb === 'get') {
|
||||
rejectSkillArgs(a, 'get');
|
||||
return skills.getSkillInfo(ctx, name);
|
||||
}
|
||||
return runSkillWithVisit(ctx, name, a.args);
|
||||
}
|
||||
|
||||
// lab:全工具统一入口(元工具,tools 数组收缩到 1 个仍可达内置 + MCP 全部能力)。
|
||||
@@ -1696,7 +1737,8 @@ const HANDLERS = {
|
||||
};
|
||||
|
||||
// ctx: {sessionId, taskDir, run, cfg, db, vision}
|
||||
// 返回值一般为字符串;read_file 读图且 ctx.vision 为 true 时返回 {text, image_rel}。
|
||||
// 返回值一般为字符串;read_file 读图时返回 {text, image_rel},text 是结构化 JSON 回执
|
||||
// (见 imageReceipt),image_rel 缺省表示本次不投递像素。
|
||||
// 内置工具优先;未命中时交给 MCP 注册表(启动时发现的服务工具)。
|
||||
async function executeTool(baseCtx, name, args) {
|
||||
const handler = HANDLERS[name];
|
||||
|
||||
@@ -24,6 +24,8 @@ services:
|
||||
# 微信接入
|
||||
- WXBOT_URL=http://agent-wxbot:8093
|
||||
- WXBOT_TOKEN=${WXBOT_TOKEN:?WXBOT_TOKEN not set - run scripts/init.sh}
|
||||
# 语音识别(可选):mcp_asr 的内网地址。置空 = 不启用,前端不显示麦克风按钮。
|
||||
- ASR_URL=http://mcp_asr:3000
|
||||
networks:
|
||||
- agent-mcp-net
|
||||
|
||||
@@ -206,6 +208,30 @@ services:
|
||||
networks:
|
||||
- agent-mcp-net
|
||||
|
||||
# 语音识别:录音在客户端(套壳 App 原生桥 / 浏览器 MediaRecorder),转写在这里。
|
||||
# 同进程两个入口(见 mcp/asr/src/server.js 头注释):
|
||||
# POST /transcribe ← agent-core 的 /api/asr 转发(输入框,低延迟、不经过模型)
|
||||
# POST /mcp ← 模型的 MCP 工具 transcribe_audio({url})(转写任务目录里的音频)
|
||||
# 转写走上游 OpenAI 兼容的 /audio/transcriptions(默认智谱 glm-asr,换平台改 ASR_API_* 即可)。
|
||||
mcp_asr:
|
||||
build: ./mcp/asr
|
||||
container_name: mcp_asr
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000/health').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 5s
|
||||
environment:
|
||||
- ASR_API_KEY=${ASR_API_KEY:-}
|
||||
- ASR_API_BASE=${ASR_API_BASE:-https://open.bigmodel.cn/api/paas/v4}
|
||||
- ASR_MODEL=${ASR_MODEL:-glm-asr}
|
||||
- ASR_LOG_LEVEL=${ASR_LOG_LEVEL:-info}
|
||||
- TZ=Asia/Shanghai
|
||||
networks:
|
||||
- agent-mcp-net
|
||||
|
||||
networks:
|
||||
# 全部服务共用一个网络;由本仓库创建(固定名字,避免被加上 compose 项目名前缀)。
|
||||
# - core 与 executor 靠它按容器名访问 MCP 服务(如 http://mcp_web_search:3000/)
|
||||
|
||||
+15
-3
@@ -25,9 +25,8 @@
|
||||
- 跨 op 共享的参数放顶层,description 说明哪些 op 用它;
|
||||
- op 专属参数也放顶层,description 说明 `op=X 时必填`;
|
||||
- 仅当参数结构在工具定义时无法确定时,才用自由对象 `{ "type": "object" }`。
|
||||
- 动词 + 单 token 标识可以折叠成「动词作键、标识作值」(技能的 `{"get":"<名>"}` 即此形)。
|
||||
多个动词互斥时 `required` 表达不了,必须在代码里校验,并在报错里回显全部合法形态;
|
||||
取值范围暂只有一个时(如 `{"list":"all"}`)仍用字符串而非布尔,便于后续扩成分类等子集。
|
||||
- 互斥的多动作一律用枚举开关 `op` + 各自顶层参数,不要把动作编码进键名:键名方案无法用
|
||||
`required` 约束(漏键只能到运行时才发现),弱模型也更容易漏写。
|
||||
|
||||
### 四、条件必填
|
||||
|
||||
@@ -66,3 +65,16 @@ system prompt 不重复 schema 已表达的内容。
|
||||
### 八、上下文成本
|
||||
|
||||
tools 数组每轮请求完整重发。新工具进来时,先问"能不能装进已有工具",再问"要不要新建"。
|
||||
|
||||
### 九、回执形态
|
||||
|
||||
- **内容型**工具(`read_file` 的文本、`fetch` 的正文、`db` 的查询结果)回执就是内容本身,不要套 JSON 信封。
|
||||
- **结果型**动作(读图、裁剪、转换)回执用**单个 JSON 对象**,把判定写进字段,不写自然语言:模型按字段分支,
|
||||
不要再让它从句子里猜。字段名跨工具保持一致:`ok`(调用是否成功)/ `reason`(稳定枚举)/ `retryable` /
|
||||
`width` `height` 之类的事实字段 / `hint`(仅解释,不承载判定)。
|
||||
- 回执可能被多层改写(发送前裁剪、同回合去重、投递失败降级):改写**只动字段**,不得在 JSON 后追加句子——
|
||||
追加会把回执变成不可解析的尾巴,等于把判定权又还给语义猜测。
|
||||
- **回执是「那次调用」的历史事实,不是当前状态**:发送侧为了省 token 省略内容时,不要改写回执
|
||||
(典型错例:把当时成功的图片投递改写成 `image_attached:false`)。模型会把历史读图当成失败,
|
||||
于是反复重读同一条;省略、重放、缓存这类传输层细节不属于回执语义。
|
||||
- `ok:true` 不等于「结果已送达」:另有字段(如 `image_attached`)描述投递状态,两者不要混成一个布尔。
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# 文件上传(选择器 / 拖拽 / 粘贴)
|
||||
|
||||
前端上传只有一个落点:`app/server/public/js/files.js` 的 `uploadFilesNow(files)`。
|
||||
文件选择器和拖拽 drop 都只负责凑出 `File[]`,之后的大图压缩询问、大文件确认、
|
||||
自动建档、逐卡进度、上传与刷新文件面板全部共用同一条链路。
|
||||
|
||||
## 入口
|
||||
|
||||
| 入口 | 位置 | 说明 |
|
||||
|---|---|---|
|
||||
| 回形针按钮 / 面板上传图标 | `#file-input` 的 `change` | 选中后快照 `input.files` → `uploadFilesNow` |
|
||||
| 拖拽文件到页面任意位置 | `window` 的 `drop` | 从 `dataTransfer` 同步取 `File[]` → `uploadFilesNow` |
|
||||
| 粘贴超长文本 | `#msg-input` 的 `paste` | 包成 `File` 放进 `state.attachments`,发送时随消息上传 |
|
||||
|
||||
发布时 `send()` 会把已成功卡片的 `dataset.name` 拼成「用户上传了文件:…」告知行;
|
||||
卡片名以服务端返回的 `uploaded[].filename` 为准(重名时后端会加 `-1`/`-2`)。
|
||||
|
||||
## 拖拽实现要点
|
||||
|
||||
- **监听在 `window` 上**,不是某个 dropZone。文件落到输入框、文件面板、抽屉上时,
|
||||
没有处理的 `drop` 会让浏览器导航去打开该文件,当前草稿和正在流式的回合都会丢。
|
||||
- **只认带 `Files` 的拖拽**:拖动页面内选中的文字同样触发 `dragenter`,不判断会误弹遮罩。
|
||||
- `dragover` 必须 `preventDefault()`,否则 `drop` 不触发;同时设 `dropEffect = 'copy'`。
|
||||
- 遮罩 `#drop-overlay` 是 `pointer-events: none` 的固定层,不参与命中测试,drop 仍由 window 接管。
|
||||
- `dragenter` / `dragleave` 在子元素间移动会成对冒泡到 window,用 150ms 兜底定时器收尾;
|
||||
`relatedTarget === null`(拖出窗口)直接收干净。
|
||||
- **不支持拖入目录**:目录在 `dataTransfer.files` 里是 0 字节条目,直接传会落一个空文件。
|
||||
实现用 `webkitGetAsEntry().isDirectory` 识别并跳过,有文件时提示「暂不支持拖入文件夹,已忽略」。
|
||||
- 触屏设备没有 drag 事件,遮罩只在真实拖拽时出现,不影响移动端布局。
|
||||
- 并发批次(上一次还在传又拖入一批)共用一个总进度条:各批登记 `{loaded,total}`,
|
||||
绘制时求和,最后一批结束后才收起。
|
||||
|
||||
## 后端约束(前端已对齐)
|
||||
|
||||
- 单文件硬限 50MB(`routes/files.js` 的 `multer.limits.filesize`),超限在预处理阶段直接拒掉。
|
||||
- 大于 `upload.confirm_size_mb`(默认 10MB)先二次确认;大图大于
|
||||
`upload.image_compress_threshold_kb`(默认 1536KB)走本地 canvas 压缩询问。
|
||||
- 每个文件一个 XHR 请求,所以即时上传路径不会撞上 `upload.array('files', 10)` 的上限;
|
||||
发送时随消息上传的附件是「一个请求带多个文件」,`uploadFiles()` 会按 10 个一批切分。
|
||||
- 落盘名经 `sanitizeName` 清洗,重名自动追加序号;前端把响应里的真实文件名回写到卡片,
|
||||
避免告知行指向同名的旧文件。
|
||||
|
||||
## 已知限制
|
||||
|
||||
- 一次拖入多张大图时,`prepareUploadFile` 会逐张弹压缩确认(串行,不做批量合并)。
|
||||
- 拖拽期间会立刻 `ensureSession()` 建档(与回形针一致),即使用户随后放弃上传;
|
||||
全部文件被放弃时不会建档。
|
||||
|
||||
## 长文本粘贴转文件
|
||||
|
||||
在 `#msg-input` 上监听 `paste`(不挂 `document`,避免劫持搜索框等输入):
|
||||
|
||||
- 剪贴板文本 trim 后超过 `upload.paste_to_file_threshold_chars`(默认 2000 字符,`0` 关闭)时
|
||||
`preventDefault()`,用 `new File([原始文本], 'pasted-text-<时间戳>.txt')` 构造附件。
|
||||
阈值判断用 trim 后的长度,写入必须用原始文本(保留首尾空白与行尾)。
|
||||
- 附件进入 `state.attachments`,**不提前建档、不落盘**,chip 可随时撤掉;发送时才随消息上传。
|
||||
- 剪贴板同时有文件时:文本超阈值则文本转 txt(保住内容),另外只补非图片文件;
|
||||
文本很短时保持默认粘贴,剪贴板里的截图/文件另行挂成附件;纯文件则拦掉文件名插入。
|
||||
- 发送约束:`send()` 允许「只有附件」的消息,告知行合并已上传卡片与本条附件(用服务端返回的
|
||||
真实文件名);纯附件发送时标题退回粘贴文本的前 24 字。
|
||||
- 上传/发送失败时附件会被还原回 chip(`send()` 的失败分支),内容不丢。
|
||||
@@ -0,0 +1,393 @@
|
||||
# 语音输入:客户端录音契约
|
||||
|
||||
链路是「按住说话 → 客户端录音 → `POST /api/asr/transcribe` → 转写结果**自动发送**」。
|
||||
服务端只收音频字节,不关心是谁录的。`public/js/voice.js` 按能力自己选录音方式:
|
||||
|
||||
| 页面环境 | 录音方式 | 说明 |
|
||||
|---|---|---|
|
||||
| Android 套壳 App(WebView) | **原生录音 + JS 桥** | WebView 没有 Web Speech API;http 页面也允许 `getUserMedia` 的替代路径 |
|
||||
| 浏览器 / PWA(HTTPS) | `MediaRecorder` | 标准 `getUserMedia`,不用改壳 |
|
||||
| 纯浏览器(`http://<IP>`) | 不可用 | 非安全上下文,浏览器拒绝开麦 |
|
||||
|
||||
**发送键的四种状态**(`sessions.js` 的 `updateSendBtn` 决定):
|
||||
|
||||
| `data-state` | 什么时候 | 点击行为 |
|
||||
|---|---|---|
|
||||
| `stop` | 生成中且输入框为空 | 停止当前回合(优先级最高,不被语音入口顶掉) |
|
||||
| `voice` | 空闲、无文字无附件、且语音可用 | 进入语音模式 |
|
||||
| `keyboard` | 已在语音模式 | 回键盘模式,聚焦输入框(移动端弹键盘) |
|
||||
| `send` | 其余情况 | 发送 |
|
||||
|
||||
「语音可用」= ①有录音能力(原生桥,或安全上下文下的 `MediaRecorder`)②`GET /api/asr/status`
|
||||
返回 `enabled !== false`。所以**套壳 App 里即使页面是 `http://`,只要桥注入成功,麦克风入口照样出现**;
|
||||
纯浏览器 `http://<IP>` 不会出现(非安全上下文,浏览器禁止开麦)。
|
||||
|
||||
语音模式下输入框被 `#voice-hold` 取代(只有居中文字,无按钮外观):按住后整块输入区被
|
||||
**全宽遮罩**盖住(`#voice-overlay`,向左右各扩到屏幕边缘),遮罩里是「实时电平声波 + 提示文案」。
|
||||
手指上滑进入文案区域 → 取消态(声波与文案变红,文案改为「松开取消」),滑回下方恢复;
|
||||
松手时取消态则**直接丢弃**(不弹 toast——它会盖住声波,取消本身已经有变色/文案反馈),否则转写并直接发送。
|
||||
松手后遮罩保持到识别结束,文案换成「正在识别…」。发送键变成键盘按钮。
|
||||
|
||||
---
|
||||
|
||||
## 一、桥的契约
|
||||
|
||||
### 1. 注入时机
|
||||
|
||||
必须在 **`loadUrl()` 之前**注册,不要在 `onPageFinished` 里用 `evaluateJavascript` 注入:
|
||||
|
||||
```java
|
||||
webView.setWebViewClient(/* ... */);
|
||||
webView.addJavascriptInterface(new VoiceBridge(activity, webView), "AgentNative");
|
||||
webView.loadUrl(startUrl);
|
||||
```
|
||||
|
||||
`addJavascriptInterface` 的绑定对所有后续导航生效;在 `onPageFinished` 里再注入,首屏那次
|
||||
加载会拿不到 `window.AgentNative`(页面脚本已经跑完了)。
|
||||
|
||||
### 2. 方法(壳实现)
|
||||
|
||||
```js
|
||||
window.AgentNative = {
|
||||
startVoice(), // 开始录音;失败/拒绝通过 onNativeVoiceError 回报
|
||||
stopVoice(), // 结束录音;结果通过 onNativeVoiceResult 回传
|
||||
cancelVoice() // 放弃本次录音(无回调;页面自己已经把状态复位了)
|
||||
};
|
||||
```
|
||||
|
||||
**线程模型**:这三个方法都跑在 WebView 的 JavaBridge 子线程上,**不是主线程**。
|
||||
`MediaRecorder` 的所有调用必须收敛到主线程:方法入口先 `mainHandler.post(...)`,
|
||||
读临时文件 + Base64 才放回子线程。
|
||||
|
||||
### 3. 回调(页面定义,壳调用)
|
||||
|
||||
```js
|
||||
window.onNativeVoiceResult(base64, mime) // mime 例:'audio/mp4'
|
||||
window.onNativeVoiceError(code, message) // 见下方错误码表
|
||||
window.onNativeVoiceState('recording') // 建议实现:真正开始录时调一次
|
||||
window.onNativeVoiceAmp(amp) // 建议实现:原始幅度 0~32767,驱动遮罩里的声波
|
||||
```
|
||||
|
||||
- `onNativeVoiceAmp(amp)`:壳只上报 `MediaRecorder.getMaxAmplitude()` 的**原始值**
|
||||
(0~32767,主线程每 100ms 一次),**不要做任何换算**。dB 归一化、静音门限、快攻慢放平滑
|
||||
全部在页面侧——浏览器(RMS)和壳(峰值)两条路的规则只存在一份。
|
||||
旧名 `onNativeVoiceLevel` 仍作为别名接受(同样按原始幅度解释)。
|
||||
不实现也能用:声波静默停在基线上(不空转、不假动)。
|
||||
JS 侧还要自己做快攻慢放平滑,壳不需要平滑。
|
||||
- `onNativeVoiceState('recording')`:**按住说话的时长判定依赖它**。页面用这个回调重置
|
||||
「60 秒上限」和「太短丢弃」的起点,避免首次授权弹窗的时间被算进录音。不实现也不会崩:
|
||||
页面按点击时刻计时,代价是首次授权那一次会被当成「太短」丢弃(用户再按一次即可)。
|
||||
- 页面收到 `onNativeVoiceResult` 后会直接发送消息(不经过输入框、不可编辑)。
|
||||
- 所有回调都必须在**主线程**调 `webView.evaluateJavascript`。
|
||||
- 字符串一律用 `JSONObject.quote(...)` 拼,不要手写 `'` 转义——异常 message 里可能有
|
||||
`\`、换行、U+2028,会把整段脚本打断。
|
||||
|
||||
### 4. 状态机与错误码
|
||||
|
||||
壳侧状态:`IDLE → (WAITING_PERMISSION) → RECORDING → ENCODING → IDLE`。
|
||||
`startVoice` 只在 `IDLE` 生效,`stopVoice` 只在 `RECORDING` 生效(其余静默忽略)。
|
||||
|
||||
| code | 触发时机 | 页面表现 |
|
||||
|---|---|---|
|
||||
| `no_permission` | 用户拒绝 / 取消授权弹窗 | toast「需要麦克风权限」 |
|
||||
| `start_failed` | `prepare()` / `start()` 抛异常(含回退 MIC 后仍失败) | toast |
|
||||
| `too_short` | `stop()` 抛异常(录音 < ~1s) | toast「说话时间太短,请按住再说」,**不上传** |
|
||||
| `read_failed` | 读临时文件失败 | toast |
|
||||
| `cancelled` | 生命周期打断(切后台 / 换页) | **静默复位,不弹提示** |
|
||||
| 其他未知 code | — | toast 里显示 `message`,没有就显示 code |
|
||||
|
||||
**取消语义**:页面在 `stopVoice()` 之后会一直等结果(直到点击后 90 秒的硬超时),
|
||||
所以壳在生命周期里取消录音时**必须**回调 `onNativeVoiceError('cancelled','')`,
|
||||
否则按钮会卡在「录音中」。`cancelVoice()` 是页面主动发起的,壳不需要回报。
|
||||
|
||||
### 5. 体积与时序
|
||||
|
||||
- 16kHz / 32kbps / 单声道 AAC:60 秒 ≈ **240KB**,Base64 后 ≈ **320KB**。
|
||||
- `evaluateJavascript` 走 Binder,单次约 1MB 上限;320KB 安全,但**不要再延长单次时长**。
|
||||
真要 2 分钟以上,改成按段回传或本地 HTTP 回传。
|
||||
- 松开后页面最多等 3 秒拿音频(等不到就复位,覆盖「壳还在等权限」的情况);
|
||||
拿到后上传+识别,正常 1~2 秒。
|
||||
- 按住时间 < 500ms 视为误触:页面直接丢弃,不上传;壳自己的 `too_short` 是第二道保险。
|
||||
|
||||
---
|
||||
|
||||
## 二、Java 参考骨架(纯 Java / minSdk 21)
|
||||
|
||||
只演示契约要求的部分;`UrlPolicy`、埋点、通知等按各自工程接。
|
||||
|
||||
```java
|
||||
// AndroidManifest.xml
|
||||
// <uses-permission android:name="android.permission.RECORD_AUDIO" />
|
||||
|
||||
public final class VoiceBridge {
|
||||
private static final int REQ_MIC = 1001;
|
||||
private static final long MAX_MS = 60_000;
|
||||
|
||||
private enum State { IDLE, WAITING_PERMISSION, RECORDING, ENCODING }
|
||||
|
||||
private final Activity activity;
|
||||
private final WebView webView;
|
||||
private final Handler main = new Handler(Looper.getMainLooper());
|
||||
private final Runnable autoStop = this::stopOnMain;
|
||||
|
||||
// 实时电平:每 100ms 读一次 getMaxAmplitude() 交给页面画声波
|
||||
private static final long LEVEL_INTERVAL_MS = 100;
|
||||
private final Runnable levelTick = this::tickLevel;
|
||||
|
||||
private void tickLevel() {
|
||||
MediaRecorder r = recorder;
|
||||
if (state != State.RECORDING || r == null) return;
|
||||
int amp = 0;
|
||||
try { amp = r.getMaxAmplitude(); } catch (Exception ignored) { }
|
||||
// 原始幅度直接给页面:dB 归一化/门限/平滑全在 JS,壳不做任何数学
|
||||
js("window.onNativeVoiceAmp && window.onNativeVoiceAmp(" + amp + ")");
|
||||
main.postDelayed(levelTick, LEVEL_INTERVAL_MS);
|
||||
}
|
||||
|
||||
private State state = State.IDLE;
|
||||
private MediaRecorder recorder;
|
||||
private File file;
|
||||
|
||||
public VoiceBridge(Activity activity, WebView webView) {
|
||||
this.activity = activity;
|
||||
this.webView = webView;
|
||||
}
|
||||
|
||||
// ---- JS 入口:JavaBridge 线程 → 主线程 ----
|
||||
@JavascriptInterface public void startVoice() { main.post(this::startOnMain); }
|
||||
@JavascriptInterface public void stopVoice() { main.post(this::stopOnMain); }
|
||||
@JavascriptInterface public void cancelVoice() { main.post(() -> cancelOnMain(true)); }
|
||||
|
||||
// ---- 生命周期:由 Activity / WebViewClient 调用 ----
|
||||
// 注意:只取消真正在录/在编码的状态。授权弹窗期间某些 ROM 会走 onPause,
|
||||
// 那里如果一并取消,首次授权的两段式流程就断了。
|
||||
public void onPause() {
|
||||
main.post(() -> {
|
||||
if (state == State.RECORDING || state == State.ENCODING) cancelOnMain(true);
|
||||
});
|
||||
}
|
||||
public void onDestroy() { main.post(() -> cancelOnMain(false)); }
|
||||
public void onPageChanged() { main.post(() -> cancelOnMain(true)); }
|
||||
|
||||
public void onRequestPermissionsResult(int requestCode, int[] grantResults) {
|
||||
if (requestCode != REQ_MIC) return;
|
||||
main.post(() -> {
|
||||
if (state != State.WAITING_PERMISSION) return;
|
||||
state = State.IDLE;
|
||||
boolean ok = grantResults != null && grantResults.length > 0
|
||||
&& grantResults[0] == PackageManager.PERMISSION_GRANTED;
|
||||
if (!ok) { fail("no_permission", "需要麦克风权限"); return; }
|
||||
beginRecording(); // 授权后直接开录(体验优先,不要求用户再点一次)
|
||||
});
|
||||
}
|
||||
|
||||
private void startOnMain() {
|
||||
if (state != State.IDLE) return;
|
||||
// 第三方 CDN 脚本同样能调这个桥:只允许自家页面 + 前台时录音
|
||||
if (!UrlPolicy.isAllowed(webView.getUrl()) || !activity.hasWindowFocus()) {
|
||||
fail("forbidden", "当前页面不允许录音");
|
||||
return;
|
||||
}
|
||||
if (ContextCompat.checkSelfPermission(activity, Manifest.permission.RECORD_AUDIO)
|
||||
!= PackageManager.PERMISSION_GRANTED) {
|
||||
state = State.WAITING_PERMISSION;
|
||||
ActivityCompat.requestPermissions(
|
||||
activity, new String[]{Manifest.permission.RECORD_AUDIO}, REQ_MIC);
|
||||
return;
|
||||
}
|
||||
beginRecording();
|
||||
}
|
||||
|
||||
@SuppressLint("NewApi")
|
||||
private void beginRecording() {
|
||||
// VOICE_RECOGNITION 关掉 AGC/降噪,对 ASR 更友好;个别机型不支持则回退 MIC
|
||||
int[] sources = {MediaRecorder.AudioSource.VOICE_RECOGNITION,
|
||||
MediaRecorder.AudioSource.MIC};
|
||||
for (int source : sources) {
|
||||
File f = null;
|
||||
MediaRecorder r = null;
|
||||
try {
|
||||
f = File.createTempFile("voice", ".m4a", activity.getCacheDir());
|
||||
r = Build.VERSION.SDK_INT >= 31
|
||||
? new MediaRecorder(activity) // 无参构造在 API 31 起废弃
|
||||
: new MediaRecorder();
|
||||
r.setAudioSource(source);
|
||||
r.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
|
||||
r.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
|
||||
r.setAudioSamplingRate(16000);
|
||||
r.setAudioEncodingBitRate(32000);
|
||||
r.setAudioChannels(1);
|
||||
r.setOutputFile(f.getAbsolutePath());
|
||||
r.prepare();
|
||||
r.start();
|
||||
|
||||
recorder = r;
|
||||
file = f;
|
||||
state = State.RECORDING;
|
||||
main.postDelayed(autoStop, MAX_MS);
|
||||
main.postDelayed(levelTick, LEVEL_INTERVAL_MS);
|
||||
// 让页面的 60s 从“真的开始录”算(授权弹窗的时间不算)
|
||||
js("window.onNativeVoiceState && window.onNativeVoiceState('recording')");
|
||||
// 不要再额外弹原生 Toast:页面遮罩已经给了全部反馈,弹窗会压住声波
|
||||
return;
|
||||
} catch (Exception e) {
|
||||
// 局部变量释放:失败时 recorder 字段还没赋值,releaseQuietly() 管不到它
|
||||
if (r != null) {
|
||||
try { r.stop(); } catch (Exception ignored) { }
|
||||
try { r.release(); } catch (Exception ignored) { }
|
||||
}
|
||||
delete(f);
|
||||
}
|
||||
}
|
||||
state = State.IDLE;
|
||||
fail("start_failed", "无法开始录音");
|
||||
}
|
||||
|
||||
private void stopOnMain() {
|
||||
if (state != State.RECORDING) return;
|
||||
main.removeCallbacks(autoStop);
|
||||
main.removeCallbacks(levelTick); // 停止上报电平
|
||||
state = State.ENCODING;
|
||||
|
||||
MediaRecorder r = recorder;
|
||||
File f = file;
|
||||
recorder = null;
|
||||
file = null;
|
||||
|
||||
boolean stopped = true;
|
||||
try { r.stop(); } catch (RuntimeException e) { stopped = false; } // 太短会抛
|
||||
try { r.release(); } catch (Exception ignored) { }
|
||||
|
||||
if (!stopped) {
|
||||
delete(f);
|
||||
state = State.IDLE;
|
||||
fail("too_short", "没有录到声音");
|
||||
return;
|
||||
}
|
||||
new Thread(() -> {
|
||||
try {
|
||||
byte[] bytes = readAll(f);
|
||||
String b64 = Base64.encodeToString(bytes, Base64.NO_WRAP);
|
||||
delete(f);
|
||||
main.post(() -> {
|
||||
state = State.IDLE;
|
||||
js("window.onNativeVoiceResult(" + JSONObject.quote(b64) + ",'audio/mp4')");
|
||||
});
|
||||
} catch (Exception e) {
|
||||
delete(f);
|
||||
main.post(() -> { state = State.IDLE; fail("read_failed", "读取录音失败"); });
|
||||
}
|
||||
}, "voice-encode").start();
|
||||
}
|
||||
|
||||
/** notify=true:页面在等结果,必须回 cancelled 让它复位;onDestroy 时页面已走,不必回。 */
|
||||
private void cancelOnMain(boolean notify) {
|
||||
main.removeCallbacks(autoStop);
|
||||
main.removeCallbacks(levelTick);
|
||||
boolean wasActive = state != State.IDLE;
|
||||
releaseQuietly();
|
||||
delete(file);
|
||||
file = null;
|
||||
state = State.IDLE;
|
||||
if (notify && wasActive) js("window.onNativeVoiceError('cancelled','')");
|
||||
}
|
||||
|
||||
private void releaseQuietly() {
|
||||
if (recorder == null) return;
|
||||
try { recorder.stop(); } catch (Exception ignored) { }
|
||||
try { recorder.release(); } catch (Exception ignored) { }
|
||||
recorder = null;
|
||||
}
|
||||
|
||||
private void fail(String code, String message) {
|
||||
js("window.onNativeVoiceError(" + JSONObject.quote(code) + ","
|
||||
+ JSONObject.quote(message) + ")");
|
||||
}
|
||||
|
||||
private void js(String script) {
|
||||
webView.post(() -> webView.evaluateJavascript(script, null));
|
||||
}
|
||||
|
||||
private static byte[] readAll(File f) throws Exception {
|
||||
try (FileInputStream in = new FileInputStream(f);
|
||||
ByteArrayOutputStream out = new ByteArrayOutputStream()) {
|
||||
byte[] buf = new byte[8192];
|
||||
int n;
|
||||
while ((n = in.read(buf)) > 0) out.write(buf, 0, n);
|
||||
return out.toByteArray();
|
||||
}
|
||||
}
|
||||
|
||||
private static void delete(File f) {
|
||||
if (f != null) f.delete();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
接线:
|
||||
|
||||
```java
|
||||
// setupWebView() 里,loadUrl 之前
|
||||
webView.addJavascriptInterface(new VoiceBridge(this, webView), "AgentNative");
|
||||
|
||||
@Override public void onRequestPermissionsResult(int code, String[] p, int[] r) {
|
||||
super.onRequestPermissionsResult(code, p, r);
|
||||
bridge.onRequestPermissionsResult(code, r);
|
||||
}
|
||||
@Override protected void onPause() { bridge.onPause(); super.onPause(); }
|
||||
@Override protected void onDestroy() { bridge.onDestroy(); super.onDestroy(); }
|
||||
// WebViewClient.onPageStarted:URL 变了就 bridge.onPageChanged()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 三、壳侧必须同步的清单
|
||||
|
||||
| 项 | 内容 |
|
||||
|---|---|
|
||||
| `AndroidManifest.xml` | `<uses-permission android:name="android.permission.RECORD_AUDIO" />` |
|
||||
| `proguard-rules.pro` | 保留桥:`-keepclassmembers class * { @android.webkit.JavascriptInterface <methods>; }`,并删掉「本项目未使用 addJavascriptInterface」的注释 |
|
||||
| README 安全表 | 从「不使用 JS 桥」改为「仅暴露语音桥:三个无参方法,无文件/存储读取能力」 |
|
||||
| README 权限行 | 「只有 INTERNET」→「INTERNET + RECORD_AUDIO」 |
|
||||
| 桥的防护 | `UrlPolicy.isAllowed(webView.getUrl())` + `activity.hasWindowFocus()` + 单实例状态机;建议再加一个原生 Toast 提示录音已开始(外部 CDN 脚本与页面同源,能调桥,但用户至少看得见) |
|
||||
| 页面导航 | `onPageStarted` 且 URL 变化时取消,避免 base64 打到新页面的回调上 |
|
||||
|
||||
> 触摸时间戳门槛(最近 N 秒内有触摸才允许录音)能进一步防静默录音,但会让无障碍操作不可用,
|
||||
> 不建议默认开。
|
||||
|
||||
---
|
||||
|
||||
## 四、iOS(以后若有 WKWebView 壳)
|
||||
|
||||
契约相同,换注入方式:
|
||||
|
||||
```swift
|
||||
// 注入:WKUserScript(source: "window.AgentNative = {...}", injectionTime: .atDocumentStart, ...)
|
||||
// 录音:AVAudioRecorder(16kHz / 单声道 / m4a)
|
||||
// 回传:webView.evaluateJavaScript("window.onNativeVoiceResult('\(b64)','audio/mp4')")
|
||||
```
|
||||
|
||||
`Info.plist` 需要 `NSMicrophoneUsageDescription`。
|
||||
|
||||
## 五、服务端接口
|
||||
|
||||
```http
|
||||
POST /api/asr/transcribe
|
||||
Content-Type: audio/mp4 # 或 audio/webm / audio/wav / application/octet-stream
|
||||
Cookie: <登录态> # 与页面同一份 HttpOnly cookie
|
||||
|
||||
<音频二进制>
|
||||
```
|
||||
|
||||
```json
|
||||
{ "text": "打开今天的日报", "ms": 842, "model": "glm-asr" }
|
||||
```
|
||||
|
||||
- 上限 20MB,超时 120s;识别失败返回 `{"error": "..."}`(前端直接 toast 出来)。
|
||||
- `GET /api/asr/status` → `{"enabled": true}`:`ASR_URL` 没接线时为 `false`。
|
||||
|
||||
## 六、为什么不用浏览器的 Web Speech API
|
||||
|
||||
- Android WebView 没有实现 `SpeechRecognition`;iOS WKWebView 同样不可用(只有 Safari 有)。
|
||||
- 桌面 Chrome 的识别走 Google 服务,国内基本报 `network`。
|
||||
- 三端行为不一致、错误码各异,最后还是要留一条服务端通路——不如只留一条。
|
||||
+65
-8
@@ -36,6 +36,11 @@ const IDLE_MS = Number(process.env.OFFICE_IDLE_MS || 6 * 3600 * 1000);
|
||||
const RESPOND_TIMEOUT_MS = 15000; // 数据请求(资源/API/SSE)等待 watch 首响应包的时限;拒连/超时走自愈重试
|
||||
const NAV_TIMEOUT_MS = 5000; // 导航请求等就绪页面首响应:更短——失败立刻转中转页,不白屏干等
|
||||
const COOLDOWN_MS = 8000; // 启动失败拉黑该文件一段时间:中转页自动刷新不成 spawn 风暴(二进制缺失/坏文件早退)
|
||||
// 探针(HEAD)等「首帧可用」的上限:watch 冷启动 = 起进程 + 首次渲染,实测 2~6s(大文档更久)。
|
||||
// 等不到就照回 200,把后续交给导航侧的中转页 + 自愈,探针不无限挂着。
|
||||
const HEAD_WAIT_MS = 15000;
|
||||
// 导航请求连续超时多少次才判死重启:单次超时只是「还没渲染完」(见 route 里的注释)。
|
||||
const NAV_TIMEOUT_RETRIES = 3;
|
||||
const BIN = process.env.OFFICECLI_BIN || '/workspace/skills/office/bin/officecli';
|
||||
const TOKEN = String(process.env.EXECUTOR_TOKEN || '');
|
||||
const EXTS = new Set(['.docx', '.xlsx', '.pptx']);
|
||||
@@ -164,10 +169,13 @@ async function spawnWatch(abs) {
|
||||
const page = pageOf(abs);
|
||||
let slot = -1;
|
||||
for (let i = 0; i < SLOTS; i++) if (!slots.has(i)) { slot = i; break; }
|
||||
if (slot < 0) { // 槽满:优先踢「无活动连接且最久未用」;全开着页面才抢最冷的
|
||||
if (slot < 0) { // 槽满:优先踢「就绪且无活动连接」里最久未用的;正在启动的槽最后才动
|
||||
// (踢掉一个还没就绪的槽 = 白白浪费数秒冷启动,且会让等它的那一路请求失败并进冷却)
|
||||
const ent = [...slots.entries()];
|
||||
const idleReady = ent.filter(([, s]) => !s.conns && s.ready);
|
||||
const idle = ent.filter(([, s]) => !s.conns);
|
||||
slot = (idle.length ? idle : ent).sort((a, b) => a[1].lastUsed - b[1].lastUsed)[0][0];
|
||||
slot = (idleReady.length ? idleReady : (idle.length ? idle : ent))
|
||||
.sort((a, b) => a[1].lastUsed - b[1].lastUsed)[0][0];
|
||||
stop(slot); // 端口按代轮转,新 watch 不会落在被踢者的旧口上,无需等其回收
|
||||
}
|
||||
const wp = await allocPort();
|
||||
@@ -293,6 +301,18 @@ function matchPage(rel) {
|
||||
return best;
|
||||
}
|
||||
|
||||
// 探针预热:把槽位等到「就绪」(= 首帧渲染完成,portUp 探的就是页面本身),复用时直接返回。
|
||||
// 与并发请求共用同一次 startWatch(starting 去重);任何失败/超时都静默——就绪与否不影响 200。
|
||||
async function warmSlot(rr) {
|
||||
let timer;
|
||||
try {
|
||||
await Promise.race([
|
||||
waitReady(rr),
|
||||
new Promise((r) => { timer = setTimeout(r, HEAD_WAIT_MS); }),
|
||||
]);
|
||||
} catch (_) {} finally { clearTimeout(timer); }
|
||||
}
|
||||
|
||||
// 未命中兜底:请求路径本身就是存在的会话文档 → 回收/挤掉后就地复活;不行 = 维持 404。
|
||||
async function revive(rel) {
|
||||
const rr = (rel || '').replace(/\/+$/, '');
|
||||
@@ -354,10 +374,12 @@ function tryForward(entry, req, res, innerPath, respondTimeoutMs) {
|
||||
res.once('close', release);
|
||||
res.once('finish', release);
|
||||
let started = false;
|
||||
let upRes = null;
|
||||
const upstream = http.request(
|
||||
{ host: '127.0.0.1', port: entry.port, path: encodeURI(inner) + search, method: req.method, headers: fwd },
|
||||
(up) => {
|
||||
started = true;
|
||||
upRes = up;
|
||||
clearTimeout(watchdog);
|
||||
const h = Object.assign({}, up.headers);
|
||||
const isSSE = String(h['content-type'] || '').includes('text/event-stream');
|
||||
@@ -385,8 +407,14 @@ function tryForward(entry, req, res, innerPath, respondTimeoutMs) {
|
||||
upstream.on('error', () => {
|
||||
clearTimeout(watchdog);
|
||||
release();
|
||||
if (!started && !res.headersSent) reject(new Error('watch 不可达'));
|
||||
else res.destroy();
|
||||
if (!started && !res.headersSent) { reject(new Error('watch 不可达')); return; }
|
||||
// 已开始回写后上游报错:不立刻销毁客户端响应。watch 会在响应其实已完整时断链
|
||||
//(对 HEAD 必现,见 route 里的注释),Node 客户端的报错早于 'end' 一拍到达,直接
|
||||
// destroy 会把马上要 flush 的响应打断。留一拍再判,真断裂照样收掉,不把半截响应当成功。
|
||||
setTimeout(() => {
|
||||
if (res.writableEnded) return;
|
||||
try { res.destroy(); } catch (_) {}
|
||||
}, upRes && upRes.complete ? 0 : 120);
|
||||
});
|
||||
req.pipe(upstream);
|
||||
});
|
||||
@@ -517,7 +545,8 @@ function fail502(res) {
|
||||
}
|
||||
|
||||
function notFound(res) {
|
||||
res.writeHead(404, { 'content-type': 'text/plain; charset=utf-8' })
|
||||
// no-store:前端拿 HEAD 当探针,404 一旦被浏览器缓存,文件补上后还会继续显示「暂无文档」
|
||||
res.writeHead(404, { 'content-type': 'text/plain; charset=utf-8', 'cache-control': 'no-store' })
|
||||
.end('无匹配预览(文档不存在/不在会话目录下,或不是 .docx/.xlsx/.pptx)');
|
||||
}
|
||||
|
||||
@@ -534,6 +563,24 @@ async function route(req, res) {
|
||||
if (u === PREFIX + '/' || u.startsWith(PREFIX + '/')) {
|
||||
const rel = u.slice(PREFIX.length) || '/';
|
||||
const rr = rel.replace(/\/+$/, '');
|
||||
// HEAD = 前端预览窗的「这份文档能不能预览」探针。文档在不在只在本地按文件系统回答,绝不上探 watch:
|
||||
// 实测 watch(Kestrel) 对 HEAD 只发响应头(带 content-length)不发 body 就断链,Node 客户端抛
|
||||
// HPE_CLOSED_CONNECTION;旧逻辑把它当上游故障 res.destroy(),同一文档十次 HEAD 里六七次变成
|
||||
// 空响应/502。
|
||||
// 但回 200 前要等槽位就绪(有上限):冷启动的首次渲染很贵,旧实现由 HEAD 自己把这段等掉,
|
||||
// iframe 一挂上就是渲染好的页面;若秒回 200,这段开销就落到随后 5s 时限的导航请求上,
|
||||
// 慢文档会被判超时,表现为冷启动转圈很久。等不到也照回 200(见 HEAD_WAIT_MS)。
|
||||
if (req.method === 'HEAD') {
|
||||
let abs = null;
|
||||
try { abs = guard(path.join(WORKSPACE, rr)); } catch (_) {}
|
||||
if (abs) {
|
||||
await warmSlot(rr);
|
||||
res.writeHead(200, { 'content-type': 'text/plain; charset=utf-8', 'cache-control': 'no-store', 'content-length': '0' }).end();
|
||||
return;
|
||||
}
|
||||
// 只有办公文档路径由本网关判定存在性;页面资源等其它 HEAD 仍走原转发逻辑
|
||||
if (EXTS.has(path.extname(rr).toLowerCase())) return notFound(res);
|
||||
}
|
||||
if (isNav(req)) {
|
||||
// 导航请求 = 用户正盯着这一刻:绝不为启动/自愈阻塞首字节。
|
||||
// 槽位不在或未就绪 → 中转页 + 后台热槽(startWatch 去重,在途就等它);就绪槽位转发失败 → 同样中转页。
|
||||
@@ -546,10 +593,20 @@ async function route(req, res) {
|
||||
if (fileChanged(m.entry)) { dropChanged(m.entry); startWatch(abs).catch(() => {}); return loading(res); }
|
||||
if (m.exact && !u.endsWith('/')) { res.writeHead(301, { location: (req.url || u) + '/' }).end(); return; } // 相对资源解析需要尾斜杠
|
||||
const cut = PREFIX.length + m.entry.pageDec.length; // 剥掉「前缀+页面路径」才是 watch 的内层路径
|
||||
try { await tryForward(m.entry, req, res, u.length > cut ? u.slice(cut) : '/', NAV_TIMEOUT_MS); return; }
|
||||
try { await tryForward(m.entry, req, res, u.length > cut ? u.slice(cut) : '/', NAV_TIMEOUT_MS); m.entry.navTimeouts = 0; return; }
|
||||
catch (e) {
|
||||
retire(m.entry, String(e && e.message || e));
|
||||
if (!res.headersSent) { startWatch(abs).catch(() => {}); return loading(res); }
|
||||
const msg = String(e && e.message || e);
|
||||
// 超时 ≠ 死:watch 可能只是首次渲染还没完(大文档 >5s)。一超时就摘掉会把在跑的渲染连坐杀掉,
|
||||
// 中转页刷新后从头再来——用户看到的就是「冷启动一直转圈」。单次超时保留槽位,
|
||||
// 连续 NAV_TIMEOUT_RETRIES 次(约 15s 无首字节)才判死重启,兼顶真卡死。
|
||||
m.entry.navTimeouts = /超时/.test(msg) ? (m.entry.navTimeouts || 0) + 1 : NAV_TIMEOUT_RETRIES;
|
||||
if (m.entry.navTimeouts < NAV_TIMEOUT_RETRIES) {
|
||||
log(`slot 保留:导航超时第 ${m.entry.navTimeouts} 次(${msg})`);
|
||||
} else {
|
||||
retire(m.entry, msg);
|
||||
startWatch(abs).catch(() => {});
|
||||
}
|
||||
if (!res.headersSent) return loading(res);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
+17
-7
@@ -1,12 +1,13 @@
|
||||
# MCP 服务集合
|
||||
|
||||
三个 MCP(Model Context Protocol)服务,各占一个目录,由仓库根 `docker-compose.yml` 统一编排:
|
||||
四个 MCP(Model Context Protocol)服务,各占一个目录,由仓库根 `docker-compose.yml` 统一编排:
|
||||
|
||||
| 目录 | 容器名 | 作用 | 容器内端口 | MCP 端点 |
|
||||
|---|---|---|---|---|
|
||||
| `web_search/` | `mcp_web_search` | 免费网页搜索(Parallel / Exa,无需 key) | `3000` | `POST /` |
|
||||
| `image_task/` | `mcp_image_task` | 智谱 CogView 图片生成(单工具 op=create) | `3000` | `POST /mcp` |
|
||||
| `image_task/` | `mcp_image_task` | 图片生成 / 编辑(智谱原生或任意 OpenAI 兼容 `/images/generations`,单工具 op=create/get) | `3000` | `POST /mcp` |
|
||||
| `video_task/` | `mcp_video_task` | 智谱 CogVideo 视频生成(异步,单工具 op=create/get) | `3000` | `POST /mcp` |
|
||||
| `asr/` | `mcp_asr` | 语音转文字(上游 OpenAI 兼容 `/audio/transcriptions`),**额外带一个给输入框用的 `POST /transcribe`** | `3000` | `POST /mcp` |
|
||||
|
||||
**这些服务一个宿主机端口都不占**——它们只挂在 `agent-mcp-net` 上,由 `agent-core` / `agent-executor`
|
||||
按容器名访问。所以整台服务器对外只需要开 agent-core 那一个端口(`AGENT_PORT`,默认 8090)。
|
||||
@@ -31,15 +32,23 @@ mcp_servers:
|
||||
url: "http://mcp_image_task:3000/mcp"
|
||||
- name: video_task
|
||||
url: "http://mcp_video_task:3000/mcp"
|
||||
# 可选:语音转写(低频,建议 skill 模式,不占每轮工具表)
|
||||
- name: asr
|
||||
url: "http://mcp_asr:3000/mcp"
|
||||
mode: skill
|
||||
```
|
||||
|
||||
服务没起来时,core 只在启动日志里告警,不影响 Web 主流程,但对应工具不会下发给模型。工具发现只在启动时做一次,失败后会自动按 2s / 20s / 1min 重试 3 次;三次都失败则需重启 core。三个服务都配了 `/health` 健康检查,`docker ps` 可直接看状态。
|
||||
服务没起来时,core 只在启动日志里告警,不影响 Web 主流程,但对应工具不会下发给模型。工具发现只在启动时做一次,失败后会自动按 2s / 20s / 1min 重试 3 次;三次都失败则需重启 core。四个服务都配了 `/health` 健康检查,`docker ps` 可直接看状态。
|
||||
|
||||
> 从旧的「agent + mcp 双仓库」部署升级:先在旧 mcp 仓库目录 `docker compose down`(只需一次,否则同名容器与网络会冲突),再 `up` 本栈。
|
||||
|
||||
`image_task` / `video_task` 需要智谱 token,但 token **不由服务端保存**——每个请求从 header 读取
|
||||
(agent 侧配在 `config.yml` 的 `mcp_servers[].headers`,`init.sh` 会用 `BIGMODEL_KEY` 自动写入)。
|
||||
所以这里不用配任何密钥。
|
||||
`image_task` / `video_task` 的 token **不由服务端保存**——每个请求从 `Authorization` header 读取
|
||||
(agent 侧配在 `config.yml` 的 `mcp_servers[].headers`)。`video_task` 用智谱 token;
|
||||
`image_task` 用对应上游(智谱或 OpenAI 兼容网关)的 token,上游地址与模型分别由
|
||||
`X-Image-Task-Base-Url` / `X-Image-Task-Model` 指定。所以这两个服务不用配任何密钥。
|
||||
|
||||
`asr` 是例外:输入框那条路(`agent-core` → `POST /transcribe`)不经过 MCP 调用,拿不到 header,
|
||||
所以它自己的上游密钥由容器 env `ASR_API_KEY` 提供(写在根 `.env`,已 gitignore,不挂给其他容器)。
|
||||
|
||||
### 调试端口
|
||||
|
||||
@@ -75,8 +84,9 @@ AI 把拿到的 URL 当 image_url 交给 video_task:3000/mcp
|
||||
| 服务 | 公网暴露 | 说明 |
|
||||
|---|---|---|
|
||||
| `mcp_web_search` | 无 | 服务端无鉴权,**永远不要映射出去**(会被白嫖搜索额度);agent 走内网访问 |
|
||||
| `mcp_image_task` | 无 | token 由调用方自带,暴露后可被拿去消耗你的智谱额度 |
|
||||
| `mcp_image_task` | 无 | token 由调用方自带,暴露后可被拿去消耗你的上游图片额度 |
|
||||
| `mcp_video_task` | 无 | 它需要的公开临时 URL 由 agent 侧集成的工具提供,自己不占端口 |
|
||||
| `mcp_asr` | 无 | `/mcp` 与 `/transcribe` 都无鉴权,只能由 core 经内网调用;它持有上游 ASR 密钥 |
|
||||
|
||||
## 单独调试
|
||||
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
node_modules
|
||||
*.md
|
||||
@@ -0,0 +1,33 @@
|
||||
FROM node:22-slim
|
||||
|
||||
# 国内构建慢:apt 默认源在国外,apt-get update 能卡十几分钟。默认换成腾讯云镜像。
|
||||
# 境外构建或想换源:docker compose build --build-arg DEBIAN_MIRROR=deb.debian.org mcp_asr
|
||||
# 腾讯云 CVM 上想更快可用内网镜像 mirrors.tencentyun.com(仅腾讯云内网可达)。
|
||||
ARG DEBIAN_MIRROR=mirrors.cloud.tencent.com
|
||||
|
||||
# 换 apt 源:bookworm/trixie 可能是 /etc/apt/sources.list,也可能是 deb822 的
|
||||
# /etc/apt/sources.list.d/*.sources,直接用 grep 扫一遍、只替换域名,两种格式都盖住
|
||||
#(路径 /debian、/debian-security 保持不变)。与 executor/Dockerfile 同一套做法。
|
||||
RUN set -eux; \
|
||||
files="$(grep -rl 'deb.debian.org' /etc/apt/ 2>/dev/null || true)"; \
|
||||
if [ -n "$files" ]; then sed -i "s|deb.debian.org|${DEBIAN_MIRROR}|g" $files; fi; \
|
||||
grep -rn "${DEBIAN_MIRROR}" /etc/apt/ | head -5 || true
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# ffmpeg 只装在这里(core 镜像保持干净):WebView 录出来是 webm/opus,iOS 是 mp4/aac,
|
||||
# 云端只认 wav/mp3——统一在容器内转成 16k 单声道 wav,前端与引擎两头都不用关心格式。
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends ffmpeg ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# 零依赖:不需要 npm install(也就没有 registry 慢/被墙的问题)
|
||||
COPY package.json ./
|
||||
COPY src ./src
|
||||
|
||||
ENV PORT=3000
|
||||
EXPOSE 3000
|
||||
|
||||
CMD ["node", "src/server.js"]
|
||||
@@ -0,0 +1,87 @@
|
||||
# mcp/asr — 语音识别服务
|
||||
|
||||
一个容器,两个入口,共用同一套转写实现:
|
||||
|
||||
| 入口 | 谁在用 | 说明 |
|
||||
|---|---|---|
|
||||
| `POST /transcribe` | `agent-core` 的 `/api/asr/transcribe`(H5 输入框) | 普通 HTTP,音频二进制进、`{text}` 出。**不经过模型、不消耗 token、1~3s** |
|
||||
| `POST /mcp` | 模型(MCP 工具 `transcribe_audio`) | 转写任务目录里已有的音频,参数是公网 URL(与 `video_task` 的 `image_url` 同构) |
|
||||
| `GET /health` | compose healthcheck | `{ok, model, configured}` |
|
||||
|
||||
端口 `3000` **只在 `agent-mcp-net` 内网**,不映射宿主机——本服务自身没有鉴权,暴露即被白嫖。
|
||||
对外唯一入口仍是 `agent-core`(`/api/asr` 走登录态鉴权)。
|
||||
|
||||
## 为什么不是「录音 → 上传附件 → 模型调工具」
|
||||
|
||||
那要额外一次完整模型往返(3~10s + token),识别结果落进聊天记录而不是输入框,用户还没法改错字。
|
||||
输入法级别的功能必须绕开模型;`/mcp` 那个口只留给「转写一段已上传的录音」这种低频需求。
|
||||
|
||||
## 部署
|
||||
|
||||
```bash
|
||||
# 1) 配上游密钥(写进仓库根 .env,已 gitignore)
|
||||
# ASR_API_KEY=你的智谱 token
|
||||
# 2) 起服务
|
||||
docker compose up -d --build mcp_asr
|
||||
docker compose restart agent-core # core 要在启动时读到 ASR_URL
|
||||
curl -s http://127.0.0.1:8090/api/asr/status # {"enabled":true,...}
|
||||
```
|
||||
|
||||
`agent-core` 的环境变量 `ASR_URL`(compose 里固定为 `http://mcp_asr:3000`)是开关:置空即
|
||||
不启用,`/api/asr/*` 回 503,前端不显示麦克风按钮。
|
||||
|
||||
## 上游
|
||||
|
||||
任何 OpenAI 兼容的 `/audio/transcriptions` 都能用;默认智谱:
|
||||
|
||||
| 变量 | 默认 |
|
||||
|---|---|
|
||||
| `ASR_API_BASE` | `https://open.bigmodel.cn/api/paas/v4` |
|
||||
| `ASR_API_KEY` | 空(空则 `/transcribe` 回 503) |
|
||||
| `ASR_MODEL` | `glm-asr` |
|
||||
|
||||
常驻内存约 35~40MB,CPU 基本为 0。没有本地模型可选:2 vCPU / 2GB 的机器跑不动能用的
|
||||
whisper(固定 30 秒窗口,`small` 单次要 15~30 秒),需要私有化就把这个容器放到另一台机器,
|
||||
只改 core 的 `ASR_URL` 指向它。
|
||||
|
||||
## HTTP 接口
|
||||
|
||||
```bash
|
||||
# 任意格式(webm / mp4 / mp3 / wav / ogg / aac / flac),服务端用 ffmpeg 归一化
|
||||
curl -s -X POST http://mcp_asr:3000/transcribe \
|
||||
-H 'Content-Type: audio/webm' \
|
||||
--data-binary @voice.webm
|
||||
# → {"text":"打开今天的日报","ms":842,"model":"glm-asr"}
|
||||
```
|
||||
|
||||
- 上限 20MB;超过 `ASR_MAX_CHUNK_SEC`(默认 25s)的音频自动切片后逐段识别再拼接
|
||||
(多数云端 ASR 单次只吃 ~30s)。
|
||||
- 同时最多 `ASR_MAX_CONCURRENT`(默认 2)个转写,超了回 429。
|
||||
|
||||
## MCP 工具
|
||||
|
||||
```yaml
|
||||
# config.yml
|
||||
mcp_servers:
|
||||
- name: asr
|
||||
url: "http://mcp_asr:3000/mcp"
|
||||
mode: skill # 低频能力,别每轮都发 schema
|
||||
```
|
||||
|
||||
`transcribe_audio({ url })`:本地文件先用内置工具 `mint_public_link` 换成公网短链,
|
||||
再把链接传进来(和 `video_task` 要 `image_url` 是同一套路)。
|
||||
|
||||
## 环境变量速查
|
||||
|
||||
| 变量 | 默认 | 说明 |
|
||||
|---|---|---|
|
||||
| `ASR_API_BASE` / `ASR_API_KEY` / `ASR_MODEL` | 见上 | 上游语音识别 |
|
||||
| `ASR_MAX_BYTES` | 20971520 | 单次上传上限 |
|
||||
| `ASR_MAX_CHUNK_SEC` | `25` | 超长音频切片长度,`0` = 不切 |
|
||||
| `ASR_MAX_CONCURRENT` | `2` | 并发转写数 |
|
||||
| `ASR_LOG_LEVEL` | `info` | debug / info / warn / error / none |
|
||||
|
||||
## 客户端怎么录音
|
||||
|
||||
浏览器侧只有 HTTPS 才允许开麦;**Android 套壳 App 用原生录音再把 base64 交给页面**,
|
||||
http 页面也能用。桥的契约与 Kotlin 示例见 [docs/voice-input.md](../../docs/voice-input.md)。
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"name": "mcp_asr",
|
||||
"version": "1.0.0",
|
||||
"description": "ASR service: POST /transcribe for the H5 composer, POST /mcp for the model (tool transcribe_audio). Zero dependencies.",
|
||||
"type": "module",
|
||||
"main": "src/server.js",
|
||||
"scripts": {
|
||||
"start": "node src/server.js"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,433 @@
|
||||
#!/usr/bin/env node
|
||||
// ASR(语音转文字)服务。一个进程两个入口,共用同一套转写实现:
|
||||
//
|
||||
// POST /transcribe 普通 HTTP —— 给 agent-core 的输入框用:录完音直接拿文字,
|
||||
// 不经过模型、不花 token、延迟 1~3s。
|
||||
// POST /mcp Streamable HTTP MCP —— 工具 transcribe_audio({url}),给模型转写
|
||||
// 会话目录里的音频(本地文件先 mint_public_link 换公网短链,与 video_task 同构)。
|
||||
//
|
||||
// 为什么不直接用 MCP 工具做语音输入:MCP 是模型协议,走一次工具调用要多一轮模型往返
|
||||
//(3~10s + token),识别结果还会落进聊天记录而不是输入框,用户没法改错字。输入法级别的
|
||||
// 功能必须绕开模型。
|
||||
//
|
||||
// 为什么零依赖(不用 express / MCP SDK):core 的 MCP 客户端只发 initialize / tools/list /
|
||||
// tools/call 三种请求,手写这几十行 JSON-RPC 就够;本机内存只有 2G,省掉 SDK 与 express
|
||||
// 的常驻开销(本服务空载约 35MB)。
|
||||
//
|
||||
// 转写走上游的 OpenAI 兼容 /audio/transcriptions(默认智谱 glm-asr,换平台改 env 即可)。
|
||||
// 音频一律先经 ffmpeg 归一化成 16kHz 单声道 wav:WebView 录出来是 webm/opus,iOS 是 mp4/aac,
|
||||
// 上游只认 wav/mp3,归一化后两端都自由。
|
||||
|
||||
import http from 'node:http';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import fs from 'node:fs/promises';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
|
||||
// ── 配置(全部走 env,不读 config.yml:本容器按仓库惯例只拿 env) ──────────────
|
||||
const PORT = Number(process.env.PORT || 3000);
|
||||
|
||||
// 上游:任何 OpenAI 兼容的 /audio/transcriptions(智谱 glm-asr、OpenAI whisper-1 等)
|
||||
const API_BASE = String(process.env.ASR_API_BASE || 'https://open.bigmodel.cn/api/paas/v4').trim().replace(/\/+$/, '');
|
||||
const API_KEY = String(process.env.ASR_API_KEY || '').trim();
|
||||
const API_MODEL = String(process.env.ASR_MODEL || 'glm-asr').trim();
|
||||
// 单次上传上限。60s 的 AAC 约 240KB、16k wav 约 1.9MB,20MB 是给长音频留的余量。
|
||||
const MAX_BYTES = Math.max(1024 * 1024, Number(process.env.ASR_MAX_BYTES || 20 * 1024 * 1024));
|
||||
// 云端多数 ASR 单次只吃 ~30s:超长音频按这个秒数切片后逐段识别再拼接(0 = 不切)。
|
||||
// 16k/16bit 单声道 wav 恒为 32000 B/s,所以用文件大小就能估时长,不用再起 ffprobe。
|
||||
const CHUNK_SEC = Math.max(0, Number(process.env.ASR_MAX_CHUNK_SEC || 25));
|
||||
// 同时转写数:2 vCPU 上并发再高只会互相拖慢。
|
||||
const MAX_CONCURRENT = Math.max(1, Number(process.env.ASR_MAX_CONCURRENT || 2));
|
||||
|
||||
const FFMPEG = String(process.env.FFMPEG_BIN || 'ffmpeg').trim();
|
||||
const LOG_LEVEL = String(process.env.ASR_LOG_LEVEL || 'info').toLowerCase();
|
||||
const LEVELS = { debug: 10, info: 20, warn: 30, error: 40, none: 99 };
|
||||
|
||||
function log(level, msg) {
|
||||
if ((LEVELS[level] || 20) < (LEVELS[LOG_LEVEL] == null ? 20 : LEVELS[LOG_LEVEL])) return;
|
||||
console.log('[asr ' + level + '] ' + msg);
|
||||
}
|
||||
|
||||
// ── 子进程:统一超时 + 收 stdout/stderr ────────────────────────────────────
|
||||
function run(cmd, args, timeoutMs) {
|
||||
return new Promise((resolve, reject) => {
|
||||
let child;
|
||||
try {
|
||||
child = spawn(cmd, args, { stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
} catch (e) {
|
||||
return reject(e);
|
||||
}
|
||||
let out = '';
|
||||
let err = '';
|
||||
let settled = false;
|
||||
const timer = setTimeout(() => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
try { child.kill('SIGKILL'); } catch (_) { /* 已退出 */ }
|
||||
reject(new Error(cmd + ' 超时(' + timeoutMs + 'ms)'));
|
||||
}, timeoutMs);
|
||||
timer.unref();
|
||||
child.stdout.on('data', (d) => { out += d; });
|
||||
child.stderr.on('data', (d) => { err += d; });
|
||||
child.on('error', (e) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
reject(e); // ENOENT:二进制没装(如没装 ffmpeg)
|
||||
});
|
||||
child.on('close', (code) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
if (code === 0) resolve({ stdout: out, stderr: err });
|
||||
else reject(new Error(cmd + ' 退出码 ' + code + ':' + ((err || out).slice(-400) || '无输出')));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// ── 音频:归一化成 16k 单声道 wav(再把超长音频切片) ───────────────────────
|
||||
async function toWav(buf, ext) {
|
||||
const id = randomUUID();
|
||||
const inPath = path.join(os.tmpdir(), 'asr-in-' + id + (ext ? '.' + ext : ''));
|
||||
const wavPath = path.join(os.tmpdir(), 'asr-out-' + id + '.wav');
|
||||
try {
|
||||
await fs.writeFile(inPath, buf);
|
||||
await run(FFMPEG, [
|
||||
'-hide_banner', '-loglevel', 'error', '-y',
|
||||
'-i', inPath,
|
||||
'-vn', '-ac', '1', '-ar', '16000', '-c:a', 'pcm_s16le', '-f', 'wav',
|
||||
wavPath,
|
||||
], 60000);
|
||||
return wavPath;
|
||||
} catch (e) {
|
||||
await fs.rm(inPath, { force: true }).catch(() => {});
|
||||
await fs.rm(wavPath, { force: true }).catch(() => {});
|
||||
// ffmpeg 的 stderr 很有用(定位是封装不对还是文件截断),但只进日志——
|
||||
// 回给前端的长英文堆栈对用户毫无意义,toast 里只会显示第一行。
|
||||
log('warn', '音频解码失败:' + String(e && e.message || e));
|
||||
const err = new Error('音频解码失败(格式不支持或文件损坏)');
|
||||
err.status = 400;
|
||||
throw err;
|
||||
} finally {
|
||||
await fs.rm(inPath, { force: true }).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
// 按秒切片:只在超过 CHUNK_SEC 时做。切片用 -c copy,纯 PCM 无损。
|
||||
async function splitWav(wavPath, chunkSec) {
|
||||
const dir = path.dirname(wavPath);
|
||||
const base = path.basename(wavPath, '.wav');
|
||||
const pattern = path.join(dir, base + '-part-%03d.wav');
|
||||
await run(FFMPEG, [
|
||||
'-hide_banner', '-loglevel', 'error', '-y',
|
||||
'-i', wavPath,
|
||||
'-f', 'segment', '-segment_time', String(chunkSec), '-c', 'copy',
|
||||
pattern,
|
||||
], 60000);
|
||||
const files = (await fs.readdir(dir))
|
||||
.filter((f) => f.startsWith(base + '-part-') && f.endsWith('.wav'))
|
||||
.sort();
|
||||
return files.map((f) => path.join(dir, f));
|
||||
}
|
||||
|
||||
// ── 引擎 ──────────────────────────────────────────────────────────────────
|
||||
function extractText(obj) {
|
||||
if (!obj) return '';
|
||||
if (typeof obj === 'string') return obj;
|
||||
if (typeof obj.text === 'string') return obj.text;
|
||||
if (obj.result && typeof obj.result.text === 'string') return obj.result.text;
|
||||
return '';
|
||||
}
|
||||
|
||||
async function runCloud(wavPath) {
|
||||
if (!API_KEY) {
|
||||
const e = new Error('语音识别未配置:容器环境变量 ASR_API_KEY 为空');
|
||||
e.status = 503;
|
||||
throw e;
|
||||
}
|
||||
const buf = await fs.readFile(wavPath);
|
||||
const form = new FormData();
|
||||
form.append('model', API_MODEL);
|
||||
form.append('file', new Blob([buf], { type: 'audio/wav' }), 'audio.wav');
|
||||
let res;
|
||||
try {
|
||||
res = await fetch(API_BASE + '/audio/transcriptions', {
|
||||
method: 'POST',
|
||||
headers: { Authorization: 'Bearer ' + API_KEY },
|
||||
body: form,
|
||||
signal: AbortSignal.timeout(60000),
|
||||
});
|
||||
} catch (e) {
|
||||
const err = new Error('调用上游语音识别失败:' + String(e && e.message || e));
|
||||
err.status = 502;
|
||||
throw err;
|
||||
}
|
||||
const raw = await res.text();
|
||||
if (!res.ok) {
|
||||
// 智谱对“解码后没有可识别语音”的音频回 400/1210 no audio segment found。
|
||||
// 这通常不是服务故障(录音是静音、或浏览器只录到空白),给一句人话,别把上游 JSON 丢给用户。
|
||||
let upstreamMsg = raw.slice(0, 300);
|
||||
let upstreamCode = '';
|
||||
try {
|
||||
const obj = JSON.parse(raw);
|
||||
if (obj && obj.error) {
|
||||
upstreamMsg = String(obj.error.message || upstreamMsg);
|
||||
upstreamCode = String(obj.error.code || '');
|
||||
}
|
||||
} catch (_) { /* 非 JSON,保留原文 */ }
|
||||
const noSpeech = /no audio segment/i.test(upstreamMsg) || upstreamCode === '1210';
|
||||
const err = new Error(noSpeech
|
||||
? '没有识别到语音内容(录到的是静音,或音频解码后为空)'
|
||||
: '上游识别失败 HTTP ' + res.status + ':' + upstreamMsg);
|
||||
err.status = noSpeech ? 400 : 502;
|
||||
err.upstream = upstreamMsg;
|
||||
throw err;
|
||||
}
|
||||
let obj = null;
|
||||
try { obj = JSON.parse(raw); } catch (_) { return raw.trim(); }
|
||||
const text = extractText(obj);
|
||||
if (!text) {
|
||||
const err = new Error('上游返回为空:' + raw.slice(0, 300));
|
||||
err.status = 502;
|
||||
throw err;
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
// 归一化后 wav 的实测音量(仅 debug 级别调用):用于区分“文件坏了”和“录到的是静音”。
|
||||
// volumedetect 跟解码共用一次遍历,开销很小,但只在排查时跑。
|
||||
async function wavLevel(wavPath) {
|
||||
const r = await run(FFMPEG, [
|
||||
'-hide_banner', '-nostats', '-i', wavPath, '-af', 'volumedetect', '-f', 'null', '-',
|
||||
], 30000);
|
||||
const mean = /mean_volume:\s*(-?[\d.]+) dB/.exec(r.stderr);
|
||||
const max = /max_volume:\s*(-?[\d.]+) dB/.exec(r.stderr);
|
||||
return (mean ? mean[1] : '?') + ' / ' + (max ? max[1] : '?') + ' dB';
|
||||
}
|
||||
|
||||
// 转写一段音频缓冲:归一化 →(超长则切片)→ 逐段识别 → 拼接。
|
||||
async function transcribeBuffer(buf, opts) {
|
||||
const wavPath = await toWav(buf, opts && opts.ext);
|
||||
const parts = [];
|
||||
try {
|
||||
const st = await fs.stat(wavPath);
|
||||
const seconds = st.size / 32000; // 16k/单声道/16bit
|
||||
// 每次请求都打一行:原始字节 → 归一化时长。排查“上游说没声音”时第一眼看这个。
|
||||
log('info', '归一化:' + buf.length + 'B (' + ((opts && opts.ext) || '?') + ') → '
|
||||
+ seconds.toFixed(1) + 's wav');
|
||||
if (CHUNK_SEC > 0 && seconds > CHUNK_SEC * 1.2) {
|
||||
parts.push(...await splitWav(wavPath, CHUNK_SEC));
|
||||
log('info', '超长音频切成 ' + parts.length + ' 段');
|
||||
}
|
||||
if (!parts.length) {
|
||||
try {
|
||||
return (await runCloud(wavPath)).trim();
|
||||
} catch (e) {
|
||||
// 只在上游判定“没有语音”时量一遍音量(正常路径零开销):
|
||||
// 静音是 -91dB,正常语音 -20dB 上下,一眼就能分清是设备没收到声还是文件坏了。
|
||||
if (e && e.status === 400) {
|
||||
log('warn', '音量(mean/max): ' + await wavLevel(wavPath).catch(() => '?'));
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
const out = [];
|
||||
for (const p of parts) {
|
||||
// 每段独立识别;单段失败不连累其他段(长录音里偶发静音段常见)
|
||||
try {
|
||||
out.push((await runCloud(p)).trim());
|
||||
} catch (e) {
|
||||
log('warn', '切片识别失败(跳过):' + String(e && e.message || e).slice(0, 200));
|
||||
}
|
||||
}
|
||||
return out.filter(Boolean).join('');
|
||||
} finally {
|
||||
await fs.rm(wavPath, { force: true }).catch(() => {});
|
||||
for (const p of parts) await fs.rm(p, { force: true }).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
// ── HTTP 小工具 ───────────────────────────────────────────────────────────
|
||||
function json(res, code, obj) {
|
||||
if (res.headersSent) return;
|
||||
const body = JSON.stringify(obj);
|
||||
res.writeHead(code, {
|
||||
'Content-Type': 'application/json; charset=utf-8',
|
||||
'Content-Length': Buffer.byteLength(body),
|
||||
'Cache-Control': 'no-store',
|
||||
});
|
||||
res.end(body);
|
||||
}
|
||||
|
||||
function readBody(req, limit) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const chunks = [];
|
||||
let n = 0;
|
||||
let failed = false;
|
||||
req.on('data', (c) => {
|
||||
if (failed) return;
|
||||
n += c.length;
|
||||
if (n > limit) {
|
||||
failed = true;
|
||||
const e = new Error('音频过大(上限 ' + Math.round(limit / 1048576) + 'MB)');
|
||||
e.status = 413;
|
||||
reject(e);
|
||||
return;
|
||||
}
|
||||
chunks.push(c);
|
||||
});
|
||||
req.on('end', () => { if (!failed) resolve(Buffer.concat(chunks)); });
|
||||
req.on('error', (e) => { if (!failed) reject(e); });
|
||||
});
|
||||
}
|
||||
|
||||
const EXT_BY_MIME = {
|
||||
'audio/webm': 'webm', 'audio/ogg': 'ogg', 'audio/mp4': 'm4a', 'audio/x-m4a': 'm4a',
|
||||
'audio/mpeg': 'mp3', 'audio/mp3': 'mp3', 'audio/wav': 'wav', 'audio/x-wav': 'wav',
|
||||
'audio/aac': 'aac', 'audio/flac': 'flac', 'audio/amr': 'amr',
|
||||
};
|
||||
|
||||
function extOf(contentType) {
|
||||
const k = String(contentType || '').split(';')[0].trim().toLowerCase();
|
||||
return EXT_BY_MIME[k] || '';
|
||||
}
|
||||
|
||||
// ── 入口 1:POST /transcribe(普通 HTTP,给输入框) ────────────────────────
|
||||
let active = 0;
|
||||
|
||||
async function handleTranscribe(req, res, url) {
|
||||
if (active >= MAX_CONCURRENT) {
|
||||
return json(res, 429, { error: '语音识别繁忙,请稍后再试' });
|
||||
}
|
||||
const buf = await readBody(req, MAX_BYTES);
|
||||
if (!buf.length) return json(res, 400, { error: '未收到音频数据' });
|
||||
|
||||
active++;
|
||||
const t0 = Date.now();
|
||||
try {
|
||||
const text = await transcribeBuffer(buf, { ext: extOf(req.headers['content-type']) });
|
||||
log('info', 'transcribed ' + buf.length + 'B → ' + text.length + ' 字,' + (Date.now() - t0) + 'ms');
|
||||
json(res, 200, { text, ms: Date.now() - t0, model: API_MODEL });
|
||||
} finally {
|
||||
active--;
|
||||
}
|
||||
}
|
||||
|
||||
// ── 入口 2:POST /mcp(Streamable HTTP,给模型) ──────────────────────────
|
||||
const TOOL = {
|
||||
name: 'transcribe_audio',
|
||||
description:
|
||||
'Transcribe an audio file to text. Provide a publicly reachable http(s) URL '
|
||||
+ '(mp3 / wav / m4a / webm / ogg / aac / flac, <= 20MB). '
|
||||
+ 'For a file inside the current task directory, first call the built-in mint_public_link tool '
|
||||
+ 'to get a temporary public URL, then pass that URL here.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
url: { type: 'string', description: 'Publicly reachable http(s) URL of the audio file.' },
|
||||
},
|
||||
required: ['url'],
|
||||
},
|
||||
};
|
||||
|
||||
async function fetchAudio(urlStr) {
|
||||
let u;
|
||||
try { u = new URL(urlStr); } catch (_) { throw new Error('url 不是合法链接'); }
|
||||
if (u.protocol !== 'http:' && u.protocol !== 'https:') throw new Error('只支持 http(s) 链接');
|
||||
const res = await fetch(u, { redirect: 'follow', signal: AbortSignal.timeout(30000) });
|
||||
if (!res.ok) throw new Error('下载音频失败 HTTP ' + res.status);
|
||||
const buf = Buffer.from(await res.arrayBuffer());
|
||||
if (!buf.length) throw new Error('下载到的音频为空');
|
||||
if (buf.length > MAX_BYTES) throw new Error('音频过大(上限 ' + Math.round(MAX_BYTES / 1048576) + 'MB)');
|
||||
return { buf, ext: extOf(res.headers.get('content-type')) };
|
||||
}
|
||||
|
||||
async function handleMcp(req, res) {
|
||||
const raw = await readBody(req, 1024 * 1024);
|
||||
let msg;
|
||||
try { msg = JSON.parse(raw.toString('utf8')); } catch (_) {
|
||||
return json(res, 400, { jsonrpc: '2.0', id: null, error: { code: -32700, message: 'Parse error' } });
|
||||
}
|
||||
const id = msg && msg.id !== undefined ? msg.id : null;
|
||||
const method = msg && msg.method;
|
||||
const params = (msg && msg.params) || {};
|
||||
const ok = (result) => json(res, 200, { jsonrpc: '2.0', id, result });
|
||||
const fail = (code, message) => json(res, 200, { jsonrpc: '2.0', id, error: { code, message } });
|
||||
|
||||
if (method === 'initialize') {
|
||||
return ok({
|
||||
protocolVersion: '2024-11-05',
|
||||
capabilities: { tools: {} },
|
||||
serverInfo: { name: 'asr', version: '1.0.0' },
|
||||
});
|
||||
}
|
||||
// 无状态实现:不返回 Mcp-Session-Id,core 也就不会带;通知直接回空结果(core 不校验)。
|
||||
if (method === 'notifications/initialized' || method === 'notifications/cancelled') return ok({});
|
||||
if (method === 'ping') return ok({});
|
||||
if (method === 'tools/list') return ok({ tools: [TOOL] });
|
||||
|
||||
if (method === 'tools/call') {
|
||||
if (params.name !== TOOL.name) return fail(-32602, 'Unknown tool: ' + params.name);
|
||||
const args = params.arguments || {};
|
||||
const urlStr = String(args.url || '').trim();
|
||||
if (!urlStr) return ok({ isError: true, content: [{ type: 'text', text: '缺少 url 参数' }] });
|
||||
try {
|
||||
const { buf, ext } = await fetchAudio(urlStr);
|
||||
const text = await transcribeBuffer(buf, { ext });
|
||||
return ok({ content: [{ type: 'text', text }] });
|
||||
} catch (e) {
|
||||
return ok({ isError: true, content: [{ type: 'text', text: '转写失败:' + String(e && e.message || e) }] });
|
||||
}
|
||||
}
|
||||
|
||||
return fail(-32601, 'Method not found: ' + method);
|
||||
}
|
||||
|
||||
// ── 服务 ──────────────────────────────────────────────────────────────────
|
||||
const server = http.createServer((req, res) => {
|
||||
let url;
|
||||
try { url = new URL(req.url, 'http://localhost'); } catch (_) {
|
||||
return json(res, 400, { error: 'bad url' });
|
||||
}
|
||||
const done = (p) => {
|
||||
Promise.resolve(p).catch((e) => {
|
||||
const code = (e && e.status) || 500;
|
||||
// 4xx 是可预期的输入问题(坏音频、没配 key),只记一行;5xx 才带堆栈
|
||||
log(code >= 500 ? 'error' : 'warn', String(e && e.message || e) + (code >= 500 && e && e.stack ? '\n' + e.stack : ''));
|
||||
if (!res.headersSent) json(res, code, { error: String(e && e.message || e) });
|
||||
else try { res.end(); } catch (_) { /* 已断 */ }
|
||||
});
|
||||
};
|
||||
|
||||
if (req.method === 'GET' && url.pathname === '/health') {
|
||||
return json(res, 200, {
|
||||
ok: true,
|
||||
model: API_MODEL,
|
||||
configured: !!API_KEY,
|
||||
});
|
||||
}
|
||||
if (req.method === 'POST' && url.pathname === '/transcribe') return done(handleTranscribe(req, res, url));
|
||||
if (url.pathname === '/mcp') {
|
||||
if (req.method !== 'POST') return json(res, 405, { error: 'only POST /mcp' });
|
||||
return done(handleMcp(req, res));
|
||||
}
|
||||
return json(res, 404, { error: 'not found' });
|
||||
});
|
||||
|
||||
let shuttingDown = false;
|
||||
function shutdown(sig) {
|
||||
if (shuttingDown) return;
|
||||
shuttingDown = true;
|
||||
log('info', '收到 ' + sig + ',退出');
|
||||
server.close(() => process.exit(0));
|
||||
setTimeout(() => process.exit(0), 3000).unref();
|
||||
}
|
||||
process.on('SIGTERM', () => shutdown('SIGTERM'));
|
||||
process.on('SIGINT', () => shutdown('SIGINT'));
|
||||
|
||||
server.listen(PORT, '0.0.0.0', () => {
|
||||
log('info', 'listening on ' + PORT + ',model=' + API_MODEL + ' @ ' + API_BASE);
|
||||
if (!API_KEY) log('warn', 'ASR_API_KEY 为空:/transcribe 会返回 503');
|
||||
});
|
||||
+197
-96
@@ -1,12 +1,18 @@
|
||||
# MCP Image Task
|
||||
|
||||
一个基于 [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) 的图片生成服务,对接智谱 BigModel 的 `CogView` 系列模型。每次调用通过 HTTP header 传入智谱 API 的 Bearer Token。只有一个工具 `image_task`,用 `op` 区分动作(`create` / `get`),对外提供与 `video_task` 一致的**异步任务**契约;上游为同步接口,异步语义由服务自维护的任务表模拟(见「架构与文件」)。
|
||||
一个基于 [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) 的图片生成 / 编辑服务。
|
||||
对外只有一个工具 `image_task`,用 `op` 区分动作(`create` / `get`),异步任务契约与 `video_task` 一致。
|
||||
|
||||
上游支持两种协议,由 `X-Image-Task-Base-Url` 选择:
|
||||
|
||||
| 通道 | 上游 | 请求体 |
|
||||
|---|---|---|
|
||||
| 智谱原生 | `https://open.bigmodel.cn/api/paas/v4/images/generations` | `{ model, prompt }` |
|
||||
| OpenAI 兼容 | 任意实现 `/images/generations` 的网关(如 `https://wb.ai.foresh.com/v1`) | `{ model, prompt, n: 1, response_format: "url", images? }` |
|
||||
|
||||
- 协议:MCP **Streamable HTTP**(单端点 `/mcp`,支持 SSE 流)
|
||||
- 上游:[智谱 BigModel Images Generations API](https://open.bigmodel.cn/dev/api#images)
|
||||
- 默认模型:`CogView-3-Flash`
|
||||
- 端口:`3000`
|
||||
- 鉴权:**无服务端密钥**,每次请求必须从 header 携带智谱 token
|
||||
- 端口:容器内 `3000`(**不映射宿主机**,只在 `agent-mcp-net` 内被 agent 访问)
|
||||
- 鉴权:**无服务端密钥**,每次请求从 `Authorization` header 读取上游 key
|
||||
|
||||
---
|
||||
|
||||
@@ -14,16 +20,13 @@
|
||||
|
||||
### 启动(docker-compose)
|
||||
|
||||
仓库根目录 `docker-compose.yml` 已包含本服务:
|
||||
|
||||
```bash
|
||||
cd /root/workspace/tools/agent
|
||||
docker compose up -d --build mcp_image_task
|
||||
```
|
||||
|
||||
服务容器名 `mcp_image_task`,容器内端口 3000,**默认不占宿主端口**:agent 走 `agent-mcp-net`
|
||||
按容器名访问(`http://mcp_image_task:3000/mcp`)。需要从宿主机 curl 调试时,去仓库根
|
||||
`docker-compose.yml` 把 `mcp_image_task` 注释掉的 `ports` 两行打开(默认写的是 8082),调试完记得注释回去。
|
||||
服务容器名 `mcp_image_task`。要从宿主机 curl 调试,把根 `docker-compose.yml` 里
|
||||
`mcp_image_task` 注释掉的 `ports` 两行打开(默认 8082),调试完记得注释回去。
|
||||
|
||||
```bash
|
||||
# 不占端口的健康检查
|
||||
@@ -33,67 +36,55 @@ docker exec mcp_image_task node -e "fetch('http://localhost:3000/health').then(r
|
||||
|
||||
### MCP 客户端配置
|
||||
|
||||
#### Claude Desktop / Cursor 等支持 `headers` 配置的客户端
|
||||
|
||||
```jsonc
|
||||
// Claude Desktop: ~/Library/Application Support/Claude/claude_desktop_config.json
|
||||
// Cursor: ~/.cursor/mcp.json
|
||||
{
|
||||
"mcpServers": {
|
||||
"image_task": {
|
||||
"url": "http://localhost:8082/mcp",
|
||||
"headers": {
|
||||
"Authorization": "Bearer <your-zhipu-api-key>"
|
||||
"Authorization": "Bearer <upstream-api-key>",
|
||||
"X-Image-Task-Base-Url": "zhipu",
|
||||
"X-Image-Task-Model": "CogView-4"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> 上面 `url` 里的端口 `8082` 需要先把 compose 里注释的 `ports` 打开(默认不占宿主端口)。
|
||||
|
||||
> **重要**:客户端必须在 **每一个 HTTP 请求**(`initialize` / `notifications/initialized` / `tools/call`)都带上 `Authorization` header。配置在 `headers` 里、由客户端自动重发即可。直连模式下 token 不在 session 间共享。
|
||||
|
||||
#### 不支持 `headers` 字段的客户端
|
||||
|
||||
如果客户端只支持 `command`/`args` 启动方式,可改用 stdio 模式(需自行包一层把 token 注入环境变量,再让工具回调从环境变量读——当前实现 HTTP 模式下不依赖环境变量,需自行扩展)。
|
||||
> 客户端必须在**每一个 HTTP 请求**(`initialize` / `notifications/initialized` / `tools/call`)都带上
|
||||
> `Authorization`。`X-Image-Task-Base-Url` / `X-Image-Task-Model` 在 `tools/call` 时读取。
|
||||
|
||||
---
|
||||
|
||||
## 工具:image_task
|
||||
|
||||
管理图片生成异步任务。典型流程:`op=create` 拿 `task_id`,然后不断 `op=get` 直到 `task_status` 为 `SUCCESS` 或 `FAIL`。
|
||||
典型流程:`op=create` 拿 `task_id`,然后不断 `op=get` 直到 `task_status` 为 `SUCCESS` 或 `FAIL`。
|
||||
|
||||
| op | 作用 | 说明 |
|
||||
|---|---|---|
|
||||
| `create` | 提交图片生成任务,**立即**返回 `task_id`,生成在后台执行 | 需要 `prompt` |
|
||||
| `create` | 提交图片生成 / 编辑任务,**立即**返回 `task_id`,生成在后台执行 | 需要 `prompt` |
|
||||
| `get` | 查询任务状态;`SUCCESS` 时返回图片 URL | 需要 `task_id` |
|
||||
|
||||
> 上游智谱 images 接口是同步阻塞的;这里的异步是对外的统一形态(对齐 `video_task`),服务内部把同步调用丢到后台跑并回写任务表。
|
||||
> 上游是同步阻塞的;这里的异步是对外统一形态(对齐 `video_task`),服务内部把同步调用丢到后台跑并回写任务表。
|
||||
|
||||
### 参数(AI 可见)
|
||||
|
||||
| 名称 | 类型 | 必填 | 说明 |
|
||||
|---|---|---|---|
|
||||
| `op` | `"create"` \| `"get"` | ✅ | 要执行的操作 |
|
||||
| `prompt` | string | op=create 时 ✅ | 图片生成提示词,描述想要的画面内容 |
|
||||
| `prompt` | string | op=create 时 ✅ | 提示词;文生图描述画面,图生图描述要改成什么样 |
|
||||
| `task_id` | string | op=get 时 ✅ | `op=create` 返回的任务 ID |
|
||||
| `image` | string | ❌ | 一张图的公网 `http(s)` URL,用于图生图 / 修图 |
|
||||
|
||||
> 图片尺寸不再透传,走上游默认;尺寸档位与成本/成功率挂钩,AI 传错值经常直接报错。
|
||||
`image` 是模式开关:
|
||||
|
||||
### Model 控制(不在工具参数中暴露)
|
||||
- 不传 → **文生图**;
|
||||
- 传一个公网 URL → **图生图 / 编辑**。
|
||||
|
||||
模型名由调用方在 **tools/call 这一次的 HTTP header** 中传递,**AI 看不到也填不了**,避免其误传。识别顺序(首个非空值生效):
|
||||
本地图片先用 `mint_public_link` 换成公网 URL,再传给 `image`。
|
||||
|
||||
| 优先级 | Header |
|
||||
|---|---|
|
||||
| 1 | `X-Image-Task-Model` |
|
||||
| 2 | `X-Mcp-Model` |
|
||||
| 3 | `X-Model` |
|
||||
|
||||
如果都没传,走代码内常量 `DEFAULT_MODEL = "CogView-3-Flash"`。
|
||||
|
||||
**为什么不放进工具参数?** 调用方一般是 AI 助手(Claude / Cursor 等),它不需要也没能力选择用哪个模型;模型选择属于部署/路由策略,由请求的发起方在 HTTP header 里注入更可控。
|
||||
> 尺寸、质量、模型、上游地址都**不是**工具参数:它们属于部署策略,由调用方在 header 里固定,
|
||||
> 避免 AI 乱填尺寸档位或模型名白烧额度。智谱原生通道也不接受尺寸,直接走上游默认值。
|
||||
|
||||
### 返回
|
||||
|
||||
@@ -102,62 +93,150 @@ docker exec mcp_image_task node -e "fetch('http://localhost:3000/health').then(r
|
||||
```jsonc
|
||||
{
|
||||
"content": [
|
||||
{ "type": "text", "text": "Model: CogView-3-Flash\ntask_id: <uuid>\ntask_status: PROCESSING\n\nImage generation is running in background. Call this tool again with op=\"get\" and task_id=\"<uuid>\" to fetch the result." }
|
||||
{ "type": "text", "text": "Model: CogView-4\nProvider: zhipu\nMode: text-to-image\ntask_id: <uuid>\ntask_status: PROCESSING\n\nImage generation is running in background. Call this tool again with op=\"get\" and task_id=\"<uuid>\" to fetch the result." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`op=get` 按任务状态返回:
|
||||
|
||||
- `PROCESSING`:提示几秒后再查;
|
||||
- `SUCCESS`:单个 `text` 块,摘要后跟图片 URL(单图直接给 `Image URL:`,多图按 `Image N:` 逐行列出):
|
||||
- `PROCESSING`:带上已等待秒数 `elapsed`,建议等待 `min(30, 3 + floor(elapsed / 5))` 秒后再查;
|
||||
- `SUCCESS`:单个 `text` 块,摘要后跟图片 URL(单图直接给 `Image URL:`,多图按 `Image N:` 逐行列出);`elapsed` 为从 `create` 到完成的总耗时:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"content": [
|
||||
{ "type": "text", "text": "task_id: <uuid>\nmodel: CogView-3-Flash\ntask_status: SUCCESS\nImage URL: https://maas-watermark-prod-new.cn-wlcb.ufileos.com/..." }
|
||||
{ "type": "text", "text": "task_id: <uuid>\nmodel: CogView-4\nprovider: zhipu\nelapsed: 12s\ntask_status: SUCCESS\nImage URL: https://..." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `FAIL`:`isError: true`,正文含 `task_status: FAIL` 与错误信息(上游 4xx/5xx、无 URL 返回等都会归档到这里)。
|
||||
- `FAIL`:`isError: true`,正文含 `task_status: FAIL`、总耗时 `elapsed` 与错误信息(上游 4xx/5xx、无 URL 返回等都会归档到这里)。
|
||||
|
||||
URL 由智谱签名下发,**带有效期**(通常 24h),如需长期保存请在收到后及时转存到自己的存储。
|
||||
URL 由上游签名下发,**带有效期**(通常 24h),需要长期保存请及时转存。
|
||||
|
||||
---
|
||||
|
||||
## 鉴权细节
|
||||
## Headers(只有三个)
|
||||
|
||||
每个 HTTP 请求必须携带智谱 API Token,server 从以下 header 字段按顺序识别,**首个非空值生效**:
|
||||
|
||||
| 优先级 | Header | 说明 |
|
||||
| Header | 必填 | 说明 |
|
||||
|---|---|---|
|
||||
| 1 | `X-Image-Task-Token: <token>` | 直传,避免与客户端自身的 Authorization 冲突 |
|
||||
| 2 | `X-Mcp-Token: <token>` | 通用兜底 |
|
||||
| 3 | `Authorization: Bearer <token>` | 标准 OAuth 风格(排最后是为避免误用) |
|
||||
| `Authorization` | ✅ | `Bearer <key>`,原样作为上游 Bearer token |
|
||||
| `X-Image-Task-Base-Url` | ❌ | 别名或完整 URL;缺省 `zhipu` |
|
||||
| `X-Image-Task-Model` | 见下 | 模型名原样透传 |
|
||||
|
||||
Token 仅在每次 tool call 时从 header 读取,不写入任何存储、不进 session。
|
||||
### Base-Url 别名
|
||||
|
||||
### 错误响应
|
||||
| 值 | 实际地址 | 协议 |
|
||||
|---|---|---|
|
||||
| `zhipu`(默认) | `https://open.bigmodel.cn/api/paas/v4` | 智谱原生 |
|
||||
|
||||
其余值必须是完整的 `http(s)` URL:
|
||||
|
||||
```yaml
|
||||
X-Image-Task-Base-Url: "https://wb.ai.foresh.com/v1"
|
||||
```
|
||||
|
||||
- 已经给到 `/images/generations` 全路径就直接用,否则自动补 `/images/generations`。
|
||||
- 解析后的 host 是 `bigmodel.cn`(含子域)→ 智谱原生请求体;其余 host → OpenAI 兼容请求体。
|
||||
- 因此把网关地址填进来即可,不需要额外指定“用哪家协议”。
|
||||
|
||||
### 模型
|
||||
|
||||
| 场景 | `X-Image-Task-Model` |
|
||||
|---|---|
|
||||
| Base-Url = `zhipu` | 可省略,缺省 `CogView-3-Flash` |
|
||||
| 其他 Base-Url | **必填**(不会把 CogView 误发到网关) |
|
||||
|
||||
模型只由 header 决定,AI 看不到也填不了。换模型 = 改 header 后重启 core。
|
||||
|
||||
### 图生图
|
||||
|
||||
`image` 只对 OpenAI 兼容通道生效:
|
||||
|
||||
| Base-Url | 传 `image` |
|
||||
|---|---|
|
||||
| `zhipu` | 直接返回错误:智谱不支持图生图(只做文生图) |
|
||||
| 网关 URL | 上游请求体带 `images: [url]` |
|
||||
|
||||
### 请求体差异
|
||||
|
||||
| 字段 | 智谱原生 | OpenAI 兼容 |
|
||||
|---|---|---|
|
||||
| `model` / `prompt` | ✅ | ✅ |
|
||||
| `n` | ❌ | 固定 `1` |
|
||||
| `response_format` | ❌ | 固定 `"url"` |
|
||||
| `size` / `quality` | ❌ | ❌(交给上游默认值) |
|
||||
| `images` | ❌ | 传图时带,数组形式 `[url]` |
|
||||
|
||||
---
|
||||
|
||||
## 配置示例
|
||||
|
||||
### 智谱直连(默认)
|
||||
|
||||
```yaml
|
||||
- name: image_task
|
||||
url: "http://mcp_image_task:3000/mcp"
|
||||
headers:
|
||||
Authorization: "Bearer <智谱 KEY>"
|
||||
X-Image-Task-Model: "CogView-4"
|
||||
mode: skill
|
||||
```
|
||||
|
||||
### OpenAI 兼容网关(文生图)
|
||||
|
||||
```yaml
|
||||
- name: image_task
|
||||
url: "http://mcp_image_task:3000/mcp"
|
||||
headers:
|
||||
Authorization: "Bearer <网关 KEY>"
|
||||
X-Image-Task-Base-Url: "https://wb.ai.foresh.com/v1"
|
||||
X-Image-Task-Model: "global:gpt-image-2.5-sunburst"
|
||||
mode: skill
|
||||
```
|
||||
|
||||
### OpenAI 兼容网关(图生图 / 修图)
|
||||
|
||||
```yaml
|
||||
- name: image_task
|
||||
url: "http://mcp_image_task:3000/mcp"
|
||||
headers:
|
||||
Authorization: "Bearer <网关 KEY>"
|
||||
X-Image-Task-Base-Url: "https://wb.ai.foresh.com/v1"
|
||||
X-Image-Task-Model: "cn:hunyuan-image-alpha"
|
||||
mode: skill
|
||||
```
|
||||
|
||||
同一个网关换模型只改 `X-Image-Task-Model`;换供应商只改 `X-Image-Task-Base-Url`。
|
||||
|
||||
---
|
||||
|
||||
## 错误响应
|
||||
|
||||
| 场景 | HTTP 状态 | 表现 |
|
||||
|---|---|---|
|
||||
| initialize 时缺 token | `401` | JSON-RPC `error.message` 提示补充 header |
|
||||
| op=create 缺 prompt / op=get 缺 task_id | `200`(MCP 工具返回) | `isError: true`,参数提示 |
|
||||
| op=get 查不到任务(ID 错/已过期/服务重启过) | `200`(MCP 工具返回) | `isError: true`,提示 unknown task_id |
|
||||
| 智谱 API 4xx/5xx | `200`(MCP 工具返回) | 在 `op=get` 中以 `isError: true` + `task_status: FAIL` 呈现,正文含智谱返回的状态码和 body |
|
||||
| 网络失败 | `200`(MCP 工具返回) | 同上,`task_status: FAIL`,正文含 `fetch` 错误信息 |
|
||||
| 上游挂住不回(超时 >120s) | `200`(MCP 工具返回) | 同上,`task_status: FAIL`,正文提示 timed out;不会出现永久 PROCESSING 的僵尸任务 |
|
||||
| 缺 `Authorization` | `200`(MCP 工具返回) | `isError: true`,提示补 header |
|
||||
| `X-Image-Task-Base-Url` 非法 | `200`(MCP 工具返回) | `isError: true`,列出可用别名 |
|
||||
| 非 zhipu 通道缺 `X-Image-Task-Model` | `200`(MCP 工具返回) | `isError: true`,提示补 header |
|
||||
| zhipu 通道传了 `image` | `200`(MCP 工具返回) | `isError: true`,提示智谱不支持图生图,改用 OpenAI 兼容端点 |
|
||||
| `image` 不是公网 http(s) URL | `200`(MCP 工具返回) | `isError: true`,提示只接受公网 URL,不接受文件上传 / base64 |
|
||||
| `op=create` 缺 prompt / `op=get` 缺 task_id | `200`(MCP 工具返回) | `isError: true`,参数提示 |
|
||||
| `op=get` 查不到任务 | `200`(MCP 工具返回) | `isError: true`,提示 unknown task_id |
|
||||
| 上游 4xx/5xx | `200`(MCP 工具返回) | 在 `op=get` 中以 `task_status: FAIL` 呈现,正文含上游状态码和 body |
|
||||
| 上游只回 `b64_json` | `200`(MCP 工具返回) | `task_status: FAIL`,提示本服务只接受 URL |
|
||||
| 网络失败 / 超时(>180s) | `200`(MCP 工具返回) | `task_status: FAIL`,不会出现永久 PROCESSING 的僵尸任务 |
|
||||
|
||||
> JSON 请求体上限保持 2MB:`image` 只接受 URL,不接收文件或 base64。
|
||||
|
||||
---
|
||||
|
||||
## 用 curl 手工调试
|
||||
|
||||
MCP Streamable HTTP 不是普通 REST,**必须按顺序**:`initialize` → `notifications/initialized` → `tools/call`。下面用一行版串起来:
|
||||
MCP Streamable HTTP **必须按顺序**:`initialize` → `notifications/initialized` → `tools/call`。
|
||||
|
||||
```bash
|
||||
# 把 <your-zhipu-api-key> 替换成你智谱的 API Key
|
||||
TOKEN="<your-zhipu-api-key>"
|
||||
TOKEN="<upstream-api-key>"
|
||||
|
||||
# 1. initialize 拿 sessionId
|
||||
SESSION=$(curl -s -D - -X POST http://localhost:8082/mcp \
|
||||
@@ -185,19 +264,12 @@ curl -s -X POST http://localhost:8082/mcp \
|
||||
-H "Mcp-Session-Id: $SESSION" \
|
||||
-d '{"jsonrpc": "2.0", "method": "notifications/initialized"}'
|
||||
|
||||
# 3. 列出工具(可选验证)
|
||||
curl -s -X POST http://localhost:8082/mcp \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Accept: application/json, text/event-stream" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Mcp-Session-Id: $SESSION" \
|
||||
-d '{"jsonrpc": "2.0", "id": 2, "method": "tools/list"}'
|
||||
|
||||
# 4. 调 image_task(op=create,立即返回 task_id)
|
||||
# 3. 文生图(智谱通道;换成网关时补 X-Image-Task-Base-Url / X-Image-Task-Model)
|
||||
curl -s -X POST http://localhost:8082/mcp \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Accept: application/json, text/event-stream" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "X-Image-Task-Model: CogView-4" \
|
||||
-H "Mcp-Session-Id: $SESSION" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
@@ -212,7 +284,29 @@ curl -s -X POST http://localhost:8082/mcp \
|
||||
}
|
||||
}'
|
||||
|
||||
# 5. 拿上一步返回的 task_id 轮询结果(直到 SUCCESS / FAIL)
|
||||
# 4. 图生图(OpenAI 兼容通道)
|
||||
curl -s -X POST http://localhost:8082/mcp \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Accept: application/json, text/event-stream" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "X-Image-Task-Base-Url: https://wb.ai.foresh.com/v1" \
|
||||
-H "X-Image-Task-Model: global:gpt-image-2.5-sunburst" \
|
||||
-H "Mcp-Session-Id: $SESSION" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 4,
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "image_task",
|
||||
"arguments": {
|
||||
"op": "create",
|
||||
"prompt": "把背景改成雪夜,其他保持不变",
|
||||
"image": "https://example.com/input.png"
|
||||
}
|
||||
}
|
||||
}'
|
||||
|
||||
# 5. 轮询结果
|
||||
curl -s -X POST http://localhost:8082/mcp \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Accept: application/json, text/event-stream" \
|
||||
@@ -220,7 +314,7 @@ curl -s -X POST http://localhost:8082/mcp \
|
||||
-H "Mcp-Session-Id: $SESSION" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 4,
|
||||
"id": 5,
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "image_task",
|
||||
@@ -250,22 +344,23 @@ image_task/
|
||||
|
||||
### 关键设计
|
||||
|
||||
- **单工具 + `op` 分发**:`image_task(op=create|get)`,与 `video_task` 的服务形态完全一致,agent 侧一套轮询逻辑通吃图片/视频
|
||||
- **同步上游 → 异步契约**:智谱 images 接口只支持同步,所以服务自维护一张任务表来模拟异步:
|
||||
- **单工具 + `op` 分发**:`image_task(op=create|get)`,与 `video_task` 的服务形态一致,agent 侧一套轮询逻辑通吃图片/视频
|
||||
- **同步上游 → 异步契约**:上游同步阻塞,所以服务自维护一张任务表来模拟异步:
|
||||
- `create` 本地生成 `task_id`(`randomUUID`),登记 `PROCESSING`,把同步调用丢后台(fire-and-forget),立即返回
|
||||
- 后台完成时回写 `SUCCESS`(含 URLs)或 `FAIL`(含错误);上游同步调用带 120s 超时(`AbortSignal.timeout`),超时记 `FAIL`,杜绝僵尸 `PROCESSING`
|
||||
- 后台完成时回写 `SUCCESS`(含 URLs)或 `FAIL`(含错误);上游同步调用带 180s 超时(`AbortSignal.timeout`),超时记 `FAIL`,杜绝僵尸 `PROCESSING`
|
||||
- `get` 只读本地表,不碰上游
|
||||
- 表为进程内存 Map:已完成任务保留 1 小时(TTL)、容量上限 1000,每次 `create` 顺带淘汰;**容器重启后任务丢失**,`get` 会报 unknown task_id;**不适用于多副本部署**(表不互通)
|
||||
- token/model 在 `create` 那一刻从请求 header 捕获进闭包,后台完成前不再读取,保持「每请求读 header」的无状态设计
|
||||
- **每 session 一个 transport + server**:通过 `Mcp-Session-Id` 复用,避免每请求重建
|
||||
- **模型参数不入 schema**:model 只从 HTTP header 读取(`X-Image-Task-Model` / `X-Mcp-Model` / `X-Model`),AI 不可见
|
||||
- **URL 提取**:`response.data[].url` 为准,若返回 `b64_json` 也会一并归档进任务表
|
||||
- 表为进程内存 Map:已完成任务保留 1 小时(TTL)、容量上限 1000,每次 `create` 顺带淘汰;**容器重启后任务丢失**,`get` 会报 unknown task_id;**不适用于多副本部署**
|
||||
- token / 上游地址 / 模型在 `create` 那一刻从请求 header 捕获进闭包,后台完成前不再读取
|
||||
- **每 session 一个 transport + server**:通过 `Mcp-Session-Id` 复用
|
||||
- **模型 / 上游地址不入 schema**:只从 HTTP header 读取,AI 不可见
|
||||
- **`image` 单一开关**:不传 = 文生图,传一个公网 URL = 图生图;`image` 是单个字符串,只接受公网 `http(s)` URL
|
||||
- **URL 提取**:只接受 `data[].url` 形式的 http(s) URL;上游若回 `b64_json` 则记 `FAIL` 并给出明确错误
|
||||
|
||||
### 默认模型
|
||||
|
||||
```js
|
||||
// src/mcp-server.js
|
||||
const DEFAULT_MODEL = "CogView-3-Flash";
|
||||
const DEFAULT_MODEL = "CogView-3-Flash"; // 仅 Base-Url=zhipu 时生效
|
||||
```
|
||||
|
||||
修改后 `docker compose up -d --build mcp_image_task` 即可生效。
|
||||
@@ -275,31 +370,37 @@ const DEFAULT_MODEL = "CogView-3-Flash";
|
||||
## 常见问题
|
||||
|
||||
**Q: AI 不知道有哪些 model 可选,怎么办?**
|
||||
A: 这是设计意图。模型选择属于路由策略,不让 AI 决策。调用方在 HTTP header `X-Image-Task-Model` 里注入即可。
|
||||
A: 这是设计意图。模型选择属于部署 / 路由策略,由调用方在 `X-Image-Task-Model` 里固定,不让 AI 决策。
|
||||
|
||||
**Q: 换模型需要重连 MCP session 吗?**
|
||||
A: 不需要。Model header 在每次 `tools/call` 都从当前请求读取,不需要重建 session。
|
||||
**Q: 换模型 / 换供应商要重连 MCP session 吗?**
|
||||
A: 不需要重建 session,但 core 的 `mcp_servers[].headers` 是启动时读取的,改完要 `docker compose restart agent-core`。
|
||||
|
||||
**Q: 为什么只支持一张参考图?**
|
||||
A: 当前上游契约只稳定支持单张;`image` 因此是单个 URL 字符串。
|
||||
|
||||
**Q: 图生图为什么在 zhipu 通道报错?**
|
||||
A: 智谱原生接口不支持图生图。把 `X-Image-Task-Base-Url` 指向支持 `images` 字段的 OpenAI 兼容网关即可。
|
||||
|
||||
**Q: 本地图片怎么传?**
|
||||
A: 先调用 agent 内置的 `mint_public_link` 工具,把会话目录里的文件换成短期公网 URL,再传给 `image`。本服务不接受文件上传或 base64,上游服务器也无法访问你的本地路径。
|
||||
|
||||
**Q: 上游是同步接口,为什么对外做成异步?**
|
||||
A: 为了和 `video_task` 保持同一套任务型契约,agent 侧只需维护一种「create 后轮询 get」的调用模式。同步调用被放到后台执行,对 AI 而言表现为标准异步任务。
|
||||
A: 和 `video_task` 保持同一套任务型契约;同时避免一次生图把 core 的 `tools/call` 拖超时。
|
||||
|
||||
**Q: 任务一直 PROCESSING 会不会没完没了?**
|
||||
A: 不会。上游同步调用封顶 120s(`AbortSignal.timeout`),超时直接记 `FAIL`,`get` 会拿到明确的 timed out 错误;不存在僵尸任务。
|
||||
A: 不会。上游调用封顶 180s,超时直接记 `FAIL`。
|
||||
|
||||
**Q: 为什么 PROCESSING 时建议的等待时间会变长?**
|
||||
A: 轮询间隔按 `min(30, 3 + floor(elapsed / 5))` 退避:前期密查尽快拿到结果,后期拉长间隔减少无意义轮询,封顶 30s。
|
||||
|
||||
**Q: create 之后服务容器重启了,get 会怎样?**
|
||||
A: 任务表在进程内存里,重启即丢失,`get` 会返回 unknown task_id 的 `isError`。需要重新 `create`。图片生成只跑几秒,受影响窗口很小;若将来要上多副本或需要可靠恢复,再把表换成持久化存储。
|
||||
|
||||
**Q: 为什么 get 不需要等上游?**
|
||||
A: `get` 只读本地任务表,上游调用只发生在 `create` 的后台一次,不会因轮询而重复计费/重复生图。
|
||||
|
||||
**Q: 为什么 Model 优先级不是「工具参数 > header」?**
|
||||
A: 因为我们故意不在 schema 暴露 model 参数,AI 根本不会传 model,所以「工具参数 > header」这条分支实际上不存在。
|
||||
A: 任务表在进程内存里,重启即丢失,`get` 会返回 unknown task_id。需要重新 `create`。
|
||||
|
||||
**Q: Token 会不会泄露到日志?**
|
||||
A: 当前实现只在 session 内存里持有,不会写入 stdout/stderr。Container 日志只打印 session id。
|
||||
A: 不会。只从 header 读取,不写入日志。
|
||||
|
||||
**Q: 收到 `Missing valid Mcp-Session-Id` 错误?**
|
||||
A: 漏发了 `Mcp-Session-Id` header,或调 `tools/call` 前没先 `initialize`。
|
||||
|
||||
**Q: URL 多久过期?**
|
||||
A: 智谱签发 URL 默认 24h 左右,详见响应里的 `Expires` 参数。需要长期使用请自行转存到 OSS/S3。
|
||||
A: 上游签发的 URL 通常 24h 左右。需要长期使用请自行转存。
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "mcp_image_task",
|
||||
"version": "1.0.0",
|
||||
"description": "MCP server for image generation via Zhipu BigModel CogView (single tool image_task with op=create)",
|
||||
"description": "MCP server for image generation and editing via Zhipu BigModel CogView or any OpenAI-compatible /images/generations endpoint (single tool image_task with op=create/get)",
|
||||
"type": "module",
|
||||
"main": "src/mcp-server.js",
|
||||
"scripts": {
|
||||
|
||||
@@ -6,8 +6,14 @@ import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/
|
||||
import { isInitializeRequest } from "@modelcontextprotocol/sdk/types.js";
|
||||
import { z } from "zod";
|
||||
|
||||
const ZHIPU_API_URL =
|
||||
"https://open.bigmodel.cn/api/paas/v4/images/generations";
|
||||
// 上游地址既支持完整 URL,也支持别名。zhipu 走智谱原生协议(body 只有
|
||||
// model/prompt);其余地址一律按 OpenAI 兼容的 /images/generations 协议
|
||||
// (body 加 n / response_format=url)。别名目前只保留 zhipu。
|
||||
const BASE_URL_ALIASES = {
|
||||
zhipu: "https://open.bigmodel.cn/api/paas/v4",
|
||||
};
|
||||
// 只有智谱需要走原生请求体,其余主机一律按 OpenAI 兼容处理。
|
||||
const ZHIPU_HOST_SUFFIX = "bigmodel.cn";
|
||||
const DEFAULT_MODEL = "CogView-3-Flash";
|
||||
|
||||
// 上游同步调用耗时上限(出图几十秒量级,留足余量)。
|
||||
@@ -49,43 +55,102 @@ function parseArgs(argv) {
|
||||
return args;
|
||||
}
|
||||
|
||||
function pickToken(headers) {
|
||||
// 对外只有三个 header:Authorization / X-Image-Task-Base-Url / X-Image-Task-Model。
|
||||
// 不再支持 X-Image-Task-Token、X-Mcp-Token、X-Mcp-Model、X-Model 等旧写法。
|
||||
function pickAuthToken(headers) {
|
||||
if (!headers) return null;
|
||||
const candidates = [
|
||||
headers["x-image-task-token"],
|
||||
headers["x-mcp-token"],
|
||||
headers["authorization"],
|
||||
];
|
||||
for (const v of candidates) {
|
||||
if (!v) continue;
|
||||
const s = String(v).trim();
|
||||
if (!s) continue;
|
||||
if (/^Bearer\s+/i.test(s)) {
|
||||
const t = s.replace(/^Bearer\s+/i, "").trim();
|
||||
if (t) return t;
|
||||
} else {
|
||||
return s;
|
||||
}
|
||||
const s = String(headers["authorization"] || "").trim();
|
||||
if (!s) return null;
|
||||
if (/^Bearer\s+/i.test(s)) {
|
||||
const t = s.replace(/^Bearer\s+/i, "").trim();
|
||||
return t || null;
|
||||
}
|
||||
return null;
|
||||
return s;
|
||||
}
|
||||
|
||||
function pickModel(headers) {
|
||||
function pickHeaderString(headers, name) {
|
||||
if (!headers) return null;
|
||||
const v =
|
||||
headers["x-image-task-model"] ||
|
||||
headers["x-mcp-model"] ||
|
||||
headers["x-model"];
|
||||
if (!v) return null;
|
||||
const v = headers[name];
|
||||
if (v === undefined || v === null) return null;
|
||||
const s = String(v).trim();
|
||||
return s || null;
|
||||
}
|
||||
|
||||
async function callZhipuImagesGenerations({ token, model, prompt }) {
|
||||
const body = { model, prompt };
|
||||
function pickBaseUrl(headers) {
|
||||
return pickHeaderString(headers, "x-image-task-base-url");
|
||||
}
|
||||
|
||||
function pickModel(headers) {
|
||||
return pickHeaderString(headers, "x-image-task-model");
|
||||
}
|
||||
|
||||
// 解析 Base-Url:别名优先,否则必须是 http(s) URL。
|
||||
// 返回 { provider, url }:url 是最终 POST 的完整地址。
|
||||
function resolveEndpoint(rawBase) {
|
||||
const raw = String(rawBase || "zhipu").trim().replace(/\/+$/, "");
|
||||
const alias = BASE_URL_ALIASES[raw.toLowerCase()];
|
||||
const base = alias || raw;
|
||||
|
||||
let host;
|
||||
try {
|
||||
if (!/^https?:\/\//i.test(base)) throw new Error("not http(s)");
|
||||
host = new URL(base).hostname.toLowerCase();
|
||||
} catch {
|
||||
throw new Error(
|
||||
`Invalid X-Image-Task-Base-Url "${raw}": use an alias (${Object.keys(BASE_URL_ALIASES).join(", ")}) ` +
|
||||
"or a full http(s) URL."
|
||||
);
|
||||
}
|
||||
|
||||
// 已经给到 /images/generations 就直接用;否则补在 base 后面。
|
||||
const url = /\/images\/generations$/i.test(base)
|
||||
? base
|
||||
: base + "/images/generations";
|
||||
const provider =
|
||||
host === ZHIPU_HOST_SUFFIX || host.endsWith("." + ZHIPU_HOST_SUFFIX)
|
||||
? "zhipu"
|
||||
: "openai";
|
||||
return { provider, url };
|
||||
}
|
||||
|
||||
// 模型只由 header 决定,AI 看不到也填不了。
|
||||
// zhipu 通道有默认模型;其他通道必须显式给 —— 否则会把 CogView 发到网关,
|
||||
// 换来的只是一个含义不明的 400。
|
||||
function resolveModel(headers, provider) {
|
||||
const model = pickModel(headers);
|
||||
if (model) return model;
|
||||
if (provider === "zhipu") return DEFAULT_MODEL;
|
||||
throw new Error(
|
||||
"X-Image-Task-Model is required when X-Image-Task-Base-Url is not the zhipu alias."
|
||||
);
|
||||
}
|
||||
|
||||
// 一次解析出本次请求的上游、凭据、模型。
|
||||
// 配置问题同步抛错,由 create 直接回给调用方,不登记注定失败的任务。
|
||||
function resolveTarget(headers) {
|
||||
const token = pickAuthToken(headers);
|
||||
if (!token) {
|
||||
throw new Error('Missing Authorization: use "Authorization: Bearer <token>".');
|
||||
}
|
||||
const { provider, url } = resolveEndpoint(pickBaseUrl(headers));
|
||||
return { provider, url, token, model: resolveModel(headers, provider) };
|
||||
}
|
||||
|
||||
async function callImagesGenerations({ provider, url, token, model, prompt, image }) {
|
||||
// 智谱原生只认 model/prompt;OpenAI 兼容通道固定单图,并要求返回 URL
|
||||
// (b64_json 几十 MB,既塞不进任务表也没法给模型看)。尺寸/质量一律不传,
|
||||
// 交给上游默认值 —— 各家(尤其 CogView)对尺寸的限制并不一致。
|
||||
// 传了 image 就是图生图;上游字段是数组,所以这里包一层。
|
||||
const body =
|
||||
provider === "zhipu"
|
||||
? { model, prompt }
|
||||
: Object.assign(
|
||||
{ model, prompt, n: 1, response_format: "url" },
|
||||
image ? { images: [image] } : null
|
||||
);
|
||||
let resp;
|
||||
try {
|
||||
resp = await fetch(ZHIPU_API_URL, {
|
||||
resp = await fetch(url, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Authorization: `Bearer ${token}`,
|
||||
@@ -110,8 +175,9 @@ async function callZhipuImagesGenerations({ token, model, prompt }) {
|
||||
data = { raw: text };
|
||||
}
|
||||
if (!resp.ok) {
|
||||
const label = provider === "zhipu" ? "Zhipu" : "OpenAI-compatible";
|
||||
const err = new Error(
|
||||
`Zhipu API error: ${resp.status} ${resp.statusText} - ${text}`
|
||||
`${label} API error: ${resp.status} ${resp.statusText} - ${text}`
|
||||
);
|
||||
err.status = resp.status;
|
||||
err.detail = data;
|
||||
@@ -120,15 +186,24 @@ async function callZhipuImagesGenerations({ token, model, prompt }) {
|
||||
return data;
|
||||
}
|
||||
|
||||
// 只认 http(s) URL。b64_json 不再当 URL 存:它会把任务表撑爆,模型也读不了。
|
||||
function extractUrls(data) {
|
||||
if (!data) return [];
|
||||
if (Array.isArray(data.data)) {
|
||||
return data.data
|
||||
.map((it) => it && (it.url || it.b64_json))
|
||||
.filter(Boolean);
|
||||
}
|
||||
if (typeof data.url === "string") return [data.url];
|
||||
return [];
|
||||
const items = Array.isArray(data.data)
|
||||
? data.data
|
||||
: typeof data.url === "string"
|
||||
? [data]
|
||||
: [];
|
||||
return items
|
||||
.map((it) => (it && typeof it.url === "string" ? it.url.trim() : ""))
|
||||
.filter((u) => /^https?:\/\//i.test(u));
|
||||
}
|
||||
|
||||
function hasInlineBase64(data) {
|
||||
return (
|
||||
Array.isArray(data && data.data) &&
|
||||
data.data.some((it) => it && typeof it.b64_json === "string")
|
||||
);
|
||||
}
|
||||
|
||||
// ===== 自维护任务表(异步语义全靠这层模拟) =====
|
||||
@@ -163,19 +238,31 @@ function pruneTasks() {
|
||||
}
|
||||
|
||||
// 后台跑同步上游调用,结果回写任务表(fire-and-forget,不 await)
|
||||
function runImageTask({ task, token, model, prompt }) {
|
||||
callZhipuImagesGenerations({ token, model, prompt })
|
||||
function runImageTask({ task, target, prompt, image }) {
|
||||
callImagesGenerations({
|
||||
provider: target.provider,
|
||||
url: target.url,
|
||||
token: target.token,
|
||||
model: target.model,
|
||||
prompt,
|
||||
image,
|
||||
})
|
||||
.then((data) => {
|
||||
task.finishedAt = Date.now();
|
||||
const urls = extractUrls(data);
|
||||
if (urls.length === 0) {
|
||||
task.status = "FAIL";
|
||||
task.error = `Zhipu returned no image URL. Raw response: ${JSON.stringify(data)}`;
|
||||
task.error = hasInlineBase64(data)
|
||||
? "Upstream returned b64_json instead of a URL; this service only returns URLs. " +
|
||||
"Use a model that supports response_format=url."
|
||||
: `Upstream returned no image URL. Raw response: ${JSON.stringify(data)}`;
|
||||
} else {
|
||||
task.status = "SUCCESS";
|
||||
task.urls = urls;
|
||||
}
|
||||
})
|
||||
.catch((e) => {
|
||||
task.finishedAt = Date.now();
|
||||
task.status = "FAIL";
|
||||
task.error = `${e.message}${
|
||||
e.detail ? ` Details: ${JSON.stringify(e.detail)}` : ""
|
||||
@@ -184,7 +271,7 @@ function runImageTask({ task, token, model, prompt }) {
|
||||
}
|
||||
|
||||
// ===== op=create:提交图片生成(立即返回 task_id,后台执行)=====
|
||||
async function handleCreate({ args, headers, token }) {
|
||||
async function handleCreate({ args, headers }) {
|
||||
// schema 只约束了 min(1)," " 这种纯空白能通过校验并白跑一次出图,所以这里 trim 后再判。
|
||||
// 顺手用 trim 过的值:上游拿到的 prompt 不带首尾空白。
|
||||
const prompt = typeof args.prompt === "string" ? args.prompt.trim() : "";
|
||||
@@ -195,26 +282,76 @@ async function handleCreate({ args, headers, token }) {
|
||||
};
|
||||
}
|
||||
|
||||
// model 完全由 HTTP header 控制(X-Image-Task-Model / X-Mcp-Model / X-Model),
|
||||
// 不存进 session,每次从本次请求的 header 读取,未设置则用默认 CogView-3-Flash。
|
||||
// AI 看不到 model 参数(schema 中已删除),避免其误传。
|
||||
const model = pickModel(headers) || DEFAULT_MODEL;
|
||||
// image 是文生图/图生图开关:不传 = 文生图;传一个公网 URL = 修图。
|
||||
// 只接受公网 http(s) URL;本地文件请先由调用方用 mint_public_link 换成公网链接。
|
||||
if (args.image !== undefined && typeof args.image !== "string") {
|
||||
return {
|
||||
isError: true,
|
||||
content: [{ type: "text", text: 'op=create: "image" must be a string.' }],
|
||||
};
|
||||
}
|
||||
const image = typeof args.image === "string" ? args.image.trim() : "";
|
||||
if (image && !/^https?:\/\//i.test(image)) {
|
||||
return {
|
||||
isError: true,
|
||||
content: [
|
||||
{
|
||||
type: "text",
|
||||
text:
|
||||
'op=create: "image" must be a public http(s) URL. ' +
|
||||
"For a local file, call mint_public_link first to get a public URL.",
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
// 上游地址、模型、凭据全部来自本次请求的 header(Authorization /
|
||||
// X-Image-Task-Base-Url / X-Image-Task-Model),AI 看不到也填不了。
|
||||
// 配置类错误直接回错,不登记一个必然失败的任务。
|
||||
let target;
|
||||
try {
|
||||
target = resolveTarget(headers);
|
||||
} catch (e) {
|
||||
return {
|
||||
isError: true,
|
||||
content: [{ type: "text", text: e.message }],
|
||||
};
|
||||
}
|
||||
|
||||
// 智谱原生接口不支持图生图;带 image 的请求只能走 OpenAI 兼容端点。
|
||||
if (image && target.provider === "zhipu") {
|
||||
return {
|
||||
isError: true,
|
||||
content: [
|
||||
{
|
||||
type: "text",
|
||||
text:
|
||||
'Image editing is not supported by the configured model. ' +
|
||||
'Remove "image" for text-to-image.',
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
pruneTasks();
|
||||
const taskId = randomUUID();
|
||||
const task = {
|
||||
status: "PROCESSING",
|
||||
model,
|
||||
provider: target.provider,
|
||||
model: target.model,
|
||||
prompt,
|
||||
createdAt: Date.now(),
|
||||
};
|
||||
tasks.set(taskId, task);
|
||||
|
||||
// token/model 在此刻从本次请求 header 捕获进闭包,后台完成前无需再取
|
||||
runImageTask({ task, token, model, prompt });
|
||||
// target 与 image 在此刻从本次请求捕获进闭包,后台完成前无需再取
|
||||
runImageTask({ task, target, prompt, image });
|
||||
|
||||
const mode = image ? "edit" : "text-to-image";
|
||||
const summary =
|
||||
`Model: ${model}\n` +
|
||||
`Model: ${target.model}\n` +
|
||||
`Provider: ${target.provider}\n` +
|
||||
`Mode: ${mode}\n` +
|
||||
`task_id: ${taskId}\n` +
|
||||
`task_status: PROCESSING\n` +
|
||||
`\n` +
|
||||
@@ -224,6 +361,20 @@ async function handleCreate({ args, headers, token }) {
|
||||
return { content: [{ type: "text", text: summary }] };
|
||||
}
|
||||
|
||||
// 从 create 到完成(未完成则到当前)的累计秒数。
|
||||
// 已完成任务用 finishedAt 定格,避免它留在表里越查越大。
|
||||
function elapsedSeconds(task) {
|
||||
const end = task.status === "PROCESSING"
|
||||
? Date.now()
|
||||
: (task.finishedAt || Date.now());
|
||||
return Math.max(0, Math.round((end - task.createdAt) / 1000));
|
||||
}
|
||||
|
||||
// 轮询退避:起步 3s,之后每多等 5s 加 1s,封顶 30s。
|
||||
function pollIntervalSeconds(elapsed) {
|
||||
return Math.min(30, 3 + Math.floor(elapsed / 5));
|
||||
}
|
||||
|
||||
// ===== op=get:查询任务状态(只读本地表,不碰上游)=====
|
||||
async function handleGet({ args }) {
|
||||
// 与 handleCreate 同理:纯空白 task_id 应报"缺参数",而不是得到 Unknown task_id " " 这种含糊错误。
|
||||
@@ -251,6 +402,8 @@ async function handleGet({ args }) {
|
||||
}
|
||||
|
||||
if (task.status === "PROCESSING") {
|
||||
const elapsed = elapsedSeconds(task);
|
||||
const retryAfter = pollIntervalSeconds(elapsed);
|
||||
return {
|
||||
content: [
|
||||
{
|
||||
@@ -258,9 +411,11 @@ async function handleGet({ args }) {
|
||||
text:
|
||||
`task_id: ${taskId}\n` +
|
||||
`model: ${task.model}\n` +
|
||||
`provider: ${task.provider}\n` +
|
||||
`elapsed: ${elapsed}s\n` +
|
||||
`task_status: PROCESSING\n` +
|
||||
`\n` +
|
||||
`Call op="get" again in a few seconds.`,
|
||||
`Task is still processing. Sleep for ${retryAfter} seconds, then call op="get" again.`,
|
||||
},
|
||||
],
|
||||
};
|
||||
@@ -272,7 +427,12 @@ async function handleGet({ args }) {
|
||||
content: [
|
||||
{
|
||||
type: "text",
|
||||
text: `task_id: ${taskId}\ntask_status: FAIL\nerror: ${task.error}`,
|
||||
text:
|
||||
`task_id: ${taskId}\n` +
|
||||
`provider: ${task.provider}\n` +
|
||||
`elapsed: ${elapsedSeconds(task)}s\n` +
|
||||
`task_status: FAIL\n` +
|
||||
`error: ${task.error}`,
|
||||
},
|
||||
],
|
||||
};
|
||||
@@ -287,6 +447,8 @@ async function handleGet({ args }) {
|
||||
const summary =
|
||||
`task_id: ${taskId}\n` +
|
||||
`model: ${task.model}\n` +
|
||||
`provider: ${task.provider}\n` +
|
||||
`elapsed: ${elapsedSeconds(task)}s\n` +
|
||||
`task_status: SUCCESS\n` +
|
||||
`${lines}`;
|
||||
return { content: [{ type: "text", text: summary }] };
|
||||
@@ -307,17 +469,18 @@ function buildServer() {
|
||||
|
||||
server.tool(
|
||||
"image_task",
|
||||
`AI image generation task.
|
||||
`AI image generation and editing task.
|
||||
|
||||
Operations:
|
||||
- \`create\`: Create a new generation task. Requires \`prompt\`. Returns \`task_id\` immediately; generation runs in background.
|
||||
- \`get\`: Check an existing task. Requires \`task_id\`. Returns status and, on SUCCESS, image URLs.
|
||||
- \`create\`: Create a generation task. Requires \`prompt\`. Returns \`task_id\` immediately; generation runs in background. Optionally pass \`image\` for image-to-image editing.
|
||||
- \`get\`: Check a task. Requires \`task_id\`. Returns status and, on SUCCESS, image URLs.
|
||||
|
||||
Examples:
|
||||
- New image request → \`{"op":"create","prompt":"A cat sitting in a garden"}\`
|
||||
- Check task result → \`{"op":"get","task_id":"<task_id from create>"}\`
|
||||
- New image → \`{"op":"create","prompt":"A cat sitting in a garden"}\`
|
||||
- Edit an image → \`{"op":"create","prompt":"Make it snow","image":"https://..."}\`
|
||||
- Check result → \`{"op":"get","task_id":"<task_id from create>"}\`
|
||||
|
||||
Workflow: call \`create\` once, then call \`get\` with the returned \`task_id\` until \`task_status\` is \`SUCCESS\` or \`FAIL\`. Never call \`get\` without a \`task_id\`, and never submit duplicate tasks just to check status.`,
|
||||
Workflow: call \`create\` once, then \`get\` with the returned \`task_id\` until \`task_status\` is \`SUCCESS\` or \`FAIL\`.`,
|
||||
{
|
||||
op: z
|
||||
.enum(["create", "get"])
|
||||
@@ -334,12 +497,20 @@ Workflow: call \`create\` once, then call \`get\` with the returned \`task_id\`
|
||||
.min(1)
|
||||
.optional()
|
||||
.describe("task_id returned by create. Required for get; omit for create."),
|
||||
image: z
|
||||
.string()
|
||||
.min(1)
|
||||
.regex(/^https?:\/\//i)
|
||||
.optional()
|
||||
.describe(
|
||||
"One public image URL to edit; omit for text-to-image. For a local file, call mint_public_link first to get a public URL."
|
||||
),
|
||||
},
|
||||
async (args, extra) => {
|
||||
// SDK 在 tool callback 的 extra.requestInfo.headers 里会透传当前 HTTP 请求的 headers
|
||||
// (参见 webStandardStreamableHttp.js: buildRequestInfo 段)
|
||||
const headers = extra?.requestInfo?.headers;
|
||||
const token = pickToken(headers);
|
||||
const token = pickAuthToken(headers);
|
||||
if (!token) {
|
||||
return {
|
||||
isError: true,
|
||||
@@ -347,7 +518,7 @@ Workflow: call \`create\` once, then call \`get\` with the returned \`task_id\`
|
||||
{
|
||||
type: "text",
|
||||
text:
|
||||
"Missing authentication token.",
|
||||
'Missing Authorization: use "Authorization: Bearer <token>".',
|
||||
},
|
||||
],
|
||||
};
|
||||
@@ -355,7 +526,7 @@ Workflow: call \`create\` once, then call \`get\` with the returned \`task_id\`
|
||||
|
||||
switch (args.op) {
|
||||
case "create":
|
||||
return await handleCreate({ args, headers, token });
|
||||
return await handleCreate({ args, headers });
|
||||
case "get":
|
||||
return await handleGet({ args });
|
||||
default:
|
||||
|
||||
+29
-4
@@ -10,12 +10,14 @@
|
||||
#
|
||||
# 输入(都可选):
|
||||
# BIGMODEL_KEY / DEEPSEEK_KEY / QWEN_KEY 模型平台 key,填几个启用几个(0~3 个)
|
||||
# image_task / video_task 复用 BIGMODEL_KEY(智谱即 BigModel);
|
||||
# 未配 BIGMODEL_KEY 时这两个服务写为 mode: disable
|
||||
# image_task / video_task / asr 复用 BIGMODEL_KEY(智谱即 BigModel);
|
||||
# 未配 BIGMODEL_KEY 时这三个服务写为 mode: disable
|
||||
# AUTH_USER / AUTH_PASSWORD 登录账号(默认 admin / admin,请自行改掉)
|
||||
# PUBLIC_IP / AGENT_PORT 对外地址与端口,写进 public_link.base_url;
|
||||
# PUBLIC_IP 不配置则自动探测公网 IP
|
||||
# AUTH_SECRET / EXECUTOR_TOKEN / WXBOT_TOKEN 一般不用给:不给就自动生成并复用(见下)
|
||||
# ASR_API_KEY 语音识别(mcp_asr)的上游 key;不给则复用 BIGMODEL_KEY,
|
||||
# 已有 .env 里的值优先(语音输入只是可选能力,留空不影响主流程)
|
||||
#
|
||||
# 微信接入(wxbot 已合入根 compose,作为栈内服务):本脚本只负责生成/复用根 .env 的 WXBOT_TOKEN;
|
||||
# wxbot 自身零配置文件(compose 注 env + 内置默认,同 executor 模式),无需任何同步动作。
|
||||
@@ -51,6 +53,8 @@ EXECUTOR_TOKEN="${EXECUTOR_TOKEN:-}"
|
||||
AUTH_SECRET="${AUTH_SECRET:-}"
|
||||
# 微信接入服务 token(core ↔ wxbot 双向共用,compose 两侧同值注入;wxbot 已并入仓库)
|
||||
WXBOT_TOKEN="${WXBOT_TOKEN:-}"
|
||||
# 语音识别(mcp_asr)的上游 key:独立于模型平台 key,但默认复用 BIGMODEL_KEY(智谱 glm-asr)
|
||||
ASR_API_KEY="${ASR_API_KEY:-}"
|
||||
|
||||
log() { echo -e "\n\033[1;32m===== $* =====\033[0m"; }
|
||||
info() { echo -e "\033[1;36m[INFO]\033[0m $*"; }
|
||||
@@ -74,6 +78,8 @@ if [ -f "$ENV_FILE" ]; then
|
||||
EXECUTOR_TOKEN="${EXECUTOR_TOKEN:-$(sed -n 's/^EXECUTOR_TOKEN=//p' "$ENV_FILE" | head -1)}"
|
||||
AUTH_SECRET="${AUTH_SECRET:-$(sed -n 's/^AUTH_SECRET=//p' "$ENV_FILE" | head -1)}"
|
||||
WXBOT_TOKEN="${WXBOT_TOKEN:-$(sed -n 's/^WXBOT_TOKEN=//p' "$ENV_FILE" | head -1)}"
|
||||
# ASR_API_KEY 同理复用:手填的 key 不会被重跑抹掉
|
||||
ASR_API_KEY="${ASR_API_KEY:-$(sed -n 's/^ASR_API_KEY=//p' "$ENV_FILE" | head -1)}"
|
||||
# AGENT_PORT 同理复用:避免不带参数重跑时把部署机自定义端口重置回 8090(显式传参优先)
|
||||
AGENT_PORT="${AGENT_PORT_INPUT:-$(sed -n 's/^AGENT_PORT=//p' "$ENV_FILE" | head -1)}"
|
||||
AGENT_PORT="${AGENT_PORT:-8090}"
|
||||
@@ -81,19 +87,22 @@ fi
|
||||
EXECUTOR_TOKEN="${EXECUTOR_TOKEN:-$(gen_hex 32)}"
|
||||
AUTH_SECRET="${AUTH_SECRET:-$(gen_hex 32)}"
|
||||
WXBOT_TOKEN="${WXBOT_TOKEN:-$(gen_hex 32)}"
|
||||
ASR_API_KEY="${ASR_API_KEY:-$BIGMODEL_KEY}"
|
||||
|
||||
cat > "$ENV_FILE" <<EOF
|
||||
# 本文件由 scripts/init.sh 生成,勿手改(改配置请重新运行该脚本,或用环境变量覆盖)
|
||||
# EXECUTOR_TOKEN: core 与 executor 之间的调用鉴权(两侧同值,compose 注入)
|
||||
# AUTH_SECRET: 登录态签名密钥(写进 config.yml 的 auth.secret)
|
||||
# WXBOT_TOKEN: core ↔ wxbot 微信接入服务 token(compose 注入两侧;wxbot 零配置文件,env 即全部接线)
|
||||
# ASR_API_KEY: 语音识别(mcp_asr)的上游 key,空 = 语音输入关闭;默认复用 BIGMODEL_KEY
|
||||
# AGENT_PORT: agent-core 映射到宿主机的端口
|
||||
EXECUTOR_TOKEN=$EXECUTOR_TOKEN
|
||||
AUTH_SECRET=$AUTH_SECRET
|
||||
WXBOT_TOKEN=$WXBOT_TOKEN
|
||||
ASR_API_KEY=$ASR_API_KEY
|
||||
AGENT_PORT=$AGENT_PORT
|
||||
EOF
|
||||
info "EXECUTOR_TOKEN / AUTH_SECRET / WXBOT_TOKEN 已就位(沿用已有 .env 的值,没有则新生成)"
|
||||
info "EXECUTOR_TOKEN / AUTH_SECRET / WXBOT_TOKEN 已就位(沿用已有 .env 的值,没有则新生成);ASR_API_KEY $([ -n "$ASR_API_KEY" ] && echo 已配置 || echo 为空(语音输入关闭))"
|
||||
|
||||
# --env-only:只刷密钥,绝不碰 config.yml(平台 key 不在 .env 里,重跑全量会清空已部署配置)
|
||||
if [ "${ENV_ONLY:-}" = "1" ]; then
|
||||
@@ -229,6 +238,21 @@ EOF
|
||||
EOF
|
||||
fi
|
||||
|
||||
# 语音转写:key 在容器 env(ASR_API_KEY),不经 headers,所以单独按 ASR_API_KEY 判断
|
||||
if [ -n "$ASR_API_KEY" ]; then
|
||||
cat <<'EOF'
|
||||
- name: asr
|
||||
url: "http://mcp_asr:3000/mcp"
|
||||
mode: skill # 语音转写是低频能力,不下发到每轮工具表
|
||||
EOF
|
||||
else
|
||||
cat <<'EOF'
|
||||
- name: asr
|
||||
url: "http://mcp_asr:3000/mcp"
|
||||
mode: disable # 未配 ASR_API_KEY:不连接、不发现
|
||||
EOF
|
||||
fi
|
||||
|
||||
echo
|
||||
cat <<EOF
|
||||
# 给外部服务(如智谱)现签公网临时链接用;改地址后重跑本脚本
|
||||
@@ -249,7 +273,8 @@ echo " 账号 : $AUTH_USER"
|
||||
echo " 密码 : $AUTH_PASSWORD"
|
||||
echo " 平台 : ${enabled_plats[*]:-(无)}"
|
||||
echo " 图像/视频: $([ -n "$BIGMODEL_KEY" ] && echo "image_task / video_task 已启用" || echo "未配置(mode: disable)")"
|
||||
echo " MCP 服务 : web_search / image_task / video_task(同栈,随 compose 启动)"
|
||||
echo " 语音输入 : $([ -n "$ASR_API_KEY" ] && echo "mcp_asr 已启用" || echo "未配置(ASR_API_KEY 为空)")"
|
||||
echo " MCP 服务 : web_search / image_task / video_task / asr(同栈,随 compose 启动)"
|
||||
echo " 微信接入 : WXBOT_TOKEN 已就位(compose 注入 core 与 wxbot 两侧;wxbot 无需配置文件)"
|
||||
echo "---------------------------------------------"
|
||||
log "完成 ✅"
|
||||
|
||||
+14
-9
@@ -8,9 +8,9 @@
|
||||
|
||||
技能是一个自包含的目录,通常存放在技能根目录(默认为 `skills/`)下。每个技能包含一个描述文件(`SKILL.md`)和一个可执行入口(`run.py` 或 `run.sh`)。Agent 通过以下流程使用技能:
|
||||
|
||||
1. 调用 `skill` 的 `{"list":"all"}` 查看所有可用技能的名称和简介;
|
||||
2. 调用 `skill` 的 `{"get":"<技能名>"}` 获取某个技能的详细说明;
|
||||
3. 调用 `skill` 的 `{"run":"<技能名>","args":{...}}` 执行技能,传入所需参数。
|
||||
1. 调用 `skill` 的 `{"op":"list"}` 查看所有可用技能的名称和简介;
|
||||
2. 调用 `skill` 的 `{"op":"get","name":"<技能名>"}` 获取某个技能的详细说明;
|
||||
3. 调用 `skill` 的 `{"op":"run","name":"<技能名>","args":{...}}` 执行技能,传入所需参数。
|
||||
|
||||
技能可以处理任何任务,例如文件格式转换、数据处理、网页抓取等。
|
||||
|
||||
@@ -53,11 +53,16 @@ input_schema:
|
||||
output_schema:
|
||||
# 输出描述
|
||||
visit_url: /app/{task_id}/ # 可选:页面类技能的访问路径模板
|
||||
required: true # 可选:清单行首标 **required**,提示该域不要手写等价实现
|
||||
thinking: true # 可选:get 时自动开启本回合的深度思考
|
||||
---
|
||||
```
|
||||
|
||||
- `name`:技能名称,应与目录名一致。
|
||||
- `description`:一句话描述技能功能,用于 `{"list":"all"}` 的清单展示。
|
||||
- `description`:一句话描述技能功能,用于 `op=list` 的清单展示。
|
||||
- `thinking`:(可选)为 `true` 时,模型调用 `get` 会自动开启深度思考(合并模型的
|
||||
`thinking_params`),开启后保持,直到用户手动关闭;模型未配置思考参数时不生效。只影响模型回合,不影响脚本本身。
|
||||
- `required`:(可选)为 `true` 时,模型可见的技能清单里该行行首标 `**required**`,表示这个领域不要手写等价脚本(如 `office` / `x2pdf`)。纯提示,不做强制校验,也不会自动调用。
|
||||
- `input_schema`:输入参数的 JSON Schema 定义,用于参数校验和文档生成。
|
||||
- `output_schema`:输出结果的描述(可选)。
|
||||
- `visit_url`:(可选)页面类技能声明的访问路径模板,`{task_id}` 占位执行时替换为会话 id。`run` 成功后由 core(tools/executor.js)拼上 `public_link.base_url`,在输出末尾追加一行「访问地址:<完整 URL>」——技能进程跑在 executor 隔离视图里不知道公网地址,完整链接只能 core 补;AI 拿到相对路径会原样转给用户,而相对路径打不开。
|
||||
@@ -80,7 +85,7 @@ visit_url: /app/{task_id}/ # 可选:页面类技能的访问路径模板
|
||||
- 自动处理某某
|
||||
|
||||
## 使用说明
|
||||
通过 `skill` 的 `{"run":"<技能名>","args":{...}}` 调用,参数:
|
||||
通过 `skill` 的 `{"op":"run","name":"<技能名>","args":{...}}` 调用,参数:
|
||||
- `input`:某某路径(必填)
|
||||
- `output`:某某路径(可选,默认与输入同名)
|
||||
```
|
||||
@@ -92,10 +97,10 @@ visit_url: /app/{task_id}/ # 可选:页面类技能的访问路径模板
|
||||
把 `config.yml` 里某个 MCP 服务配成 `mode: skill` 即可:它的每个工具会自动成为技能,不需要
|
||||
`SKILL.md`,也不需要入口脚本。
|
||||
|
||||
- `{"list":"all"}` 里出现工具名,描述取 MCP `description` 的第一行;
|
||||
- `{"get":"<工具名>"}` 直接返回 MCP 原始的 `description` + `input_schema`(单一来源,不会与
|
||||
- `{"op":"list"}` 里出现工具名,描述取 MCP `description` 的第一行;
|
||||
- `{"op":"get","name":"<工具名>"}` 直接返回 MCP 原始的 `description` + `input_schema`(单一来源,不会与
|
||||
实现漂移);
|
||||
- `{"run":"<工具名>","args":{…}}` 把 `args` 原样透传成 MCP 的 `arguments`,由 core 直接转发。
|
||||
- `{"op":"run","name":"<工具名>","args":{…}}` 把 `args` 原样透传成 MCP 的 `arguments`,由 core 直接转发。
|
||||
|
||||
几个约束:
|
||||
|
||||
@@ -251,7 +256,7 @@ input_schema:
|
||||
---
|
||||
|
||||
## 使用说明
|
||||
通过 `skill` 的 `{"run":"<技能名>","args":{...}}` 调用,参数:
|
||||
通过 `skill` 的 `{"op":"run","name":"<技能名>","args":{...}}` 调用,参数:
|
||||
- `text`:要回显的文本(必填)
|
||||
```
|
||||
|
||||
|
||||
@@ -31,14 +31,14 @@ visit_url: /app/{task_id}/
|
||||
|
||||
## 初始化
|
||||
|
||||
调用 `skill {"run":"data-page"}`,无需参数。它会把以下文件复制到 `.app/`:
|
||||
调用 `skill {"op":"run","name":"data-page"}`,无需参数。它会把以下文件发布到 `.app/`:
|
||||
|
||||
- `table.html` / `dashboard.html` / `kanban.html` —— 三个数据页模板
|
||||
- `nav.html` —— 多页导航首页模板(静态卡片,无 config)
|
||||
- `base.css` —— 共用样式
|
||||
- `config-table.js` / `config-dashboard.js` / `config-kanban.js` —— 配置骨架
|
||||
|
||||
重跑行为按资产所有权分流:`table/dashboard/kanban/nav.html` 与 `base.css` 每次**覆盖刷新**(模板修复自动到达旧会话;这些文件契约禁止修改,覆盖无副作用);`config-*.js` 仅缺失时补默认骨架,不覆盖已写好的配置;`.app/index.html` 永不被碰。
|
||||
重跑行为按资产所有权分流:`table/dashboard/kanban/nav.html` 与 `base.css` 是指向技能的符号链接,技能更新立即生效(这些文件契约禁止修改);`config-*.js` 仅缺失时补默认骨架,不覆盖已写好的配置;`.app/index.html` 永不被碰。
|
||||
|
||||
## 数据 API
|
||||
|
||||
@@ -80,7 +80,7 @@ const taskId = location.pathname.split('/').filter(Boolean)[1];
|
||||
|
||||
1. **明确数据结构**:与用户确认字段。如果描述模糊,可以建议合理的结构。
|
||||
2. **创建表并录入数据**:用 `db` 工具完成。
|
||||
3. **初始化**:调用 `skill {"run":"data-page"}`。
|
||||
3. **初始化**:调用 `skill {"op":"run","name":"data-page"}`。
|
||||
4. **为每个要生成的页面写 config 文件**(见下一节)。**不要读也不要改模板本身**。
|
||||
5. **选定入口页 `.app/index.html`**(index 是副本,允许编辑;模板本体不动):
|
||||
- **用到 ≥2 个模板页** → `cp .app/nav.html .app/index.html`,改标题为业务名,并把未启用页面的卡片删掉(没配过的页点开会报 no such table)。子页标题左侧的「‹」返回键链回此导航页。
|
||||
|
||||
+27
-8
@@ -1,13 +1,13 @@
|
||||
#!/usr/bin/env python3
|
||||
# data-page 技能入口:把技能自带 templates/ 下的文件复制到会话任务目录的 .app/。
|
||||
# data-page 技能入口:把技能自带 templates/ 下的文件发布到会话任务目录的 .app/。
|
||||
# 目标目录固定为 <任务目录>/.app/(不带 dest 参数:manager 对绝对路径参数不做沙箱
|
||||
# 校验,开放 dest 等于开放任意目录写入;路径也已写死在文档流程里,可配置只会造成漂移)。
|
||||
# 复制内容:
|
||||
# *.html / base.css → .app/ 页面模板与共用样式(每次覆盖刷新)
|
||||
# config-*.js → .app/ 配置骨架(仅缺失时补,AI 只改这些)
|
||||
# 复制语义按资产所有权分流:
|
||||
# *.html / *.css → .app/ 模板与共用样式——每次运行覆盖刷新(SKILL 契约禁止改这些文件,
|
||||
# 覆盖无副作用;不覆盖则旧会话永远吃不到模板修复)。index.html 不在复制清单,永不碰。
|
||||
# 发布内容:
|
||||
# *.html / base.css → .app/ 页面模板与共用样式(符号链接,技能是唯一真源)
|
||||
# config-*.js → .app/ 配置骨架(实体文件,仅缺失时补,AI 只改这些)
|
||||
# 发布语义按资产所有权分流:
|
||||
# *.html / *.css → .app/ 模板与共用样式——技能只读,链接后永远是最新版本;
|
||||
# SKILL 契约本来就禁止改这些文件,所以不必留副本。
|
||||
# config-*.js → .app/ 配置骨架——仅缺失时补默认,不覆盖 AI 已写好的配置。
|
||||
# 纯 stdlib,无 requirements。
|
||||
# cwd 契约:生产模式下 cwd 即会话任务目录;本地直跑时 cwd 是技能目录——
|
||||
@@ -19,6 +19,25 @@ import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def link_asset(src, dst):
|
||||
"""模板/样式是技能的只读资产:用符号链接发布,永远跟随技能最新版本。
|
||||
|
||||
core 的 /app 静态托管只放行指向技能根目录的链接(其余 404),
|
||||
所以这里必须链到技能的绝对路径。config-*.js 是每会话的数据,仍然复制。
|
||||
"""
|
||||
src = str(Path(src).resolve())
|
||||
if dst.is_symlink():
|
||||
try:
|
||||
if os.path.realpath(dst) == os.path.realpath(src):
|
||||
return
|
||||
except OSError:
|
||||
pass
|
||||
dst.unlink()
|
||||
elif dst.exists():
|
||||
dst.unlink() # 旧会话的实体副本:删掉换成链接
|
||||
os.symlink(src, str(dst))
|
||||
|
||||
|
||||
def main():
|
||||
# ── 参数校验:本技能无参数 ─────────────────────────────
|
||||
if len(sys.argv) >= 2 and sys.argv[1].strip():
|
||||
@@ -60,7 +79,7 @@ def main():
|
||||
# ── 模板/样式:无条件覆盖刷新;config:仅补缺 ────────────
|
||||
refreshed = []
|
||||
for f in assets:
|
||||
shutil.copy2(f, app_dir / f.name)
|
||||
link_asset(f, app_dir / f.name)
|
||||
refreshed.append(f.name)
|
||||
created, kept = [], []
|
||||
for f in cfgs:
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
# 审核页内部协议(维护者文档)
|
||||
|
||||
> 本文件刻意不在 SKILL.md 里引用,避免模型顺着指针来读;只有维护者需要看。
|
||||
|
||||
`SKILL.md` 是给 AI 的调用契约;本文件记录审核页/重生成/数据表的实现细节。**AI 日常调用
|
||||
不需要读这里**,`skill get` 也只返回 SKILL.md。
|
||||
|
||||
## 文件与目录
|
||||
|
||||
```
|
||||
skills/document-photo-to-pdf/
|
||||
SKILL.md ← AI 契约(只有 images/output)
|
||||
REVIEW.md ← 本文件
|
||||
run.py ← 入口
|
||||
doc_deskew/ ← 检测/几何/出页/检验
|
||||
review_store.py ← data.db 的 doc_review_* 表
|
||||
review_page.py ← 模板链接 + 缩略图发布
|
||||
web/index.html ← 审核页模板的单一事实源
|
||||
|
||||
tasks/<session-id>/
|
||||
out/ ← 原始校正页图(PNG,自然尺寸,文件名与 PDF 同名)
|
||||
.app/doc-review/
|
||||
index.html ← 指向 web/index.html 的符号链接(不含数据注入)
|
||||
<batch_id>/p1.jpg ... ← 本批缩略图(长边 1200,JPEG)
|
||||
data.db ← doc_review_batch / doc_review_page / doc_review_job
|
||||
<output>.pdf
|
||||
<output>.state.json ← state v2,供恢复自动/调试
|
||||
```
|
||||
|
||||
core 只托管会话 `.app/`,不托管 skill 目录,所以会话里要有一个入口文件;用符号链接而不是
|
||||
副本——源仍只有 `web/index.html` 一份,改模板即刻对所有会话生效。`/app` 静态托管会
|
||||
realpath,只放行落在会话目录或技能根目录内的链接(其余 404,防止链接把 config.yml 带出来)。
|
||||
|
||||
## 对 AI 隐藏的参数
|
||||
|
||||
`run.py` 仍接受以下参数,但不出现在 SKILL.md / input_schema 里;审核页和 apply 接口使用:
|
||||
|
||||
| 参数 | 用途 |
|
||||
|---|---|
|
||||
| `paper` | a4/letter/16k/b5/数字比例/auto |
|
||||
| `orientation` | auto/none/portrait/landscape |
|
||||
| `enhance` | original/normal/strong/bw |
|
||||
| `dpi` | standard/high/72~600 |
|
||||
| `review` | `always`(默认)/`auto`(仅有 needs 时发布)/`off`(自动化用) |
|
||||
| `batch_id` | 重生成:从 data.db 读源图/输出/编辑,覆盖同名 PDF |
|
||||
| `edits` | `review/1` 对象或 JSON 路径;优先级高于显式参数 |
|
||||
| `state` | 上一份 state.json;仅用于恢复自动/调试 |
|
||||
|
||||
优先级:`edits.batch` > 显式参数 > 默认值。
|
||||
|
||||
## data.db 表
|
||||
|
||||
所有表以 `doc_review_` 前缀,避免和用户业务表混淆。技能自己 `CREATE TABLE IF NOT EXISTS`;
|
||||
`busy_timeout=5000`,WAL 由 core 建库时设置,技能侧 best-effort。
|
||||
|
||||
### doc_review_batch
|
||||
|
||||
```sql
|
||||
id TEXT PRIMARY KEY, -- drYYYYMMDD-HHMMSS-xxxx,不含点/斜杠
|
||||
output TEXT, -- 相对会话目录的 PDF 路径
|
||||
paper TEXT, enhance TEXT, dpi TEXT, orientation TEXT,
|
||||
status TEXT, -- running/ready/review/error
|
||||
summary TEXT, warnings TEXT,-- warnings 是 JSON 数组
|
||||
schema_version INTEGER,
|
||||
created_at TEXT, updated_at TEXT
|
||||
```
|
||||
|
||||
### doc_review_page
|
||||
|
||||
```sql
|
||||
batch_id TEXT, idx INTEGER, -- idx 永远是原始 images 的 1-based 下标
|
||||
source TEXT, -- 相对会话目录的原图路径
|
||||
source_w INTEGER, source_h INTEGER,
|
||||
w INTEGER, h INTEGER, -- 缩略图尺寸(页面画布用)
|
||||
auto_corners TEXT, -- 原图像素,自动基线
|
||||
auto_corners_norm TEXT, -- 归一化,页面“恢复自动”用
|
||||
corners TEXT, -- 归一化,当前角点(未旋转)
|
||||
auto_rotate INTEGER, -- 自动方向基线(0/90)
|
||||
rotate INTEGER, -- 当前应用旋转(用户覆盖后是绝对值)
|
||||
deleted INTEGER, sort INTEGER,
|
||||
status TEXT, reason TEXT, thumb TEXT, error TEXT,
|
||||
PRIMARY KEY(batch_id, idx)
|
||||
```
|
||||
|
||||
### doc_review_job
|
||||
|
||||
```sql
|
||||
batch_id TEXT PRIMARY KEY,
|
||||
edits TEXT, -- review/1 完整快照
|
||||
status TEXT, -- queued/running/done/error
|
||||
message TEXT,
|
||||
requested_at TEXT, started_at TEXT, finished_at TEXT
|
||||
```
|
||||
|
||||
## 审核页如何工作
|
||||
|
||||
1. 页面从 `/app/<task_id>/doc-review/` 加载;无 `?b=` 时查 `doc_review_batch` 最新一行。
|
||||
2. 通过 `POST /api/db/<task_id>/query` 读 `doc_review_batch` / `doc_review_page`。
|
||||
3. 用户拖角/旋转/删页/排序/改四项参数,只改浏览器内状态并写 localStorage 草稿。
|
||||
4. 点「重新生成 PDF」:
|
||||
- `INSERT ... ON CONFLICT` 写 `doc_review_job`(`status=queued`,`edits` 是完整快照);
|
||||
- 调通用执行接口
|
||||
`POST /api/sessions/<task_id>/skills/document-photo-to-pdf/run`,body `{args:{batch_id}}`;
|
||||
- 技能读 job → 应用 edits → 覆盖同名 PDF → 更新 batch/page/job;
|
||||
- 页面重新读 data.db,刷新警告、状态、PDF 链接。
|
||||
5. 全程不经过模型,也不需要用户复制 JSON。
|
||||
|
||||
通用接口实现在 `app/server/routes/skill-run.js`:core 只做会话/并发/错误映射,执行仍走
|
||||
`tools/skills.runSkill`,不感知技能名或参数语义。
|
||||
|
||||
## 页面交互约定
|
||||
|
||||
- 顶部标题条(白底,参考 H5 全屏预览):左侧 PDF 名,右侧下载 PDF,最右 X 返回会话。
|
||||
- 加载失败(无批次 / `data.db` 缺 `doc_review_batch` 表 / data.db 被删)只保留标题条:
|
||||
X 仍可返回会话、下载置灰,编辑区与底栏隐藏。所以 X 的绑定在 `start()` 里,不能放 `boot()`。
|
||||
- 重新生成 PDF 不在顶栏:放在照片区底部(stage 与预览各一个,按当前视图显示;桌面只用
|
||||
stage),仅在存在未保存修改时出现。
|
||||
- 底部只保留翻页和当前页名,不再放操作按钮。
|
||||
- 页码不占工具栏:`1/n` 浮在图片右上角(裁切区、预览区各一个,按当前视图显示);
|
||||
左上角常驻当前模式标签(裁切区「手动定位」、预览区「校正预览」)。
|
||||
- 桌面端左右分栏常显(左裁切、右预览),翻页按钮浮在裁切图左右两侧(48px 圆形),
|
||||
顶栏只留旋转 / 重置 / 设置;移动端没有分栏,翻页靠滑动 + 底栏箭头。
|
||||
- 移动端顶栏按钮带文字而非只靠图标:切换 / 旋转 / 重置 / 设置;视图按钮固定叫「切换」,
|
||||
当前模式看图片左上角标签。
|
||||
- 裁切区/预览区支持左右滑动翻页:左滑下一页、右滑上一页;仅触摸/笔生效,拖角优先,
|
||||
按钮点击不触发。
|
||||
- 是否 dirty 用 `collect()` 与加载时的基线 `S.savedEdits` 比较;拖角、旋转、删页/排序、
|
||||
改四项参数都会触发。
|
||||
- 重新生成成功后重读 DB、清本地草稿、把新状态设为基线,按钮自动消失;失败则保留以便重试。
|
||||
- X 返回 `/?s=<task_id>`;独立页面里也适用。
|
||||
- 手机默认进校正预览,点左上角切到裁切。
|
||||
|
||||
## edits / 旋转语义
|
||||
|
||||
```json
|
||||
{"schema":"review/1",
|
||||
"batch":{"paper":"a4","enhance":"normal","dpi":"standard","orientation":"auto"},
|
||||
"pages":[{"i":3,"c":[[0.03,0.05],[0.97,0.04],[0.96,0.95],[0.04,0.96]],"r":90}],
|
||||
"order":[1,2,3],"deleted":[]}
|
||||
```
|
||||
|
||||
- `i` 是原始 images 下标,不随页序变化。
|
||||
- `c` 是 exif_transpose 后原图的归一化坐标,顺序左上/右上/右下/左下。
|
||||
- `r` 是该页最终顺时针旋转(0/90/180/270),覆盖自动方向;页面总会为“人工页”带上
|
||||
`r`(包括 0),服务端用 `user_r is not None` 判断是否覆盖。
|
||||
- `pages` 只放与自动基线不同的页;未改的页由服务端重新检测/评级,避免把 fallback 页
|
||||
误升成 ready。
|
||||
- `order` 缺省按 images 原序;`deleted` 的页不进出件。
|
||||
- 人工旋转 90/270 时,`orientation=auto` 下该页画布按旋转后的有效方向走,避免把横向内容
|
||||
压进竖向画布;`portrait`/`landscape` 强制方向时仍按强制值。
|
||||
|
||||
## state v2
|
||||
|
||||
每次成功出件写 `<output>.state.json`:
|
||||
|
||||
```json
|
||||
{"schema":"document-photo-to-pdf/2",
|
||||
"batch":{"doc_type":null,"paper":"auto","enhance":"normal","dpi":150,"orientation":"auto"},
|
||||
"pages":[{"index":1,"source":"p1.jpg","source_size":[1400,1700],
|
||||
"auto_corners":[[...]],"corners_norm":[[...]],
|
||||
"rotate":0,"rotate_source":"auto","status":"ready","reason":null}]}
|
||||
```
|
||||
|
||||
`rotate` 是自动基线(不是用户覆盖值);用户覆盖值在 data.db 的 `rotate` 列。state 只用于
|
||||
恢复自动/调试,审核页不读它。
|
||||
|
||||
## 状态与失败
|
||||
|
||||
- `ready`:可直接用。
|
||||
- `review`:建议看一眼(掩膜回退、覆盖率低、纸型存疑)。
|
||||
- `needs_corners`:检测为空或校验不过,已用“画面内缩 3%”兜底框出件,需人工补角。
|
||||
- `needs_orientation`:接近正方形,横竖判不出来,需人在页面旋转。
|
||||
- `render_failed` / `failed`:出页失败/原图打不开;后者会被跳过。
|
||||
- `deleted`:用户删除,不进出件。
|
||||
|
||||
单页检测/校验失败不再让整批失败;只有一页都出不了才 `ok:false`。检测到的无效四边形会
|
||||
退回兜底框,并把 `auto_corners` 一起换成兜底框,避免“恢复自动”把无效角点拖回来。
|
||||
|
||||
## 清理
|
||||
|
||||
审核页发布只保留最近 5 个批次的缩略图目录;新批次完成后删除更早的目录和 DB 记录。
|
||||
`batch_id` 重生成(apply)不会触发清理,避免把正在编辑的旧批次裁掉。
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
name: document-photo-to-pdf
|
||||
description: 纸质文档(合同/发票/证件/书页)拍照/扫描件校正后,导出 PDF 和 out/ 下的单页 PNG 原图(未调色未缩放);自动找纸边、透视拉正、多页合并,附审核页微调
|
||||
visit_url: /app/{task_id}/doc-review/
|
||||
input_schema:
|
||||
type: object
|
||||
properties:
|
||||
images:
|
||||
type: array
|
||||
items: {type: string}
|
||||
description: 图片路径,按合并顺序;可只传一张
|
||||
output:
|
||||
type: string
|
||||
description: 输出 PDF 文件名,默认 文档照片_校正.pdf
|
||||
---
|
||||
|
||||
# 文档照片转 PDF
|
||||
|
||||
```json
|
||||
{"images": ["图1.jpg", "图2.jpg"], "output": "合同_校正.pdf"}
|
||||
```
|
||||
|
||||
`images` 必填,按顺序合并;`output` 可选,默认 `文档照片_校正.pdf`。纸张、方向、调色、
|
||||
清晰度不要传,交给审核页。
|
||||
|
||||
## 返回值
|
||||
|
||||
- `ok:true`:`summary` 一句话摘要,`pdf` 文件名,`images` 是 `out/` 下的原图路径;
|
||||
末尾附完整「访问地址」。把 PDF、地址和图片路径一起给用户。
|
||||
- `ok:false`:转述 `error.message` / `error.hint`,忽略访问地址。
|
||||
|
||||
每次运行还会把校正后的单页原图(未调色、未缩放、无白边)导出到 `out/`,文件名与 PDF
|
||||
同名:`合同_校正.pdf` → `out/合同_校正-01.png`、`out/合同_校正-02.png`。不需要图片时
|
||||
忽略 `images` 即可。
|
||||
|
||||
```json
|
||||
{"ok":true,"summary":"3 页已生成;1 页建议在审核页确认;原始页图见 out/","pdf":"合同_校正.pdf","images":["out/合同_校正-01.png","out/合同_校正-02.png","out/合同_校正-03.png"]}
|
||||
访问地址:https://agent.ai.foresh.com/app/task-20261023-001/doc-review/
|
||||
```
|
||||
|
||||
审核页可拖角、旋转、删页/排序、改纸张与清晰度;改动后点「重新生成 PDF」覆盖同名文件。
|
||||
|
||||
## 失败与拍摄
|
||||
|
||||
- 图片打不开:确认路径与格式(jpg/png/webp/bmp/tif)。
|
||||
- 检测不到纸张:整张纸入画,四边留 3%~5% 背景,换反差大的桌面。
|
||||
- 只有全部页都出不了才 `ok:false`;单页失败会用兜底框出件,审核页会标出来。
|
||||
|
||||
## 边界
|
||||
|
||||
- 纸出画面无法还原;角点可外扩。
|
||||
- 自动方向只判 90°,180° 在审核页旋转。
|
||||
- 默认按同批中位比例;严格 A4/16K/B5 在审核页选。
|
||||
- 支持 jpg/png/webp/bmp/tif;HEIC 不支持。
|
||||
- 一图一纸;同框多纸只取最大。
|
||||
- 建议 ≤20 页;更多或 300dpi 可能超 120s,可分批。
|
||||
- `out/` 里的原图是自然尺寸(不缩放、不调色),页与页尺寸可能不同;PDF 仍按纸张/批次统一。
|
||||
@@ -0,0 +1,608 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""纸张边线检测:从一张已转正的 PIL 图里估四个纸角。
|
||||
|
||||
对外只有 detect(img, P) -> (corners, info):
|
||||
- img 是已做 exif_transpose 的 RGB 图;
|
||||
- corners 是原图像素坐标 [(x, y)],顺序 左上、右上、右下、左下;
|
||||
- info 里带覆盖率、纸面占比、回退标记,供上层做置信度告警。
|
||||
|
||||
设计要点:
|
||||
- 不打印、不落盘、不画图。检测失败返回 ([], info),由调用方决定是否终止。
|
||||
- 纸面掩膜保留两份:腐蚀过的只用于限定边缘搜索环带;回退几何用未腐蚀的
|
||||
原始掩膜,避免角点系统性内收(旧实现两次 erode 会让角点切进纸内)。
|
||||
- 回退时不再只用 4 个极值点连线,而是用整条掩膜边界按边拟合,再求交点。
|
||||
"""
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 基础工具
|
||||
|
||||
def _odd(size):
|
||||
size = int(size)
|
||||
if size < 3:
|
||||
size = 3
|
||||
if size % 2 == 0:
|
||||
size += 1
|
||||
return size
|
||||
|
||||
|
||||
def _slide_reduce(a, k, op, axis):
|
||||
"""沿 axis 做窗口 k 的滑动 max/min,越界按中性值填充(与 PIL 的边缘复制等价)。
|
||||
|
||||
van Herk / Gil-Werman:块内前缀极值 + 块内后缀极值,O(1)/像素。
|
||||
二维 max/min 可分离,故行列两遍与二维滤波逐像素等价,
|
||||
但避开了 PIL rankfilter 的 O(k^2)/像素(k=35 时慢 30 倍以上)。
|
||||
"""
|
||||
a = np.moveaxis(np.asarray(a, dtype=bool).astype(np.uint8) * 255,
|
||||
axis, -1)
|
||||
M, N = a.shape
|
||||
r = k // 2
|
||||
n = N + 2 * r
|
||||
nb = (n + k - 1) // k
|
||||
m = nb * k
|
||||
p = np.full((M, m), 0 if op == "max" else 255, dtype=np.uint8)
|
||||
p[:, r:r + N] = a
|
||||
blk = p.reshape(M, nb, k)
|
||||
if op == "max":
|
||||
fwd = np.maximum.accumulate(blk, axis=-1).reshape(M, m)
|
||||
bwd = np.maximum.accumulate(blk[:, :, ::-1], axis=-1)[:, :, ::-1]
|
||||
bwd = bwd.reshape(M, m)
|
||||
out = np.maximum(bwd[:, :N], fwd[:, k - 1:k - 1 + N])
|
||||
else:
|
||||
fwd = np.minimum.accumulate(blk, axis=-1).reshape(M, m)
|
||||
bwd = np.minimum.accumulate(blk[:, :, ::-1], axis=-1)[:, :, ::-1]
|
||||
bwd = bwd.reshape(M, m)
|
||||
out = np.minimum(bwd[:, :N], fwd[:, k - 1:k - 1 + N])
|
||||
return np.moveaxis(out, -1, axis) > 127
|
||||
|
||||
|
||||
def dilate(mask, size):
|
||||
k = _odd(size)
|
||||
return _slide_reduce(_slide_reduce(mask, k, "max", 1), k, "max", 0)
|
||||
|
||||
|
||||
def erode(mask, size):
|
||||
k = _odd(size)
|
||||
return _slide_reduce(_slide_reduce(mask, k, "min", 1), k, "min", 0)
|
||||
|
||||
|
||||
def closing(mask, size):
|
||||
return erode(dilate(mask, size), size)
|
||||
|
||||
|
||||
def opening(mask, size):
|
||||
return dilate(erode(mask, size), size)
|
||||
|
||||
|
||||
def rgb_to_sv(arr):
|
||||
maxc = arr.max(axis=-1)
|
||||
minc = arr.min(axis=-1)
|
||||
delta = maxc - minc
|
||||
S = np.zeros_like(maxc, dtype=np.float32)
|
||||
np.divide(delta, maxc, out=S, where=maxc > 1e-6)
|
||||
return S.astype(np.float32), maxc.astype(np.float32)
|
||||
|
||||
|
||||
def luminance(arr):
|
||||
return (0.299 * arr[..., 0] + 0.587 * arr[..., 1]
|
||||
+ 0.114 * arr[..., 2]).astype(np.float32)
|
||||
|
||||
|
||||
def box_mean(img, r):
|
||||
"""O(1)/像素的方框均值(积分图),r 为半径。"""
|
||||
img = np.asarray(img, dtype=np.float32)
|
||||
H, W = img.shape
|
||||
pad = np.zeros((H + 1, W + 1), dtype=np.float32)
|
||||
pad[1:, 1:] = img
|
||||
ii = pad.cumsum(0).cumsum(1)
|
||||
ys = np.arange(H)
|
||||
xs = np.arange(W)
|
||||
y0 = np.clip(ys - r, 0, H)
|
||||
y1 = np.clip(ys + r + 1, 0, H)
|
||||
x0 = np.clip(xs - r, 0, W)
|
||||
x1 = np.clip(xs + r + 1, 0, W)
|
||||
# 边界裁剪等价于把积分图向外按边复制 r 行/列,这样四个角全部可用切片取,
|
||||
# 避开 np.ix_ 花式索引的 4 次全图 gather(快 3 倍以上)。
|
||||
k = 2 * r + 1
|
||||
pe = np.pad(ii, ((r, r + 1), (r, r + 1)), mode="edge")
|
||||
S = (pe[k:k + H, k:k + W] - pe[0:H, k:k + W]
|
||||
- pe[k:k + H, 0:W] + pe[0:H, 0:W])
|
||||
cnt = (y1 - y0)[:, None] * (x1 - x0)[None, :]
|
||||
return S / np.maximum(cnt, 1).astype(np.float32)
|
||||
|
||||
|
||||
def gaussian_blur_np(img, sigma):
|
||||
radius = int(np.ceil(sigma * 3))
|
||||
if radius < 1:
|
||||
return img.astype(np.float32)
|
||||
x = np.arange(-radius, radius + 1, dtype=np.float32)
|
||||
k = np.exp(-x * x / (2 * sigma * sigma))
|
||||
k /= k.sum()
|
||||
H, W = img.shape
|
||||
p = np.pad(img, radius, mode="edge")
|
||||
tmp = np.zeros((H, W), dtype=np.float32)
|
||||
for i, kv in enumerate(k):
|
||||
tmp += kv * p[radius:radius + H, i:i + W]
|
||||
p = np.pad(tmp, ((radius, radius), (0, 0)), mode="edge")
|
||||
out = np.zeros((H, W), dtype=np.float32)
|
||||
for i, kv in enumerate(k):
|
||||
out += kv * p[i:i + H, :]
|
||||
return out
|
||||
|
||||
|
||||
def convolve3x3(img, kernel):
|
||||
H, W = img.shape
|
||||
p = np.pad(img, 1, mode="edge")
|
||||
out = np.zeros((H, W), dtype=np.float32)
|
||||
for i in range(3):
|
||||
for j in range(3):
|
||||
kv = kernel[i, j]
|
||||
if kv != 0:
|
||||
out += kv * p[i:i + H, j:j + W]
|
||||
return out
|
||||
|
||||
|
||||
def sobel_magnitude(gray):
|
||||
kx = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], dtype=np.float32)
|
||||
ky = np.array([[-1, -2, -1], [0, 0, 0], [1, 2, 1]], dtype=np.float32)
|
||||
gx = convolve3x3(gray, kx)
|
||||
gy = convolve3x3(gray, ky)
|
||||
return np.sqrt(gx * gx + gy * gy)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 连通域 & 填孔
|
||||
|
||||
def downsample_any(mask, factor):
|
||||
if factor <= 1:
|
||||
return mask
|
||||
H, W = mask.shape
|
||||
H2, W2 = H // factor, W // factor
|
||||
return mask[:H2 * factor, :W2 * factor].reshape(
|
||||
H2, factor, W2, factor).any(axis=(1, 3))
|
||||
|
||||
|
||||
def upsample_nearest(mask_small, target_h, target_w):
|
||||
h, w = mask_small.shape
|
||||
yi = np.linspace(0, h - 1, target_h).astype(np.int32)
|
||||
xi = np.linspace(0, w - 1, target_w).astype(np.int32)
|
||||
return mask_small[np.ix_(yi, xi)]
|
||||
|
||||
|
||||
def _row_runs(mask):
|
||||
"""把二值掩膜拆成每行的连续段,返回 (row, start, end),end 不含。"""
|
||||
H, W = mask.shape
|
||||
pm = np.zeros((H, W + 2), dtype=np.int8)
|
||||
pm[:, 1:-1] = mask
|
||||
d = np.diff(pm, axis=1)
|
||||
ry, rs = np.nonzero(d == 1)
|
||||
_, re_ = np.nonzero(d == -1)
|
||||
return ry, rs, re_
|
||||
|
||||
|
||||
def _run_labels(ry, rs, re_, H, eight):
|
||||
"""行程级并查集:4/8 邻接,返回每个行程的连通分量标号。"""
|
||||
n = len(ry)
|
||||
parent = np.arange(n)
|
||||
|
||||
def find(a):
|
||||
while parent[a] != a:
|
||||
parent[a] = parent[parent[a]]
|
||||
a = parent[a]
|
||||
return a
|
||||
|
||||
row_start = np.searchsorted(ry, np.arange(H + 1))
|
||||
for y in range(H - 1):
|
||||
a0, a1 = row_start[y], row_start[y + 1]
|
||||
b0, b1 = row_start[y + 1], row_start[y + 2]
|
||||
i, j = a0, b0
|
||||
while i < a1 and j < b1:
|
||||
s1, e1 = rs[i], re_[i]
|
||||
s2, e2 = rs[j], re_[j]
|
||||
hit = (s2 <= e1 and s1 <= e2) if eight else (s2 < e1 and s1 < e2)
|
||||
if hit:
|
||||
ra, rb = find(i), find(j)
|
||||
if ra != rb:
|
||||
parent[rb] = ra
|
||||
if e1 <= e2:
|
||||
i += 1
|
||||
else:
|
||||
j += 1
|
||||
return np.array([find(i) for i in range(n)], dtype=np.int64)
|
||||
|
||||
|
||||
def fill_holes(mask_small):
|
||||
"""填掉不与画面边界连通的背景(4 邻接),行程 + 并查集实现。"""
|
||||
m = np.asarray(mask_small, dtype=bool)
|
||||
H, W = m.shape
|
||||
bg = ~m
|
||||
if not bg.any():
|
||||
return m.copy()
|
||||
ry, rs, re_ = _row_runs(bg)
|
||||
if len(ry) == 0:
|
||||
return m.copy()
|
||||
lab = _run_labels(ry, rs, re_, H, eight=False)
|
||||
touch = (ry == 0) | (ry == H - 1) | (rs == 0) | (re_ >= W)
|
||||
nlab = int(lab.max()) + 1
|
||||
edge = np.zeros(nlab, dtype=bool)
|
||||
np.logical_or.at(edge, lab, touch)
|
||||
sel = ~edge[lab]
|
||||
out = m.copy()
|
||||
for y, s, e in zip(ry[sel], rs[sel], re_[sel]):
|
||||
out[y, s:e] = True
|
||||
return out
|
||||
|
||||
|
||||
def largest_connected(mask_small):
|
||||
"""取最大 8 邻接连通域(行程 + 并查集,逐像素计数)。"""
|
||||
m = np.asarray(mask_small, dtype=bool)
|
||||
H, W = m.shape
|
||||
if not m.any():
|
||||
return np.zeros_like(m)
|
||||
ry, rs, re_ = _row_runs(m)
|
||||
lab = _run_labels(ry, rs, re_, H, eight=True)
|
||||
w = (re_ - rs).astype(np.int64)
|
||||
nlab = int(lab.max()) + 1
|
||||
size = np.bincount(lab, weights=w, minlength=nlab)
|
||||
best = int(np.argmax(size))
|
||||
sel = lab == best
|
||||
out = np.zeros((H, W), dtype=bool)
|
||||
for y, s, e in zip(ry[sel], rs[sel], re_[sel]):
|
||||
out[y, s:e] = True
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 纸面掩膜
|
||||
|
||||
def otsu_threshold(score):
|
||||
hist, bin_edges = np.histogram(score, bins=256, range=(0.0, 1.0))
|
||||
hist = hist.astype(np.float64)
|
||||
total = hist.sum()
|
||||
if total <= 0:
|
||||
return 0.5
|
||||
p = hist / total
|
||||
centers = (bin_edges[:-1] + bin_edges[1:]) * 0.5
|
||||
w0 = np.cumsum(p)
|
||||
w1 = 1.0 - w0
|
||||
m0 = np.cumsum(p * centers)
|
||||
mt = m0[-1]
|
||||
with np.errstate(divide="ignore", invalid="ignore"):
|
||||
mu0 = m0 / np.maximum(w0, 1e-12)
|
||||
mu1 = (mt - m0) / np.maximum(w1, 1e-12)
|
||||
between = w0 * w1 * (mu0 - mu1) ** 2
|
||||
between = np.nan_to_num(between, nan=-1.0)
|
||||
return float(centers[int(np.argmax(between))])
|
||||
|
||||
|
||||
def paper_masks(arr, P):
|
||||
"""返回 (raw, eroded)。
|
||||
|
||||
raw:未腐蚀的纸面掩膜,用于回退几何,避免角点内收。
|
||||
eroded:腐蚀后的掩膜,只用于限定边缘搜索环带。
|
||||
"""
|
||||
d = P["detect"]
|
||||
H, W = arr.shape[:2]
|
||||
S, V = rgb_to_sv(arr)
|
||||
score = (V * (1.0 - S)).astype(np.float32)
|
||||
th = max(otsu_threshold(score), d["paper_otsu_floor"])
|
||||
bright = (score > th) & (V > d["paper_v_min"]) & (S < d["paper_s_max"])
|
||||
|
||||
factor = max(1, min(H, W) // d["paper_mask_factor_div"])
|
||||
bs = downsample_any(bright, factor)
|
||||
|
||||
m = closing(bs, _odd(max(d["paper_close_min"],
|
||||
min(bs.shape) // d["paper_close_div"])))
|
||||
m = fill_holes(m)
|
||||
m = largest_connected(m)
|
||||
m = closing(m, _odd(max(d["paper_close_min"],
|
||||
min(bs.shape) // d["paper_close2_div"])))
|
||||
m = fill_holes(m)
|
||||
|
||||
raw_small = m
|
||||
eroded_small = erode(m, _odd(d["mask_erode_kernel"]))
|
||||
|
||||
if factor > 1:
|
||||
raw = upsample_nearest(raw_small, H, W)
|
||||
eroded = upsample_nearest(eroded_small, H, W)
|
||||
else:
|
||||
raw, eroded = raw_small, eroded_small
|
||||
return raw, eroded
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 边缘图
|
||||
|
||||
def boundary_band(paper, k):
|
||||
k = _odd(k)
|
||||
return dilate(paper, k) & ~erode(paper, k)
|
||||
|
||||
|
||||
def edge_map(arr, P):
|
||||
d = P["detect"]
|
||||
H = arr.shape[0]
|
||||
gray = luminance(arr)
|
||||
sigma = max(1.0, H / float(d["edge_sigma_div"]))
|
||||
blurred = gaussian_blur_np(gray, sigma)
|
||||
mag = sobel_magnitude(blurred)
|
||||
local = box_mean(mag, max(3, H // d["edge_box_div"]))
|
||||
th = np.maximum(d["edge_low"], local * d["edge_local_gain"])
|
||||
return mag > th
|
||||
|
||||
|
||||
def boundary_step_filter(arr, paper, P):
|
||||
"""跨纸边亮度阶跃:比较掩膜内侧/外侧小窗的 V 均值。
|
||||
|
||||
纸边是真实的明暗台阶,木纹/阴影纹理阶跃小。两级窗口:掩膜准确处用
|
||||
r_small,掩膜略有偏移处依靠 r_big 补上。
|
||||
"""
|
||||
d = P["detect"]
|
||||
_, V = rgb_to_sv(arr)
|
||||
p = paper.astype(np.float32)
|
||||
q = 1.0 - p
|
||||
|
||||
def side_means(r):
|
||||
din = box_mean(p, r)
|
||||
dout = box_mean(q, r)
|
||||
vin = box_mean(V * p, r) / np.maximum(din, 1e-6)
|
||||
vout = box_mean(V * q, r) / np.maximum(dout, 1e-6)
|
||||
ok = (din > 0.08) & (dout > 0.08)
|
||||
return np.where(ok, np.abs(vin - vout), 0.0)
|
||||
|
||||
step = np.maximum(side_means(d["step_r_small"]),
|
||||
side_means(d["step_r_big"]))
|
||||
return step > d["step_th"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 四角提取
|
||||
|
||||
def mask_boundary_points(mask, P):
|
||||
"""掩膜边界像素坐标 (N, 2),顺序 (x, y)。"""
|
||||
inner = erode(mask, _odd(P["detect"]["mask_erode_kernel"]))
|
||||
ys, xs = np.where(mask & ~inner)
|
||||
if len(xs) == 0:
|
||||
return np.zeros((0, 2), dtype=np.float32)
|
||||
return np.stack([xs, ys], axis=1).astype(np.float32)
|
||||
|
||||
|
||||
def _fit_line(pts):
|
||||
"""总体最小二乘拟直线,返回法向量 n 与 n·p = c。"""
|
||||
c0 = pts.mean(axis=0)
|
||||
q = pts - c0
|
||||
cov = (q.T @ q) / max(len(pts) - 1, 1)
|
||||
w, v = np.linalg.eigh(cov)
|
||||
direction = v[:, int(np.argmax(w))]
|
||||
n = np.array([-direction[1], direction[0]], dtype=np.float64)
|
||||
nn = np.linalg.norm(n)
|
||||
if nn < 1e-9:
|
||||
return np.array([0.0, 1.0]), float(c0[1])
|
||||
n = n / nn
|
||||
return n, float(n @ c0)
|
||||
|
||||
|
||||
def _robust_fit(pts, robust_px):
|
||||
n, c = _fit_line(pts)
|
||||
sel = pts
|
||||
for _ in range(2):
|
||||
res = np.abs(sel @ n - c)
|
||||
keep = sel[res <= max(robust_px, 1.0) * 2.5]
|
||||
if len(keep) < 40:
|
||||
break
|
||||
n, c = _fit_line(keep)
|
||||
sel = keep
|
||||
return n, c
|
||||
|
||||
|
||||
def _intersect(n1, c1, n2, c2):
|
||||
A = np.vstack([n1, n2])
|
||||
b = np.array([c1, c2], dtype=np.float64)
|
||||
try:
|
||||
return np.linalg.solve(A, b)
|
||||
except np.linalg.LinAlgError:
|
||||
return None
|
||||
|
||||
|
||||
def _side_line(p0, p1):
|
||||
dv = np.asarray(p1, dtype=np.float64) - np.asarray(p0, dtype=np.float64)
|
||||
L = float(np.linalg.norm(dv))
|
||||
if L < 1e-6:
|
||||
return None
|
||||
u = dv / L
|
||||
n = np.array([-u[1], u[0]], dtype=np.float64)
|
||||
return n, float(n @ np.asarray(p0, dtype=np.float64)), u, L
|
||||
|
||||
|
||||
def detect_quad_corners(paper_raw, points, P):
|
||||
"""从掩膜边界 + 边缘点估四角,返回 (corners, fallback_sides)。"""
|
||||
d = P["detect"]
|
||||
base = mask_boundary_points(paper_raw, P)
|
||||
if len(base) < 40:
|
||||
return [], [0, 1, 2, 3]
|
||||
|
||||
bx, by = base[:, 0], base[:, 1]
|
||||
anchors = np.array([
|
||||
base[int(np.argmin(bx + by))], # 左上
|
||||
base[int(np.argmax(bx - by))], # 右上
|
||||
base[int(np.argmax(bx + by))], # 右下
|
||||
base[int(np.argmin(bx - by))], # 左下
|
||||
], dtype=np.float64)
|
||||
|
||||
init = [_side_line(anchors[i], anchors[(i + 1) % 4]) for i in range(4)]
|
||||
|
||||
# 用整条掩膜边界重拟合每一边(排除两端,避开角点圆角)
|
||||
base_lines = []
|
||||
for i in range(4):
|
||||
sl = init[i]
|
||||
if sl is None:
|
||||
base_lines.append(None)
|
||||
continue
|
||||
n, c, u, L = sl
|
||||
p0 = anchors[i]
|
||||
dist = np.abs(base @ n - c)
|
||||
proj = (base - p0) @ u
|
||||
m = d["boundary_side_margin"]
|
||||
sel = base[(dist <= d["group_tol_px"]) &
|
||||
(proj >= L * m) & (proj <= L * (1.0 - m))]
|
||||
if len(sel) >= 40:
|
||||
n2, c2 = _robust_fit(sel, d["robust_px"])
|
||||
base_lines.append((n2, c2))
|
||||
else:
|
||||
base_lines.append((n, c))
|
||||
|
||||
final_lines = list(base_lines)
|
||||
fallback_sides = [0, 1, 2, 3]
|
||||
|
||||
if points is not None and len(points) >= d["min_edge_points"]:
|
||||
pts = np.asarray(points, dtype=np.float64)
|
||||
dists = []
|
||||
for line in base_lines:
|
||||
if line is None:
|
||||
dists.append(np.full(len(pts), 1e9))
|
||||
else:
|
||||
dists.append(np.abs(pts @ line[0] - line[1]))
|
||||
D = np.stack(dists, axis=1)
|
||||
idx = np.argmin(D, axis=1)
|
||||
dmin = D[np.arange(len(pts)), idx]
|
||||
|
||||
fallback_sides = []
|
||||
for i in range(4):
|
||||
line = base_lines[i]
|
||||
if line is None:
|
||||
fallback_sides.append(i)
|
||||
continue
|
||||
sel = pts[(idx == i) & (dmin <= d["group_tol_px"])]
|
||||
sl = init[i]
|
||||
if sl is not None:
|
||||
u, L = sl[2], sl[3]
|
||||
p0 = anchors[i]
|
||||
m = d["boundary_side_margin"]
|
||||
proj = (sel - p0) @ u
|
||||
sel = sel[(proj >= L * m) & (proj <= L * (1.0 - m))]
|
||||
if len(sel) >= d["min_edge_points"]:
|
||||
n2, c2 = _robust_fit(sel, d["robust_px"])
|
||||
final_lines[i] = (n2, c2)
|
||||
else:
|
||||
fallback_sides.append(i)
|
||||
|
||||
H, W = paper_raw.shape
|
||||
oob = d["corner_oob_frac"]
|
||||
out = []
|
||||
for i in range(4):
|
||||
chosen = None
|
||||
for lines in (final_lines, base_lines):
|
||||
l1, l2 = lines[i - 1], lines[i]
|
||||
if l1 is None or l2 is None:
|
||||
continue
|
||||
det = l1[0][0] * l2[0][1] - l2[0][0] * l1[0][1]
|
||||
if abs(det) < 1e-3:
|
||||
continue
|
||||
p = _intersect(l1[0], l1[1], l2[0], l2[1])
|
||||
if p is None:
|
||||
continue
|
||||
if (-oob * W <= p[0] <= (1.0 + oob) * W and
|
||||
-oob * H <= p[1] <= (1.0 + oob) * H):
|
||||
chosen = (float(p[0]), float(p[1]))
|
||||
break
|
||||
if chosen is None:
|
||||
chosen = (float(anchors[i][0]), float(anchors[i][1]))
|
||||
if i not in fallback_sides:
|
||||
fallback_sides.append(i)
|
||||
out.append(chosen)
|
||||
|
||||
return out, sorted(set(fallback_sides))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 覆盖率
|
||||
|
||||
def border_coverage_exact(edges, paper, tol):
|
||||
"""沿掩膜逐行/逐列的真实边界测边缘覆盖率。"""
|
||||
H, W = paper.shape
|
||||
col_has = paper.any(axis=0)
|
||||
row_has = paper.any(axis=1)
|
||||
top_hit = top_n = 0
|
||||
bot_hit = bot_n = 0
|
||||
for x in np.where(col_has)[0]:
|
||||
ys = np.where(paper[:, x])[0]
|
||||
y0, y1 = ys.min(), ys.max()
|
||||
if edges[max(0, y0 - tol):min(H, y0 + tol + 1), x].any():
|
||||
top_hit += 1
|
||||
top_n += 1
|
||||
if edges[max(0, y1 - tol):min(H, y1 + tol + 1), x].any():
|
||||
bot_hit += 1
|
||||
bot_n += 1
|
||||
left_hit = left_n = 0
|
||||
right_hit = right_n = 0
|
||||
for y in np.where(row_has)[0]:
|
||||
xs = np.where(paper[y, :])[0]
|
||||
x0, x1 = xs.min(), xs.max()
|
||||
if edges[y, max(0, x0 - tol):min(W, x0 + tol + 1)].any():
|
||||
left_hit += 1
|
||||
left_n += 1
|
||||
if edges[y, max(0, x1 - tol):min(W, x1 + tol + 1)].any():
|
||||
right_hit += 1
|
||||
right_n += 1
|
||||
return {
|
||||
"top": top_hit / max(top_n, 1),
|
||||
"bottom": bot_hit / max(bot_n, 1),
|
||||
"left": left_hit / max(left_n, 1),
|
||||
"right": right_hit / max(right_n, 1),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 对外接口
|
||||
|
||||
def detect(img, P):
|
||||
"""检测纸张四角,返回 (corners_原图坐标, info)。失败返回 ([], info)。"""
|
||||
d = P["detect"]
|
||||
src_w, src_h = img.size
|
||||
scale = min(1.0, d["max_side"] / float(max(src_w, src_h)))
|
||||
if scale < 1.0:
|
||||
pil = img.resize((int(round(src_w * scale)), int(round(src_h * scale))),
|
||||
Image.LANCZOS)
|
||||
else:
|
||||
pil = img
|
||||
arr = np.asarray(pil).astype(np.float32) / 255.0
|
||||
H, W = arr.shape[:2]
|
||||
|
||||
paper_raw, paper_eroded = paper_masks(arr, P)
|
||||
frac = float(paper_raw.sum()) / float(H * W)
|
||||
|
||||
info = {
|
||||
"source_size": [int(src_w), int(src_h)],
|
||||
"processed_size": [int(W), int(H)],
|
||||
"scale": round(scale, 6),
|
||||
"paper_frac": round(frac, 4),
|
||||
"edge_pixels": 0,
|
||||
"coverage": {},
|
||||
"fallback": True,
|
||||
"fallback_sides": [0, 1, 2, 3],
|
||||
}
|
||||
|
||||
if frac < 0.005 or not paper_raw.any():
|
||||
return [], info
|
||||
|
||||
band = boundary_band(paper_eroded,
|
||||
_odd(max(d["band_min"], H // d["band_div"])))
|
||||
edges = edge_map(arr, P)
|
||||
step = boundary_step_filter(arr, paper_eroded, P)
|
||||
cand = closing(edges & band & step,
|
||||
_odd(max(d["cand_close_min"], H // d["cand_close_div"])))
|
||||
cand = opening(cand,
|
||||
_odd(max(d["cand_open_min"], H // d["cand_open_div"])))
|
||||
|
||||
ey, ex = np.nonzero(cand)
|
||||
pts = (np.stack([ex, ey], axis=1).astype(np.float32)
|
||||
if len(ex) >= d["min_edge_points"] else None)
|
||||
|
||||
corners, fallback_sides = detect_quad_corners(paper_raw, pts, P)
|
||||
info["edge_pixels"] = int(cand.sum())
|
||||
info["fallback_sides"] = fallback_sides
|
||||
info["fallback"] = bool(fallback_sides)
|
||||
if not corners:
|
||||
return [], info
|
||||
|
||||
info["coverage"] = {
|
||||
k: float(v) for k, v in border_coverage_exact(
|
||||
cand, paper_eroded,
|
||||
tol=max(d["coverage_tol_min"], H // d["coverage_tol_div"])).items()
|
||||
}
|
||||
src_pts = [(float(x / scale), float(y / scale)) for x, y in corners]
|
||||
return src_pts, info
|
||||
@@ -0,0 +1,286 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""PDF 产出的数值检验:渲染真实页面后评估,不画图、不修改文件。
|
||||
|
||||
关键修正(相对旧版):
|
||||
- “边缘背景残留 / 内容贴边”改为在 paper_box 裁出的纸面区域上量。
|
||||
旧版量的是 PDF 页面最外圈,而正常出件四周是白边,这两项恒为通过。
|
||||
- 渲染页按文件名数字自然排序,≥10 页不会出现 1,10,2 的顺序。
|
||||
- 渲染后端增加 PyMuPDF(executor 镜像已装 pymupdf),pdftoppm 不在时也能跑。
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
from params import P # noqa: E402
|
||||
|
||||
|
||||
def _natural_key(path):
|
||||
name = os.path.basename(path)
|
||||
out = []
|
||||
cur = ""
|
||||
for ch in name:
|
||||
if ch.isdigit():
|
||||
cur += ch
|
||||
else:
|
||||
if cur:
|
||||
out.append((1, int(cur)))
|
||||
cur = ""
|
||||
out.append((0, ch.lower()))
|
||||
if cur:
|
||||
out.append((1, int(cur)))
|
||||
return out
|
||||
|
||||
|
||||
def otsu(vals):
|
||||
v = np.asarray(vals, dtype=np.float64).ravel()
|
||||
if v.size == 0:
|
||||
return 128.0
|
||||
hist, edges = np.histogram(v, bins=64)
|
||||
centers = (edges[:-1] + edges[1:]) / 2.0
|
||||
total = hist.sum()
|
||||
w0 = np.cumsum(hist)
|
||||
w1 = total - w0
|
||||
m0 = np.cumsum(hist * centers)
|
||||
mt = m0[-1]
|
||||
with np.errstate(divide="ignore", invalid="ignore"):
|
||||
score = (mt * w0 / total - m0) ** 2 / (w0 * w1 / total ** 2)
|
||||
score = np.nan_to_num(score)
|
||||
return float(centers[int(np.argmax(score))])
|
||||
|
||||
|
||||
def render(pdf, dpi=100, outdir=None):
|
||||
"""把 PDF 逐页渲染成图片,返回按页序排好的文件列表。
|
||||
|
||||
pdftoppm 默认输出 PPM(无损,且不压缩)。之前用 -png 时 zlib 压缩占了
|
||||
整个自检 2 秒里的 2 秒;改 PPM 后像素完全一致,耗时降到 0.1 秒量级。
|
||||
"""
|
||||
outdir = outdir or P["check"]["tmp_dir"]
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
for f in os.listdir(outdir):
|
||||
if f.startswith("page") and (f.endswith(".png") or f.endswith(".ppm")):
|
||||
try:
|
||||
os.remove(os.path.join(outdir, f))
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
base = os.path.join(outdir, "page")
|
||||
done = False
|
||||
if shutil.which("pdftoppm"):
|
||||
try:
|
||||
subprocess.run(["pdftoppm", "-r", str(dpi), pdf, base],
|
||||
check=True, stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL)
|
||||
done = True
|
||||
except Exception:
|
||||
done = False
|
||||
|
||||
if not done:
|
||||
try:
|
||||
import fitz # PyMuPDF
|
||||
doc = fitz.open(pdf)
|
||||
scale = float(dpi) / 72.0
|
||||
for i, page in enumerate(doc):
|
||||
pix = page.get_pixmap(matrix=fitz.Matrix(scale, scale))
|
||||
pix.save("%s-%d.png" % (base, i + 1))
|
||||
doc.close()
|
||||
except Exception:
|
||||
import pypdfium2 as pdfium
|
||||
doc = pdfium.PdfDocument(pdf)
|
||||
scale = float(dpi) / 72.0
|
||||
for i in range(len(doc)):
|
||||
doc[i].render(scale=scale).to_pil().save(
|
||||
"%s-%d.png" % (base, i + 1))
|
||||
|
||||
files = [os.path.join(outdir, f) for f in os.listdir(outdir)
|
||||
if f.startswith("page") and (f.endswith(".png")
|
||||
or f.endswith(".ppm"))]
|
||||
files.sort(key=_natural_key)
|
||||
return files
|
||||
|
||||
|
||||
def analyse(img, params=None):
|
||||
"""分析一张渲染页,返回纸面框、亮度、边缘与贴边指标。"""
|
||||
c = (params or P)["check"]
|
||||
g = np.asarray(img.convert("L")).astype(np.float32)
|
||||
h, w = g.shape
|
||||
res = {"w": int(w), "h": int(h), "ratio": float(w) / max(1, h)}
|
||||
|
||||
# 先用“非纯白”找纸面框:PDF 页面四周是白边,纸面被提亮到 ~245
|
||||
flat = float(np.percentile(g, c["flat_pct"]))
|
||||
nonwhite = g < (flat - c["nonwhite_delta"])
|
||||
row_frac = nonwhite.mean(axis=1)
|
||||
col_frac = nonwhite.mean(axis=0)
|
||||
rows = np.where(row_frac > c["paper_row_frac"])[0]
|
||||
cols = np.where(col_frac > c["paper_row_frac"])[0]
|
||||
box = None
|
||||
if rows.size > 4 and cols.size > 4:
|
||||
box = (int(cols[0]), int(rows[0]), int(cols[-1]), int(rows[-1]))
|
||||
res["paper_box"] = box
|
||||
res["box_fallback"] = box is None
|
||||
|
||||
if box:
|
||||
x0, y0, x1, y1 = box
|
||||
gp = g[y0:y1 + 1, x0:x1 + 1]
|
||||
else:
|
||||
gp = g
|
||||
ph, pw = gp.shape
|
||||
|
||||
# 透视采样在纸边常带 1~2px 背景毛边,先内缩再量边缘;
|
||||
# 否则会被误判成“边缘背景残留”或“内容贴边”。
|
||||
inset = max(2, int(round(min(pw, ph) * c["edge_inset_frac"])))
|
||||
if pw > 2 * inset + 4 and ph > 2 * inset + 4:
|
||||
gc = gp[inset:-inset, inset:-inset]
|
||||
else:
|
||||
gc = gp
|
||||
gh, gw = gc.shape
|
||||
|
||||
res["paper"] = float(np.percentile(gc, c["paper_pct"]))
|
||||
res["back"] = float(np.percentile(gc, c["back_pct"]))
|
||||
res["ink"] = min(otsu(gc), res["paper"] * c["ink_ratio"])
|
||||
res["paper_ratio"] = float(pw) / max(1, ph)
|
||||
res["paper_px"] = (int(pw), int(ph))
|
||||
|
||||
# 边缘背景残留:在纸面区域的最外圈量,而不是 PDF 白边
|
||||
b = max(4, int(min(gw, gh) * c["border_band_frac"]))
|
||||
border = {
|
||||
"top": float(gc[:b, :].mean()),
|
||||
"bottom": float(gc[-b:, :].mean()),
|
||||
"left": float(gc[:, :b].mean()),
|
||||
"right": float(gc[:, -b:].mean()),
|
||||
}
|
||||
res["border"] = border
|
||||
res["border_bad"] = [k for k, v in border.items()
|
||||
if v < res["paper"] * c["border_ratio"]]
|
||||
|
||||
edge_dark = {}
|
||||
for k, band in (("top", gc[:b, :]), ("bottom", gc[-b:, :]),
|
||||
("left", gc[:, :b]), ("right", gc[:, -b:])):
|
||||
edge_dark[k] = float(
|
||||
(band < res["paper"] * c["edge_dark_ratio"]).mean())
|
||||
res["edge_dark"] = edge_dark
|
||||
|
||||
# 内容贴边:纸面最外窄带内成片文字级暗像素
|
||||
m = max(2, int(min(gw, gh) * c["cut_band_frac"]))
|
||||
cut = []
|
||||
for k, band in (("top", gc[:m, :]), ("bottom", gc[-m:, :]),
|
||||
("left", gc[:, :m]), ("right", gc[:, -m:])):
|
||||
frac = float((band < res["ink"]).mean())
|
||||
if (frac > c["cut_frac"] and
|
||||
float(band.mean()) < res["paper"] * c["cut_mean_ratio"]):
|
||||
cut.append((k, frac))
|
||||
res["cut"] = cut
|
||||
return res
|
||||
|
||||
|
||||
def inspect_pdf(pdf, expect_pages=None, dpi=None, params=None):
|
||||
"""渲染 + 逐页检验,返回结构化结果;渲染失败时 ok=None。"""
|
||||
p = params or P
|
||||
c = p["check"]
|
||||
dpi = int(dpi or c["dpi"])
|
||||
outdir = os.path.join(os.path.dirname(os.path.abspath(pdf)), c["tmp_dir"])
|
||||
try:
|
||||
files = render(pdf, dpi, outdir)
|
||||
except Exception as e:
|
||||
return {"ok": None, "error": str(e), "pages": [], "page_count": None}
|
||||
|
||||
actual = len(files)
|
||||
count_ok = (expect_pages is None) or (actual == expect_pages)
|
||||
out = {
|
||||
"ok": bool(count_ok),
|
||||
"page_count": {"expected": expect_pages, "actual": actual,
|
||||
"ok": bool(count_ok)},
|
||||
"pages": [],
|
||||
}
|
||||
|
||||
# 阈值以 150dpi 出件为基准;check 用的是较低的渲染 dpi,要同时按出件 dpi
|
||||
# 折算,否则选 300dpi 会白白放宽、选 72dpi 会误报“分辨率不足”。
|
||||
render_dpi = float(p["render"]["dpi"])
|
||||
min_short = c["min_short"] * dpi / render_dpi
|
||||
try:
|
||||
for i, path in enumerate(files, 1):
|
||||
with Image.open(path) as im:
|
||||
r = analyse(im, p)
|
||||
pw, ph = r["paper_px"]
|
||||
short = min(pw, ph)
|
||||
short_ok = short >= min_short
|
||||
border_ok = ((not r["border_bad"]) and
|
||||
all(v <= c["edge_dark_frac"]
|
||||
for v in r["edge_dark"].values()))
|
||||
cut_ok = not r["cut"]
|
||||
dev = (abs(r["paper_ratio"] - r["ratio"]) / r["ratio"]
|
||||
if r["ratio"] else 0.0)
|
||||
|
||||
item = {
|
||||
"index": i,
|
||||
"paper_px": [int(pw), int(ph)],
|
||||
"short_side_px": int(short),
|
||||
"short_side_ok": bool(short_ok),
|
||||
"paper_ratio": round(float(r["paper_ratio"]), 4),
|
||||
"page_ratio": round(float(r["ratio"]), 4),
|
||||
"ratio_dev_pct": round(float(dev * 100), 1),
|
||||
"border_ok": bool(border_ok),
|
||||
"cut_ok": bool(cut_ok),
|
||||
"border": {k: round(float(v), 1) for k, v in r["border"].items()},
|
||||
"edge_dark": {k: round(float(v), 3)
|
||||
for k, v in r["edge_dark"].items()},
|
||||
"cut": [[k, round(float(v), 3)] for k, v in r["cut"]],
|
||||
"box_fallback": bool(r["box_fallback"]),
|
||||
"paper_luma": round(float(r["paper"]), 1),
|
||||
"back_luma": round(float(r["back"]), 1),
|
||||
}
|
||||
out["pages"].append(item)
|
||||
if not (short_ok and border_ok and cut_ok):
|
||||
out["ok"] = False
|
||||
finally:
|
||||
# 渲染图只是自检的中间产物:PPM 未压缩,100dpi 的 A4 一页约 3MB,
|
||||
# 检验完立刻删(量已经记录在返回值里),不要留在会话目录占空间。
|
||||
shutil.rmtree(outdir, ignore_errors=True)
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="复检已有 PDF(只渲染检验)")
|
||||
ap.add_argument("pdf")
|
||||
ap.add_argument("--expect-pages", type=int, default=None)
|
||||
ap.add_argument("--dpi", type=int, default=P["check"]["dpi"])
|
||||
args = ap.parse_args()
|
||||
|
||||
if not os.path.exists(args.pdf):
|
||||
raise SystemExit("找不到 PDF:%s" % args.pdf)
|
||||
|
||||
r = inspect_pdf(args.pdf, args.expect_pages, args.dpi)
|
||||
print("PDF: %s" % args.pdf)
|
||||
print("文件大小: %.0f KB" % (os.path.getsize(args.pdf) / 1024.0))
|
||||
if r.get("ok") is None:
|
||||
print("渲染失败:%s" % r.get("error"))
|
||||
return 1
|
||||
print("页数: %d" % r["page_count"]["actual"])
|
||||
for item in r["pages"]:
|
||||
print("\n第 %d 页 纸面 %dx%d" % (
|
||||
item["index"], item["paper_px"][0], item["paper_px"][1]))
|
||||
print(" [%s] 纸面短边 %d px"
|
||||
% ("OK" if item["short_side_ok"] else "XX",
|
||||
item["short_side_px"]))
|
||||
print(" [%s] 边缘背景残留 %s"
|
||||
% ("OK" if item["border_ok"] else "XX",
|
||||
"无" if item["border_ok"] else "有"))
|
||||
print(" [%s] 内容贴边 %s"
|
||||
% ("OK" if item["cut_ok"] else "XX",
|
||||
"无" if item["cut_ok"] else
|
||||
"、".join("%s %.1f%%" % (k, v * 100)
|
||||
for k, v in item["cut"])))
|
||||
print(" [--] 纸面比例 %.3f(页面 %.3f)"
|
||||
% (item["paper_ratio"], item["page_ratio"]))
|
||||
print("\n总判定:%s" % ("全部通过" if r["ok"] else "存在不达标项"))
|
||||
return 0 if r["ok"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,224 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""出页与合成:批次方向、目标尺寸、页面拼版、PDF 保存。
|
||||
|
||||
坐标与尺寸约定:
|
||||
- 所有 quad 都是“转正后原图”坐标;
|
||||
- 目标尺寸把 PDF 画布内容区(页面尺寸 - 四周留白)的一条边填满,可放大低分辨率照片,
|
||||
避免同一批页面因原图像素多少而大小不一;
|
||||
- raw 页图(裁剪 + 透视拉正、自然尺寸、不调色)由 warp_raw 产出,PDF 页从它派生。
|
||||
"""
|
||||
import os
|
||||
import statistics
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
from homography import brighten, warp_perspective, warp_perspective_pil
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 纸张
|
||||
|
||||
def resolve_paper(paper, P):
|
||||
"""把 paper 参数解析成 {name, canvas_mm, ratio}。
|
||||
|
||||
auto 不指定标准纸面比例;画布用 A4,比例由调用方按批次中位数决定
|
||||
a4/letter… 画布和纸面比例都用该标准纸
|
||||
数字 画布用 A4,纸面比例用该数字(宽/高)
|
||||
"""
|
||||
r = P["render"]
|
||||
name = str(paper or r["default_paper"]).strip().lower()
|
||||
a4 = r["papers_mm"]["a4"]
|
||||
if name in ("", "auto"):
|
||||
return {"name": "auto", "canvas_mm": a4, "ratio": None}
|
||||
if name in r["papers_mm"]:
|
||||
w_mm, h_mm = r["papers_mm"][name]
|
||||
return {"name": name, "canvas_mm": (w_mm, h_mm),
|
||||
"ratio": float(w_mm) / float(h_mm)}
|
||||
try:
|
||||
ratio = float(name)
|
||||
except ValueError:
|
||||
raise ValueError("未知纸张:%s(可选 %s / 数字宽高比 / auto)"
|
||||
% (paper, "/".join(r["papers_mm"])))
|
||||
if not (0.02 < ratio < 50.0):
|
||||
raise ValueError("纸张比例超出合理范围:%s" % paper)
|
||||
return {"name": "custom", "canvas_mm": a4, "ratio": ratio}
|
||||
|
||||
|
||||
def canvas_px(paper, orientation, P):
|
||||
"""按 dpi 换算画布像素;orientation=landscape 时交换宽高。"""
|
||||
w_mm, h_mm = paper["canvas_mm"]
|
||||
dpi = float(P["render"]["dpi"])
|
||||
cw = int(round(w_mm / 25.4 * dpi))
|
||||
ch = int(round(h_mm / 25.4 * dpi))
|
||||
if orientation == "landscape":
|
||||
cw, ch = ch, cw
|
||||
return max(1, cw), max(1, ch)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 方向
|
||||
|
||||
def aspect_of(quad):
|
||||
"""四边形在屏幕坐标下的自然宽高(像素)。"""
|
||||
q = np.asarray(quad, dtype=np.float64)
|
||||
w = (np.linalg.norm(q[1] - q[0]) + np.linalg.norm(q[2] - q[3])) / 2.0
|
||||
h = (np.linalg.norm(q[3] - q[0]) + np.linalg.norm(q[2] - q[1])) / 2.0
|
||||
return max(float(w), 1e-6), max(float(h), 1e-6)
|
||||
|
||||
|
||||
def classify(aspect, deadband):
|
||||
if aspect > 1.0 + deadband:
|
||||
return "landscape"
|
||||
if aspect < 1.0 - deadband:
|
||||
return "portrait"
|
||||
return "square"
|
||||
|
||||
|
||||
def decide_orientation(aspects, mode, P):
|
||||
"""返回 (batch_orientation, source, median_aspect)。"""
|
||||
dead = P["gate"]["aspect_deadband"]
|
||||
median = float(statistics.median(aspects)) if aspects else 1.0
|
||||
if mode == "none":
|
||||
return "mixed", "none", median
|
||||
if mode in ("portrait", "landscape"):
|
||||
return mode, "forced", median
|
||||
classes = [classify(a, dead) for a in aspects]
|
||||
n_land = sum(1 for c in classes if c == "landscape")
|
||||
n_port = sum(1 for c in classes if c == "portrait")
|
||||
if n_land > n_port:
|
||||
return "landscape", "majority", median
|
||||
if n_port > n_land:
|
||||
return "portrait", "majority", median
|
||||
if n_land == 0 and n_port == 0:
|
||||
return "portrait", "default", median
|
||||
return "portrait", "tie", median
|
||||
|
||||
|
||||
def rotate_quad_cw(quad):
|
||||
"""把角点序列旋转一位,等价于把页面顺时针转 90°。"""
|
||||
q = list(quad)
|
||||
return [q[3], q[0], q[1], q[2]]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 目标尺寸
|
||||
|
||||
def target_size(quad, paper, page_orientation, P, out_ratio=None):
|
||||
"""返回 (target_w, target_h, (canvas_w, canvas_h))。
|
||||
|
||||
out_ratio 是 auto 模式下按批次中位比例算出的输出宽高比;None 表示按本页
|
||||
自然比例。两种分支都把内容区的一条边填满:低分辨率照片会被放大,从而
|
||||
避免同一批页面因原图尺寸/分辨率不同而大小不一。
|
||||
"""
|
||||
w_src, h_src = aspect_of(quad)
|
||||
cw, ch = canvas_px(paper, page_orientation, P)
|
||||
margin = int(P["render"]["margin_px"])
|
||||
cont_w = max(1, cw - 2 * margin)
|
||||
cont_h = max(1, ch - 2 * margin)
|
||||
|
||||
if paper["ratio"] is not None:
|
||||
# 强制纸面比例:宽高比纠正透视压缩。
|
||||
r = (paper["ratio"] if page_orientation == "portrait"
|
||||
else 1.0 / paper["ratio"])
|
||||
else:
|
||||
r = out_ratio
|
||||
|
||||
if r is not None:
|
||||
tw = float(cont_w)
|
||||
th = tw / r
|
||||
if th > cont_h:
|
||||
th = float(cont_h)
|
||||
tw = th * r
|
||||
else:
|
||||
# auto 且未形成批次比例:按四边形自然比例填满内容区(可放大)。
|
||||
s = min(float(cont_w) / w_src, float(cont_h) / h_src)
|
||||
tw, th = w_src * s, h_src * s
|
||||
|
||||
return max(1, int(round(tw))), max(1, int(round(th))), (cw, ch)
|
||||
|
||||
|
||||
def warp_raw(img, quad):
|
||||
"""只做裁剪 + 透视拉正:自然尺寸、不调色。raw 页图的唯一来源。"""
|
||||
return warp_perspective_pil(img, quad, None)
|
||||
|
||||
|
||||
def page_for_pdf(raw, size, P):
|
||||
"""从 raw 页图派生 PDF 页:先调色,再按目标尺寸缩放(LANCZOS)。
|
||||
|
||||
与旧 warp_page 的差别只在采样顺序:先在自然尺寸做透视(超采样),
|
||||
再缩到画布内容区,缩小时比直接按目标尺寸采样更少混叠。
|
||||
"""
|
||||
r = P["render"]
|
||||
spec = P["enhance"].get(r.get("enhance") or r["default_enhance"])
|
||||
page = tone(raw, spec, r["brighten_target"], r["brighten_pct"])
|
||||
if page.size != tuple(size):
|
||||
page = page.resize(tuple(size), Image.LANCZOS)
|
||||
return page
|
||||
|
||||
|
||||
def warp_page(img, quad, size, P):
|
||||
"""旧路径:直接按目标尺寸 warp + 调色。PIL 实现出问题时回退用。"""
|
||||
r = P["render"]
|
||||
page = warp_perspective(img, quad, size,
|
||||
chunk_rows=r["warp_chunk_rows"])
|
||||
spec = P["enhance"].get(r.get("enhance") or r["default_enhance"])
|
||||
return tone(page, spec, r["brighten_target"], r["brighten_pct"])
|
||||
|
||||
|
||||
def tone(img, spec, target=245, pct=97):
|
||||
"""按调色档做整体映射;spec=None 表示原样返回。
|
||||
|
||||
只做“分位数 -> 目标亮度”的等比缩放(加可选灰度),不碰像素间的
|
||||
局部关系,所以同一张图重跑结果一致,也不会重绘文字边缘。
|
||||
"""
|
||||
if not spec:
|
||||
return img
|
||||
if spec.get("gray"):
|
||||
img = img.convert("L").convert("RGB")
|
||||
return brighten(img, spec.get("target", target), spec.get("pct", pct))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- PDF
|
||||
|
||||
def build_pdf(pages, out_pdf, canvases, P):
|
||||
"""把已校正的页面图居中放进各自画布,按顺序写 PDF。
|
||||
|
||||
pages / canvases 一一对应;画布可能因 orientation=none 而不同。
|
||||
"""
|
||||
r = P["render"]
|
||||
margin = int(r["margin_px"])
|
||||
dpi = int(r["dpi"])
|
||||
composed = []
|
||||
meta = []
|
||||
|
||||
for img, (cw, ch) in zip(pages, canvases):
|
||||
cw, ch = max(1, int(cw)), max(1, int(ch))
|
||||
cont_w = max(1, cw - 2 * margin)
|
||||
cont_h = max(1, ch - 2 * margin)
|
||||
s = min(1.0, float(cont_w) / img.width, float(cont_h) / img.height)
|
||||
if s < 1.0:
|
||||
nw = max(1, int(round(img.width * s)))
|
||||
nh = max(1, int(round(img.height * s)))
|
||||
img = img.resize((nw, nh), Image.LANCZOS)
|
||||
else:
|
||||
nw, nh = img.width, img.height
|
||||
canvas = Image.new("RGB", (cw, ch), "white")
|
||||
canvas.paste(img, ((cw - nw) // 2, (ch - nh) // 2))
|
||||
composed.append(canvas)
|
||||
meta.append({"canvas": [cw, ch], "pasted": [nw, nh]})
|
||||
|
||||
if not composed:
|
||||
raise ValueError("没有可合成的页面")
|
||||
|
||||
out_dir = os.path.dirname(os.path.abspath(out_pdf))
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
tmp = out_pdf + ".part.pdf"
|
||||
try:
|
||||
composed[0].save(tmp, "PDF", resolution=dpi, save_all=True,
|
||||
append_images=composed[1:])
|
||||
os.replace(tmp, out_pdf)
|
||||
finally:
|
||||
if os.path.exists(tmp):
|
||||
try:
|
||||
os.remove(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
return meta
|
||||
@@ -0,0 +1,118 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""纯 numpy 单应变换:等价于 cv2.warpPerspective,只依赖 Pillow + numpy。
|
||||
|
||||
与旧实现的区别:
|
||||
- 按行分块采样,峰值内存从“整页 × 多份 float64”降到几十 MB;
|
||||
- 只接受 PIL 图,坐标/方向由调用方统一(exif_transpose 已在入口做过);
|
||||
- 不再提供 CLI,出件统一走技能入口。
|
||||
"""
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
|
||||
def _solve_h(src, dst):
|
||||
A = []
|
||||
for (x, y), (u, v) in zip(src, dst):
|
||||
A.append([x, y, 1, 0, 0, 0, -u * x, -u * y, -u])
|
||||
A.append([0, 0, 0, x, y, 1, -v * x, -v * y, -v])
|
||||
_, _, vt = np.linalg.svd(np.asarray(A, dtype=np.float64))
|
||||
return vt[-1].reshape(3, 3)
|
||||
|
||||
|
||||
def natural_size(quad):
|
||||
"""四边形按自然边长对应的输出尺寸(与 warp_perspective 的 out_size=None 一致)。"""
|
||||
q = np.asarray(quad, dtype=np.float64)
|
||||
tl, tr, br, bl = q
|
||||
w = int(round(max(np.linalg.norm(tr - tl), np.linalg.norm(br - bl))))
|
||||
h = int(round(max(np.linalg.norm(bl - tl), np.linalg.norm(br - tr))))
|
||||
return max(1, w), max(1, h)
|
||||
|
||||
|
||||
def warp_perspective(img, quad, out_size=None, fill=(255, 255, 255),
|
||||
chunk_rows=256):
|
||||
"""把 quad 指定的四边形透视校正到 out_size。
|
||||
|
||||
quad 顺序:左上、右上、右下、左下(屏幕坐标)。
|
||||
out_size=None 时按四边形自然边长输出(保留照片原始比例)。
|
||||
"""
|
||||
arr = np.asarray(img if img.mode == "RGB" else img.convert("RGB"))
|
||||
src_h, src_w = arr.shape[:2]
|
||||
quad = np.asarray(quad, dtype=np.float64)
|
||||
|
||||
if out_size is None:
|
||||
out_size = natural_size(quad)
|
||||
out_w = max(1, int(out_size[0]))
|
||||
out_h = max(1, int(out_size[1]))
|
||||
|
||||
dst = np.array([[0, 0], [out_w - 1, 0],
|
||||
[out_w - 1, out_h - 1], [0, out_h - 1]],
|
||||
dtype=np.float64)
|
||||
Hinv = np.linalg.inv(_solve_h(quad, dst))
|
||||
|
||||
out = np.empty((out_h, out_w, 3), dtype=np.uint8)
|
||||
step = max(1, int(chunk_rows))
|
||||
for y0 in range(0, out_h, step):
|
||||
y1 = min(y0 + step, out_h)
|
||||
ys, xs = np.mgrid[y0:y1, 0:out_w]
|
||||
dst_h = np.stack([xs.ravel(), ys.ravel(),
|
||||
np.ones(xs.size)], axis=0).astype(np.float64)
|
||||
sp = Hinv @ dst_h
|
||||
den = sp[2]
|
||||
den = np.where(np.abs(den) < 1e-9, 1e-9, den)
|
||||
sx = sp[0] / den
|
||||
sy = sp[1] / den
|
||||
|
||||
valid = (sx >= 0) & (sx <= src_w - 1) & (sy >= 0) & (sy <= src_h - 1)
|
||||
sx = np.clip(sx, 0, src_w - 1)
|
||||
sy = np.clip(sy, 0, src_h - 1)
|
||||
|
||||
x0 = np.floor(sx).astype(np.int32)
|
||||
x1 = np.minimum(x0 + 1, src_w - 1)
|
||||
y0i = np.floor(sy).astype(np.int32)
|
||||
y1i = np.minimum(y0i + 1, src_h - 1)
|
||||
fx = (sx - x0)[:, None]
|
||||
fy = (sy - y0i)[:, None]
|
||||
|
||||
a = arr[y0i, x0].astype(np.float32)
|
||||
b = arr[y0i, x1].astype(np.float32)
|
||||
c = arr[y1i, x0].astype(np.float32)
|
||||
d = arr[y1i, x1].astype(np.float32)
|
||||
block = (a * (1 - fx) * (1 - fy) + b * fx * (1 - fy)
|
||||
+ c * (1 - fx) * fy + d * fx * fy)
|
||||
block[~valid] = fill
|
||||
out[y0:y1] = np.clip(block, 0, 255).astype(np.uint8).reshape(
|
||||
y1 - y0, out_w, 3)
|
||||
|
||||
return Image.fromarray(out)
|
||||
|
||||
|
||||
def warp_perspective_pil(img, quad, out_size=None, fill=(255, 255, 255)):
|
||||
"""PIL(C) 实现的透视校正,采样约定与 warp_perspective 一致。
|
||||
|
||||
out_size=None 时按四边形自然边长输出。10MP 约 0.3s,比上面的 numpy
|
||||
逐像素实现快约 10 倍;实测两者平均差 0.26 灰度级、89% 像素完全一致。
|
||||
只用于 raw 页图导出;PDF 出件仍走 numpy 版本,避免改变既有输出的像素。
|
||||
"""
|
||||
if out_size is None:
|
||||
out_size = natural_size(quad)
|
||||
ow = max(1, int(out_size[0]))
|
||||
oh = max(1, int(out_size[1]))
|
||||
dst = np.array([[0, 0], [ow - 1, 0], [ow - 1, oh - 1], [0, oh - 1]],
|
||||
dtype=np.float64)
|
||||
H = np.linalg.inv(_solve_h(np.asarray(quad, dtype=np.float64), dst))
|
||||
H = H / H[2, 2]
|
||||
coeffs = (H[0, 0], H[0, 1], H[0, 2],
|
||||
H[1, 0], H[1, 1], H[1, 2],
|
||||
H[2, 0], H[2, 1])
|
||||
src = img if img.mode == "RGB" else img.convert("RGB")
|
||||
return src.transform((ow, oh), Image.PERSPECTIVE, coeffs, Image.BILINEAR,
|
||||
fillcolor=fill)
|
||||
|
||||
|
||||
def brighten(img, target=245, pct=97):
|
||||
"""全局等比提亮:把 pct 分位映射到 target,保持相对对比度。"""
|
||||
arr = np.asarray(img if img.mode == "RGB" else img.convert("RGB")).astype(np.float32)
|
||||
hi = float(np.percentile(arr, pct))
|
||||
if hi > 1.0:
|
||||
arr = np.clip(arr * (float(target) / hi), 0, 255)
|
||||
return Image.fromarray(arr.astype(np.uint8))
|
||||
@@ -0,0 +1,120 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""document-photo-to-pdf 的全部可调参数。
|
||||
|
||||
改这里之前先读这段:
|
||||
- 这些值大多沿用旧流程(人工读网格图)的经验值,**没有用真实样张重新标定**。
|
||||
没有样张时只改行为开关,不要凭感觉调阈值。
|
||||
- 像素类的值默认指“检测处理图”坐标(检测会先缩到 detect.max_side),不是原图像素;
|
||||
带 _frac / _pct / _div 的是比例。少数原图/画布坐标的值会在字段名里写明 _px。
|
||||
- 纸张尺寸用毫米,画布像素由 render.dpi 换算,避免 1240/1754 这类数字散落各处。
|
||||
"""
|
||||
P = {
|
||||
"schema": "document-photo-to-pdf/2",
|
||||
|
||||
# ---------------------------------------------------------------- 检测
|
||||
"detect": {
|
||||
"max_side": 1600, # 检测降采样长边;越大越准越慢
|
||||
"min_edge_points": 40, # 单边拟合所需最少点数,低于则回退掩膜线
|
||||
"group_tol_px": 22.0, # 边线点归属容差(处理图像素)
|
||||
"robust_px": 2.0, # 稳健拟合的残差基准(处理图像素)
|
||||
"mask_erode_kernel": 3, # 掩膜内收核;3 = 半径 1 像素
|
||||
"paper_otsu_floor": 0.50, # 纸面阈值下限,防止亮桌面被并入
|
||||
"paper_v_min": 0.50, # 纸面亮度硬下限
|
||||
"paper_s_max": 0.45, # 纸面饱和度上限
|
||||
"paper_mask_factor_div": 400, # 掩膜降采样倍数 = min(H,W)//该值
|
||||
"paper_close_div": 60, # 第一次闭运算核 = min(H,W)//该值
|
||||
"paper_close_min": 5, # 形态学核最小值
|
||||
"paper_close2_div": 80, # 第二次闭运算核 = min(H,W)//该值
|
||||
"band_div": 45, # 搜索环带宽度 = H//该值
|
||||
"band_min": 15, # 环带宽度下限
|
||||
"edge_local_gain": 1.55, # 边缘图局部阈值增益
|
||||
"edge_low": 0.035, # 边缘图绝对下限
|
||||
"edge_sigma_div": 800, # 高斯模糊 sigma = H//该值
|
||||
"edge_box_div": 50, # 局部均值窗口 = H//该值
|
||||
"step_r_small": 10, # 阶跃检测小窗半径
|
||||
"step_r_big": 24, # 阶跃检测大窗半径
|
||||
"step_th": 0.055, # 跨纸边亮度阶跃阈值
|
||||
"cand_close_div": 160, # 候选边线闭运算 = H//该值
|
||||
"cand_close_min": 5,
|
||||
"cand_open_div": 500, # 候选边线开运算 = H//该值
|
||||
"cand_open_min": 3,
|
||||
"coverage_tol_div": 110, # 覆盖率检测容差 = H//该值
|
||||
"coverage_tol_min": 10,
|
||||
"corner_oob_frac": 0.15, # 角点越界允许幅度(相对画面长边)
|
||||
"boundary_side_margin": 0.10, # 拟合单边时排除两端各 10%,避开角点
|
||||
},
|
||||
|
||||
# ---------------------------------------------------------------- 几何门槛
|
||||
"gate": {
|
||||
"oob_max_pct": 15.0, # 越界超过此值判失败
|
||||
"inner_paper_min": 0.60, # 框内纸面占比下限
|
||||
"inner_edge_min": 0.75, # 每边内侧纸面占比下限
|
||||
"outer_dark_min": 0.35, # 外侧暗像素占比低于此值只告警
|
||||
"coverage_warn": 0.40, # 四边覆盖率低于此值告警(未标定)
|
||||
"aspect_deadband": 0.10, # 0.9~1.1 视为接近正方形,不参与方向投票
|
||||
"paper_mismatch_pct": 20.0, # 与批次中位宽高比偏差超过此值告警
|
||||
"sample_n": 60, # 校验采样网格边长
|
||||
"sample_shrink": 0.88, # 框内采样内收比例
|
||||
"fallback_inset": 0.03, # 检测为空时的兜底矩形:画面短边向内缩的比例
|
||||
},
|
||||
|
||||
# ---------------------------------------------------------------- 调色档
|
||||
# 全局四档,对应审核页的“图像处理”。都是“整体等比映射 + 可选灰度”:
|
||||
# 不做二值化——二值化会吃拮抗锯齿,笔画变胖变瘦,等于重绘文字。
|
||||
"enhance": {
|
||||
"original": None, # 原样:不提亮
|
||||
"normal": {"target": 245, "pct": 97}, # 标准
|
||||
"strong": {"target": 252, "pct": 90}, # 加强:背景白得更彻底
|
||||
"bw": {"target": 252, "pct": 60, "gray": True}, # 黑白(保留灰阶)
|
||||
},
|
||||
|
||||
# ---------------------------------------------------------------- 出页
|
||||
"render": {
|
||||
"dpi": 150, # PDF 输出 dpi
|
||||
"margin_px": 40, # PDF 页面四周留白(150dpi 下的像素)
|
||||
"brighten_target": 245, # 提亮目标:97 分位映射到该亮度
|
||||
"brighten_pct": 97,
|
||||
"warp_chunk_rows": 256, # 透视变换分块行数,控制峰值内存
|
||||
"default_paper": "auto", # auto=按批次中位比例统一版式,A4 画布内留白
|
||||
"default_enhance": "normal", # 调色档默认值,可选 key 见 P["enhance"]
|
||||
"enhance": "normal", # 本次生效的调色档(run.py 按参数覆写副本)
|
||||
"dpi_options": {"standard": 150, "high": 300}, # 清晰度档名 -> dpi
|
||||
"papers_mm": {
|
||||
"a4": (210.0, 297.0),
|
||||
"letter": (215.9, 279.4),
|
||||
"16k": (185.0, 260.0),
|
||||
"b5": (176.0, 250.0),
|
||||
},
|
||||
},
|
||||
|
||||
# ---------------------------------------------------------------- PDF 检验
|
||||
"check": {
|
||||
"dpi": 100, # 检验渲染 dpi
|
||||
"min_short": 800, # 纸面短边下限(按 150dpi 基准,随 dpi 换算)
|
||||
"ratio_tol": 0.08, # 纸面比例偏差提示阈值(仅提示)
|
||||
"border_ratio": 0.80, # 边带亮度 / 纸面亮度 下限
|
||||
"edge_warnings": False, # 边缘背景残留/内容贴边是否升为 warning
|
||||
# (启发式、未标定,误报多;默认只留在 checks)
|
||||
"edge_dark_ratio": 0.60, # 边带“暗像素”亮度 / 纸面亮度
|
||||
"edge_dark_frac": 0.35, # 边带暗像素占比上限
|
||||
"cut_frac": 0.05, # 最外带内文字级暗像素占比上限
|
||||
"cut_mean_ratio": 0.70, # 贴边带均值低于纸面该比例才算成片暗
|
||||
"edge_inset_frac": 0.002, # 纸面框内缩比例,避开透视采样毛边
|
||||
"border_band_frac": 0.012, # 边带宽度 = 纸面短边 * 该值
|
||||
"cut_band_frac": 0.003, # 贴边检测带宽度 = 纸面短边 * 该值
|
||||
"paper_pct": 80, # 纸面亮度分位
|
||||
"back_pct": 2, # 背景亮度分位
|
||||
"ink_ratio": 0.80, # 墨色阈值上限 = 纸面亮度 * 该值
|
||||
"flat_pct": 99.5, # 页面纯白分位
|
||||
"nonwhite_delta": 8.0, # 低于纯白该值视为非白(用于找纸面框)
|
||||
"paper_row_frac": 0.05, # 行列非白占比阈值
|
||||
"tmp_dir": ".tmp/pdfcheck", # 检验渲染临时目录(相对输出目录,检验完即删)
|
||||
},
|
||||
|
||||
# ---------------------------------------------------------------- 输出
|
||||
"output": {
|
||||
"default_name": "文档照片_校正.pdf",
|
||||
"state_suffix": ".state.json",
|
||||
"max_logs": 20,
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""审核页发布:模板链接 + 每批缩略图,落会话 .app/doc-review/。
|
||||
|
||||
模板的单一事实源在 skill 目录的 web/;会话里的 index.html 只是指向它的符号链接,
|
||||
不含任何数据注入。core 的 /app 静态托管放行指向技能根目录的链接(其余位置 404),
|
||||
所以改一次模板就对所有会话生效,不用重新生成。页面运行时通过 /api/db 读 doc_review_*
|
||||
表,因此同一份 HTML 可服务所有批次。缩略图是每会话的数据,仍写真目录。
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
WEB_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "web")
|
||||
APP_SUBDIR = os.path.join(".app", "doc-review")
|
||||
THUMB_LONG = 1200
|
||||
THUMB_Q = 80
|
||||
|
||||
|
||||
def _link(src, dst):
|
||||
"""把技能里的静态文件以符号链接发布到会话目录。
|
||||
|
||||
旧版本是复制副本,模板改动到不了已有会话;改链接后技能就是唯一真源。
|
||||
链接目标用绝对路径:executor 与 core 都把技能挂在同一位置,两边都能解析。
|
||||
"""
|
||||
src = os.path.abspath(src)
|
||||
if os.path.islink(dst):
|
||||
try:
|
||||
if os.path.realpath(dst) == os.path.realpath(src):
|
||||
return
|
||||
except OSError:
|
||||
pass
|
||||
os.unlink(dst)
|
||||
elif os.path.exists(dst):
|
||||
os.unlink(dst) # 旧会话的实体副本:删掉换成链接
|
||||
os.symlink(src, dst)
|
||||
|
||||
|
||||
def publish(session_root, batch_id, pages):
|
||||
"""发布模板与缩略图。
|
||||
|
||||
pages: [{idx, source_abs}];返回 (url, thumbs),thumbs[idx] = (相对路径, w, h)。
|
||||
"""
|
||||
app_dir = os.path.join(session_root, APP_SUBDIR)
|
||||
os.makedirs(app_dir, exist_ok=True)
|
||||
for name in sorted(os.listdir(WEB_DIR)):
|
||||
src = os.path.join(WEB_DIR, name)
|
||||
if os.path.isfile(src):
|
||||
_link(src, os.path.join(app_dir, name))
|
||||
|
||||
thumb_dir = os.path.join(app_dir, batch_id)
|
||||
os.makedirs(thumb_dir, exist_ok=True)
|
||||
thumbs = {}
|
||||
for p in pages:
|
||||
src = p.get("source_abs")
|
||||
idx = p.get("idx")
|
||||
if not src or idx is None or not os.path.isfile(src):
|
||||
continue
|
||||
try:
|
||||
with Image.open(src) as raw:
|
||||
im = ImageOps.exif_transpose(raw).convert("RGB")
|
||||
w, h = im.size
|
||||
s = min(1.0, float(THUMB_LONG) / max(w, h))
|
||||
if s < 1.0:
|
||||
im = im.resize((max(1, int(round(w * s))), max(1, int(round(h * s)))),
|
||||
Image.LANCZOS)
|
||||
name = "p%d.jpg" % int(idx)
|
||||
im.save(os.path.join(thumb_dir, name), "JPEG", quality=THUMB_Q, optimize=True)
|
||||
thumbs[int(idx)] = (batch_id + "/" + name, im.width, im.height)
|
||||
except Exception:
|
||||
continue
|
||||
task_id = os.path.basename(os.path.abspath(session_root))
|
||||
return ("/app/%s/doc-review/?b=%s" % (task_id, batch_id), thumbs)
|
||||
|
||||
|
||||
def prune_dirs(session_root, keep_ids):
|
||||
"""删除不在 keep_ids 里的旧批次缩略图目录(模板文件不动)。"""
|
||||
root = os.path.join(session_root, APP_SUBDIR)
|
||||
if not os.path.isdir(root):
|
||||
return
|
||||
keep = set(keep_ids)
|
||||
for name in os.listdir(root):
|
||||
path = os.path.join(root, name)
|
||||
if os.path.isdir(path) and name not in keep:
|
||||
shutil.rmtree(path, ignore_errors=True)
|
||||
@@ -0,0 +1,223 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""审核页运行记录:会话 data.db 里的 doc_review_* 表。
|
||||
|
||||
为什么由技能自己写:
|
||||
- core 对 data.db 只有通用 SQL API(/api/db)和 db 工具,不认识任何业务表;
|
||||
- 技能自己拥有 schema、写入与清理,core 保持零感知(松耦合)。
|
||||
约定:
|
||||
- 时间一律 ISO8601 UTC 字符串;
|
||||
- 坐标/结构一律 JSON 文本,页面用 JSON.parse;
|
||||
- 表名统一 doc_review_ 前缀,避免和用户业务表混淆。
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import sqlite3
|
||||
from datetime import datetime, timezone
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
DB_NAME = "data.db"
|
||||
|
||||
|
||||
def now_iso():
|
||||
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def new_batch_id():
|
||||
# 不含点/斜杠/空格:避免 skills.absolutizeArgs 把 batch_id 误判成路径。
|
||||
return "dr" + datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S") + "-" + secrets.token_hex(2)
|
||||
|
||||
|
||||
def _json(v):
|
||||
return None if v is None else json.dumps(v, ensure_ascii=False, separators=(",", ":"))
|
||||
|
||||
|
||||
def _loads(s, default=None):
|
||||
if not s:
|
||||
return default
|
||||
try:
|
||||
return json.loads(s)
|
||||
except Exception:
|
||||
return default
|
||||
|
||||
|
||||
def connect(session_root):
|
||||
path = os.path.join(session_root, DB_NAME)
|
||||
conn = sqlite3.connect(path, timeout=5.0)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA busy_timeout=5000")
|
||||
# core 建库时已是 WAL;这里 best-effort,不强行切换(并发打开时切换会拿写锁)。
|
||||
try:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
except sqlite3.DatabaseError:
|
||||
pass
|
||||
ensure_schema(conn)
|
||||
return conn
|
||||
|
||||
|
||||
def ensure_schema(conn):
|
||||
conn.executescript("""
|
||||
CREATE TABLE IF NOT EXISTS doc_review_batch (
|
||||
id TEXT PRIMARY KEY,
|
||||
output TEXT NOT NULL,
|
||||
paper TEXT,
|
||||
enhance TEXT,
|
||||
dpi TEXT,
|
||||
orientation TEXT,
|
||||
status TEXT,
|
||||
summary TEXT,
|
||||
warnings TEXT,
|
||||
schema_version INTEGER,
|
||||
created_at TEXT,
|
||||
updated_at TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS doc_review_page (
|
||||
batch_id TEXT NOT NULL,
|
||||
idx INTEGER NOT NULL,
|
||||
source TEXT,
|
||||
source_w INTEGER,
|
||||
source_h INTEGER,
|
||||
w INTEGER,
|
||||
h INTEGER,
|
||||
auto_corners TEXT,
|
||||
auto_corners_norm TEXT,
|
||||
corners TEXT,
|
||||
auto_rotate INTEGER,
|
||||
rotate INTEGER,
|
||||
deleted INTEGER DEFAULT 0,
|
||||
sort INTEGER,
|
||||
status TEXT,
|
||||
reason TEXT,
|
||||
thumb TEXT,
|
||||
error TEXT,
|
||||
PRIMARY KEY (batch_id, idx)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS doc_review_job (
|
||||
batch_id TEXT PRIMARY KEY,
|
||||
edits TEXT,
|
||||
status TEXT,
|
||||
message TEXT,
|
||||
requested_at TEXT,
|
||||
started_at TEXT,
|
||||
finished_at TEXT
|
||||
);
|
||||
""")
|
||||
conn.commit()
|
||||
|
||||
|
||||
def upsert_batch(conn, batch_id, output, opts):
|
||||
ts = now_iso()
|
||||
conn.execute("""
|
||||
INSERT INTO doc_review_batch
|
||||
(id, output, paper, enhance, dpi, orientation, status, summary, warnings,
|
||||
schema_version, created_at, updated_at)
|
||||
VALUES (?,?,?,?,?,?,'running',NULL,NULL,?,?,?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
output=excluded.output, paper=excluded.paper, enhance=excluded.enhance,
|
||||
dpi=excluded.dpi, orientation=excluded.orientation,
|
||||
schema_version=excluded.schema_version, updated_at=excluded.updated_at
|
||||
""", (batch_id, output, opts.get("paper"), opts.get("enhance"),
|
||||
str(opts.get("dpi")), opts.get("orientation"),
|
||||
SCHEMA_VERSION, ts, ts))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def replace_pages(conn, batch_id, pages):
|
||||
conn.execute("DELETE FROM doc_review_page WHERE batch_id = ?", (batch_id,))
|
||||
for p in pages:
|
||||
conn.execute("""
|
||||
INSERT INTO doc_review_page
|
||||
(batch_id, idx, source, source_w, source_h, w, h,
|
||||
auto_corners, auto_corners_norm, corners, auto_rotate, rotate,
|
||||
deleted, sort, status, reason, thumb, error)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
""", (batch_id, int(p.get("idx")), p.get("source"),
|
||||
p.get("source_w"), p.get("source_h"), p.get("w"), p.get("h"),
|
||||
_json(p.get("auto_corners")), _json(p.get("auto_corners_norm")),
|
||||
_json(p.get("corners")), int(p.get("auto_rotate") or 0),
|
||||
int(p.get("rotate") or 0), 1 if p.get("deleted") else 0,
|
||||
int(p.get("sort") or 0), p.get("status"), p.get("reason"),
|
||||
p.get("thumb"), p.get("error")))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _row(r):
|
||||
return dict(r) if r is not None else None
|
||||
|
||||
|
||||
def get_batch(conn, batch_id):
|
||||
return _row(conn.execute("SELECT * FROM doc_review_batch WHERE id = ?", (batch_id,)).fetchone())
|
||||
|
||||
|
||||
def get_latest_batch(conn):
|
||||
return _row(conn.execute(
|
||||
"SELECT * FROM doc_review_batch ORDER BY created_at DESC, updated_at DESC LIMIT 1").fetchone())
|
||||
|
||||
|
||||
def get_pages(conn, batch_id):
|
||||
rows = [_row(r) for r in conn.execute(
|
||||
"SELECT * FROM doc_review_page WHERE batch_id = ? ORDER BY sort, idx", (batch_id,))]
|
||||
for r in rows:
|
||||
r["auto_corners"] = _loads(r.get("auto_corners"))
|
||||
r["auto_corners_norm"] = _loads(r.get("auto_corners_norm"))
|
||||
r["corners"] = _loads(r.get("corners"))
|
||||
return rows
|
||||
|
||||
|
||||
def get_job(conn, batch_id):
|
||||
r = _row(conn.execute("SELECT * FROM doc_review_job WHERE batch_id = ?", (batch_id,)).fetchone())
|
||||
if r:
|
||||
r["edits_obj"] = _loads(r.get("edits"))
|
||||
return r
|
||||
|
||||
|
||||
def upsert_job(conn, batch_id, edits, status="queued", message=None):
|
||||
ts = now_iso()
|
||||
conn.execute("""
|
||||
INSERT INTO doc_review_job(batch_id, edits, status, message, requested_at, started_at, finished_at)
|
||||
VALUES (?,?,?,?,?,NULL,NULL)
|
||||
ON CONFLICT(batch_id) DO UPDATE SET
|
||||
edits=excluded.edits, status=excluded.status, message=excluded.message,
|
||||
requested_at=excluded.requested_at, started_at=NULL, finished_at=NULL
|
||||
""", (batch_id, _json(edits), status, message, ts))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def set_job(conn, batch_id, status, message=None):
|
||||
ts = now_iso()
|
||||
if status == "running":
|
||||
conn.execute("UPDATE doc_review_job SET status=?, message=?, started_at=?, finished_at=NULL WHERE batch_id=?",
|
||||
(status, message, ts, batch_id))
|
||||
elif status in ("done", "error"):
|
||||
conn.execute("UPDATE doc_review_job SET status=?, message=?, finished_at=? WHERE batch_id=?",
|
||||
(status, message, ts, batch_id))
|
||||
else:
|
||||
conn.execute("UPDATE doc_review_job SET status=?, message=? WHERE batch_id=?",
|
||||
(status, message, batch_id))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def finish_batch(conn, batch_id, ok, summary, warnings, status=None):
|
||||
ts = now_iso()
|
||||
if status is None:
|
||||
status = "ready" if ok else "error"
|
||||
conn.execute("UPDATE doc_review_batch SET status=?, summary=?, warnings=?, updated_at=? WHERE id=?",
|
||||
(status, summary, _json(warnings or []), ts, batch_id))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def prune(conn, keep=5, protect=None):
|
||||
"""只保留最近 keep 个批次;protect 指定的批次永不删(防止在旧批次上重生成时被裁掉)。"""
|
||||
rows = conn.execute(
|
||||
"SELECT id FROM doc_review_batch ORDER BY created_at DESC, updated_at DESC").fetchall()
|
||||
kept = [r["id"] for r in rows[:keep]]
|
||||
if protect and protect not in kept:
|
||||
kept.append(protect)
|
||||
old = [r["id"] for r in rows if r["id"] not in kept]
|
||||
for bid in old:
|
||||
conn.execute("DELETE FROM doc_review_page WHERE batch_id=?", (bid,))
|
||||
conn.execute("DELETE FROM doc_review_job WHERE batch_id=?", (bid,))
|
||||
conn.execute("DELETE FROM doc_review_batch WHERE id=?", (bid,))
|
||||
if old:
|
||||
conn.commit()
|
||||
return old, kept
|
||||
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""合成样张回归测试(不依赖真实照片)。
|
||||
|
||||
覆盖:
|
||||
1. 单页透视校正 + JSON 输出 + PDF 生成 + state 文件
|
||||
2. EXIF Orientation=6 的转正
|
||||
3. 批次方向不一致时的旋转告警
|
||||
4. paper=a4 的画布尺寸
|
||||
5. orientation=none 的混排
|
||||
6. orientation=landscape 的强制方向
|
||||
|
||||
运行:
|
||||
python3 skills/document-photo-to-pdf/doc_deskew/smoke_test.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
SKILL = os.path.dirname(HERE)
|
||||
sys.path.insert(0, HERE)
|
||||
from homography import _solve_h # noqa: E402
|
||||
import check # noqa: E402
|
||||
from params import P # noqa: E402
|
||||
|
||||
W_PAGE, H_PAGE = 900, 1272
|
||||
QUAD = [(200, 240), (1080, 190), (1120, 1420), (150, 1380)]
|
||||
PHOTO_SIZE = (1200, 1600)
|
||||
|
||||
|
||||
def make_page():
|
||||
im = Image.new("RGB", (W_PAGE, H_PAGE), "white")
|
||||
d = ImageDraw.Draw(im)
|
||||
for i in range(18):
|
||||
y = 90 + i * 58
|
||||
d.line([(90, y), (W_PAGE - 90, y)], fill=(30, 30, 30), width=8)
|
||||
if i % 4 == 0:
|
||||
d.line([(90, y + 18), (W_PAGE - 300, y + 18)],
|
||||
fill=(60, 60, 60), width=5)
|
||||
return im
|
||||
|
||||
|
||||
def make_photo(page, quad=QUAD, size=PHOTO_SIZE):
|
||||
bg = Image.new("RGB", size, (38, 42, 48))
|
||||
dst = [(0, 0), (W_PAGE - 1, 0), (W_PAGE - 1, H_PAGE - 1),
|
||||
(0, H_PAGE - 1)]
|
||||
H = _solve_h(quad, dst)
|
||||
H = H / H[2, 2]
|
||||
coeffs = (H[0, 0], H[0, 1], H[0, 2],
|
||||
H[1, 0], H[1, 1], H[1, 2],
|
||||
H[2, 0], H[2, 1])
|
||||
warped = page.transform(size, Image.PERSPECTIVE, coeffs, Image.BICUBIC)
|
||||
mask = Image.new("L", (W_PAGE, H_PAGE), 255)
|
||||
mask = mask.transform(size, Image.PERSPECTIVE, coeffs, Image.BICUBIC)
|
||||
mask = mask.point(lambda v: 255 if v > 127 else 0)
|
||||
photo = bg.copy()
|
||||
photo.paste(warped, mask=mask)
|
||||
return photo
|
||||
|
||||
|
||||
def run_skill(args, cwd):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, os.path.join(SKILL, "run.py"),
|
||||
json.dumps(args, ensure_ascii=False)],
|
||||
cwd=cwd, capture_output=True, text=True)
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
# stdout 必须是纯 JSON(不含进度日志)
|
||||
data = json.loads(proc.stdout)
|
||||
assert isinstance(data, dict)
|
||||
return data
|
||||
|
||||
|
||||
def assert_corners(got, want, tol=35):
|
||||
assert len(got) == 4, got
|
||||
for (gx, gy), (ex, ey) in zip(got, want):
|
||||
assert abs(gx - ex) <= tol and abs(gy - ey) <= tol, (
|
||||
"角点偏差过大:got=%s want=%s" % (got, want))
|
||||
|
||||
|
||||
def assert_checks(res):
|
||||
checks = res.get("checks") or {}
|
||||
if checks.get("ok") is None:
|
||||
print("[skip] PDF 渲染不可用:%s" % checks.get("error"))
|
||||
return
|
||||
assert checks.get("ok") is True, checks
|
||||
assert checks["page_count"]["ok"], checks
|
||||
|
||||
|
||||
def verbose(label, res):
|
||||
if not os.environ.get("SMOKE_VERBOSE"):
|
||||
return
|
||||
print("[%s] %s" % (label, json.dumps({
|
||||
"summary": res.get("summary"),
|
||||
"batch": res.get("batch"),
|
||||
"warnings": res.get("warnings"),
|
||||
"checks": res.get("checks"),
|
||||
}, ensure_ascii=False, indent=2)))
|
||||
|
||||
|
||||
def make_composed_page(strip=None, edge_text=False):
|
||||
"""造一张“PDF 渲染页”:白画布 + 居中纸面(~245 亮度)。"""
|
||||
canvas = Image.new("RGB", (1240, 1754), "white")
|
||||
paper = Image.new("RGB", (1160, 1640), (245, 245, 245))
|
||||
d = ImageDraw.Draw(paper)
|
||||
for i in range(12):
|
||||
y = 120 + i * 90
|
||||
d.line([(100, y), (1060, y)], fill=(30, 30, 30), width=8)
|
||||
if strip is not None:
|
||||
d.rectangle([0, 0, 1159, strip - 1], fill=(60, 60, 60))
|
||||
if edge_text:
|
||||
d.rectangle([60, 0, 1100, 4], fill=(0, 0, 0))
|
||||
canvas.paste(paper, (40, 57))
|
||||
return canvas
|
||||
|
||||
|
||||
def test_analyse():
|
||||
"""直接验证 check.analyse 量的是纸边,不是 PDF 白边。"""
|
||||
normal = check.analyse(make_composed_page(), P)
|
||||
assert not normal["border_bad"], normal["border_bad"]
|
||||
assert not normal["cut"], normal["cut"]
|
||||
|
||||
residue = check.analyse(make_composed_page(strip=12), P)
|
||||
assert "top" in residue["border_bad"], residue["border_bad"]
|
||||
assert residue["edge_dark"]["top"] > P["check"]["edge_dark_frac"], residue
|
||||
|
||||
cut = check.analyse(make_composed_page(edge_text=True), P)
|
||||
assert any(k == "top" for k, _ in cut["cut"]), cut["cut"]
|
||||
|
||||
|
||||
def main():
|
||||
test_analyse()
|
||||
page = make_page()
|
||||
photo = make_photo(page)
|
||||
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
# 1. 单页 + auto
|
||||
src = os.path.join(td, "图1.jpg")
|
||||
photo.save(src, quality=92)
|
||||
res = run_skill({"images": ["图1.jpg"], "output": "结果.pdf",
|
||||
"paper": "auto"}, td)
|
||||
assert res["ok"], res
|
||||
assert res["output"]["pages"] == 1, res
|
||||
assert os.path.exists(res["output"]["path"])
|
||||
assert_corners(res["pages"][0]["corners"], QUAD)
|
||||
assert_checks(res)
|
||||
# 低分辨率纸面也要放大到内容区,避免各页大小不一
|
||||
assert res["pages"][0]["out_size"][0] == 1160, res["pages"][0]
|
||||
assert os.path.exists(res["output"]["state"])
|
||||
verbose("auto", res)
|
||||
|
||||
# check.py CLI 也跑一遍(executor 有渲染后端时)
|
||||
if (res.get("checks") or {}).get("ok") is not None:
|
||||
cli = subprocess.run(
|
||||
[sys.executable, os.path.join(HERE, "check.py"),
|
||||
res["output"]["path"], "--expect-pages", "1"],
|
||||
capture_output=True, text=True)
|
||||
assert cli.returncode == 0, cli.stdout + cli.stderr
|
||||
|
||||
# 2. EXIF Orientation=6:存 90° CCW + 标记 6,读入应转回原方向
|
||||
stored = photo.transpose(Image.ROTATE_90)
|
||||
exif = Image.Exif()
|
||||
exif[274] = 6
|
||||
exif_src = os.path.join(td, "exif.jpg")
|
||||
stored.save(exif_src, quality=92, exif=exif)
|
||||
res2 = run_skill({"images": ["exif.jpg"], "output": "exif.pdf"}, td)
|
||||
assert res2["ok"], res2
|
||||
assert_corners(res2["pages"][0]["corners"], QUAD)
|
||||
assert_checks(res2)
|
||||
verbose("exif", res2)
|
||||
|
||||
# 3. 批次方向:第二页横拍,应被旋转并告警
|
||||
sideways = photo.transpose(Image.ROTATE_270)
|
||||
side_src = os.path.join(td, "图2.jpg")
|
||||
sideways.save(side_src, quality=92)
|
||||
res3 = run_skill({"images": ["图1.jpg", "图2.jpg"],
|
||||
"output": "批次.pdf"}, td)
|
||||
assert res3["ok"], res3
|
||||
assert res3["output"]["pages"] == 2, res3
|
||||
assert res3["batch"]["rotated_pages"] == [2], res3["batch"]
|
||||
assert res3["pages"][1]["warnings"], res3["pages"][1]
|
||||
# 同批页面即使一竖一横,也应放大/缩放到同一版式宽度
|
||||
s1, s2 = res3["pages"][0]["out_size"], res3["pages"][1]["out_size"]
|
||||
assert s1[0] == s2[0] == 1160, (s1, s2)
|
||||
assert abs(s1[1] - s2[1]) <= 2, (s1, s2)
|
||||
assert_checks(res3)
|
||||
verbose("batch", res3)
|
||||
|
||||
# 4. paper=a4 画布
|
||||
res4 = run_skill({"images": ["图1.jpg"], "output": "a4.pdf",
|
||||
"paper": "a4"}, td)
|
||||
assert res4["ok"], res4
|
||||
assert res4["batch"]["canvas_px"] == [1240, 1754], res4["batch"]
|
||||
assert res4["batch"]["content_px"] == [1160, 1674], res4["batch"]
|
||||
assert_checks(res4)
|
||||
verbose("a4", res4)
|
||||
|
||||
# 5. orientation=none:不旋转,允许横竖混排
|
||||
res5 = run_skill({"images": ["图1.jpg", "图2.jpg"],
|
||||
"output": "none.pdf", "orientation": "none"}, td)
|
||||
assert res5["ok"], res5
|
||||
assert res5["batch"]["orientation"] == "mixed", res5["batch"]
|
||||
assert res5["batch"]["rotated_pages"] == [], res5["batch"]
|
||||
assert res5["batch"]["canvas_px"] is None, res5["batch"]
|
||||
assert_checks(res5)
|
||||
verbose("none", res5)
|
||||
|
||||
# 6. orientation=landscape:强制横版,第一页(竖版)被旋转
|
||||
res6 = run_skill({"images": ["图1.jpg", "图2.jpg"],
|
||||
"output": "land.pdf",
|
||||
"orientation": "landscape"}, td)
|
||||
assert res6["ok"], res6
|
||||
assert res6["batch"]["rotated_pages"] == [1], res6["batch"]
|
||||
assert_checks(res6)
|
||||
verbose("landscape", res6)
|
||||
|
||||
print("SMOKE OK")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,236 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""state.json v2 与 edits.json:归一化坐标、页面状态分级、编辑覆盖。
|
||||
|
||||
坐标约定
|
||||
--------
|
||||
corners_norm / edits 里的 "c" 一律是归一化 0~1,相对 exif_transpose 之后的
|
||||
**原图**:[[x_tl,y_tl],[x_tr,y_tr],[x_br,y_br],[x_bl,y_bl]]。
|
||||
原图像素坐标只在检测、渲染时用;进出 state 一律换成归一化,
|
||||
换预览分辨率、换 UI 画布都不会错位。
|
||||
|
||||
页面状态(取代"整批失败")
|
||||
--------------------------
|
||||
ready 已就绪,无需人看
|
||||
review 建议看一眼(掩膜回退角点、覆盖率低、纸型存疑)
|
||||
needs_orientation 宽高比接近正方形,横竖判不出来,需要人点旋转
|
||||
needs_corners 必须补角(检测为空、框内纸面占比过低)
|
||||
render_failed 出页失败,已用白页占位(极少见)
|
||||
failed 原图打不开,本页已跳过(传错文件)
|
||||
deleted 人主动删掉的页,不进出件
|
||||
|
||||
严重度只用来决定"一页命中多条规则时记哪一条",不参与任何几何计算。
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
|
||||
STATE_SCHEMA = "document-photo-to-pdf/2"
|
||||
EDITS_SCHEMA = "review/1"
|
||||
|
||||
# 顺序即严重度,越靠后越需要人介入
|
||||
STATUS_ORDER = ["ready", "review", "needs_orientation",
|
||||
"needs_corners", "render_failed", "failed", "deleted"]
|
||||
_SEV = {s: i for i, s in enumerate(STATUS_ORDER)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 坐标
|
||||
|
||||
def norm_quad(quad, size):
|
||||
"""原图像素坐标 -> 归一化 0~1(4 位小数,300dpi 往返误差远小于 1px)。"""
|
||||
w = float(size[0]) or 1.0
|
||||
h = float(size[1]) or 1.0
|
||||
return [[round(float(p[0]) / w, 4), round(float(p[1]) / h, 4)]
|
||||
for p in quad]
|
||||
|
||||
|
||||
def denorm_quad(norm, size):
|
||||
"""归一化 0~1 -> 原图像素坐标。"""
|
||||
w, h = float(size[0]), float(size[1])
|
||||
return [[float(p[0]) * w, float(p[1]) * h] for p in norm]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 状态
|
||||
|
||||
def escalate(status, reason, cand, cand_reason):
|
||||
"""取更严重的一对 (status, reason);同级保留先到的。"""
|
||||
if _SEV.get(cand, 0) > _SEV.get(status, 0):
|
||||
return cand, cand_reason
|
||||
return status, reason
|
||||
|
||||
|
||||
def grade_page(meta, P, ref_ratio=None):
|
||||
"""按方案第 6 节的表,把一页的检测数字转成"要不要人看"。
|
||||
|
||||
纯路由:不重新检测、不改数字,只决定这一页在审核页上标不标 `!`。
|
||||
ref_ratio 是这张纸应有的宽高比(竖版为正);None 表示未知,跳过纸型判断。
|
||||
"""
|
||||
g = P["gate"]
|
||||
dead = g["aspect_deadband"]
|
||||
status, reason = "ready", None
|
||||
|
||||
if meta.get("fallback"):
|
||||
status, reason = escalate(status, reason, "review", "mask_fallback")
|
||||
|
||||
cov = meta.get("coverage") or {}
|
||||
if cov and min(float(v) for v in cov.values()) < g["coverage_warn"]:
|
||||
status, reason = escalate(status, reason, "review", "low_coverage")
|
||||
|
||||
metrics = (meta.get("validation") or {}).get("metrics") or {}
|
||||
frac = metrics.get("inner_paper_frac")
|
||||
if frac is not None and float(frac) <= g["inner_paper_min"]:
|
||||
status, reason = escalate(status, reason,
|
||||
"needs_corners", "inner_paper_low")
|
||||
|
||||
aspect = meta.get("aspect")
|
||||
if aspect:
|
||||
if abs(float(aspect) - 1.0) <= dead:
|
||||
# 横竖看不出来:不猜,交给人点旋转
|
||||
status, reason = escalate(status, reason,
|
||||
"needs_orientation", "aspect_ambiguous")
|
||||
elif ref_ratio:
|
||||
# 同向折算后再比:横放的 A4 不该被判成异形纸
|
||||
r = float(ref_ratio)
|
||||
if float(aspect) > 1.0:
|
||||
r = 1.0 / r if r > 0 else r
|
||||
dev = abs(float(aspect) - r) / r if r > 0 else 0.0
|
||||
if dev > g["paper_mismatch_pct"] / 100.0:
|
||||
status, reason = escalate(status, reason,
|
||||
"review", "paper_mismatch")
|
||||
return status, reason
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- edits
|
||||
|
||||
def _quad_ok(c):
|
||||
"""校验归一化四边形:4 点、每点 2 数、范围放宽到 -1~2(角点允许拖出画面)。"""
|
||||
if not isinstance(c, (list, tuple)) or len(c) != 4:
|
||||
return None
|
||||
out = []
|
||||
for p in c:
|
||||
if not isinstance(p, (list, tuple)) or len(p) != 2:
|
||||
return None
|
||||
try:
|
||||
x, y = float(p[0]), float(p[1])
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
# 方案第 11 节:纸被切掉一块时角点必须能拖到画面外,所以下限放到 -1
|
||||
if not (-1.0 <= x <= 2.0 and -1.0 <= y <= 2.0):
|
||||
return None
|
||||
out.append([x, y])
|
||||
return out
|
||||
|
||||
|
||||
def _rot_ok(v):
|
||||
"""旋转角归一到 0/90/180/270,认不出来按 0(不转)处理。"""
|
||||
try:
|
||||
r = int(round(float(v))) % 360
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
return r if r in (0, 90, 180, 270) else 0
|
||||
|
||||
|
||||
def load_edits(value):
|
||||
"""把 edits.json 读成内部结构;value 是路径、dict 或 None。
|
||||
|
||||
只校验结构与范围,不认识的多余字段一律忽略——前端的版本总比后端快,
|
||||
不该因为多了一个字段就拒绝出件。
|
||||
"""
|
||||
if value is None or value == "" or value == {}:
|
||||
return None
|
||||
if isinstance(value, str):
|
||||
path = os.path.abspath(os.path.expanduser(value.strip()))
|
||||
if not os.path.isfile(path):
|
||||
raise ValueError("edits 文件不存在:%s" % path)
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
value = json.load(f)
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("edits 必须是 JSON 对象或 edits.json 路径")
|
||||
|
||||
batch = value.get("batch")
|
||||
batch = dict(batch) if isinstance(batch, dict) else {}
|
||||
|
||||
pages = {}
|
||||
for item in (value.get("pages") or []):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
try:
|
||||
idx = int(item.get("i"))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if idx < 1:
|
||||
continue
|
||||
rec = {}
|
||||
q = _quad_ok(item.get("c"))
|
||||
if q:
|
||||
rec["corners_norm"] = q
|
||||
if item.get("r") is not None:
|
||||
rec["rotate"] = _rot_ok(item.get("r"))
|
||||
if rec:
|
||||
pages[idx] = rec
|
||||
|
||||
order = []
|
||||
for v in (value.get("order") or []):
|
||||
try:
|
||||
order.append(int(v))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
# order 缺省/为空 = 按 images 传入顺序出件,调用方自己兜底
|
||||
|
||||
deleted = set()
|
||||
for v in (value.get("deleted") or []):
|
||||
try:
|
||||
deleted.add(int(v))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
|
||||
return {"schema": EDITS_SCHEMA, "batch": batch, "pages": pages,
|
||||
"order": order, "deleted": deleted}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- state v2
|
||||
|
||||
_STATE_PAGE_KEYS = ("index", "source", "source_size", "auto_corners",
|
||||
"corners_norm", "rotate", "rotate_source",
|
||||
"status", "reason")
|
||||
|
||||
|
||||
def build_state(batch, pages):
|
||||
"""state.json v2:只放重建这次结果需要的东西,不放 PDF 检验数据。
|
||||
|
||||
pages 项由调用方组装好,这里只裁字段、保顺序,不再做判断。
|
||||
"""
|
||||
return {
|
||||
"schema": STATE_SCHEMA,
|
||||
"batch": {
|
||||
"doc_type": batch.get("doc_type"),
|
||||
"paper": batch.get("paper"),
|
||||
"enhance": batch.get("enhance"),
|
||||
"dpi": batch.get("dpi"),
|
||||
"orientation": batch.get("orientation"),
|
||||
},
|
||||
"pages": [{k: p.get(k) for k in _STATE_PAGE_KEYS} for p in pages],
|
||||
}
|
||||
|
||||
|
||||
def load_state(value):
|
||||
"""读上一份 state v2(可选输入):按 index 取 auto_corners,供"恢复自动"。
|
||||
|
||||
缺文件、旧 schema、字段不全都当成没传——state 是参考,不是契约。
|
||||
"""
|
||||
if value in (None, ""):
|
||||
return {}
|
||||
try:
|
||||
if isinstance(value, str):
|
||||
path = os.path.abspath(os.path.expanduser(value.strip()))
|
||||
if not os.path.isfile(path):
|
||||
return {}
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
value = json.load(f)
|
||||
if not isinstance(value, dict) or value.get("schema") != STATE_SCHEMA:
|
||||
return {}
|
||||
out = {}
|
||||
for item in (value.get("pages") or []):
|
||||
if isinstance(item, dict) and item.get("index") is not None:
|
||||
out[int(item["index"])] = item
|
||||
return out
|
||||
except Exception:
|
||||
return {}
|
||||
@@ -0,0 +1,149 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""角点几何校验:纯函数,返回结论和指标,不打印、不落盘。
|
||||
|
||||
判定只看几何与亮度关系,不依赖固定像素常量:
|
||||
1. 四边形凸,且顺序为 左上→右上→右下→左下
|
||||
2. 越界幅度在允许范围内(角点出画面属正常)
|
||||
3. 框内以纸面(亮像素)为主
|
||||
4. 每边内侧都是纸面(混入桌面说明取点落在纸外)
|
||||
外侧偏亮只作提示:可能是取点偏内,也可能是纸边倾斜,亮度无法区分。
|
||||
"""
|
||||
import numpy as np
|
||||
|
||||
|
||||
def _sample_map(a, pts, n, shrink=1.0, outward=0.0):
|
||||
"""在四边形内部取网格点(shrink<1 向内收),或在外部环带取点(outward>0)。"""
|
||||
h, w = a.shape
|
||||
tl, tr, br, bl = [np.asarray(p, dtype=np.float64) for p in pts]
|
||||
c = (tl + tr + br + bl) / 4.0
|
||||
if shrink != 1.0:
|
||||
tl, tr, br, bl = [c + (p - c) * shrink for p in (tl, tr, br, bl)]
|
||||
if outward > 0:
|
||||
tl, tr, br, bl = [c + (p - c) * (1.0 + outward)
|
||||
for p in (tl, tr, br, bl)]
|
||||
vals = []
|
||||
for u in np.linspace(0.03, 0.97, n):
|
||||
p1 = tl + (tr - tl) * u
|
||||
p2 = bl + (br - bl) * u
|
||||
for v in np.linspace(0.03, 0.97, n):
|
||||
x, y = p1 + (p2 - p1) * v
|
||||
xi, yi = int(round(x)), int(round(y))
|
||||
if 0 <= xi < w and 0 <= yi < h:
|
||||
vals.append(a[yi, xi])
|
||||
return np.asarray(vals)
|
||||
|
||||
|
||||
def _cross(o, a, b):
|
||||
return ((a[0] - o[0]) * (b[1] - o[1])
|
||||
- (a[1] - o[1]) * (b[0] - o[0]))
|
||||
|
||||
|
||||
def validate(img, pts, P):
|
||||
"""返回 {ok, metrics, failures, warnings}。"""
|
||||
g = P["gate"]
|
||||
a = np.asarray(img.convert("L")).astype(np.float32)
|
||||
h, w = a.shape
|
||||
paper = float(np.percentile(a, 80))
|
||||
back = float(np.percentile(a, 2))
|
||||
thr = back + (paper - back) * 0.5
|
||||
|
||||
failures = []
|
||||
warnings = []
|
||||
metrics = {}
|
||||
|
||||
# 1. 凸性与顺序
|
||||
signs = [_cross(pts[i], pts[(i + 1) % 4], pts[(i + 2) % 4])
|
||||
for i in range(4)]
|
||||
convex = all(s > 0 for s in signs)
|
||||
metrics["convex"] = bool(convex)
|
||||
if not convex:
|
||||
failures.append("角点顺序/凸性不对,应为 左上→右上→右下→左下")
|
||||
|
||||
# 2. 越界幅度
|
||||
xs = [p[0] for p in pts]
|
||||
ys = [p[1] for p in pts]
|
||||
out = [max(0.0, -min(xs)), max(0.0, max(xs) - w),
|
||||
max(0.0, -min(ys)), max(0.0, max(ys) - h)]
|
||||
oob = max(out) / float(max(w, h)) * 100.0
|
||||
metrics["oob_pct"] = round(float(oob), 2)
|
||||
if oob >= g["oob_max_pct"]:
|
||||
failures.append("角点越界 %.1f%%,超过 %.0f%%"
|
||||
% (oob, g["oob_max_pct"]))
|
||||
|
||||
# 3. 框内纸面占比
|
||||
inner = _sample_map(a, pts, g["sample_n"], shrink=g["sample_shrink"])
|
||||
if inner.size == 0:
|
||||
frac_paper = 0.0
|
||||
else:
|
||||
frac_paper = float((inner > thr).mean())
|
||||
metrics["inner_paper_frac"] = round(frac_paper, 4)
|
||||
if frac_paper <= g["inner_paper_min"]:
|
||||
failures.append("框内纸面占比 %.0f%%,偏低(可能取到桌面)"
|
||||
% (frac_paper * 100))
|
||||
|
||||
# 4. 四边外侧:只提示
|
||||
names = ["上边", "右边", "下边", "左边"]
|
||||
outer_dark = {}
|
||||
span = float(max(w, h))
|
||||
for i in range(4):
|
||||
p1, p2 = np.asarray(pts[i], float), np.asarray(pts[(i + 1) % 4], float)
|
||||
mid = (p1 + p2) / 2.0
|
||||
c = sum(np.asarray(p, float) for p in pts) / 4.0
|
||||
d = mid - c
|
||||
norm = float(np.linalg.norm(d))
|
||||
if norm < 1e-6:
|
||||
continue
|
||||
d = d / norm
|
||||
vals = []
|
||||
for t in np.linspace(0.15, 0.85, 12):
|
||||
base = p1 + (p2 - p1) * t
|
||||
for k in (span * 0.02, span * 0.035, span * 0.05):
|
||||
x, y = base + d * k
|
||||
xi, yi = int(round(x)), int(round(y))
|
||||
if 0 <= xi < w and 0 <= yi < h:
|
||||
vals.append(a[yi, xi])
|
||||
if len(vals) < 8:
|
||||
continue
|
||||
vals = np.asarray(vals)
|
||||
dark_frac = float((vals < thr).mean())
|
||||
outer_dark[names[i]] = round(dark_frac, 3)
|
||||
if dark_frac < g["outer_dark_min"]:
|
||||
warnings.append("%s外侧仍偏亮(可能取点偏内,也可能纸边倾斜)"
|
||||
% names[i])
|
||||
metrics["outer_dark"] = outer_dark
|
||||
|
||||
# 5. 四边内侧:必须是纸面
|
||||
inner_paper = {}
|
||||
for i in range(4):
|
||||
p1, p2 = np.asarray(pts[i], float), np.asarray(pts[(i + 1) % 4], float)
|
||||
mid = (p1 + p2) / 2.0
|
||||
c = sum(np.asarray(p, float) for p in pts) / 4.0
|
||||
d = mid - c
|
||||
norm = float(np.linalg.norm(d))
|
||||
if norm < 1e-6:
|
||||
continue
|
||||
d = d / norm
|
||||
vals = []
|
||||
for t in np.linspace(0.15, 0.85, 12):
|
||||
base = p1 + (p2 - p1) * t
|
||||
for k in (span * 0.015, span * 0.03, span * 0.045):
|
||||
x, y = base - d * k
|
||||
xi, yi = int(round(x)), int(round(y))
|
||||
if 0 <= xi < w and 0 <= yi < h:
|
||||
vals.append(a[yi, xi])
|
||||
if len(vals) < 8:
|
||||
continue
|
||||
vals = np.asarray(vals)
|
||||
paper_frac = float((vals > thr).mean())
|
||||
inner_paper[names[i]] = round(paper_frac, 3)
|
||||
if paper_frac < g["inner_edge_min"]:
|
||||
failures.append("%s内侧混入桌面(纸面仅 %.0f%%),取点落在纸外"
|
||||
% (names[i], paper_frac * 100))
|
||||
metrics["inner_paper"] = inner_paper
|
||||
|
||||
return {
|
||||
"ok": not failures,
|
||||
"metrics": metrics,
|
||||
"failures": failures,
|
||||
"warnings": warnings,
|
||||
}
|
||||
@@ -0,0 +1,951 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""document-photo-to-pdf 技能入口。
|
||||
|
||||
两种用法共用一条出件内核:
|
||||
全自动:{"images": [...], "output": "..."}
|
||||
出 PDF,并把校正后的单页原图导出到 out/(不碰 .app/、不写 data.db)。
|
||||
半自动:{"images": [...], "review": "always|auto"}
|
||||
出 PDF 和 out/ 页图,并把审核页发布到 .app/doc-review/、运行记录写进 data.db。
|
||||
重生成:{"batch_id": "dr..."}
|
||||
从 data.db 读上次的源图/输出/编辑,覆盖同名 PDF;args 带 edits 时以 edits
|
||||
为准,不带时读 doc_review_job.edits,再退化为当前页记录(按现在的角点再渲染)。
|
||||
|
||||
stdout 只输出一个 JSON 对象;进度、警告、异常都在字段里,不写 stderr。
|
||||
失败也返回 exit 0,用 ok=false 表达(平台非 0 会给输出追加 [exit code],破坏 JSON)。
|
||||
"""
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, os.path.join(HERE, "doc_deskew"))
|
||||
|
||||
from params import P # noqa: E402
|
||||
from auto_quad import detect # noqa: E402
|
||||
from validate import validate # noqa: E402
|
||||
import compose # noqa: E402
|
||||
import check # noqa: E402
|
||||
import state # noqa: E402
|
||||
import review_store # noqa: E402
|
||||
import review_page # noqa: E402
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 基础
|
||||
|
||||
def _err(code, message, page=None, hint=None):
|
||||
e = {"code": code, "message": message}
|
||||
if page is not None:
|
||||
e["page"] = int(page)
|
||||
if hint:
|
||||
e["hint"] = hint
|
||||
return e
|
||||
|
||||
|
||||
def _json_default(o):
|
||||
try:
|
||||
import numpy as np
|
||||
if isinstance(o, np.integer):
|
||||
return int(o)
|
||||
if isinstance(o, np.floating):
|
||||
return float(o)
|
||||
if isinstance(o, np.ndarray):
|
||||
return o.tolist()
|
||||
except Exception:
|
||||
pass
|
||||
return str(o)
|
||||
|
||||
|
||||
def load_image(path):
|
||||
"""唯一的读图入口:转正 EXIF、压白透明底、转 RGB。"""
|
||||
with Image.open(path) as im:
|
||||
im = ImageOps.exif_transpose(im)
|
||||
if (im.mode in ("RGBA", "LA")
|
||||
or (im.mode == "P" and "transparency" in im.info)):
|
||||
im = im.convert("RGBA")
|
||||
bg = Image.new("RGB", im.size, "white")
|
||||
bg.paste(im, mask=im.split()[-1])
|
||||
im = bg
|
||||
else:
|
||||
im = im.convert("RGB")
|
||||
im.load()
|
||||
return im
|
||||
|
||||
|
||||
def parse_images(args):
|
||||
raw = args.get("images")
|
||||
if raw is None:
|
||||
return []
|
||||
if isinstance(raw, str):
|
||||
return [raw.strip()] if raw.strip() else []
|
||||
if isinstance(raw, (list, tuple)):
|
||||
out, seen = [], set()
|
||||
for v in raw:
|
||||
if not isinstance(v, str) or not v.strip():
|
||||
raise ValueError("images 里只能放路径字符串")
|
||||
p = v.strip()
|
||||
if p not in seen:
|
||||
seen.add(p)
|
||||
out.append(p)
|
||||
return out
|
||||
raise ValueError("images 必须是字符串或字符串数组")
|
||||
|
||||
|
||||
def write_state(path, data):
|
||||
if not path:
|
||||
return False
|
||||
try:
|
||||
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
|
||||
tmp = path + ".tmp"
|
||||
with open(tmp, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, ensure_ascii=False, indent=2)
|
||||
os.replace(tmp, path)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 参数/路径
|
||||
|
||||
def _review_mode(args):
|
||||
if args.get("batch_id") and "review" not in args:
|
||||
return "always"
|
||||
v = args.get("review", "always")
|
||||
if isinstance(v, bool):
|
||||
return "always" if v else "off"
|
||||
s = str(v).strip().lower()
|
||||
if s in ("1", "true", "yes", "on"):
|
||||
s = "always"
|
||||
if s in ("0", "false", "no", ""):
|
||||
s = "off"
|
||||
if s not in ("off", "always", "auto"):
|
||||
raise ValueError("review 只能是 off/always/auto")
|
||||
return s
|
||||
|
||||
|
||||
def _resolve_dpi(v, P):
|
||||
if v is None or str(v).strip() == "":
|
||||
return int(P["render"]["dpi"])
|
||||
s = str(v).strip().lower()
|
||||
opts = P["render"]["dpi_options"]
|
||||
if s in opts:
|
||||
return int(opts[s])
|
||||
try:
|
||||
n = int(float(s))
|
||||
except ValueError:
|
||||
raise ValueError("dpi 只能是 standard/high 或 72~600 的数字")
|
||||
if not (72 <= n <= 600):
|
||||
raise ValueError("dpi 超出范围(72~600):%s" % v)
|
||||
return n
|
||||
|
||||
|
||||
def _dpi_label(v, dpi):
|
||||
# 150/300 归一到审核页的两个档名,页面分段控件才能正确高亮;其它数值原样返回。
|
||||
if int(dpi) == 150:
|
||||
return "standard"
|
||||
if int(dpi) == 300:
|
||||
return "high"
|
||||
return str(dpi)
|
||||
|
||||
|
||||
def _batch_opts(args, ed, P):
|
||||
b = (ed or {}).get("batch") or {}
|
||||
paper_input = b.get("paper", args.get("paper"))
|
||||
if paper_input is None or str(paper_input).strip() == "":
|
||||
paper_input = P["render"]["default_paper"]
|
||||
paper_input = str(paper_input).strip()
|
||||
paper = compose.resolve_paper(paper_input, P)
|
||||
|
||||
orientation = str(b.get("orientation", args.get("orientation") or "auto")).strip().lower()
|
||||
if orientation not in ("auto", "none", "portrait", "landscape"):
|
||||
raise ValueError("orientation 只能是 auto/none/portrait/landscape")
|
||||
|
||||
enhance = str(b.get("enhance", args.get("enhance") or P["render"]["default_enhance"])).strip().lower()
|
||||
if enhance not in P["enhance"]:
|
||||
raise ValueError("enhance 只能是 " + "/".join(P["enhance"].keys()))
|
||||
P["render"]["enhance"] = enhance
|
||||
|
||||
dpi_arg = b.get("dpi", args.get("dpi"))
|
||||
dpi = _resolve_dpi(dpi_arg, P)
|
||||
P["render"]["dpi"] = dpi
|
||||
return paper_input, paper, orientation, enhance, _dpi_label(dpi_arg, dpi), dpi
|
||||
|
||||
|
||||
def _inside(root, path):
|
||||
try:
|
||||
rel = os.path.relpath(os.path.abspath(path), os.path.abspath(root))
|
||||
return rel != ".." and not rel.startswith(".." + os.sep)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _rel(root, path):
|
||||
try:
|
||||
rel = os.path.relpath(os.path.abspath(path), os.path.abspath(root))
|
||||
if rel == ".." or rel.startswith(".." + os.sep):
|
||||
return os.path.abspath(path)
|
||||
return rel
|
||||
except Exception:
|
||||
return path
|
||||
|
||||
|
||||
def _abs(root, path):
|
||||
p = str(path or "")
|
||||
return p if os.path.isabs(p) else os.path.join(root, p)
|
||||
|
||||
|
||||
def _norm(quad, size):
|
||||
w = float(size[0]) or 1.0
|
||||
h = float(size[1]) or 1.0
|
||||
return [[round(float(p[0]) / w, 4), round(float(p[1]) / h, 4)]
|
||||
for p in quad]
|
||||
|
||||
|
||||
def _denorm(norm, size):
|
||||
w, h = float(size[0]), float(size[1])
|
||||
return [[float(p[0]) * w, float(p[1]) * h] for p in norm]
|
||||
|
||||
|
||||
def _aspect(quad):
|
||||
w, h = compose.aspect_of(quad)
|
||||
return round(float(w / h), 4)
|
||||
|
||||
|
||||
def _fallback_quad(size, P):
|
||||
inset = float(P["gate"]["fallback_inset"])
|
||||
w, h = float(size[0]), float(size[1])
|
||||
return [[inset * w, inset * h], [(1 - inset) * w, inset * h],
|
||||
[(1 - inset) * w, (1 - inset) * h], [inset * w, (1 - inset) * h]]
|
||||
|
||||
|
||||
def _prev_page(rows, idx):
|
||||
for r in rows or []:
|
||||
try:
|
||||
if int(r.get("idx")) == int(idx):
|
||||
return r
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _edits_from_pages(rows, batch):
|
||||
pages, deleted, order = [], [], []
|
||||
for r in rows or []:
|
||||
try:
|
||||
idx = int(r.get("idx"))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if r.get("deleted"):
|
||||
deleted.append(idx)
|
||||
continue
|
||||
c = r.get("corners")
|
||||
if c:
|
||||
pages.append({"i": idx, "c": c, "r": int(r.get("rotate") or 0)})
|
||||
order.append(idx)
|
||||
b = {}
|
||||
if batch:
|
||||
b = {"paper": batch.get("paper"), "enhance": batch.get("enhance"),
|
||||
"dpi": batch.get("dpi"), "orientation": batch.get("orientation")}
|
||||
return state.load_edits({"schema": "review/1", "batch": b, "pages": pages,
|
||||
"order": order, "deleted": deleted})
|
||||
|
||||
|
||||
def _targets(args, batch_id):
|
||||
output_arg = args.get("output")
|
||||
if output_arg is not None:
|
||||
output_arg = str(output_arg).strip()
|
||||
cwd = os.getcwd()
|
||||
if batch_id:
|
||||
if args.get("db"):
|
||||
root = os.path.dirname(os.path.abspath(os.path.expanduser(str(args["db"]))))
|
||||
elif output_arg:
|
||||
root = os.path.dirname(os.path.abspath(os.path.expanduser(output_arg)))
|
||||
else:
|
||||
root = cwd
|
||||
conn = review_store.connect(root)
|
||||
batch = review_store.get_batch(conn, batch_id)
|
||||
if not batch:
|
||||
conn.close()
|
||||
raise ValueError("batch_id 不存在:%s" % batch_id)
|
||||
output = _abs(root, batch["output"])
|
||||
if not output.lower().endswith(".pdf"):
|
||||
output += ".pdf"
|
||||
return output, root, conn, batch
|
||||
|
||||
output = os.path.abspath(os.path.expanduser(
|
||||
str(output_arg or P["output"]["default_name"])))
|
||||
if not output.lower().endswith(".pdf"):
|
||||
output += ".pdf"
|
||||
# 生产环境 cwd 就是会话目录;输出在会话内时以 cwd 为根,子目录输出也不会把
|
||||
# .app/data.db 建到子目录里。本地从别的目录跑时退化为输出目录。
|
||||
root = cwd if _inside(cwd, output) else os.path.dirname(output)
|
||||
return output, root, None, None
|
||||
|
||||
|
||||
def _png_dir(cwd):
|
||||
d = os.path.join(cwd, "out")
|
||||
if os.path.exists(d) and not os.path.isdir(d):
|
||||
raise ValueError("out 已存在且不是目录:%s" % d)
|
||||
os.makedirs(d, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
def _clean_pngs(d, stem, keep):
|
||||
"""删掉本 PDF 前缀下编号大于本次出页数的旧页图;不动目录里其它文件。"""
|
||||
pat = re.compile(r"^" + re.escape(stem) + r"-(\d+)\.png$")
|
||||
for name in os.listdir(d):
|
||||
m = pat.match(name)
|
||||
if m and int(m.group(1)) > keep:
|
||||
try:
|
||||
os.remove(os.path.join(d, name))
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _mark_error(conn, batch_id, message):
|
||||
if not conn or not batch_id:
|
||||
return
|
||||
try:
|
||||
review_store.set_job(conn, batch_id, "error", str(message))
|
||||
review_store.finish_batch(conn, batch_id, False, str(message), [], status="error")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 主流程
|
||||
|
||||
def process(args, t0):
|
||||
review_mode = _review_mode(args)
|
||||
batch_id_arg = args.get("batch_id")
|
||||
batch_id_arg = str(batch_id_arg).strip() if batch_id_arg else None
|
||||
|
||||
output, session_root, conn, batch_row = _targets(args, batch_id_arg)
|
||||
# 审核页由 core 从会话 .app/ 托管;输出不在会话目录内时页面/下载都不可达。
|
||||
if review_mode != "off" and not _inside(os.getcwd(), output):
|
||||
raise ValueError("审核页要求 PDF 位于会话目录内(当前工作目录):%s" % output)
|
||||
|
||||
use_db = bool(batch_id_arg) or review_mode != "off"
|
||||
if use_db and conn is None:
|
||||
conn = review_store.connect(session_root)
|
||||
|
||||
batch_id = batch_id_arg
|
||||
is_new_batch = batch_id is None
|
||||
if use_db and not batch_id:
|
||||
batch_id = review_store.new_batch_id()
|
||||
|
||||
state_path = os.path.splitext(output)[0] + P["output"]["state_suffix"]
|
||||
|
||||
try:
|
||||
old_pages = []
|
||||
if batch_id_arg:
|
||||
if not batch_row:
|
||||
batch_row = review_store.get_batch(conn, batch_id)
|
||||
old_pages = review_store.get_pages(conn, batch_id)
|
||||
if not old_pages:
|
||||
raise ValueError("批次没有页面记录:%s" % batch_id)
|
||||
|
||||
ed = state.load_edits(args.get("edits"))
|
||||
job = None
|
||||
if ed is None and batch_id:
|
||||
job = review_store.get_job(conn, batch_id)
|
||||
if job and job.get("edits_obj"):
|
||||
ed = state.load_edits(job["edits_obj"])
|
||||
else:
|
||||
ed = _edits_from_pages(old_pages, batch_row)
|
||||
if batch_id:
|
||||
if not job:
|
||||
job = review_store.get_job(conn, batch_id)
|
||||
if job:
|
||||
review_store.set_job(conn, batch_id, "running")
|
||||
|
||||
paper_input, paper, orientation, enhance, dpi_label, dpi = \
|
||||
_batch_opts(args, ed, P)
|
||||
|
||||
# ---------------- 源图清单
|
||||
if batch_id_arg:
|
||||
source_map = {}
|
||||
for r in old_pages:
|
||||
try:
|
||||
source_map[int(r["idx"])] = _abs(session_root, r["source"])
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
all_indices = sorted(source_map)
|
||||
else:
|
||||
images = parse_images(args)
|
||||
if not images:
|
||||
raise ValueError("必须提供 images")
|
||||
all_indices = list(range(1, len(images) + 1))
|
||||
source_map = {i: os.path.abspath(os.path.expanduser(str(images[i - 1])))
|
||||
for i in all_indices}
|
||||
|
||||
deleted = set(ed.get("deleted") or []) if ed else set()
|
||||
run_indices = [i for i in all_indices if i not in deleted]
|
||||
if not run_indices:
|
||||
raise ValueError("所有页面都被删除,没有可处理的页")
|
||||
|
||||
pages = []
|
||||
plan = []
|
||||
unreadable = []
|
||||
|
||||
for idx in all_indices:
|
||||
src = source_map.get(idx)
|
||||
meta = {"index": idx, "source_abs": src, "warnings": [],
|
||||
"deleted": idx in deleted}
|
||||
if meta["deleted"]:
|
||||
meta.update(status="deleted", reason="user_deleted",
|
||||
corners=None, corners_norm=None, auto_corners=None,
|
||||
auto_corners_norm=None, auto_rotate=0, rotated_deg=0)
|
||||
pages.append(meta)
|
||||
continue
|
||||
|
||||
try:
|
||||
img = load_image(src)
|
||||
except Exception as e:
|
||||
err = _err(
|
||||
"IMAGE_UNREADABLE",
|
||||
"第 %d 页 %s 打不开:%s" % (idx, os.path.basename(src or ""), e),
|
||||
idx,
|
||||
"确认文件存在且是 jpg/png/webp/bmp/tif 等常见格式")
|
||||
meta.update(status="failed", reason="unreadable", error=err,
|
||||
corners=None, corners_norm=None,
|
||||
auto_corners=None, auto_corners_norm=None,
|
||||
auto_rotate=0, rotated_deg=0)
|
||||
meta["warnings"].append(err["message"])
|
||||
unreadable.append(err)
|
||||
pages.append(meta)
|
||||
continue
|
||||
|
||||
meta["source_size"] = [int(img.width), int(img.height)]
|
||||
edit = (ed or {}).get("pages", {}).get(idx) if ed else None
|
||||
manual = bool(edit and edit.get("corners_norm"))
|
||||
|
||||
if manual:
|
||||
meta["manual"] = True
|
||||
meta["corners"] = _denorm(edit["corners_norm"], meta["source_size"])
|
||||
meta["corners_norm"] = [[float(x), float(y)]
|
||||
for x, y in edit["corners_norm"]]
|
||||
meta["aspect"] = _aspect(meta["corners"])
|
||||
meta["coverage"] = {}
|
||||
meta["paper_frac"] = None
|
||||
meta["fallback"] = False
|
||||
meta["fallback_sides"] = []
|
||||
meta["validation"] = {"ok": True, "metrics": {},
|
||||
"warnings": [], "failures": []}
|
||||
prev = _prev_page(old_pages, idx)
|
||||
if prev and prev.get("auto_corners"):
|
||||
meta["auto_corners"] = prev["auto_corners"]
|
||||
meta["auto_corners_norm"] = prev.get("auto_corners_norm")
|
||||
meta["auto_rotate"] = int(prev.get("auto_rotate") or 0)
|
||||
else:
|
||||
meta["auto_corners"] = [list(map(float, q))
|
||||
for q in meta["corners"]]
|
||||
meta["auto_corners_norm"] = meta["corners_norm"]
|
||||
meta["auto_rotate"] = 0
|
||||
meta["status"] = "ready"
|
||||
meta["reason"] = None
|
||||
else:
|
||||
try:
|
||||
corners, info = detect(img, P)
|
||||
except Exception as e:
|
||||
corners, info = [], {"error": str(e)}
|
||||
|
||||
if not corners:
|
||||
corners = _fallback_quad(meta["source_size"], P)
|
||||
info = {"fallback": True, "fallback_sides": [0, 1, 2, 3],
|
||||
"coverage": {}, "paper_frac": None}
|
||||
meta["status"] = "needs_corners"
|
||||
meta["reason"] = "detect_failed"
|
||||
else:
|
||||
meta["status"] = "ready"
|
||||
meta["reason"] = None
|
||||
|
||||
meta["corners"] = [[float(x), float(y)] for x, y in corners]
|
||||
meta["corners_norm"] = _norm(meta["corners"], meta["source_size"])
|
||||
meta["auto_corners"] = [list(map(float, q))
|
||||
for q in meta["corners"]]
|
||||
meta["auto_corners_norm"] = meta["corners_norm"]
|
||||
meta["auto_rotate"] = 0
|
||||
meta["coverage"] = {k: round(float(v), 3)
|
||||
for k, v in (info.get("coverage") or {}).items()}
|
||||
pf = info.get("paper_frac")
|
||||
meta["paper_frac"] = None if pf is None else round(float(pf), 4)
|
||||
meta["fallback"] = bool(info.get("fallback", False))
|
||||
meta["fallback_sides"] = [int(s)
|
||||
for s in (info.get("fallback_sides") or [])]
|
||||
try:
|
||||
v = validate(img, meta["corners"], P)
|
||||
except Exception as e:
|
||||
v = {"ok": False, "metrics": {},
|
||||
"failures": ["校验异常:%s" % e], "warnings": []}
|
||||
meta["validation"] = v
|
||||
if not v["ok"]:
|
||||
# 检测到了但几何不可信:退回兜底框,交给人补角,而不是让整批失败。
|
||||
# auto_corners 也要跟着换成兜底框,否则“恢复自动”会把无效角点拖回来。
|
||||
meta["detected_corners"] = meta["corners"]
|
||||
meta["corners"] = _fallback_quad(meta["source_size"], P)
|
||||
meta["corners_norm"] = _norm(meta["corners"],
|
||||
meta["source_size"])
|
||||
meta["auto_corners"] = [list(map(float, q))
|
||||
for q in meta["corners"]]
|
||||
meta["auto_corners_norm"] = meta["corners_norm"]
|
||||
meta["status"] = "needs_corners"
|
||||
meta["reason"] = "invalid_quad"
|
||||
meta["aspect"] = _aspect(meta["corners"])
|
||||
|
||||
pages.append(meta)
|
||||
plan.append((idx, src, meta))
|
||||
img.close()
|
||||
|
||||
if not plan:
|
||||
failures = unreadable or [
|
||||
_err("NO_PAGES", "没有可处理的页面", None,
|
||||
"检查 images 是否都删除了/是否都打不开")]
|
||||
res = finish_failure(pages, failures, output, state_path, t0)
|
||||
_mark_error(conn, batch_id, res["summary"])
|
||||
return res
|
||||
|
||||
# 出件顺序先定下来:raw 页图按最终 PDF 页序编号(文件名与 PDF 同名)。
|
||||
plan_by_idx = {m["index"]: (i, s, m) for (i, s, m) in plan}
|
||||
ordered_idx = []
|
||||
if ed and ed.get("order"):
|
||||
for v in ed["order"]:
|
||||
if v in plan_by_idx and v not in ordered_idx:
|
||||
ordered_idx.append(v)
|
||||
for i in sorted(plan_by_idx):
|
||||
if i not in ordered_idx:
|
||||
ordered_idx.append(i)
|
||||
plan = [plan_by_idx[i] for i in ordered_idx]
|
||||
|
||||
# raw 页图固定落会话根下的 out/;重生成页数变少时,成功后清掉多余编号。
|
||||
png_dir = _png_dir(os.getcwd())
|
||||
pdf_stem = os.path.splitext(os.path.basename(output))[0]
|
||||
png_pad = max(2, len(str(len(plan))))
|
||||
png_n = 0
|
||||
png_paths = []
|
||||
|
||||
# ---------------- 批次方向
|
||||
aspects = [m["aspect"] for (_, _, m) in plan]
|
||||
batch_orientation, _, median_aspect = \
|
||||
compose.decide_orientation(aspects, orientation, P)
|
||||
dead = P["gate"]["aspect_deadband"]
|
||||
if batch_orientation in ("portrait", "landscape"):
|
||||
same = [m["aspect"] for (_, _, m) in plan
|
||||
if compose.classify(m["aspect"], dead) == batch_orientation]
|
||||
batch_median = float(statistics.median(same)) if same else float(median_aspect)
|
||||
else:
|
||||
batch_median = float(median_aspect)
|
||||
|
||||
batch_ratio = None
|
||||
if paper["ratio"] is None and orientation != "none":
|
||||
m_class = compose.classify(batch_median, dead)
|
||||
if m_class == batch_orientation:
|
||||
batch_ratio = batch_median
|
||||
elif m_class != "square":
|
||||
batch_ratio = 1.0 / batch_median
|
||||
|
||||
# 人工改过的页不再重新评级;自动页按方案表分级后与已有状态取较严重者
|
||||
# grade_page 的 ref_ratio 约定是「竖版为正」,横版批次要先折回竖版,否则
|
||||
# 所有横页都会被判成 paper_mismatch。
|
||||
ref_ratio = paper["ratio"]
|
||||
if ref_ratio is None:
|
||||
ref_ratio = batch_median
|
||||
if ref_ratio > 1.0:
|
||||
ref_ratio = 1.0 / ref_ratio
|
||||
for (_, _, m) in plan:
|
||||
if m.get("manual"):
|
||||
continue
|
||||
st2, r2 = state.grade_page(m, P, ref_ratio)
|
||||
m["status"], m["reason"] = state.escalate(
|
||||
m.get("status") or "ready", m.get("reason"), st2, r2)
|
||||
|
||||
# ---------------- 出页
|
||||
out_meta = {}
|
||||
for (idx, src, meta) in plan:
|
||||
page_class = compose.classify(meta["aspect"], dead)
|
||||
auto_rotate = 0
|
||||
if orientation == "auto":
|
||||
if page_class not in ("square", batch_orientation):
|
||||
auto_rotate = 90
|
||||
elif orientation in ("portrait", "landscape"):
|
||||
if page_class not in ("square", orientation):
|
||||
auto_rotate = 90
|
||||
|
||||
edit = (ed or {}).get("pages", {}).get(idx) if ed else None
|
||||
user_r = edit.get("rotate") if edit else None
|
||||
applied = int(user_r) if user_r is not None else auto_rotate
|
||||
applied = ((applied % 360) + 360) % 360
|
||||
if applied not in (0, 90, 180, 270):
|
||||
applied = 0
|
||||
|
||||
out_corners = [list(map(float, q)) for q in meta["corners"]]
|
||||
for _ in range(applied // 90):
|
||||
out_corners = compose.rotate_quad_cw(out_corners)
|
||||
|
||||
meta["rotated_deg"] = applied
|
||||
if not meta.get("manual"):
|
||||
meta["auto_rotate"] = auto_rotate
|
||||
|
||||
if orientation == "none":
|
||||
out_aspect = _aspect(out_corners)
|
||||
page_orient = compose.classify(out_aspect, dead)
|
||||
if page_orient == "square":
|
||||
page_orient = "portrait"
|
||||
elif orientation in ("portrait", "landscape"):
|
||||
page_orient = orientation
|
||||
else:
|
||||
# auto 模式下,用户显式旋转过就按旋转后的有效方向出画布;
|
||||
# 否则跟批次统一方向。否则 90° 旋转会把内容塞进反向画布压扁。
|
||||
if user_r is not None:
|
||||
page_orient = compose.classify(_aspect(out_corners), dead)
|
||||
if page_orient == "square":
|
||||
page_orient = batch_orientation \
|
||||
if batch_orientation in ("portrait", "landscape") \
|
||||
else "portrait"
|
||||
else:
|
||||
page_orient = batch_orientation
|
||||
|
||||
out_ratio = None
|
||||
if batch_ratio is not None:
|
||||
eff_w, eff_h = compose.aspect_of(out_corners)
|
||||
eff = eff_w / eff_h
|
||||
if abs(eff - batch_ratio) / batch_ratio \
|
||||
<= P["gate"]["paper_mismatch_pct"] / 100.0:
|
||||
out_ratio = batch_ratio
|
||||
|
||||
tw, th, canvas = compose.target_size(
|
||||
out_corners, paper, page_orient, P, out_ratio)
|
||||
|
||||
try:
|
||||
img = load_image(src)
|
||||
try:
|
||||
raw = compose.warp_raw(img, out_corners)
|
||||
finally:
|
||||
img.close()
|
||||
except Exception as e:
|
||||
err = _err("INTERNAL",
|
||||
"第 %d 页重新读取/透视变换失败:%s" % (idx, e), idx,
|
||||
"重试;若持续出现请反馈")
|
||||
meta["status"] = "render_failed"
|
||||
meta["reason"] = "render_failed"
|
||||
meta["error"] = err
|
||||
meta["warnings"].append(err["message"])
|
||||
out_meta[idx] = {"page": None, "canvas": None, "error": err}
|
||||
continue
|
||||
|
||||
try:
|
||||
page = compose.page_for_pdf(raw, (tw, th), P)
|
||||
except Exception as e:
|
||||
err = _err("INTERNAL",
|
||||
"第 %d 页生成 PDF 页面失败:%s" % (idx, e), idx,
|
||||
"重试;若持续出现请反馈")
|
||||
meta["status"] = "render_failed"
|
||||
meta["reason"] = "render_failed"
|
||||
meta["error"] = err
|
||||
meta["warnings"].append(err["message"])
|
||||
out_meta[idx] = {"page": None, "canvas": None, "error": err}
|
||||
continue
|
||||
|
||||
# raw 页图:自然尺寸、不调色、无白边。编号跟最终页序走。
|
||||
png_n += 1
|
||||
png_name = "%s-%0*d.png" % (pdf_stem, png_pad, png_n)
|
||||
png_path = os.path.join(png_dir, png_name)
|
||||
try:
|
||||
raw.save(png_path, "PNG", compress_level=6)
|
||||
png_paths.append(_rel(os.getcwd(), png_path))
|
||||
except Exception as e:
|
||||
meta["warnings"].append("第 %d 页原图导出失败:%s" % (idx, e))
|
||||
|
||||
meta["out_size"] = [int(tw), int(th)]
|
||||
meta["canvas_size"] = [int(canvas[0]), int(canvas[1])]
|
||||
meta["page_orientation"] = page_orient
|
||||
out_meta[idx] = {"page": page, "canvas": canvas, "error": None}
|
||||
|
||||
cov = meta.get("coverage") or {}
|
||||
if cov:
|
||||
cmin = min(float(v) for v in cov.values())
|
||||
if cmin < P["gate"]["coverage_warn"]:
|
||||
meta["warnings"].append(
|
||||
"四边覆盖率最低 %.2f(低于 %.2f),角点可能不准"
|
||||
% (cmin, P["gate"]["coverage_warn"]))
|
||||
if meta.get("reason") == "detect_failed":
|
||||
meta["warnings"].append(
|
||||
"第 %d 页未检测到纸张四角,已用画面内缩兜底框出件;需人工补角"
|
||||
% idx)
|
||||
elif meta.get("reason") == "invalid_quad":
|
||||
meta["warnings"].append(
|
||||
"第 %d 页角点校验未通过,已用兜底框出件;需人工补角" % idx)
|
||||
elif meta.get("fallback"):
|
||||
names = ["上", "右", "下", "左"]
|
||||
sides = "、".join(names[s] for s in meta.get("fallback_sides", [])
|
||||
if 0 <= s < 4)
|
||||
if sides:
|
||||
meta["warnings"].append(
|
||||
"第 %s 边使用了掩膜回退角点,精度可能略低" % sides)
|
||||
for w in (meta.get("validation") or {}).get("warnings", []):
|
||||
meta["warnings"].append(w)
|
||||
if (not meta.get("manual") and batch_orientation != "mixed"
|
||||
and batch_median > 0 and page_class == batch_orientation):
|
||||
dev = abs(meta["aspect"] - batch_median) / batch_median * 100.0
|
||||
if dev > P["gate"]["paper_mismatch_pct"]:
|
||||
meta["warnings"].append(
|
||||
"宽高比与批次中位相差 %.0f%%,可能不是同一种纸;"
|
||||
"可改 paper=auto 或单独处理" % dev)
|
||||
if applied != 0 and user_r is None:
|
||||
meta["warnings"].append(
|
||||
"方向与批次不一致,已旋转 %d°;文字方向需人工确认" % applied)
|
||||
|
||||
# ---------------- 页序 / PDF
|
||||
# plan 已按 ed.order + 源序号排好;只保留真正出页成功的。
|
||||
order = [i for (i, _s, _m) in plan
|
||||
if out_meta.get(i, {}).get("page") is not None]
|
||||
|
||||
ordered_pages = [out_meta[i]["page"] for i in order]
|
||||
ordered_canvases = [out_meta[i]["canvas"] for i in order]
|
||||
if not ordered_pages:
|
||||
failures = unreadable or [
|
||||
_err("RENDER_FAILED", "所有页面都未能出页", None, "重试或检查原图")]
|
||||
res = finish_failure(pages, failures, output, state_path, t0)
|
||||
_mark_error(conn, batch_id, res["summary"])
|
||||
return res
|
||||
|
||||
try:
|
||||
compose.build_pdf(ordered_pages, output, ordered_canvases, P)
|
||||
except Exception as e:
|
||||
err = _err("PDF_WRITE_FAILED", "写 PDF 失败:%s" % e, None,
|
||||
"确认输出目录可写、磁盘空间足够")
|
||||
res = finish_failure(pages, [err], output, state_path, t0)
|
||||
_mark_error(conn, batch_id, res["summary"])
|
||||
return res
|
||||
|
||||
# PDF 落盘成功后再清多余页图:失败时保留上一份,避免两边都不完整。
|
||||
_clean_pngs(png_dir, pdf_stem, png_n)
|
||||
|
||||
# ---------------- 检验
|
||||
try:
|
||||
checks = check.inspect_pdf(output, len(ordered_pages),
|
||||
P["check"]["dpi"], P)
|
||||
except Exception as e:
|
||||
checks = {"ok": None, "error": str(e), "pages": [],
|
||||
"page_count": None}
|
||||
|
||||
warnings = []
|
||||
for m in pages:
|
||||
warnings.extend(m.get("warnings", []))
|
||||
if checks.get("ok") is None:
|
||||
warnings.append("PDF 自检无法执行:%s" % checks.get("error"))
|
||||
else:
|
||||
for item in checks.get("pages", []):
|
||||
orig = order[item["index"] - 1] if 0 < item["index"] <= len(order) \
|
||||
else item["index"]
|
||||
if item.get("box_fallback"):
|
||||
warnings.append(
|
||||
"第 %d 页未能稳定识别纸面区域,边缘检验仅供参考" % orig)
|
||||
if not item["short_side_ok"]:
|
||||
warnings.append(
|
||||
"第 %d 页纸面短边 %dpx 偏低,可能分辨率不足"
|
||||
% (orig, item["short_side_px"]))
|
||||
if not item["border_ok"]:
|
||||
warnings.append(
|
||||
"第 %d 页边缘有背景残留,角点可能落在背景上" % orig)
|
||||
if not item["cut_ok"]:
|
||||
warnings.append(
|
||||
"第 %d 页内容贴边,可能裁到了内容" % orig)
|
||||
if (paper["ratio"] is not None
|
||||
and item["ratio_dev_pct"] > P["check"]["ratio_tol"] * 100):
|
||||
warnings.append(
|
||||
"第 %d 页纸面比例与页面相差 %.1f%%"
|
||||
% (orig, item["ratio_dev_pct"]))
|
||||
pc = checks.get("page_count") or {}
|
||||
if pc and not pc.get("ok"):
|
||||
warnings.append(
|
||||
"PDF 页数与出页数不一致(期望 %s,实际 %s)"
|
||||
% (pc.get("expected"), pc.get("actual")))
|
||||
|
||||
# ---------------- 记录顺序 / 审核状态
|
||||
sort_map = {idx: pos for pos, idx in enumerate(order)}
|
||||
next_sort = len(order)
|
||||
for idx in all_indices:
|
||||
if idx not in sort_map:
|
||||
sort_map[idx] = next_sort
|
||||
next_sort += 1
|
||||
|
||||
needs = []
|
||||
for m in pages:
|
||||
if m.get("status") in ("review", "needs_corners", "needs_orientation",
|
||||
"render_failed", "failed"):
|
||||
needs.append(m)
|
||||
|
||||
# 发布审核页:always 总是发布;auto 只在有页需要人看时发布
|
||||
publish = review_mode == "always" or (review_mode == "auto" and bool(needs))
|
||||
thumbs = {}
|
||||
if publish and batch_id:
|
||||
_, thumbs = review_page.publish(
|
||||
session_root, batch_id,
|
||||
[{"idx": m["index"], "source_abs": m.get("source_abs")}
|
||||
for m in pages
|
||||
if not m.get("deleted") and m.get("source_abs")])
|
||||
|
||||
# ---------------- state v2
|
||||
order_all = sorted(pages, key=lambda m: (sort_map.get(m["index"], m["index"]),
|
||||
m["index"]))
|
||||
state_pages = []
|
||||
for m in order_all:
|
||||
src_rel = _rel(session_root, m.get("source_abs") or "")
|
||||
auto_r = int(m.get("auto_rotate") or 0)
|
||||
state_pages.append({
|
||||
"index": m["index"],
|
||||
"source": src_rel,
|
||||
"source_size": m.get("source_size"),
|
||||
"auto_corners": m.get("auto_corners"),
|
||||
"corners_norm": m.get("corners_norm"),
|
||||
"rotate": auto_r,
|
||||
"rotate_source": "manual" if m.get("manual") else "auto",
|
||||
"status": m.get("status"),
|
||||
"reason": m.get("reason"),
|
||||
})
|
||||
|
||||
state_obj = state.build_state({
|
||||
"paper": paper_input,
|
||||
"enhance": enhance,
|
||||
"dpi": int(P["render"]["dpi"]),
|
||||
"orientation": orientation,
|
||||
}, state_pages)
|
||||
state_ok = write_state(state_path, state_obj)
|
||||
if not state_ok:
|
||||
warnings.append("state 文件写入失败:%s" % state_path)
|
||||
|
||||
if checks.get("ok") is False:
|
||||
summary = "%d 页已生成;自检有未通过项" % len(ordered_pages)
|
||||
elif needs:
|
||||
summary = "%d 页已生成;%d 页建议在审核页确认" % (
|
||||
len(ordered_pages), len(needs))
|
||||
elif warnings:
|
||||
summary = "%d 页已生成;%d 项提示,见审核页" % (
|
||||
len(ordered_pages), len(warnings))
|
||||
else:
|
||||
summary = "%d 页已生成" % len(ordered_pages)
|
||||
if png_paths:
|
||||
summary += ";原始页图见 out/"
|
||||
|
||||
# ---------------- 写 data.db
|
||||
if use_db:
|
||||
output_rel = _rel(session_root, output)
|
||||
review_store.upsert_batch(conn, batch_id, output_rel, {
|
||||
"paper": paper_input, "enhance": enhance,
|
||||
"dpi": dpi_label, "orientation": orientation,
|
||||
})
|
||||
recs = []
|
||||
for m in pages:
|
||||
ti = thumbs.get(m["index"])
|
||||
size = m.get("source_size") or [None, None]
|
||||
rec = {
|
||||
"idx": m["index"],
|
||||
"source": _rel(session_root, m.get("source_abs") or ""),
|
||||
"source_w": size[0], "source_h": size[1],
|
||||
"auto_corners": m.get("auto_corners"),
|
||||
"auto_corners_norm": m.get("auto_corners_norm"),
|
||||
"corners": m.get("corners_norm"),
|
||||
"auto_rotate": int(m.get("auto_rotate") or 0),
|
||||
"rotate": int(m.get("rotated_deg") or 0),
|
||||
"deleted": 1 if m.get("deleted") else 0,
|
||||
"sort": sort_map.get(m["index"], m["index"]),
|
||||
"status": m.get("status"),
|
||||
"reason": m.get("reason"),
|
||||
"thumb": ti[0] if ti else None,
|
||||
"w": ti[1] if ti else (size[0] or None),
|
||||
"h": ti[2] if ti else (size[1] or None),
|
||||
"error": ((m.get("error") or {}).get("message")
|
||||
if isinstance(m.get("error"), dict)
|
||||
else m.get("error")),
|
||||
}
|
||||
recs.append(rec)
|
||||
review_store.replace_pages(conn, batch_id, recs)
|
||||
batch_status = "ready" if not needs else "review"
|
||||
review_store.finish_batch(conn, batch_id, True, summary,
|
||||
warnings, status=batch_status)
|
||||
if review_store.get_job(conn, batch_id):
|
||||
review_store.set_job(conn, batch_id, "done", summary)
|
||||
if is_new_batch:
|
||||
old, kept = review_store.prune(conn, keep=5, protect=batch_id)
|
||||
if old:
|
||||
review_page.prune_dirs(session_root, kept)
|
||||
|
||||
# AI 侧只回三件事:是否成功、一句摘要、PDF 文件名。
|
||||
# 完整访问地址由 core 的 visit_url 追加;pages/checks/批次 ID 都在 data.db/state 里。
|
||||
return {
|
||||
"ok": True,
|
||||
"summary": summary,
|
||||
"pdf": _rel(session_root, output),
|
||||
"images": png_paths,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
_mark_error(conn, batch_id, e)
|
||||
raise
|
||||
finally:
|
||||
if conn:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def finish_failure(pages, failures, output, state_path, t0):
|
||||
err = failures[0] if failures else _err("INTERNAL", "未知错误")
|
||||
return {
|
||||
"ok": False,
|
||||
"summary": "失败:%s" % err.get("message", "未知错误"),
|
||||
"error": err,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 入口
|
||||
|
||||
def main():
|
||||
t0 = time.time()
|
||||
buf = io.StringIO()
|
||||
old_stderr = sys.stderr
|
||||
sys.stderr = buf
|
||||
try:
|
||||
if len(sys.argv) != 2:
|
||||
raise ValueError(
|
||||
'用法:python3 run.py \'{"images": ["图1.jpg"]}\'')
|
||||
try:
|
||||
args = json.loads(sys.argv[1])
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError("参数不是合法 JSON:%s" % e)
|
||||
if not isinstance(args, dict):
|
||||
raise ValueError("参数必须是 JSON 对象")
|
||||
res = process(args, t0)
|
||||
except Exception as e:
|
||||
code = "BAD_ARGS" if isinstance(e, ValueError) else "INTERNAL"
|
||||
hint = ("检查参数后重试" if code == "BAD_ARGS"
|
||||
else "可用 DOC_DESKEW_DEBUG=1 重跑查看堆栈")
|
||||
res = {
|
||||
"ok": False,
|
||||
"summary": "失败:%s" % e,
|
||||
"error": _err(code, str(e), None, hint),
|
||||
}
|
||||
if os.environ.get("DOC_DESKEW_DEBUG"):
|
||||
res["logs"] = traceback.format_exc().splitlines()
|
||||
finally:
|
||||
sys.stderr = old_stderr
|
||||
|
||||
logs = [line for line in buf.getvalue().splitlines() if line.strip()]
|
||||
if logs:
|
||||
res["logs"] = (res.get("logs", []) + logs)[:P["output"]["max_logs"]]
|
||||
|
||||
print(json.dumps(res, ensure_ascii=False, separators=(",", ":"),
|
||||
default=_json_default))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: image-process
|
||||
description: 图像处理(Pillow)。查看图片信息、格式转换、缩放、裁剪、压缩、文字水印、九宫格切片、多图拼图。仅做确定性图像处理,不做 AI 生成/修图
|
||||
description: 普通图片/截图的确定性编辑(Pillow):查看信息、转格式、缩放、裁剪、压缩、加水印、九宫格切片、多图拼图
|
||||
input_schema:
|
||||
type: object
|
||||
properties:
|
||||
@@ -87,7 +87,7 @@ output_schema:
|
||||
- `grid`:把同一目录下多张图拼成一张 contact sheet(缩略图网格)
|
||||
|
||||
## 使用说明
|
||||
通过 `skill` 的 `{"run":"image-process","args":{...}}` 调用,参数:
|
||||
通过 `skill` 的 `{"op":"run","name":"image-process","args":{...}}` 调用,参数:
|
||||
- `action`:操作类型(必填)
|
||||
- `input`:输入图片或目录(必填;相对会话工作目录的路径即可,会自动转绝对路径)
|
||||
- `output`:输出路径(建议给全;不给时自动生成 `<原名>_<动作>.<原扩展名>`,绝不覆盖原图)
|
||||
|
||||
@@ -66,7 +66,7 @@ output_schema:
|
||||
|
||||
## 调用方式
|
||||
|
||||
通过 `skill` 的 `{"run":"x2pdf","args":{...}}` 调用,参数:
|
||||
通过 `skill` 的 `{"op":"run","name":"x2pdf","args":{...}}` 调用,参数:
|
||||
- `input`:输入文件路径(必填)
|
||||
- `output`:PDF 输出路径(可选,默认与输入同名)
|
||||
- `css`:自定义 CSS 文件路径(可选,仅 md/html/txt)
|
||||
|
||||
Reference in New Issue
Block a user