llm: 工具回执夹图改批次后 user 合并投递 + 同回合去重护栏(线上死循环复盘)

多模态上游兼容性事故:tool 角色消息带 image_url 块是非标准形态,大量
OpenAI 兼容上游整批忽略或只认批次最后一张——模型看不到图、却见回执写着
「已回传」,加上旧催读文案「如需重读请再次 read_file」火上浇油,模型一轮轮
重读同一批图烧 token 死循环。改为:

- llm.js buildMessages:未消费的 tool 回执图不塞回执内(回执降级纯文本),
  image_ref 现签外链后统一并入该批回执之后的一条 user 消息(user 角色各家
  通吃);每张图前配一行「[图片 N,对应回执] <回执首行>」标签,模型可对号;
  旧库 data: URL 块原样并入;签不到降级文字占位。顺序天然满足 assistant→
  tool 紧邻配对约束(合并消息落在整段历史尾部)。
- 两处催读文案(已省略/外链不可用)改为「请告知用户、勿反复重读同一文件」。
- engine.js runLoopInner 死循环护栏:run.imgDelivered 按图 rel 去重,同回合
  再读同图只回文字提示不再签新图;run 随回合销毁,新回合自动重置,正常
  追问重看不受影响。
- definitions.json read_file 描述同步新形态(图在批次回执后的独立 user
  消息到达;同回合重读不再带像素,勿重试)。
- ARCHITECTURE.md §5.1 图片约定、§14 陷阱 7 改写,记录本次复盘。

验证:容器内构造 history 直调 buildMessages 六场景单测全过(合并/标签/
现签/签失败占位/已消费省略/同批多图序号/旧 data: 透传/user 行原位现签);
JSON 与 node --check 通过;restart agent-core 启动干净、微信对账正常。
This commit is contained in:
2026-09-28 12:56:39 +08:00
parent 3d8477e7e0
commit e1bee18cd0
4 changed files with 57 additions and 13 deletions
+15 -2
View File
@@ -689,8 +689,9 @@ async function runLoopInner(sessionId, run) {
}
}
// 工具可返回 {text, image_rel}(read_file 读图且模型支持 vision,压缩产物在 .imgcache/):
// 落库时以 image_ref 内容块拼进 tool 消息(哨兵 JSON,见 llm.js encodeBlocks),
// buildMessages 发请求前现签成公网外链的 image_url 块;前端工具卡走带 cookie 的下载路由渲染。
// 落库时以 image_ref 内容块拼进 tool 消息(哨兵 JSON,见 llm.js encodeBlocks);
// buildMessages 发请求前把未消费的图现签成公网外链 image_url 块,并入该批回执之后的
// 一条 user 消息发送(tool 角色带图非标准形态,上游支持度差);前端工具卡走带 cookie 的下载路由渲染。
let outText;
let imageRel = null;
if (output && typeof output === 'object' && output.image_rel) {
@@ -700,6 +701,18 @@ async function runLoopInner(sessionId, run) {
outText = String(output == null ? '' : output);
}
if (run.cancelled && !isError) outText = (outText ? outText + '\n' : '') + '(用户已终止任务)';
// 死循环护栏:同一回合内同一张图只投递一次——若上游没认图,模型会一轮轮重读同一文件、
// 每轮都签新图烧 token;同回合再读同图只回文字提示,不再携带图片(run 随回合结束销毁,
// 新回合/新用户消息自动重置额度,正常追问重看不受影响)。
if (imageRel) {
if (!run.imgDelivered) run.imgDelivered = new Set();
if (run.imgDelivered.has(imageRel)) {
outText += '\n[该图片本回合已投递过,不再重复携带;若你未看到它,请直接告知用户,勿反复重读同一文件]';
imageRel = null;
} else {
run.imgDelivered.add(imageRel);
}
}
log.debug('tool', sessionId + ' ' + tc.function.name + (isError ? ' 失败: ' : ' 完成: ') + outText);
const toolContent = imageRel
? encodeBlocks([
+39 -8
View File
@@ -123,12 +123,16 @@ function cachedDecodeBlocks(session, row) {
// 将 DB 消息行转为 OpenAI messages 数组(不含 system,由调用方拼接)。
// 图片两种形态:旧库/用户粘贴附件的 image_url 块(data: URL)原样透传;
// 新版 read_file 读图存的是 image_ref 块({rel},指向任务目录 .imgcache/ 产物),
// 发送前用 mintUrl(rel)(engine 传入的公网链现签回调)换成 image_url 外链块,
// 新版 read_file 读图存的是 image_ref 块({rel},指向任务目录 .imgcache/ 产物)。
// tool 回执里的未消费图不放在回执内发送——tool 角色带 image_url 非标准形态,
// 大量 OpenAI 兼容上游会整批忽略或只认批次最后一张,模型看不到图却见回执写着
// 「已回传」,会反复 read_file 补看(死循环烧 token,见多模态回退复盘);
// 故回执降级为纯文本,图块现签成 image_url(mintUrl 公网链回调)后统一并入
// 该批回执之后的一条 user 消息(user 角色各家通吃),每张图前配一行对应回执文字;
// 签不到(base_url 未配等)降级为文字占位,绝不把 image_ref 这种非标准块发上游。
// 省 token:图块仅在其位于最后一条 assistant 消息之后(模型尚未对它回复)时保留;
// 之前的视为已消费,发请求时替换为文字标记。DB 始终存全量:UI 历史照常渲染图片,
// 模型如需重看可再次 read_file(新消息在尾部,会重新携带图片)。
// 省 token:图块仅在其位于最后一条 assistant 消息之后(模型尚未对它回复)时携带;
// 之前的视为已消费,发请求时替换为文字标记(措辞不再催促重读)。DB 始终存全量:
// UI 历史照常渲染图片,模型如需重看可再次 read_file(新回执在尾部,会重新携带图片)。
// 压缩摘要相关行(见 engine.runLoopInner / summarizeContext):
// - type='summary'(assistant,收尾答复):不跳过。它是压缩后后续轮次的上下文起点,
// 正常以 assistant 历史行发给模型;压缩收尾时打此标。
@@ -171,6 +175,11 @@ function buildMessages(session, language, mintUrl, profile, withProfile) {
history.forEach((row, i) => {
if (row.role === 'assistant' && !row.tool_calls) lastClosingAssistant = i;
});
// 未消费图片的暂存区:tool 回执降级为纯文本,图块收集到这里,
// 循环结束后合并成一条 user 消息发送(顺序约束:天然落在该批全部回执之后、
// 且是模型下一次的直接输入,assistant→tool 回执的紧邻约束不被打断)。
const pendingImages = [];
let imgSeq = 0;
history.forEach((row, i) => {
// archived:UI 可见,但不进入模型上下文(节省 token)
if (row.type === 'archived') return;
@@ -215,13 +224,27 @@ function buildMessages(session, language, mintUrl, profile, withProfile) {
if (blocks) {
if (i < lastAssistant) {
content = blocks.filter(b => b && b.type === 'text').map(b => b.text).join('\n') +
'\n[图片数据已省略:模型此前已识别过本图;如需重新查看请再次调用 read_file]';
'\n[图片数据已省略:本图已随上一条回执发送过,不再重复携带;若你当时未能识别其内容,请直接告知用户,勿反复重读同一文件]';
} else if (row.role === 'tool') {
// 未消费的图:回执只留文字,图块挪进批次后的 user 合并消息(见 pendingImages)
const texts = blocks.filter(b => b && b.type === 'text').map(b => b.text);
content = texts.join('\n');
for (const b of blocks) {
if (!b || (b.type !== 'image_ref' && b.type !== 'image_url')) continue;
// 每张图配一行对应回执说明(回执首行含文件名与压缩信息),模型能对上号
pendingImages.push({ type: 'text', text: '[图片 ' + (++imgSeq) + ',对应回执] ' + ((texts[0] || 'read_file 图片回传').split('\n')[0]) });
if (b.type === 'image_url') { pendingImages.push(b); continue; } // 旧库内联 data: URL 原样
const url = mintUrl ? mintUrl(b.rel) : null;
pendingImages.push(url ? { type: 'image_url', image_url: { url } }
: { type: 'text', text: '[图片外链不可用:未能取得图片内容,请直接告知用户,勿反复重读同一文件]' });
}
} else {
// 未消费:image_ref → 现签公网外链(每次请求重签,不过期);其余块(含旧 data: URL)原样
// user 行(粘贴附件/旧形态):image_url 块(data: URL)原样;image_ref → 现签公网外链
//(每次请求重签,不过期)
content = blocks.map(b => {
if (!b || b.type !== 'image_ref') return b;
const url = mintUrl ? mintUrl(b.rel) : null;
return url ? { type: 'image_url', image_url: { url } } : { type: 'text', text: '[图片外链不可用:请重新读取]' };
return url ? { type: 'image_url', image_url: { url } } : { type: 'text', text: '[图片外链不可用:请直接告知用户,勿反复重读同一文件]' };
});
}
}
@@ -231,6 +254,14 @@ function buildMessages(session, language, mintUrl, profile, withProfile) {
messages.push({ role: row.role, content });
}
});
// 该批未消费图片合并发送:一条 user 消息带全部图块,逐图前置对应回执标签
if (pendingImages.length) {
messages.push({
role: 'user',
content: [{ type: 'text', text: '[系统] 以上工具回执对应的图片内容如下(read_file 回传,按回执顺序):' }]
.concat(pendingImages),
});
}
return messages;
}
+1 -1
View File
@@ -202,7 +202,7 @@
"type": "function",
"function": {
"name": "read_file",
"description": "Reads the content of a file.\n\nText files: each returned line is prefixed with a line number right-aligned in a 6-character field, followed by a tab, then the line content.\n\nLarge text files (roughly over 16 KB) are not returned in full: you get an outline listing top-level symbols (or Markdown headings) with line numbers instead. The outline IS a successful response — do NOT retry the same call without a line range; use the outline line numbers with `start_line` / `end_line` to read specific sections. If no outline can be extracted, a first-1KB excerpt is returned instead. Supplying `start_line` or `end_line` disables outline mode.\n\nImage files (PNG, JPEG, WebP, GIF, BMP, TIFF) are returned as visual content you can analyze directly, but only if the current model supports vision; otherwise a text placeholder is returned — that placeholder is not an error. Very large images (roughly over 50 MB) are skipped, and images may be downscaled (long edge ~2000 px, re-encoded as JPEG) before being returned.\n\nBinary files are not returned as text; a short notice is returned instead. Text files over 32 MB cannot be read by this tool — use `terminal` (grep -n / head / sed -n).",
"description": "Reads the content of a file.\n\nText files: each returned line is prefixed with a line number right-aligned in a 6-character field, followed by a tab, then the line content.\n\nLarge text files (roughly over 16 KB) are not returned in full: you get an outline listing top-level symbols (or Markdown headings) with line numbers instead. The outline IS a successful response — do NOT retry the same call without a line range; use the outline line numbers with `start_line` / `end_line` to read specific sections. If no outline can be extracted, a first-1KB excerpt is returned instead. Supplying `start_line` or `end_line` disables outline mode.\n\nImage files (PNG, JPEG, WebP, GIF, BMP, TIFF) are returned as visual content you can analyze directly, but only if the current model supports vision; otherwise a text placeholder is returned — that placeholder is not an error. The tool result itself carries a text notice only; the actual image pixels arrive in a separate user message right after the batch of tool results, each image preceded by a label naming the read_file result it belongs to. Re-reading the same image within one turn returns a text-only notice without pixels — if an image never arrives, tell the user instead of retrying the read. Very large images (roughly over 50 MB) are skipped, and images may be downscaled (long edge ~2000 px, re-encoded as JPEG) before being returned.\n\nBinary files are not returned as text; a short notice is returned instead. Text files over 32 MB cannot be read by this tool — use `terminal` (grep -n / head / sed -n).",
"parameters": {
"type": "object",
"required": [