图片回执不在重放时改写:已消费只省略像素,保留当时成功的读图事实

This commit is contained in:
2026-10-22 00:18:30 +08:00
parent 49a3d35227
commit c59ddf5eb4
6 changed files with 23 additions and 21 deletions
+5 -4
View File
@@ -138,11 +138,12 @@ POST /api/sessions/:id/messages
- 含图消息:DB 里存 `{"__blocks__":[...]}` 哨兵 JSON(`encodeBlocks`/`decodeBlocks`);
发请求时 `image_ref` 现签公网短链(`mintUrl`,每次请求重签、不会中途过期)。
`tool` 回执里的未消费图片会**挪出回执**,合并成该批回执之后的一条 `user` 消息
(`tool` 角色带图是非标准形态,很多上游会吞图);已消费的图降级为文字占位。
(`tool` 角色带图是非标准形态,很多上游会吞图);已消费的图只省略像素块,回执原样保留。
- `read_file` 读图的回执是**结构化 JSON**(`tools/executor.js` 的 `imageReceipt`:`ok` /
`image_attached` / `reason` / `width` / `height` …),不是散文。发送侧三处改写(engine 同回合去重、
`buildMessages` 签外链失败与已消费重放)一律走 `patchImageReceipt` **只改字段**;
追加句子会让 JSON 不可解析,模型又退回从自然语言猜「像素到底到没到」。
`image_attached` / `reason` / `width` / `height` …),不是散文。发送侧改写只限两种真事件
(engine 同回合/同批去重、`buildMessages` 签外链失败),一律走 `patchImageReceipt` **只改字段**。
- **已消费的图重放时只省略像素块,不改写回执**:回执是「那次读取」的历史事实,把当时成功的
读图改写成 `image_attached:false` 会让模型以为当时就没拿到画面 → 反复重读同一条(实测循环)。
- 图片注入消息的两条硬约束(踩过循环):①**消费判据是「最后一条写过文本的 assistant」**
(不要求无 `tool_calls`)——旧判据只认收尾回复,多步任务里图永远「未消费」;②**只有本批新增的图
带「一句话说明」的祈使句**,早前批次静默携带。两者合起来才断得掉「每轮要求描述 → 模型描述并继续
+2 -3
View File
@@ -616,8 +616,7 @@ async function runLoopInner(sessionId, run) {
}
const txt = String(m.content || '');
return m.role + (m.tool_calls ? '[calls]' : '')
+ (txt.includes('图片数据已省略') || txt.includes('"omitted_already_sent"') ? '[图已省]' : '')
+ (txt.includes('本回合已投递过') || txt.includes('"already_delivered"') ? '[图重复读挡回]' : '');
+ (txt.includes('"already_delivered"') ? '[图重复读挡回]' : '');
}).join(' '));
}
// 内置工具 + 启动时发现的 MCP 工具(按 profile 白名单过滤),一起随 system prompt 发给模型(压缩回合不发)
@@ -794,7 +793,7 @@ async function runLoopInner(sessionId, run) {
if (!run.imgDelivered) run.imgDelivered = new Map();
if (run.imgDelivered.get(imageRel) === iter) {
const patched = patchImageReceipt(outText, {
image_attached: false, reason: 'already_delivered', retryable: false,
image_attached: false, reason: 'already_delivered', retryable: true,
hint: '同一批 tool call 里已为这个文件投递过图片,本次不再重复;需要重看时在下一轮再读即可',
});
outText = patched !== null
+10 -11
View File
@@ -170,6 +170,8 @@ function imageLabel(text) {
// 同一批图被反复描述直到人工终止)。没写任何文本的中间轮仍算「没看过」,图继续随请求
// 携带(14:33 实测:图已进上下文而模型仍自述没看到,行为鲁棒优先于极致省钱)。
// DB 始终存全量:UI 历史照常渲染图片,模型如需重看可再次 read_file(新回执在尾部)。
// 重放时只省略像素块、**不改写回执字段**:回执是历史事实,改写成「未投递」会让模型
// 把当时成功的读图误判为失败,进而反复重读同一条(2026-10 实机循环的直接成因)。
// 压缩摘要相关行(见 engine.runLoopInner / summarizeContext):
// - type='summary'(assistant,收尾答复):不跳过。它是压缩后后续轮次的上下文起点,
// 正常以 assistant 历史行发给模型;压缩收尾时打此标。
@@ -305,15 +307,12 @@ function buildMessages(session, language, mintUrl, profile, withProfile, trimOn)
let content = row.content || '';
if (blocks) {
if (i < lastAck) {
// 已消费的图重放时**只省略像素,不改写回执**:回执记的是「那次读取发生了什么」
//(当时确实投递了);「这次重放不重复发」只是传输细节,不该写进历史事实。
// 若改写成 image_attached:false / reason:omitted_already_sent / retryable:false,
// 模型会把历史读图当成失败、认为「这次没拿到画面」,于是反复重读同一条(实测循环)。
// 确实需要重看时它自己再 read_file 即可,那条新回执会是完整的成功回执。
content = blocks.filter(b => b && b.type === 'text').map(b => b.text).join('\n');
// 已消费的图重放时不再携带像素:把回执的 image_attached 降为 false 并给出原因,
// 模型读字段就知道「不是又丢了」,不会因叙事惯性重读。旧库散文回执保持原有尾巴。
const consumed = patchImageReceipt(content, {
image_attached: false, reason: 'omitted_already_sent', retryable: false,
hint: '该图已在此前的轮次投递过,重放时不再重复携带像素;若你当时未能识别,请直接告知用户,勿反复重读同一文件',
});
content = consumed !== null ? consumed
: content + '\n[图片数据已省略:本图已随上一条回执发送过,不再重复携带;若你当时未能识别其内容,请直接告知用户,勿反复重读同一文件]';
} else if (row.role === 'tool') {
// 未消费的图:回执只留文字,图块挪进批次后的 user 合并消息(见 pendingImages/Labels)
const texts = blocks.filter(b => b && b.type === 'text').map(b => b.text);
@@ -340,7 +339,7 @@ function buildMessages(session, language, mintUrl, profile, withProfile, trimOn)
if (mintFailed) {
const degraded = patchImageReceipt(content, {
image_attached: false, reason: 'link_unavailable', retryable: false,
hint: '服务端签不出图片外链,本张图的像素未能随本次请求送达;请直接告知用户,勿反复重读同一文件',
hint: '服务端签不出图片外链,本张图的像素未能随本次请求送达;重读也无法送达,请直接告知用户',
});
if (degraded !== null) content = degraded;
}
@@ -368,8 +367,8 @@ function buildMessages(session, language, mintUrl, profile, withProfile, trimOn)
if (pendingImages.length) {
const tail = newCount
? '\n标注「本批新增」的图片是本轮新投递的:各用一句话说明里面有什么(确认像素已收到),'
+ '然后继续你手头的工具流程。本条不是新的用户任务,不要在这里展开分析,也不要重读同一文件。'
: '\n以上均为早前批次已投递过的图片(像素仍在上下文中):不要重复描述,也不要重读同一文件;专注完成用户的任务。';
+ '然后继续你手头的工具流程。本条不是新的用户任务,不要在这里展开分析。'
: '\n以上均为早前批次已投递过的图片(像素仍在上下文中):不要重复描述,专注完成用户的任务。';
messages.push({
role: 'user',
content: [{ type: 'text',
+1 -1
View File
@@ -219,7 +219,7 @@
"type": "function",
"function": {
"name": "read_file",
"description": "Reads the content of a file.\n\nText files: each returned line is prefixed with a line number right-aligned in a 6-character field, followed by a tab, then the line content.\n\nCross-session peek: `path` accepts '@<session-id>/<rel>' to read a text file from your OWN past session, read-only (find the id with recall op=search, browse with list_directory). It is a look, not an import: nothing lands in your session directory, so `terminal`, grep and `edit_file` cannot see it, and past images are refused. When you need to work with a file or view a past image, `copy_path` it first (its `source_path` takes the same '@' form). '@' is never accepted by `write_file` or `edit_file`.\n\nLarge text files (roughly over 16 KB) are not returned in full: you get an outline listing top-level symbols (or Markdown headings) with line numbers instead. The outline IS a successful response — do NOT retry the same call without a line range; use the outline line numbers with `start_line` / `end_line` to read specific sections. If no outline can be extracted, a first-1KB excerpt is returned instead. Supplying `start_line` or `end_line` disables outline mode.\n\nImage files (PNG, JPEG, WebP, GIF, BMP, TIFF): the receipt is a single JSON object, not prose — read its fields instead of interpreting the text. Success shape: {\"ok\":true,\"tool\":\"read_file\",\"kind\":\"image\",\"path\":\"...\",\"image_attached\":true,\"width\":1080,\"height\":1920,\"format\":\"jpeg\",\"bytes_in\":3550000,\"bytes_out\":210000,\"resized\":true,\"source_width\":4000,\"source_height\":7111}. `width`/`height` are the delivered pixel size; `source_width`/`source_height` appear only when the image was downscaled. `image_attached:true` means the server produced the image and it will arrive as an image block in the separate user message that immediately follows the batch of tool results; that message labels each image `[图片 N] read_file <path> <w>×<h>` and asks you to describe it before continuing. The tool result itself never carries pixels. Refusal or failure shape: {\"ok\":false,\"kind\":\"image\",\"path\":\"...\",\"image_attached\":false,\"reason\":\"...\",\"retryable\":false,\"hint\":\"...\"} where `reason` is one of no_vision (this model cannot see images), too_large (roughly over 50 MB), past_session (import it with copy_path first), decode_failed, sharp_missing, already_delivered (this same image was already delivered by an earlier call in the SAME batch of tool calls, so this duplicate was not attached again — reading it again in a later batch always delivers the current pixels, which may have changed since), link_unavailable (no public URL could be minted, so the pixels did not arrive for that request), omitted_already_sent (an older receipt whose image was already delivered; replay omits the pixels). `ok:true` with `image_attached:false` is still a successful call, e.g. a deduplicated re-read. When `retryable` is false do not retry — tell the user instead. Images may be downscaled (long edge ~2000 px, re-encoded as JPEG) before delivery.\n\nBinary files are not returned as text; a short notice is returned instead. Text files over 32 MB cannot be read by this tool — use `terminal` (grep -n / head / sed -n).",
"description": "Reads the content of a file.\n\nText files: each returned line is prefixed with a line number right-aligned in a 6-character field, followed by a tab, then the line content.\n\nCross-session peek: `path` accepts '@<session-id>/<rel>' to read a text file from your OWN past session, read-only (find the id with recall op=search, browse with list_directory). It is a look, not an import: nothing lands in your session directory, so `terminal`, grep and `edit_file` cannot see it, and past images are refused. When you need to work with a file or view a past image, `copy_path` it first (its `source_path` takes the same '@' form). '@' is never accepted by `write_file` or `edit_file`.\n\nLarge text files (roughly over 16 KB) are not returned in full: you get an outline listing top-level symbols (or Markdown headings) with line numbers instead. The outline IS a successful response — do NOT retry the same call without a line range; use the outline line numbers with `start_line` / `end_line` to read specific sections. If no outline can be extracted, a first-1KB excerpt is returned instead. Supplying `start_line` or `end_line` disables outline mode.\n\nImage files (PNG, JPEG, WebP, GIF, BMP, TIFF): the receipt is a single JSON object, not prose — read its fields instead of interpreting the text. Success shape: {\"ok\":true,\"tool\":\"read_file\",\"kind\":\"image\",\"path\":\"...\",\"image_attached\":true,\"width\":1080,\"height\":1920,\"format\":\"jpeg\",\"bytes_in\":3550000,\"bytes_out\":210000,\"resized\":true,\"source_width\":4000,\"source_height\":7111}. `width`/`height` are the delivered pixel size; `source_width`/`source_height` appear only when the image was downscaled. `image_attached:true` means the server produced the image and it will arrive as an image block in the separate user message that immediately follows the batch of tool results; that message labels each image `[图片 N] read_file <path> <w>×<h>` and asks you to describe it before continuing. The tool result itself never carries pixels. Refusal or failure shape: {\"ok\":false,\"kind\":\"image\",\"path\":\"...\",\"image_attached\":false,\"reason\":\"...\",\"retryable\":false,\"hint\":\"...\"} where `reason` is one of no_vision (this model cannot see images), too_large (roughly over 50 MB), past_session (import it with copy_path first), decode_failed, sharp_missing, already_delivered (this same image was already delivered by an earlier call in the SAME batch of tool calls, so this duplicate was not attached again — reading it again in a later batch always delivers the current pixels, which may have changed since), link_unavailable (no public URL could be minted, so the pixels did not arrive for that request). `ok:true` with `image_attached:false` is still a successful call, e.g. a deduplicated re-read. When `retryable` is false do not retry — tell the user instead. Images may be downscaled (long edge ~2000 px, re-encoded as JPEG) before delivery.\n\nBinary files are not returned as text; a short notice is returned instead. Text files over 32 MB cannot be read by this tool — use `terminal` (grep -n / head / sed -n).",
"parameters": {
"type": "object",
"required": [
+2 -2
View File
@@ -186,8 +186,8 @@ function outlineWrapper(outlineText) {
// path 模型请求的路径(@ 历史形原样保留);投递产物路径只在服务端内部用,不回执
// image_attached 服务端是否产出了图块、会随本批回执之后的那条 user 消息投递(≠上游已收到)
// reason 仅 image_attached=false 时有值,稳定枚举(no_vision / too_large / past_session /
// decode_failed / sharp_missing / already_delivered / link_unavailable /
// omitted_already_sent)——模型按它分支,不要读 hint 猜
// decode_failed / sharp_missing / already_delivered / link_unavailable)——
// 模型按它分支,不要读 hint 猜。注意:重放省略像素不属于回执事件,不写 reason。
// retryable 重试同一调用是否有意义(图片这条路上恒 false:同样的字节只会得到同样的结果)
// width/height 实际投递的像素尺寸;source_width/height + resized 只在确实缩放过时出现
// hint 一句人话补充,仅供解释,不承载判定
+3
View File
@@ -74,4 +74,7 @@ tools 数组每轮请求完整重发。新工具进来时,先问"能不能装
`width` `height` 之类的事实字段 / `hint`(仅解释,不承载判定)。
- 回执可能被多层改写(发送前裁剪、同回合去重、投递失败降级):改写**只动字段**,不得在 JSON 后追加句子——
追加会把回执变成不可解析的尾巴,等于把判定权又还给语义猜测。
- **回执是「那次调用」的历史事实,不是当前状态**:发送侧为了省 token 省略内容时,不要改写回执
(典型错例:把当时成功的图片投递改写成 `image_attached:false`)。模型会把历史读图当成失败,
于是反复重读同一条;省略、重放、缓存这类传输层细节不属于回执语义。
- `ok:true` 不等于「结果已送达」:另有字段(如 `image_attached`)描述投递状态,两者不要混成一个布尔。