feat(fetch): 下载/返回双上限解耦 + offset 回合缓存续读 + SSRF 防护 + HTML 剥壳增强

- 返回窗收紧:max_bytes(下载,1MB)与 max_chars(返回,默认 12000/硬帽 64000)拆分;单次抓取上下文开销从最坏几十万 token 掉到数千
- offset 续读:超窗尾部提示全文长度与本段区间,转换后全文挂 ctx.run.fetchCache(回合级 LRU 3 条),续读纯本地切片不重抓;字节截断如实标注不完整,不承诺无效续读
- SSRF 防护(默认开):主机名解析结果含回环/私有/链路本地即拒,redirect:manual 逐跳复查至多 5 次;fetch.allow_private: true 可放行;DNS 重绑定为标注的残余风险
- htmlToMarkdown 补剥 nav/header/footer/aside/form 整块与 class/id 样板词启发式丢块,<title> 提成首行;字节截断尾 U+FFFD 去噪
- schema 增 offset/max_chars;旧 config.yml 经 deepMerge 自动获得新默认,无需迁移
- 验证:临时功能脚本 22/22(真 HTTP server 打 executeTool),dep-check 通过;smoke 的 held 超时经 stash 对照确认为既有环境问题,与本改动无关
This commit is contained in:
2026-09-29 16:27:02 +08:00
parent 8ffaac5f64
commit 147f0654c5
5 changed files with 168 additions and 16 deletions
+12 -1
View File
@@ -90,7 +90,7 @@
"type": "function",
"function": {
"name": "fetch",
"description": "Fetches a URL and returns its content as Markdown. Only http and https are supported; if the scheme is omitted, https is assumed.\n\nHTML pages are converted to Markdown. JSON, XML, plain text, and Markdown responses are returned as-is. Binary responses are not returned — only their HTTP status, content-type, and size.\n\nThe response always starts with a line `HTTP <status>`, followed by the body. Check the status line: a non-2xx status means the fetch did not succeed, even if body content is present.\n\nRequests are subject to a timeout, and oversized responses are truncated. Redirects are followed automatically.",
"description": "Fetches a URL and returns its content as Markdown. Only http and https are supported; if the scheme is omitted, https is assumed.\n\nHTML pages are converted to Markdown. JSON, XML, plain text, and Markdown responses are returned as-is. Binary responses are not returned — only their HTTP status, content-type, and size.\n\nThe response always starts with a line `HTTP <status>`, followed by the body. Check the status line: a non-2xx status means the fetch did not succeed, even if body content is present.\n\nReturned text is capped per call (default is modest; raise one call with `max_chars` up to a hard cap). When truncated, the tail hint gives the total length and the `offset` to continue from — re-call with that `offset` to page through WITHOUT re-downloading. Most pages fit in the first window; only page further when you actually need more.\n\nTargets that resolve to private/loopback addresses are refused (SSRF guard). Redirects are followed up to 5 hops, each hop checked.",
"parameters": {
"type": "object",
"required": [
@@ -100,6 +100,17 @@
"url": {
"type": "string",
"description": "The URL to fetch. Must be http or https; if no scheme is given, https is assumed."
},
"offset": {
"type": "integer",
"minimum": 0,
"default": 0,
"description": "Character offset into the converted page body, for continuing a truncated fetch. Use the offset value given in the previous truncation hint. Requires no re-download."
},
"max_chars": {
"type": "integer",
"minimum": 1,
"description": "Max characters to return for this call. Optional; raise only when the default window clearly cut off content you need. Values above the hard cap are clamped."
}
}
}