Files
Obsidian/X-Articles-CN/永续 Agent 的最后一环:Self-Improving Skills.md
coco 8b94d76799 a
2026-07-21 13:57:15 +08:00

495 lines
16 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: "\"永续\" Agent 的最后一环:Self-Improving Skills"
source: "https://x.com/xxx111god/status/2033261636195373123"
author:
- "[[Unknown]]"
published: 2026-03-16
created: 2026-03-16
description:
tags:
- "clippings"
---
过去2个月我使用OpenClaw🦞过程中一直在听到“永续工作”的概念,也写了几篇我自己思考和实践的文章
[任务管理系统](https://x.com/xxx111god/status/...)让 agent 能 24/7 自主工作,issue 系统追踪每个任务的状态
[记忆系统](https://x.com/xxx111god/status/...)实现了 L0/L1/L2 分层管理,daily logs 自动归档,semantic search 快速检索
这些系统让 agent 能持续运行,不会因为"忘了之前做什么"而中断。
但有个盲点我一直没解决:Skills 本身。
我现在在用 30 多个技能,覆盖几乎所有日常工作:XHS/X 内容制作流程、播客全自动化(collector → editor → tts → publisher)、预测市场交易(auto-research + monitor + execution)、发邮件、写代码、代码审查、查天气、管理日程、搜索文档。它们每天被调用几十上百次。
但这些 skills 是静态的,它们不在"永续系统"里。高频使用时经常出现各种问题:"诶,这个工具路径怎么找不到了?""怎么又报错了?上次不是能跑吗?""换到 writer session 怎么不知道怎么用这个 skill 了?"每次都要手动排查:是 skill 写得不够清楚?是环境变了工具路径变了?是跨 session 没有足够的上下文?然后改 SKILL.md,重启,重新加载。
这不对。如果 agent 要永续工作,Skills 也要永续改进。任务可以追踪,记忆可以归档,Skills 为什么不能自己进化?
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 第一次接触到"Skills 迭代"
第一次接触到"Skills 迭代"的概念,是看 [@baoyu](https://github.com/dotey/openai-quickstart/blob/main/writing_style.md) 老师分享他的写作风格 skill。他提了一个观点:写作风格不应该是静态 prompt 文件,应该让 AI 根据反馈不断进化,否则所谓的去AI味的写作风格也会沦为千篇一律的新AI味。
我用宝玉老师skill创建的我个人的写作风格 skill 现在是 v1.11.0,已经迭代了 11 个版本,但每次都要我主动触发更新。用了这么久,一直在想:这个过程能自动化吗?
其实答案是显而易见的。不是手动告诉 AI 怎么改,而是让系统自己观察、分析、提出改进。这才是永续 Agent 的最后一环。
核心是一个 5 步循环:Ingest → Observe → Inspect → Improve → Evaluate。下面分享我是怎么理解这 5 步的。
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## Step 1: Ingest(结构化存储)
目标:把 SKILL.md 变成可查询的结构。 静态 markdown 文件没法程序化分析,需要结构化才能做后续的 Inspect 和 Improve。
数据结构示例
```json
{
"id":"gog-email",
"name":"Send Email via Google Workspace",
"trigger_patterns":[
"send email to *",
"email * about *"
],
"tools":[
"gog mail send"
],
"dependencies":[
"SMTP credentials",
"OAuth token"
],
"constraints":{
"timeout":30,
"max_attachment_size":25
},
"last_updated":"2026-03-10T15:30:00Z",
"version":"1.3.0"
}
```
实现方式
可以用现成的工具(比如 [Cognee](https://github.com/topoteretes/cognee) 的结构化存储功能),或者自己写 parser 把 SKILL.md → JSON → 存 SQLite。关键是要把 trigger patterns 做 semantic embedding(方便后续匹配),把 tools、dependencies、constraints 分开存,方便后续分析到底是哪个部分出问题。
一开始我犯的错误
不要只存 prompt text,要把每个字段都结构化,不然后面 Inspect 时没法细分问题。
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## Step 2: Observe(记录执行)
目标:每次 skill 调用留下可审计的痕迹。 没有执行记录就没法知道哪里失败,这是整个循环的基础。
数据结构示例
```json
{
"run_id":"8f3a9c2b",
"timestamp":"2026-03-14T16:20:15Z",
"skill_id":"gog-email",
"skill_version":"1.3.0",
"task":"send email to john@example.com about meeting",
"selected_by":"semantic_match",
"match_score":0.87,
"success":false,
"error":{
"type":"SMTPTimeout",
"message":"Connection timeout after 30s",
"traceback":"..."
},
"context":{
"attachment_size_mb":8.5,
"time_of_day":"16:20",
"network_latency_ms":450
},
"user_feedback":"manually sent via Gmail web"
}
```
实现方式
在 skill wrapper 里加一段 logging:
```python
# ~/clawd/lib/skill_executor.py
from uuid import uuid4
from datetime import datetime, date
import json
def execute_skill(skill_id, task):
run_id = uuid4()
try:
result = _run_skill(skill_id, task)
log_run(
run_id=run_id,
skill_id=skill_id,
task=task,
success=True,
result=result
)
return result
except Exception as e:
log_run(
run_id=run_id,
skill_id=skill_id,
task=task,
success=False,
error=e
)
raise
def log_run(run_id, skill_id, task, success, error=None, result=None):
run_data = {
"run_id": str(run_id),
"timestamp": datetime.utcnow().isoformat(),
"skill_id": skill_id,
"task": task,
"success": success,
"error": serialize_error(error) if error else None,
"context": get_runtime_context(),
}
with open(f"~/memory/skill-runs/{date.today()}.jsonl", "a") as f:
f.write(json.dumps(run_data) + "\n")
```
三个教训
一开始只记录 true/false,后来发现没用,要记录错误类型和 context。只在最外层捕获异常,工具调用失败的细节丢了,要在各层都记录。忘记记录 skill version,改进后没法对比新旧版本效果。
额外有用的数据
用户有没有手动 retry?(说明 skill 不可靠)用户有没有改 task 重新提交?(说明 trigger pattern 不准)调用这个 skill 前后还用了什么?(说明 routing 问题)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## Step 3: Inspect(失败模式分析)
目标:从执行日志里提取 actionable insights。 有日志不够,要能看出模式。"gog skill 失败率 30%" 没用,"gog skill 在发 >5MB 附件时 80% 超时" 才有用。
分析维度
```python
# ~/clawd/scripts/skill-inspector.py
from collections import Counter
def analyze_failures(skill_id, days=7):
runs = load_runs(skill_id, days)
# 1. 总体失败率 (overall failure rate)
total = len(runs)
failed = [r for r in runs if not r["success"]]
failure_rate = len(failed) / total if total else 0
# 2. 按错误类型分组 (group by error type)
error_types = Counter(r["error"]["type"] for r in failed if r.get("error"))
# 3. 按 context 维度分组 (group by attachment size)
by_attachment_size = group_by(
failed,
key=lambda r: r["context"].get("attachment_size_mb", 0),
buckets=[0, 1, 5, 10, float("inf")]
)
# 4. 关联分析 (correlation analysis)
high_latency = [
r for r in failed
if r["context"].get("network_latency_ms", 0) > 300
]
network_issue_ratio = len(high_latency) / len(failed) if failed else 0
return {
"failure_rate": failure_rate,
"error_types": error_types,
"patterns": {
"large_attachments": by_attachment_size,
"network_issues": network_issue_ratio,
}
}
```
输出示例
```markdown
Skill: gog-email
Failure Rate:
28% (14 / 50 runs)
Error Types:
- SMTPTimeout: 10 times (71%)
- AuthError: 3 times (21%)
- InvalidRecipient: 1 time (7%)
Patterns:
Attachment Size
- > 5MB: 80% failure (8 / 10)
- < 5MB: 10% failure (2 / 40)
Network Conditions
- Latency > 300ms: 90% failure (9 / 10)
Time Patterns
- Hour 16–18 (peak traffic): 40% failure (6 / 15)
```
关键洞察
SMTP timeout 不是随机的,是大附件 + 高延迟的组合导致的。修复方向:增加 timeout、改用 Drive link for large files。
为什么一开始分析不出东西
一开始只看总失败率,没发现是特定 context 下才失败。只看单一维度,错过了"大附件 × 高延迟"这种交叉模式。忽略成功案例,对比成功/失败的 context 差异才能找到根因。
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## Step 4: Improve(提出改进)
目标:基于 Inspect 的分析,生成具体的改进方案。 分析完不改进等于白分析,但改进不能瞎改,要有证据支撑。
Improvement 数据结构
```json
{
"improvement_id":"gog-email-v1.4.0",
"skill_id":"gog-email",
"base_version":"1.3.0",
"proposed_at":"2026-03-14T17:00:00Z",
"rationale":{
"failure_rate":0.28,
"root_cause":"SMTP timeout on large attachments + high network latency",
"evidence":{
"large_attachment_failure_rate":0.80,
"small_attachment_failure_rate":0.10,
"high_latency_correlation":0.90
}
},
"changes":[
{
"type":"constraint_update",
"field":"timeout",
"old_value":30,
"new_value":60,
"reason":"80% of failures are SMTP timeout with >5MB attachments"
},
{
"type":"logic_addition",
"description":"Use Google Drive link for attachments >5MB",
"pseudo_code":"if attachment_size > 5MB: upload_to_drive() + send_link()"
},
{
"type":"retry_policy",
"config":{
"max_retries":3,
"backoff":"exponential",
"retry_on":[
"SMTPTimeout"
]
}
}
],
"status":"proposed",
"approval_required":true
}
```
实现方式
```python
# ~/clawd/scripts/skill-improver.py
def propose_improvement(skill_id, analysis):
improvements = []
# Rule 1: Timeout issues
if "SMTPTimeout" in analysis["error_types"]:
timeout_failures = analysis["error_types"]["SMTPTimeout"]
if timeout_failures / analysis["total_failures"] > 0.5:
improvements.append({
"type": "constraint_update",
"field": "timeout",
"new_value": current_timeout * 2,
"reason": f"{timeout_failures} timeout failures"
})
# Rule 2: Large attachment pattern
if analysis["patterns"]["large_attachments"][">5MB"]["failure_rate"] > 0.7:
improvements.append({
"type": "logic_addition",
"description": "Use Drive link for >5MB attachments",
"reason": "80% failure rate on large attachments"
})
# Rule 3: Network latency correlation
if analysis["patterns"]["network_issues"] > 0.8:
improvements.append({
"type": "retry_policy",
"config": {
"max_retries": 3,
"backoff": "exponential"
},
"reason": "90% correlation with high network latency"
})
return create_improvement(skill_id, improvements, analysis)
```
什么情况下改进会适得其反
一次改太多,搞不清楚到底是哪个改动起作用。没证据就改,瞎改一通可能引入新问题。改完直接 apply,应该先 proposed 状态人工 review。
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## Step 5: Evaluate(A/B 测试)
目标:验证改进是否真的有效。 改进可能引入新问题,"修了 A 问题,搞出 B 问题"很常见。
Evaluation 数据结构
```json
{
"evaluation_id":"eval-8a3f2c1b",
"improvement_id":"gog-email-v1.4.0",
"evaluation_period":{
"start":"2026-03-15T00:00:00Z",
"end":"2026-03-22T00:00:00Z",
"duration_days":7
},
"baseline":{
"version":"1.3.0",
"runs":50,
"failure_rate":0.28,
"avg_execution_time_s":12.5,
"error_types":{
"SMTPTimeout":10,
"AuthError":3
}
},
"treatment":{
"version":"1.4.0",
"runs":52,
"failure_rate":0.08,
"avg_execution_time_s":15.2,
"error_types":{
"SMTPTimeout":2,
"AuthError":2
}
},
"metrics":{
"failure_rate_improvement":-0.20,
"execution_time_regression":+2.7,
"new_errors_introduced":0
},
"verdict":"approve",
"reason":"Failure rate 降低 71%,execution time 增加可接受"
}
```
实现方式
A/B 流量分配:10% 流量走新版本,90% 走旧版本。
```python
import random
def route_to_version(skill_id, task):
# 10% traffic to new version (1.4.0), 90% to baseline (1.3.0)
if random.random() < 0.1:
return execute_skill(skill_id, task, version="1.4.0")
else:
return execute_skill(skill_id, task, version="1.3.0")
# Evaluation criteria
def evaluate_improvement(baseline, treatment):
# Must-have: failure rate must decrease
if treatment["failure_rate"] >= baseline["failure_rate"]:
return "reject", "No failure rate improvement"
# Nice-to-have: execution time should not regress too much
time_regression = (
treatment["avg_execution_time_s"]
- baseline["avg_execution_time_s"]
)
if time_regression > baseline["avg_execution_time_s"] * 0.5:
return "reject", f"Execution time +{time_regression:.1f}s (>50%)"
# Must-not: cannot introduce new error types
new_errors = set(treatment["error_types"]) - set(baseline["error_types"])
if new_errors:
return "reject", f"New errors: {new_errors}"
# Compute improvement percentage
improvement = (
baseline["failure_rate"] - treatment["failure_rate"]
) / baseline["failure_rate"]
return "approve", f"Failure rate improved {improvement:.1%}"
```
A/B 测试容易踩的三个坑
只看失败率,执行时间变慢 50% 也是问题。样本太小,10 次测试就下结论统计意义不够。改完立刻全量,应该先 10% → 50% → 100% 逐步放量。
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 我的 3 个实例
1\. OpenClaw Skills(30+ 个)
现状是全静态,失败不可见。我已经加了 Observation Layer,现在每次 skill 调用都有 JSONL 日志。下一步是写 Inspector 脚本,看看 gog/coding-agent/github 的失败模式。
2\. Trading Autoresearch(30 个策略)
现状是已有 Observe(autoresearch\_results.jsonl),缺的是 Improve(自动提出策略变体)。
计划实现:
\# 当 neighbor\_t2\_d1 在高波下失败
if brier\_score > 0.15 and volatility > 0.03:
propose\_variants(\[
"neighbor\_t2\_d2", # distance 更保守
"neighbor\_t2\_d1\_v10" # 加 volatility filter
\])
然后 paper trades 跑 2 周,Brier score 对比,更好的替换原策略。
3\. Writing Style(v1.11.0)
现状是手动迭代。可以自动化的部分:Observe 捕获我的修改 diff,Inspect 统计最常违反的规则,Improve 自动提出新禁止词。示例:过去 7 天修改记录显示"综上所述"出现 12 次、"不得不说"出现 8 次,自动提案 v1.12.0 新增这两个禁止词,然后测试新版本看我的修改次数是否减少。
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 结尾
Static Skills = Copilot(人工维护),Self-Improving Skills = Autopilot(自己优化)。但 Autopilot 必须有 safeguards:Improve 必须有证据(Inspect 的分析),Evaluate 必须验证改进(A/B 测试),无效就回滚(不能盲目自改)。不然就变成 Gilfoyle 的 "Son of Anton",删了所有代码说 "efficient debugging
从 [@baoyu](https://x.com/@baoyu) 老师学到"技能可以迭代",后来发现"迭代可以自动化"
核心是 5 步循环:Ingest → Observe → Inspect → Improve → Evaluate。我已经开始加 observation layer,让失败变得可见,让改进变得自动。这才是永续 Agent 的最后一环。
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 相关链接
- Cognee Self-Improving Skills: [https://github.com/topoteretes/cognee](https://github.com/topoteretes/cognee)
- [@baoyu](https://x.com/@baoyu) skill: [https://github.com/JimLiu/baoyu-skills](https://github.com/JimLiu/baoyu-skills)