495 lines
16 KiB
Markdown
495 lines
16 KiB
Markdown
---
|
||
title: "\"永续\" Agent 的最后一环:Self-Improving Skills"
|
||
source: "https://x.com/xxx111god/status/2033261636195373123"
|
||
author:
|
||
- "[[Unknown]]"
|
||
published: 2026-03-16
|
||
created: 2026-03-16
|
||
description:
|
||
tags:
|
||
- "clippings"
|
||
---
|
||
过去2个月我使用OpenClaw🦞过程中一直在听到“永续工作”的概念,也写了几篇我自己思考和实践的文章
|
||
|
||
[任务管理系统](https://x.com/xxx111god/status/...)让 agent 能 24/7 自主工作,issue 系统追踪每个任务的状态
|
||
|
||
[记忆系统](https://x.com/xxx111god/status/...)实现了 L0/L1/L2 分层管理,daily logs 自动归档,semantic search 快速检索
|
||
|
||
这些系统让 agent 能持续运行,不会因为"忘了之前做什么"而中断。
|
||
|
||
但有个盲点我一直没解决:Skills 本身。
|
||
|
||
我现在在用 30 多个技能,覆盖几乎所有日常工作:XHS/X 内容制作流程、播客全自动化(collector → editor → tts → publisher)、预测市场交易(auto-research + monitor + execution)、发邮件、写代码、代码审查、查天气、管理日程、搜索文档。它们每天被调用几十上百次。
|
||
|
||
但这些 skills 是静态的,它们不在"永续系统"里。高频使用时经常出现各种问题:"诶,这个工具路径怎么找不到了?""怎么又报错了?上次不是能跑吗?""换到 writer session 怎么不知道怎么用这个 skill 了?"每次都要手动排查:是 skill 写得不够清楚?是环境变了工具路径变了?是跨 session 没有足够的上下文?然后改 SKILL.md,重启,重新加载。
|
||
|
||
这不对。如果 agent 要永续工作,Skills 也要永续改进。任务可以追踪,记忆可以归档,Skills 为什么不能自己进化?
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## 第一次接触到"Skills 迭代"
|
||
|
||
第一次接触到"Skills 迭代"的概念,是看 [@baoyu](https://github.com/dotey/openai-quickstart/blob/main/writing_style.md) 老师分享他的写作风格 skill。他提了一个观点:写作风格不应该是静态 prompt 文件,应该让 AI 根据反馈不断进化,否则所谓的去AI味的写作风格也会沦为千篇一律的新AI味。
|
||
|
||
我用宝玉老师skill创建的我个人的写作风格 skill 现在是 v1.11.0,已经迭代了 11 个版本,但每次都要我主动触发更新。用了这么久,一直在想:这个过程能自动化吗?
|
||
|
||
其实答案是显而易见的。不是手动告诉 AI 怎么改,而是让系统自己观察、分析、提出改进。这才是永续 Agent 的最后一环。
|
||
|
||
核心是一个 5 步循环:Ingest → Observe → Inspect → Improve → Evaluate。下面分享我是怎么理解这 5 步的。
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## Step 1: Ingest(结构化存储)
|
||
|
||
目标:把 SKILL.md 变成可查询的结构。 静态 markdown 文件没法程序化分析,需要结构化才能做后续的 Inspect 和 Improve。
|
||
|
||
数据结构示例
|
||
|
||
```json
|
||
{
|
||
"id":"gog-email",
|
||
"name":"Send Email via Google Workspace",
|
||
"trigger_patterns":[
|
||
"send email to *",
|
||
"email * about *"
|
||
],
|
||
"tools":[
|
||
"gog mail send"
|
||
],
|
||
"dependencies":[
|
||
"SMTP credentials",
|
||
"OAuth token"
|
||
],
|
||
"constraints":{
|
||
"timeout":30,
|
||
"max_attachment_size":25
|
||
},
|
||
"last_updated":"2026-03-10T15:30:00Z",
|
||
"version":"1.3.0"
|
||
}
|
||
```
|
||
|
||
实现方式
|
||
|
||
可以用现成的工具(比如 [Cognee](https://github.com/topoteretes/cognee) 的结构化存储功能),或者自己写 parser 把 SKILL.md → JSON → 存 SQLite。关键是要把 trigger patterns 做 semantic embedding(方便后续匹配),把 tools、dependencies、constraints 分开存,方便后续分析到底是哪个部分出问题。
|
||
|
||
一开始我犯的错误
|
||
|
||
不要只存 prompt text,要把每个字段都结构化,不然后面 Inspect 时没法细分问题。
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## Step 2: Observe(记录执行)
|
||
|
||
目标:每次 skill 调用留下可审计的痕迹。 没有执行记录就没法知道哪里失败,这是整个循环的基础。
|
||
|
||
数据结构示例
|
||
|
||
```json
|
||
{
|
||
"run_id":"8f3a9c2b",
|
||
"timestamp":"2026-03-14T16:20:15Z",
|
||
"skill_id":"gog-email",
|
||
"skill_version":"1.3.0",
|
||
"task":"send email to john@example.com about meeting",
|
||
"selected_by":"semantic_match",
|
||
"match_score":0.87,
|
||
"success":false,
|
||
"error":{
|
||
"type":"SMTPTimeout",
|
||
"message":"Connection timeout after 30s",
|
||
"traceback":"..."
|
||
},
|
||
"context":{
|
||
"attachment_size_mb":8.5,
|
||
"time_of_day":"16:20",
|
||
"network_latency_ms":450
|
||
},
|
||
"user_feedback":"manually sent via Gmail web"
|
||
}
|
||
```
|
||
|
||
实现方式
|
||
|
||
在 skill wrapper 里加一段 logging:
|
||
|
||
```python
|
||
# ~/clawd/lib/skill_executor.py
|
||
|
||
from uuid import uuid4
|
||
from datetime import datetime, date
|
||
import json
|
||
|
||
def execute_skill(skill_id, task):
|
||
run_id = uuid4()
|
||
|
||
try:
|
||
result = _run_skill(skill_id, task)
|
||
|
||
log_run(
|
||
run_id=run_id,
|
||
skill_id=skill_id,
|
||
task=task,
|
||
success=True,
|
||
result=result
|
||
)
|
||
|
||
return result
|
||
|
||
except Exception as e:
|
||
log_run(
|
||
run_id=run_id,
|
||
skill_id=skill_id,
|
||
task=task,
|
||
success=False,
|
||
error=e
|
||
)
|
||
raise
|
||
|
||
def log_run(run_id, skill_id, task, success, error=None, result=None):
|
||
run_data = {
|
||
"run_id": str(run_id),
|
||
"timestamp": datetime.utcnow().isoformat(),
|
||
"skill_id": skill_id,
|
||
"task": task,
|
||
"success": success,
|
||
"error": serialize_error(error) if error else None,
|
||
"context": get_runtime_context(),
|
||
}
|
||
|
||
with open(f"~/memory/skill-runs/{date.today()}.jsonl", "a") as f:
|
||
f.write(json.dumps(run_data) + "\n")
|
||
```
|
||
|
||
三个教训
|
||
|
||
一开始只记录 true/false,后来发现没用,要记录错误类型和 context。只在最外层捕获异常,工具调用失败的细节丢了,要在各层都记录。忘记记录 skill version,改进后没法对比新旧版本效果。
|
||
|
||
额外有用的数据
|
||
|
||
用户有没有手动 retry?(说明 skill 不可靠)用户有没有改 task 重新提交?(说明 trigger pattern 不准)调用这个 skill 前后还用了什么?(说明 routing 问题)
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## Step 3: Inspect(失败模式分析)
|
||
|
||
目标:从执行日志里提取 actionable insights。 有日志不够,要能看出模式。"gog skill 失败率 30%" 没用,"gog skill 在发 >5MB 附件时 80% 超时" 才有用。
|
||
|
||
分析维度
|
||
|
||
```python
|
||
# ~/clawd/scripts/skill-inspector.py
|
||
|
||
from collections import Counter
|
||
|
||
def analyze_failures(skill_id, days=7):
|
||
runs = load_runs(skill_id, days)
|
||
|
||
# 1. 总体失败率 (overall failure rate)
|
||
total = len(runs)
|
||
failed = [r for r in runs if not r["success"]]
|
||
failure_rate = len(failed) / total if total else 0
|
||
|
||
# 2. 按错误类型分组 (group by error type)
|
||
error_types = Counter(r["error"]["type"] for r in failed if r.get("error"))
|
||
|
||
# 3. 按 context 维度分组 (group by attachment size)
|
||
by_attachment_size = group_by(
|
||
failed,
|
||
key=lambda r: r["context"].get("attachment_size_mb", 0),
|
||
buckets=[0, 1, 5, 10, float("inf")]
|
||
)
|
||
|
||
# 4. 关联分析 (correlation analysis)
|
||
high_latency = [
|
||
r for r in failed
|
||
if r["context"].get("network_latency_ms", 0) > 300
|
||
]
|
||
|
||
network_issue_ratio = len(high_latency) / len(failed) if failed else 0
|
||
|
||
return {
|
||
"failure_rate": failure_rate,
|
||
"error_types": error_types,
|
||
"patterns": {
|
||
"large_attachments": by_attachment_size,
|
||
"network_issues": network_issue_ratio,
|
||
}
|
||
}
|
||
```
|
||
|
||
输出示例
|
||
|
||
```markdown
|
||
Skill: gog-email
|
||
|
||
Failure Rate:
|
||
28% (14 / 50 runs)
|
||
|
||
Error Types:
|
||
- SMTPTimeout: 10 times (71%)
|
||
- AuthError: 3 times (21%)
|
||
- InvalidRecipient: 1 time (7%)
|
||
|
||
Patterns:
|
||
|
||
Attachment Size
|
||
- > 5MB: 80% failure (8 / 10)
|
||
- < 5MB: 10% failure (2 / 40)
|
||
|
||
Network Conditions
|
||
- Latency > 300ms: 90% failure (9 / 10)
|
||
|
||
Time Patterns
|
||
- Hour 16–18 (peak traffic): 40% failure (6 / 15)
|
||
```
|
||
|
||
关键洞察
|
||
|
||
SMTP timeout 不是随机的,是大附件 + 高延迟的组合导致的。修复方向:增加 timeout、改用 Drive link for large files。
|
||
|
||
为什么一开始分析不出东西
|
||
|
||
一开始只看总失败率,没发现是特定 context 下才失败。只看单一维度,错过了"大附件 × 高延迟"这种交叉模式。忽略成功案例,对比成功/失败的 context 差异才能找到根因。
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## Step 4: Improve(提出改进)
|
||
|
||
目标:基于 Inspect 的分析,生成具体的改进方案。 分析完不改进等于白分析,但改进不能瞎改,要有证据支撑。
|
||
|
||
Improvement 数据结构
|
||
|
||
```json
|
||
{
|
||
"improvement_id":"gog-email-v1.4.0",
|
||
"skill_id":"gog-email",
|
||
"base_version":"1.3.0",
|
||
"proposed_at":"2026-03-14T17:00:00Z",
|
||
"rationale":{
|
||
"failure_rate":0.28,
|
||
"root_cause":"SMTP timeout on large attachments + high network latency",
|
||
"evidence":{
|
||
"large_attachment_failure_rate":0.80,
|
||
"small_attachment_failure_rate":0.10,
|
||
"high_latency_correlation":0.90
|
||
}
|
||
},
|
||
"changes":[
|
||
{
|
||
"type":"constraint_update",
|
||
"field":"timeout",
|
||
"old_value":30,
|
||
"new_value":60,
|
||
"reason":"80% of failures are SMTP timeout with >5MB attachments"
|
||
},
|
||
{
|
||
"type":"logic_addition",
|
||
"description":"Use Google Drive link for attachments >5MB",
|
||
"pseudo_code":"if attachment_size > 5MB: upload_to_drive() + send_link()"
|
||
},
|
||
{
|
||
"type":"retry_policy",
|
||
"config":{
|
||
"max_retries":3,
|
||
"backoff":"exponential",
|
||
"retry_on":[
|
||
"SMTPTimeout"
|
||
]
|
||
}
|
||
}
|
||
],
|
||
"status":"proposed",
|
||
"approval_required":true
|
||
}
|
||
```
|
||
|
||
实现方式
|
||
|
||
```python
|
||
# ~/clawd/scripts/skill-improver.py
|
||
|
||
def propose_improvement(skill_id, analysis):
|
||
improvements = []
|
||
|
||
# Rule 1: Timeout issues
|
||
if "SMTPTimeout" in analysis["error_types"]:
|
||
timeout_failures = analysis["error_types"]["SMTPTimeout"]
|
||
|
||
if timeout_failures / analysis["total_failures"] > 0.5:
|
||
improvements.append({
|
||
"type": "constraint_update",
|
||
"field": "timeout",
|
||
"new_value": current_timeout * 2,
|
||
"reason": f"{timeout_failures} timeout failures"
|
||
})
|
||
|
||
# Rule 2: Large attachment pattern
|
||
if analysis["patterns"]["large_attachments"][">5MB"]["failure_rate"] > 0.7:
|
||
improvements.append({
|
||
"type": "logic_addition",
|
||
"description": "Use Drive link for >5MB attachments",
|
||
"reason": "80% failure rate on large attachments"
|
||
})
|
||
|
||
# Rule 3: Network latency correlation
|
||
if analysis["patterns"]["network_issues"] > 0.8:
|
||
improvements.append({
|
||
"type": "retry_policy",
|
||
"config": {
|
||
"max_retries": 3,
|
||
"backoff": "exponential"
|
||
},
|
||
"reason": "90% correlation with high network latency"
|
||
})
|
||
|
||
return create_improvement(skill_id, improvements, analysis)
|
||
```
|
||
|
||
什么情况下改进会适得其反
|
||
|
||
一次改太多,搞不清楚到底是哪个改动起作用。没证据就改,瞎改一通可能引入新问题。改完直接 apply,应该先 proposed 状态人工 review。
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## Step 5: Evaluate(A/B 测试)
|
||
|
||
目标:验证改进是否真的有效。 改进可能引入新问题,"修了 A 问题,搞出 B 问题"很常见。
|
||
|
||
Evaluation 数据结构
|
||
|
||
```json
|
||
{
|
||
"evaluation_id":"eval-8a3f2c1b",
|
||
"improvement_id":"gog-email-v1.4.0",
|
||
"evaluation_period":{
|
||
"start":"2026-03-15T00:00:00Z",
|
||
"end":"2026-03-22T00:00:00Z",
|
||
"duration_days":7
|
||
},
|
||
"baseline":{
|
||
"version":"1.3.0",
|
||
"runs":50,
|
||
"failure_rate":0.28,
|
||
"avg_execution_time_s":12.5,
|
||
"error_types":{
|
||
"SMTPTimeout":10,
|
||
"AuthError":3
|
||
}
|
||
},
|
||
"treatment":{
|
||
"version":"1.4.0",
|
||
"runs":52,
|
||
"failure_rate":0.08,
|
||
"avg_execution_time_s":15.2,
|
||
"error_types":{
|
||
"SMTPTimeout":2,
|
||
"AuthError":2
|
||
}
|
||
},
|
||
"metrics":{
|
||
"failure_rate_improvement":-0.20,
|
||
"execution_time_regression":+2.7,
|
||
"new_errors_introduced":0
|
||
},
|
||
"verdict":"approve",
|
||
"reason":"Failure rate 降低 71%,execution time 增加可接受"
|
||
}
|
||
```
|
||
|
||
实现方式
|
||
|
||
A/B 流量分配:10% 流量走新版本,90% 走旧版本。
|
||
|
||
```python
|
||
import random
|
||
|
||
def route_to_version(skill_id, task):
|
||
# 10% traffic to new version (1.4.0), 90% to baseline (1.3.0)
|
||
if random.random() < 0.1:
|
||
return execute_skill(skill_id, task, version="1.4.0")
|
||
else:
|
||
return execute_skill(skill_id, task, version="1.3.0")
|
||
|
||
# Evaluation criteria
|
||
def evaluate_improvement(baseline, treatment):
|
||
|
||
# Must-have: failure rate must decrease
|
||
if treatment["failure_rate"] >= baseline["failure_rate"]:
|
||
return "reject", "No failure rate improvement"
|
||
|
||
# Nice-to-have: execution time should not regress too much
|
||
time_regression = (
|
||
treatment["avg_execution_time_s"]
|
||
- baseline["avg_execution_time_s"]
|
||
)
|
||
|
||
if time_regression > baseline["avg_execution_time_s"] * 0.5:
|
||
return "reject", f"Execution time +{time_regression:.1f}s (>50%)"
|
||
|
||
# Must-not: cannot introduce new error types
|
||
new_errors = set(treatment["error_types"]) - set(baseline["error_types"])
|
||
|
||
if new_errors:
|
||
return "reject", f"New errors: {new_errors}"
|
||
|
||
# Compute improvement percentage
|
||
improvement = (
|
||
baseline["failure_rate"] - treatment["failure_rate"]
|
||
) / baseline["failure_rate"]
|
||
|
||
return "approve", f"Failure rate improved {improvement:.1%}"
|
||
```
|
||
|
||
A/B 测试容易踩的三个坑
|
||
|
||
只看失败率,执行时间变慢 50% 也是问题。样本太小,10 次测试就下结论统计意义不够。改完立刻全量,应该先 10% → 50% → 100% 逐步放量。
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## 我的 3 个实例
|
||
|
||
1\. OpenClaw Skills(30+ 个)
|
||
|
||
现状是全静态,失败不可见。我已经加了 Observation Layer,现在每次 skill 调用都有 JSONL 日志。下一步是写 Inspector 脚本,看看 gog/coding-agent/github 的失败模式。
|
||
|
||
2\. Trading Autoresearch(30 个策略)
|
||
|
||
现状是已有 Observe(autoresearch\_results.jsonl),缺的是 Improve(自动提出策略变体)。
|
||
|
||
计划实现:
|
||
|
||
\# 当 neighbor\_t2\_d1 在高波下失败
|
||
|
||
if brier\_score > 0.15 and volatility > 0.03:
|
||
|
||
propose\_variants(\[
|
||
|
||
"neighbor\_t2\_d2", # distance 更保守
|
||
|
||
"neighbor\_t2\_d1\_v10" # 加 volatility filter
|
||
|
||
\])
|
||
|
||
然后 paper trades 跑 2 周,Brier score 对比,更好的替换原策略。
|
||
|
||
3\. Writing Style(v1.11.0)
|
||
|
||
现状是手动迭代。可以自动化的部分:Observe 捕获我的修改 diff,Inspect 统计最常违反的规则,Improve 自动提出新禁止词。示例:过去 7 天修改记录显示"综上所述"出现 12 次、"不得不说"出现 8 次,自动提案 v1.12.0 新增这两个禁止词,然后测试新版本看我的修改次数是否减少。
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## 结尾
|
||
|
||
Static Skills = Copilot(人工维护),Self-Improving Skills = Autopilot(自己优化)。但 Autopilot 必须有 safeguards:Improve 必须有证据(Inspect 的分析),Evaluate 必须验证改进(A/B 测试),无效就回滚(不能盲目自改)。不然就变成 Gilfoyle 的 "Son of Anton",删了所有代码说 "efficient debugging
|
||
|
||
从 [@baoyu](https://x.com/@baoyu) 老师学到"技能可以迭代",后来发现"迭代可以自动化"
|
||
|
||
核心是 5 步循环:Ingest → Observe → Inspect → Improve → Evaluate。我已经开始加 observation layer,让失败变得可见,让改进变得自动。这才是永续 Agent 的最后一环。
|
||
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
## 相关链接
|
||
|
||
- Cognee Self-Improving Skills: [https://github.com/topoteretes/cognee](https://github.com/topoteretes/cognee)
|
||
- [@baoyu](https://x.com/@baoyu) skill: [https://github.com/JimLiu/baoyu-skills](https://github.com/JimLiu/baoyu-skills) |