a
This commit is contained in:
@@ -0,0 +1,228 @@
|
||||
# Anti-AI Detector - 去AI化技能
|
||||
|
||||
检测并去除AI生成文本中的AI痕迹,支持中文、英文和小红书风格优化。
|
||||
|
||||
## 🎯 功能特性
|
||||
|
||||
- ✅ **中文去AI化** - 去除中文AI文本的典型痕迹
|
||||
- ✅ **英文去AI化** - 基于Stop Slop的英文优化
|
||||
- ✅ **小红书风格** - 针对小红书平台的特殊优化
|
||||
- ✅ **自动检测** - 智能识别文本语言和场景
|
||||
- ✅ **IT项目集成** - 直接集成到IT资讯自动化流程
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 命令行使用
|
||||
|
||||
```bash
|
||||
# 处理单段文本
|
||||
python3 humanize.py "你的AI生成文本"
|
||||
|
||||
# 处理文件
|
||||
python3 humanize.py -f input.txt -o output.txt
|
||||
|
||||
# 指定风格
|
||||
python3 humanize.py -t zh "中文文本"
|
||||
python3 humanize.py -t en "English text"
|
||||
python3 humanize.py -t xhs "小红书文本"
|
||||
|
||||
# 分析AI痕迹(不处理)
|
||||
python3 humanize.py --analyze "文本"
|
||||
```
|
||||
|
||||
### Python API
|
||||
|
||||
```python
|
||||
from humanize import AntiAIDetector
|
||||
|
||||
detector = AntiAIDetector()
|
||||
|
||||
# 中文去AI化
|
||||
result = detector.humanize_cn("值得注意的是...")
|
||||
|
||||
# 英文去AI化
|
||||
result = detector.humanize_en("Here's the thing...")
|
||||
|
||||
# 小红书风格
|
||||
result = detector.humanize_xhs("姐妹们!这个神器绝绝子!")
|
||||
```
|
||||
|
||||
## 📋 检测规则
|
||||
|
||||
### 中文AI痕迹
|
||||
|
||||
| 类型 | 示例 | 替换为 |
|
||||
|------|------|--------|
|
||||
| 标志性短语 | "值得注意的是" | "" |
|
||||
| | "我们可以看到" | "" |
|
||||
| | "不难发现" | "可以看出" |
|
||||
| | "显而易见" | "很明显" |
|
||||
| 结构模式 | "首先...其次...最后" | "第一...第二...最后" |
|
||||
| | "综上所述" | "总结来说" |
|
||||
| 过度修饰 | "非常" | "很" |
|
||||
| | "极其" | "特别" |
|
||||
| 模板化 | "随着...的发展" | "" |
|
||||
| | "在当今社会" | "" |
|
||||
|
||||
### 英文AI痕迹(基于Stop Slop)
|
||||
|
||||
| 类型 | 示例 | 替换为 |
|
||||
|------|------|--------|
|
||||
| 标志性短语 | "Here's the thing" | "" |
|
||||
| | "Let that sink in" | "" |
|
||||
| | "The uncomfortable truth is" | "Truth is" |
|
||||
| | "Let me be clear" | "" |
|
||||
| 过度动词 | "navigate" | "handle" |
|
||||
| | "unpack" | "explain" |
|
||||
| | "leverage" | "use" |
|
||||
| 固定结构 | "Not because X. Because Y." | "Not from X, but from Y." |
|
||||
| | "That's it. That's the..." | "" |
|
||||
|
||||
### 小红书风格优化
|
||||
|
||||
| 类型 | 示例 | 替换为 |
|
||||
|------|------|--------|
|
||||
| 营销词汇 | "绝绝子" | "很好" |
|
||||
| | "YYDS" | "很棒" |
|
||||
| | "闭眼入" | "推荐" |
|
||||
| | "神器" | "好用" |
|
||||
| 称呼 | "姐妹们" | "大家" |
|
||||
| 术语 | "干货" | "内容" |
|
||||
| | "码住" | "收藏" |
|
||||
|
||||
## 🔧 集成到IT资讯系统
|
||||
|
||||
### 方式1:直接调用
|
||||
|
||||
```python
|
||||
from it_integration import humanize_for_it_news
|
||||
|
||||
# 翻译后处理
|
||||
translated = ai_processor.translate_article(content)
|
||||
humanized = humanize_for_it_news(translated, content_type='translation')
|
||||
|
||||
# 总结后处理
|
||||
summary = ai_processor.summarize_article(content)
|
||||
humanized = humanize_for_it_news(summary, content_type='summary')
|
||||
```
|
||||
|
||||
### 方式2:完整管道
|
||||
|
||||
```python
|
||||
from it_integration import ITContentHumanizer
|
||||
|
||||
humanizer = ITContentHumanizer()
|
||||
|
||||
# 完整处理
|
||||
content = {
|
||||
'title': '原标题',
|
||||
'translated': '翻译内容',
|
||||
'summary': '总结内容',
|
||||
'platform': 'xiaohongshu' # 或 'email'
|
||||
}
|
||||
|
||||
result = humanizer.full_pipeline(content)
|
||||
print(result['title'])
|
||||
print(result['summary'])
|
||||
```
|
||||
|
||||
### 方式3:风险分析
|
||||
|
||||
```python
|
||||
from it_integration import ITContentHumanizer
|
||||
|
||||
humanizer = ITContentHumanizer()
|
||||
risk = humanizer.analyze_risk(text)
|
||||
|
||||
print(f"风险等级: {risk['risk_level']}") # low/medium/high
|
||||
print(f"得分: {risk['score']}")
|
||||
print(f"建议: {risk['suggestions']}")
|
||||
```
|
||||
|
||||
## 📝 修改IT资讯处理器
|
||||
|
||||
在 `process_articles.py` 中添加:
|
||||
|
||||
```python
|
||||
from skills.anti_ai_detector.it_integration import ITContentHumanizer
|
||||
|
||||
class ArticleProcessor:
|
||||
def __init__(self):
|
||||
self.ai_processor = AIProcessor()
|
||||
self.humanizer = ITContentHumanizer() # 新增
|
||||
|
||||
def process_single_article(self, article_id: int):
|
||||
# ... 原有代码 ...
|
||||
|
||||
# AI处理后,添加去AI化
|
||||
translated = self.ai_processor.translate_article(content)
|
||||
translated = self.humanizer.process_translation(translated) # 新增
|
||||
|
||||
summary = self.ai_processor.summarize_article(content)
|
||||
summary = self.humanizer.process_summary(summary) # 新增
|
||||
|
||||
# ... 后续代码 ...
|
||||
```
|
||||
|
||||
## 🎨 效果示例
|
||||
|
||||
### 中文示例
|
||||
|
||||
**原文:**
|
||||
```
|
||||
值得注意的是,这项技术非常重要。我们可以看到,随着AI的发展,
|
||||
越来越多的应用场景被开发出来。首先,它提高了效率;其次,
|
||||
它降低了成本;最后,它改变了工作方式。
|
||||
```
|
||||
|
||||
**去AI化后:**
|
||||
```
|
||||
这项技术非常重要。
|
||||
越来越多的应用场景被开发出来。第一,它提高了效率;第二,
|
||||
它降低了成本;最后,它改变了工作方式。
|
||||
```
|
||||
|
||||
### 英文示例
|
||||
|
||||
**原文:**
|
||||
```
|
||||
Here's the thing: building great products is hard.
|
||||
Not because they lack talent. Because they lack clarity.
|
||||
Let that sink in.
|
||||
```
|
||||
|
||||
**去AI化后:**
|
||||
```
|
||||
building great products is hard.
|
||||
Not from they lack talent, but from they lack clarity.
|
||||
```
|
||||
|
||||
### 小红书示例
|
||||
|
||||
**原文:**
|
||||
```
|
||||
姐妹们!今天给大家分享一个宝藏神器!🔥🔥🔥
|
||||
强烈推荐大家码住这个干货!
|
||||
```
|
||||
|
||||
**优化后:**
|
||||
```
|
||||
大家!今天给大家分享一个不错好用!🔥🔥🔥
|
||||
推荐大家收藏这个内容!
|
||||
```
|
||||
|
||||
## 📊 性能指标
|
||||
|
||||
- 中文检测:30+ 种AI痕迹模式
|
||||
- 英文检测:40+ 种Stop Slop模式
|
||||
- 小红书优化:20+ 种营销词汇
|
||||
- 处理速度:~1000字/秒
|
||||
|
||||
## 🔗 参考
|
||||
|
||||
- Stop Slop: https://github.com/hardikpandya/stop-slop
|
||||
- Inspired by: Hardik Pandya
|
||||
|
||||
## 📝 License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,80 @@
|
||||
# Anti-AI Detector - 去AI化技能
|
||||
|
||||
检测并去除AI生成文本中的AI痕迹,支持中文、英文和小红书风格优化。
|
||||
|
||||
## 🎯 功能特性
|
||||
|
||||
- ✅ **中文去AI化** - 去除中文AI文本的典型痕迹
|
||||
- ✅ **英文去AI化** - 基于Stop Slop的英文优化
|
||||
- ✅ **小红书风格** - 针对小红书平台的特殊优化
|
||||
- ✅ **自动检测** - 智能识别文本语言和场景
|
||||
- ✅ **批量处理** - 支持文件批量处理
|
||||
|
||||
## 🚀 使用方法
|
||||
|
||||
### 命令行
|
||||
|
||||
```bash
|
||||
# 处理单段文本
|
||||
python3 /home/ubuntu/.nanobot/workspace/skills/anti-ai-detector/humanize.py "你的AI生成文本"
|
||||
|
||||
# 处理文件
|
||||
python3 /home/ubuntu/.nanobot/workspace/skills/anti-ai-detector/humanize.py -f input.txt -o output.txt
|
||||
|
||||
# 指定风格
|
||||
python3 /home/ubuntu/.nanobot/workspace/skills/anti-ai-detector/humanize.py -t xiaohongshu "文本"
|
||||
|
||||
# 批量处理
|
||||
python3 /home/ubuntu/.nanobot/workspace/skills/anti-ai-detector/humanize.py -d ./articles/ --style zh
|
||||
```
|
||||
|
||||
### Python API
|
||||
|
||||
```python
|
||||
from anti_ai_detector import AntiAIDetector
|
||||
|
||||
detector = AntiAIDetector()
|
||||
|
||||
# 中文去AI化
|
||||
result = detector.humanize_cn("值得注意的是...")
|
||||
|
||||
# 英文去AI化
|
||||
result = detector.humanize_en("Here's the thing...")
|
||||
|
||||
# 小红书风格
|
||||
result = detector.humanize_xhs("这个产品有三大优点...")
|
||||
```
|
||||
|
||||
## 📋 检测规则
|
||||
|
||||
### 中文AI痕迹
|
||||
- "值得注意的是..."
|
||||
- "我们可以看到..."
|
||||
- "换句话说..."
|
||||
- "首先...其次...最后..."
|
||||
- "不难发现..."
|
||||
- "显而易见..."
|
||||
|
||||
### 英文AI痕迹(Stop Slop)
|
||||
- "Here's the thing..."
|
||||
- "Let that sink in..."
|
||||
- "The uncomfortable truth is..."
|
||||
- "Navigate the landscape"
|
||||
- "Not because X. Because Y."
|
||||
|
||||
### 小红书风格优化
|
||||
- 避免过于规整的排版
|
||||
- 加入个人化口语
|
||||
- emoji自然分布
|
||||
- 去除营销感词汇
|
||||
|
||||
## 🔧 集成到IT资讯系统
|
||||
|
||||
在IT资讯项目的AI处理器中调用:
|
||||
|
||||
```python
|
||||
from anti_ai_detector import AntiAIDetector
|
||||
|
||||
detector = AntiAIDetector()
|
||||
translated = detector.humanize_cn(ai_output)
|
||||
```
|
||||
@@ -0,0 +1,446 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Anti-AI Detector - 去AI化工具
|
||||
支持中文、英文和小红书风格优化
|
||||
"""
|
||||
|
||||
import re
|
||||
import os
|
||||
import sys
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Tuple
|
||||
|
||||
|
||||
class AntiAIDetector:
|
||||
"""去AI化检测器"""
|
||||
|
||||
def __init__(self):
|
||||
self.init_cn_patterns()
|
||||
self.init_en_patterns()
|
||||
self.init_xhs_patterns()
|
||||
|
||||
# ==================== 中文AI痕迹模式 ====================
|
||||
def init_cn_patterns(self):
|
||||
"""初始化中文AI检测模式"""
|
||||
self.cn_patterns = {
|
||||
# 标志性短语
|
||||
"值得注意的是": ["需要关注的是", "关键点在于", ""],
|
||||
"我们可以看到": ["从数据看", "实际情况是", ""],
|
||||
"换句话说": ["也就是说", "简单来说", ""],
|
||||
"不难发现": ["可以看出", "", ""],
|
||||
"显而易见": ["很明显", "显然", ""],
|
||||
"众所周知": ["大家都知道", "", ""],
|
||||
"综上所述": ["总结来说", "", ""],
|
||||
"归根结底": ["根本原因是", "说到底", ""],
|
||||
"由此可见": ["这说明", "", ""],
|
||||
|
||||
# 结构模式
|
||||
"首先": ["第一", "开始", ""],
|
||||
"其次": ["第二", "然后", ""],
|
||||
"最后": ["最终", "", ""],
|
||||
"总而言之": ["总的来说", "", ""],
|
||||
|
||||
# 过度修饰
|
||||
"非常": ["很", "挺", ""],
|
||||
"极其": ["很", "特别", ""],
|
||||
"毫无疑问": ["确实", "", ""],
|
||||
"一定程度上": ["", "", ""],
|
||||
|
||||
# 模板化表达
|
||||
"随着...的发展": ["", "", ""],
|
||||
"在当今社会": ["", "", ""],
|
||||
"引起了广泛关注": ["", "", ""],
|
||||
"具有重要的意义": ["", "", ""],
|
||||
}
|
||||
|
||||
self.cn_structures = [
|
||||
# 排比结构
|
||||
(r'不仅[^,]+,而且[^,]+,还[^。]+', "避免三重排比"),
|
||||
(r'一方面[^,]+,另一方面[^,]+', "减少固定结构"),
|
||||
|
||||
# 过渡词堆砌
|
||||
(r'首先.*?其次.*?最后.*?(综上所述|总而言之)', "去除模板化结构"),
|
||||
|
||||
# 学术腔
|
||||
(r'从[^,]+角度(来看|分析|出发)', "口语化表达"),
|
||||
(r'基于[^,]+(的)?(背景|基础|之上)', "简化表达"),
|
||||
]
|
||||
|
||||
# ==================== 英文AI痕迹模式(Stop Slop)====================
|
||||
def init_en_patterns(self):
|
||||
"""初始化英文AI检测模式(基于Stop Slop)"""
|
||||
self.en_patterns = {
|
||||
# 标志性短语
|
||||
"Here's the thing": ["The point is", "Fact is", ""],
|
||||
"Here's the deal": ["The reality is", "", ""],
|
||||
"Let that sink in": ["", "", ""],
|
||||
"The uncomfortable truth is": ["Truth is", "", ""],
|
||||
"The reality is": ["", "", ""],
|
||||
"This matters because": ["Because", "", ""],
|
||||
"What matters is": ["", "", ""],
|
||||
"Full stop": ["", "", ""],
|
||||
"Period": ["", "", ""],
|
||||
"Let me be clear": ["", "", ""],
|
||||
"I'm going to be honest": ["", "", ""],
|
||||
"Make no mistake": ["", "", ""],
|
||||
|
||||
# 过度使用的动词
|
||||
"navigate": ["handle", "deal with", "work through"],
|
||||
"unpack": ["explain", "break down", ""],
|
||||
"leverage": ["use", "take advantage of", ""],
|
||||
"delve": ["explore", "look at", ""],
|
||||
"dive into": ["explore", "examine", ""],
|
||||
"peel back the layers": ["examine", "analyze", ""],
|
||||
|
||||
# 固定结构
|
||||
"Not because": ["Not from", "", ""],
|
||||
"But because": ["But from", "", ""],
|
||||
"That's it": ["", "", ""],
|
||||
"That's the": ["", "", ""],
|
||||
"And that's okay": ["", "", ""],
|
||||
|
||||
# 过度修饰
|
||||
"In today's world": ["Today", "Now", ""],
|
||||
"In the digital age": ["", "", ""],
|
||||
"Rapidly evolving": ["changing", "", ""],
|
||||
"Fast-paced": ["busy", "", ""],
|
||||
|
||||
# 模糊表达
|
||||
"It's important to note": ["Note", "", ""],
|
||||
"It's worth mentioning": ["", "", ""],
|
||||
"As mentioned earlier": ["", "", ""],
|
||||
}
|
||||
|
||||
self.en_structures = [
|
||||
# "Not because X. Because Y."
|
||||
(r'Not because[^.]+\. Because[^.]+\.', "合并为一句"),
|
||||
|
||||
# "[Noun]. That's it. That's the [thing]."
|
||||
(r'(\w+)\. That\'s it\. That\'s the[^.]+\.', "简化表达"),
|
||||
|
||||
# 三重形容词
|
||||
(r'\w+, \w+, and \w+ (landscape|environment|space)', "减少形容词"),
|
||||
|
||||
# 过度使用hedging
|
||||
(r'arguably|potentially|likely|generally|typically', "直接表达"),
|
||||
]
|
||||
|
||||
# ==================== 小红书风格优化 ====================
|
||||
def init_xhs_patterns(self):
|
||||
"""初始化小红书风格检测模式"""
|
||||
self.xhs_patterns = {
|
||||
# 去除营销感
|
||||
"强烈推荐": ["推荐", "我觉得", ""],
|
||||
"必买": ["值得买", "可以考虑", ""],
|
||||
"神器": ["好用", "不错", ""],
|
||||
"绝绝子": ["很好", "不错", ""],
|
||||
"YYDS": ["很棒", "", ""],
|
||||
"宝藏": ["好", "不错", ""],
|
||||
"闭眼入": ["推荐", "", ""],
|
||||
"天花板": ["很好", "顶级", ""],
|
||||
|
||||
# 去模板化
|
||||
"今天给大家分享": ["分享", "说说", ""],
|
||||
"姐妹们": ["大家", "", ""],
|
||||
"干货": ["内容", "", ""],
|
||||
"码住": ["收藏", "记住", ""],
|
||||
"抄作业": ["参考", "", ""],
|
||||
|
||||
# 过度emoji提示
|
||||
"emoji堆砌": ["适量使用emoji", "", ""],
|
||||
"标签堆砌": ["减少标签", "", ""],
|
||||
}
|
||||
|
||||
self.xhs_structures = [
|
||||
# 过于规整的排版
|
||||
(r'^[0-9]+[.、\s]', "避免过于规整的序号"),
|
||||
(r'✅|❌|⚠️|⭐|🌟{3,}', "emoji过多"),
|
||||
(r'#[^#\s]+#[^#\s]+#[^#\s]+', "标签过多"),
|
||||
]
|
||||
|
||||
# ==================== 检测方法 ====================
|
||||
def detect_cn(self, text: str) -> List[Tuple[str, str, List[str]]]:
|
||||
"""检测中文AI痕迹"""
|
||||
findings = []
|
||||
for pattern, replacements in self.cn_patterns.items():
|
||||
if pattern in text or re.search(pattern.replace('...', '.*?'), text):
|
||||
findings.append(("phrase", pattern, replacements))
|
||||
|
||||
for pattern, suggestion in self.cn_structures:
|
||||
matches = re.findall(pattern, text, re.DOTALL)
|
||||
if matches:
|
||||
findings.append(("structure", pattern, [suggestion]))
|
||||
|
||||
return findings
|
||||
|
||||
def detect_en(self, text: str) -> List[Tuple[str, str, List[str]]]:
|
||||
"""检测英文AI痕迹"""
|
||||
findings = []
|
||||
text_lower = text.lower()
|
||||
|
||||
for pattern, replacements in self.en_patterns.items():
|
||||
if pattern.lower() in text_lower:
|
||||
findings.append(("phrase", pattern, replacements))
|
||||
|
||||
for pattern, suggestion in self.en_structures:
|
||||
matches = re.findall(pattern, text, re.IGNORECASE)
|
||||
if matches:
|
||||
findings.append(("structure", pattern, [suggestion]))
|
||||
|
||||
return findings
|
||||
|
||||
def detect_xhs(self, text: str) -> List[Tuple[str, str, List[str]]]:
|
||||
"""检测小红书风格问题"""
|
||||
findings = []
|
||||
|
||||
for pattern, replacements in self.xhs_patterns.items():
|
||||
if pattern in text:
|
||||
findings.append(("xhs_phrase", pattern, replacements))
|
||||
|
||||
for pattern, suggestion in self.xhs_structures:
|
||||
matches = re.findall(pattern, text, re.MULTILINE)
|
||||
if matches:
|
||||
findings.append(("xhs_structure", pattern, [suggestion]))
|
||||
|
||||
# 检测emoji密度
|
||||
emoji_count = len(re.findall(r'[\U0001F600-\U0001F64F\U0001F300-\U0001F5FF\U0001F680-\U0001F6FF\U0001F1E0-\U0001F1FF]', text))
|
||||
if emoji_count > 5:
|
||||
findings.append(("emoji_density", f"{emoji_count}个emoji", ["建议减少到3-5个"]))
|
||||
|
||||
return findings
|
||||
|
||||
# ==================== 处理方法 ====================
|
||||
def humanize_cn(self, text: str, aggressive: bool = False) -> str:
|
||||
"""中文去AI化"""
|
||||
result = text
|
||||
|
||||
# 替换标志性短语
|
||||
replacements = {
|
||||
"值得注意的是": "",
|
||||
"我们可以看到": "",
|
||||
"不难发现": "",
|
||||
"显而易见": "很明显",
|
||||
"众所周知": "大家都知道",
|
||||
"综上所述": "总结来说",
|
||||
"归根结底": "说到底",
|
||||
"由此可见": "这说明",
|
||||
"换句话说": "也就是说",
|
||||
}
|
||||
|
||||
for old, new in replacements.items():
|
||||
if new:
|
||||
result = result.replace(old, new)
|
||||
else:
|
||||
# 删除或根据上下文调整
|
||||
result = result.replace(old + ",", "")
|
||||
result = result.replace(old + ",", "")
|
||||
|
||||
# 简化过渡词
|
||||
result = re.sub(r'首先[,,]?', "第一,", result)
|
||||
result = re.sub(r'其次[,,]?', "第二,", result)
|
||||
result = re.sub(r'最后[,,]?', "最后,", result)
|
||||
|
||||
# 去除过度修饰
|
||||
result = re.sub(r'非常([的地])', r'很\1', result)
|
||||
result = re.sub(r'极其([的地])', r'特别\1', result)
|
||||
|
||||
# 去除模板化开头
|
||||
result = re.sub(r'随着[^,]+的发展[,,]', "", result)
|
||||
result = re.sub(r'在当今社会[,,]', "", result)
|
||||
|
||||
if aggressive:
|
||||
# 更激进的优化
|
||||
result = re.sub(r'[,,]+', ",", result) # 合并多个逗号
|
||||
result = re.sub(r'。+', "。", result) # 合并多个句号
|
||||
|
||||
return result.strip()
|
||||
|
||||
def humanize_en(self, text: str, aggressive: bool = False) -> str:
|
||||
"""英文去AI化(基于Stop Slop)"""
|
||||
result = text
|
||||
|
||||
# 替换标志性短语
|
||||
replacements = {
|
||||
r"Here's the thing[:,.]*\s*": "",
|
||||
r"Here's the deal[:,.]*\s*": "",
|
||||
r"Let that sink in[:,.]*\s*": "",
|
||||
r"The uncomfortable truth is[:,.]*\s*": "",
|
||||
r"Let me be clear[:,.]*\s*": "",
|
||||
r"I'm going to be honest[:,.]*\s*": "",
|
||||
r"Make no mistake[:,.]*\s*": "",
|
||||
r"Full stop[.]*": ".",
|
||||
r"Period[.]*": ".",
|
||||
|
||||
# 过度使用的动词
|
||||
r"navigate\s+(the\s+)?": "handle ",
|
||||
r"unpack\s*": "explain ",
|
||||
r"leverage\s*": "use ",
|
||||
r"delve\s+(into\s+)?": "explore ",
|
||||
r"dive into\s*": "explore ",
|
||||
}
|
||||
|
||||
for pattern, replacement in replacements.items():
|
||||
result = re.sub(pattern, replacement, result, flags=re.IGNORECASE)
|
||||
|
||||
# 简化结构
|
||||
result = re.sub(r'Not because([^,.]+)[,.]\s*Because([^,.]+)[,.]',
|
||||
r'Not from\1, but from\2.', result, flags=re.IGNORECASE)
|
||||
|
||||
result = re.sub(r"(\w+)\.\s*That's it\.\s*That's the[^.]+\.",
|
||||
r'\1 is what matters.', result, flags=re.IGNORECASE)
|
||||
|
||||
# 去除过度修饰
|
||||
result = re.sub(r'In today\'s world[,\s]*', '', result, flags=re.IGNORECASE)
|
||||
result = re.sub(r'In the digital age[,\s]*', '', result, flags=re.IGNORECASE)
|
||||
result = re.sub(r'rapidly evolving', 'changing', result, flags=re.IGNORECASE)
|
||||
result = re.sub(r'fast-paced', 'busy', result, flags=re.IGNORECASE)
|
||||
|
||||
# 简化hedging
|
||||
result = re.sub(r'It\'s important to note that\s*', '', result, flags=re.IGNORECASE)
|
||||
result = re.sub(r'It\'s worth mentioning that\s*', '', result, flags=re.IGNORECASE)
|
||||
|
||||
if aggressive:
|
||||
# 去除更多学术腔
|
||||
result = re.sub(r'arguably\s*[,]*\s*', '', result, flags=re.IGNORECASE)
|
||||
result = re.sub(r'potentially\s*[,]*\s*', '', result, flags=re.IGNORECASE)
|
||||
|
||||
return result.strip()
|
||||
|
||||
def humanize_xhs(self, text: str) -> str:
|
||||
"""小红书风格优化"""
|
||||
result = text
|
||||
|
||||
# 替换营销词汇
|
||||
replacements = {
|
||||
"强烈推荐": "推荐",
|
||||
"必买": "值得买",
|
||||
"闭眼入": "推荐",
|
||||
"神器": "好用",
|
||||
"绝绝子": "很好",
|
||||
"YYDS": "很棒",
|
||||
"宝藏": "不错",
|
||||
"天花板": "顶级",
|
||||
"干货": "内容",
|
||||
"码住": "收藏",
|
||||
"抄作业": "参考",
|
||||
}
|
||||
|
||||
for old, new in replacements.items():
|
||||
result = result.replace(old, new)
|
||||
|
||||
# 简化称呼
|
||||
result = result.replace("姐妹们", "大家")
|
||||
result = result.replace("集美们", "大家")
|
||||
result = result.replace("宝子们", "大家")
|
||||
|
||||
# 优化emoji分布(保留但不过度)
|
||||
# 如果emoji过多,提示用户手动调整
|
||||
|
||||
# 减少标签堆砌
|
||||
result = re.sub(r'#{3,}', '#', result)
|
||||
|
||||
# 去除过于规整的序号
|
||||
result = re.sub(r'^\d+[.、]\s*', '', result, flags=re.MULTILINE)
|
||||
|
||||
return result.strip()
|
||||
|
||||
def auto_detect_and_humanize(self, text: str) -> Tuple[str, str]:
|
||||
"""自动检测语言并去AI化"""
|
||||
# 简单语言检测
|
||||
chinese_chars = len(re.findall(r'[\u4e00-\u9fff]', text))
|
||||
english_chars = len(re.findall(r'[a-zA-Z]', text))
|
||||
|
||||
if chinese_chars > english_chars:
|
||||
# 中文为主
|
||||
return self.humanize_cn(text), "zh"
|
||||
else:
|
||||
# 英文为主
|
||||
return self.humanize_en(text), "en"
|
||||
|
||||
def analyze(self, text: str) -> Dict:
|
||||
"""全面分析文本的AI痕迹"""
|
||||
cn_findings = self.detect_cn(text)
|
||||
en_findings = self.detect_en(text)
|
||||
xhs_findings = self.detect_xhs(text)
|
||||
|
||||
return {
|
||||
"cn_findings": cn_findings,
|
||||
"en_findings": en_findings,
|
||||
"xhs_findings": xhs_findings,
|
||||
"cn_score": len(cn_findings),
|
||||
"en_score": len(en_findings),
|
||||
"xhs_score": len(xhs_findings),
|
||||
"total_risk": len(cn_findings) + len(en_findings) + len(xhs_findings),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
"""命令行入口"""
|
||||
parser = argparse.ArgumentParser(description='去AI化工具')
|
||||
parser.add_argument('text', nargs='?', help='输入文本')
|
||||
parser.add_argument('-f', '--file', help='输入文件')
|
||||
parser.add_argument('-o', '--output', help='输出文件')
|
||||
parser.add_argument('-t', '--type', choices=['zh', 'en', 'xhs', 'auto'],
|
||||
default='auto', help='处理类型')
|
||||
parser.add_argument('-a', '--aggressive', action='store_true',
|
||||
help='激进模式')
|
||||
parser.add_argument('--analyze', action='store_true',
|
||||
help='仅分析不处理')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
detector = AntiAIDetector()
|
||||
|
||||
# 获取输入文本
|
||||
if args.file:
|
||||
with open(args.file, 'r', encoding='utf-8') as f:
|
||||
text = f.read()
|
||||
elif args.text:
|
||||
text = args.text
|
||||
else:
|
||||
print("请提供文本或使用-f指定文件")
|
||||
return
|
||||
|
||||
# 分析模式
|
||||
if args.analyze:
|
||||
result = detector.analyze(text)
|
||||
print("=" * 50)
|
||||
print("AI痕迹分析报告")
|
||||
print("=" * 50)
|
||||
print(f"中文AI痕迹: {result['cn_score']} 处")
|
||||
print(f"英文AI痕迹: {result['en_score']} 处")
|
||||
print(f"小红书风格问题: {result['xhs_score']} 处")
|
||||
print(f"总体风险: {result['total_risk']} 处")
|
||||
print("\n详细发现:")
|
||||
for finding in result['cn_findings'][:5]:
|
||||
print(f" [CN] {finding[1]}")
|
||||
for finding in result['en_findings'][:5]:
|
||||
print(f" [EN] {finding[1]}")
|
||||
return
|
||||
|
||||
# 处理模式
|
||||
if args.type == 'auto':
|
||||
result, lang = detector.auto_detect_and_humanize(text)
|
||||
print(f"检测到语言: {lang}")
|
||||
elif args.type == 'zh':
|
||||
result = detector.humanize_cn(text, args.aggressive)
|
||||
elif args.type == 'en':
|
||||
result = detector.humanize_en(text, args.aggressive)
|
||||
elif args.type == 'xhs':
|
||||
result = detector.humanize_xhs(text)
|
||||
|
||||
# 输出结果
|
||||
if args.output:
|
||||
with open(args.output, 'w', encoding='utf-8') as f:
|
||||
f.write(result)
|
||||
print(f"结果已保存到: {args.output}")
|
||||
else:
|
||||
print("\n" + "=" * 50)
|
||||
print("处理结果:")
|
||||
print("=" * 50)
|
||||
print(result)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Anti-AI Detector - IT资讯系统集成模块
|
||||
用于AI处理器的去AI化后处理
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# 添加技能目录到路径
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from humanize import AntiAIDetector
|
||||
|
||||
|
||||
class ITContentHumanizer:
|
||||
"""IT资讯内容人性化处理器"""
|
||||
|
||||
def __init__(self):
|
||||
self.detector = AntiAIDetector()
|
||||
|
||||
def process_translation(self, text: str, source_lang: str = 'en') -> str:
|
||||
"""
|
||||
处理翻译后的内容
|
||||
|
||||
Args:
|
||||
text: AI翻译后的文本
|
||||
source_lang: 源语言 ('en' 或 'zh')
|
||||
|
||||
Returns:
|
||||
去AI化后的文本
|
||||
"""
|
||||
# 检测并去除AI痕迹
|
||||
if source_lang == 'en':
|
||||
# 英文文章翻译成中文后
|
||||
result = self.detector.humanize_cn(text)
|
||||
else:
|
||||
# 中文文章保持中文优化
|
||||
result = self.detector.humanize_cn(text)
|
||||
|
||||
return result
|
||||
|
||||
def process_summary(self, text: str, platform: str = 'default') -> str:
|
||||
"""
|
||||
处理总结内容
|
||||
|
||||
Args:
|
||||
text: AI总结后的文本
|
||||
platform: 目标平台 ('default', 'xiaohongshu', 'email')
|
||||
|
||||
Returns:
|
||||
优化后的文本
|
||||
"""
|
||||
if platform == 'xiaohongshu':
|
||||
# 小红书风格优化
|
||||
result = self.detector.humanize_xhs(text)
|
||||
# 再进行中文去AI化
|
||||
result = self.detector.humanize_cn(result)
|
||||
elif platform == 'email':
|
||||
# 邮件风格 - 只去AI化
|
||||
result = self.detector.humanize_cn(text)
|
||||
else:
|
||||
# 默认处理
|
||||
result = self.detector.humanize_cn(text)
|
||||
|
||||
return result
|
||||
|
||||
def process_title(self, title: str) -> str:
|
||||
"""
|
||||
优化标题
|
||||
|
||||
Args:
|
||||
title: 原标题
|
||||
|
||||
Returns:
|
||||
优化后的标题
|
||||
"""
|
||||
# 去除标题中的AI痕迹
|
||||
result = self.detector.humanize_cn(title)
|
||||
|
||||
# 去除常见的AI标题套路
|
||||
result = result.replace("值得一提的是", "")
|
||||
result = result.replace("不得不说", "")
|
||||
result = result.replace("揭秘", "")
|
||||
result = result.replace("深度解析", "")
|
||||
|
||||
return result.strip()
|
||||
|
||||
def full_pipeline(self, content: dict) -> dict:
|
||||
"""
|
||||
完整处理管道
|
||||
|
||||
Args:
|
||||
content: {
|
||||
'title': '原标题',
|
||||
'translated': '翻译内容',
|
||||
'summary': '总结内容',
|
||||
'platform': 'xiaohongshu' 或 'email'
|
||||
}
|
||||
|
||||
Returns:
|
||||
处理后的内容
|
||||
"""
|
||||
return {
|
||||
'title': self.process_title(content.get('title', '')),
|
||||
'translated': self.process_translation(
|
||||
content.get('translated', ''),
|
||||
content.get('source_lang', 'en')
|
||||
),
|
||||
'summary': self.process_summary(
|
||||
content.get('summary', ''),
|
||||
content.get('platform', 'default')
|
||||
),
|
||||
}
|
||||
|
||||
def analyze_risk(self, text: str) -> dict:
|
||||
"""
|
||||
分析AI痕迹风险
|
||||
|
||||
Returns:
|
||||
{
|
||||
'risk_level': 'low'|'medium'|'high',
|
||||
'score': int,
|
||||
'suggestions': list
|
||||
}
|
||||
"""
|
||||
analysis = self.detector.analyze(text)
|
||||
total = analysis['total_risk']
|
||||
|
||||
if total < 3:
|
||||
level = 'low'
|
||||
elif total < 8:
|
||||
level = 'medium'
|
||||
else:
|
||||
level = 'high'
|
||||
|
||||
suggestions = []
|
||||
if analysis['cn_score'] > 0:
|
||||
suggestions.append(f"发现 {analysis['cn_score']} 处中文AI痕迹")
|
||||
if analysis['en_score'] > 0:
|
||||
suggestions.append(f"发现 {analysis['en_score']} 处英文AI痕迹")
|
||||
if analysis['xhs_score'] > 0:
|
||||
suggestions.append(f"发现 {analysis['xhs_score']} 处小红书风格问题")
|
||||
|
||||
return {
|
||||
'risk_level': level,
|
||||
'score': total,
|
||||
'suggestions': suggestions,
|
||||
'details': analysis
|
||||
}
|
||||
|
||||
|
||||
# 便捷函数,供IT项目直接调用
|
||||
def humanize_for_it_news(text: str, content_type: str = 'translation') -> str:
|
||||
"""
|
||||
IT资讯项目的便捷调用函数
|
||||
|
||||
Args:
|
||||
text: AI生成的文本
|
||||
content_type: 'translation'|'summary'|'title'
|
||||
|
||||
Returns:
|
||||
去AI化后的文本
|
||||
"""
|
||||
humanizer = ITContentHumanizer()
|
||||
|
||||
if content_type == 'translation':
|
||||
return humanizer.process_translation(text)
|
||||
elif content_type == 'summary':
|
||||
return humanizer.process_summary(text, platform='default')
|
||||
elif content_type == 'title':
|
||||
return humanizer.process_title(text)
|
||||
else:
|
||||
return humanizer.detector.humanize_cn(text)
|
||||
|
||||
|
||||
# 小红书专用函数
|
||||
def humanize_for_xiaohongshu(text: str) -> str:
|
||||
"""
|
||||
小红书内容专用去AI化
|
||||
"""
|
||||
humanizer = ITContentHumanizer()
|
||||
return humanizer.process_summary(text, platform='xiaohongshu')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# 测试
|
||||
test_text = """
|
||||
值得注意的是,这项技术非常重要。我们可以看到,随着AI的发展,
|
||||
越来越多的应用场景被开发出来。首先,它提高了效率;其次,
|
||||
它降低了成本;最后,它改变了工作方式。综上所述,
|
||||
这项技术毫无疑问具有极其重要的意义。
|
||||
"""
|
||||
|
||||
humanizer = ITContentHumanizer()
|
||||
|
||||
print("原文:")
|
||||
print(test_text)
|
||||
print("\n去AI化后:")
|
||||
result = humanizer.process_translation(test_text)
|
||||
print(result)
|
||||
|
||||
print("\n风险分析:")
|
||||
risk = humanizer.analyze_risk(test_text)
|
||||
print(f"风险等级: {risk['risk_level']}")
|
||||
print(f"得分: {risk['score']}")
|
||||
print("建议:", risk['suggestions'])
|
||||
@@ -0,0 +1,163 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Anti-AI Detector 测试脚本
|
||||
"""
|
||||
|
||||
import sys
|
||||
sys.path.insert(0, '/home/ubuntu/.nanobot/workspace/skills/anti-ai-detector')
|
||||
|
||||
from humanize import AntiAIDetector
|
||||
from it_integration import ITContentHumanizer, humanize_for_it_news, humanize_for_xiaohongshu
|
||||
|
||||
|
||||
def test_chinese():
|
||||
"""测试中文去AI化"""
|
||||
print("="*60)
|
||||
print("测试1: 中文去AI化")
|
||||
print("="*60)
|
||||
|
||||
text = """
|
||||
值得注意的是,这项技术非常重要。我们可以看到,随着AI的发展,
|
||||
越来越多的应用场景被开发出来。首先,它提高了效率;其次,
|
||||
它降低了成本;最后,它改变了工作方式。综上所述,
|
||||
这项技术毫无疑问具有极其重要的意义。
|
||||
"""
|
||||
|
||||
detector = AntiAIDetector()
|
||||
result = detector.humanize_cn(text)
|
||||
|
||||
print("原文:")
|
||||
print(text.strip())
|
||||
print("\n去AI化后:")
|
||||
print(result)
|
||||
print()
|
||||
|
||||
|
||||
def test_english():
|
||||
"""测试英文去AI化"""
|
||||
print("="*60)
|
||||
print("测试2: 英文去AI化 (Stop Slop)")
|
||||
print("="*60)
|
||||
|
||||
text = """
|
||||
Here's the thing: building great products is hard. It turns out that
|
||||
most teams struggle with alignment. Not because they lack talent.
|
||||
Because they lack clarity. Let that sink in.
|
||||
|
||||
The uncomfortable truth is that most advice is generic. Let me be clear:
|
||||
speed without direction is just chaos. We need to navigate the complex
|
||||
landscape of product development and leverage our resources effectively.
|
||||
"""
|
||||
|
||||
detector = AntiAIDetector()
|
||||
result = detector.humanize_en(text)
|
||||
|
||||
print("原文:")
|
||||
print(text.strip())
|
||||
print("\n去AI化后:")
|
||||
print(result)
|
||||
print()
|
||||
|
||||
|
||||
def test_xiaohongshu():
|
||||
"""测试小红书风格"""
|
||||
print("="*60)
|
||||
print("测试3: 小红书风格优化")
|
||||
print("="*60)
|
||||
|
||||
text = """
|
||||
姐妹们!今天给大家分享一个宝藏神器!🔥🔥🔥
|
||||
|
||||
1️⃣ 首先,这个产品真的绝绝子!
|
||||
2️⃣ 其次,效果YYDS!
|
||||
3️⃣ 最后,闭眼入不踩雷!
|
||||
|
||||
强烈推荐大家码住这个干货!✅✅✅
|
||||
#好物推荐 #必买清单 #宝藏神器
|
||||
"""
|
||||
|
||||
detector = AntiAIDetector()
|
||||
result = detector.humanize_xhs(text)
|
||||
|
||||
print("原文:")
|
||||
print(text.strip())
|
||||
print("\n优化后:")
|
||||
print(result)
|
||||
print()
|
||||
|
||||
|
||||
def test_it_integration():
|
||||
"""测试IT资讯集成"""
|
||||
print("="*60)
|
||||
print("测试4: IT资讯项目集成")
|
||||
print("="*60)
|
||||
|
||||
content = {
|
||||
'title': '深度解析:揭秘AI技术的十大不为人知的秘密',
|
||||
'translated': """
|
||||
值得注意的是,这项技术非常重要。我们可以看到,随着AI的发展,
|
||||
越来越多的应用场景被开发出来。
|
||||
""",
|
||||
'summary': """
|
||||
首先,AI提高了效率;其次,它降低了成本;最后,它改变了工作方式。
|
||||
综上所述,这项技术毫无疑问具有极其重要的意义。
|
||||
""",
|
||||
'platform': 'xiaohongshu'
|
||||
}
|
||||
|
||||
humanizer = ITContentHumanizer()
|
||||
result = humanizer.full_pipeline(content)
|
||||
|
||||
print("原标题:", content['title'])
|
||||
print("优化标题:", result['title'])
|
||||
print("\n优化总结:")
|
||||
print(result['summary'])
|
||||
print()
|
||||
|
||||
|
||||
def test_analysis():
|
||||
"""测试AI痕迹分析"""
|
||||
print("="*60)
|
||||
print("测试5: AI痕迹风险分析")
|
||||
print("="*60)
|
||||
|
||||
text = """
|
||||
Here's the thing: 值得注意的是,这项技术非常重要。
|
||||
我们可以看到,随着AI的发展,越来越多的应用场景被开发出来。
|
||||
Let that sink in. 首先,它提高了效率;其次,它降低了成本。
|
||||
"""
|
||||
|
||||
humanizer = ITContentHumanizer()
|
||||
risk = humanizer.analyze_risk(text)
|
||||
|
||||
print("分析文本:")
|
||||
print(text.strip())
|
||||
print(f"\n风险等级: {risk['risk_level']}")
|
||||
print(f"得分: {risk['score']}")
|
||||
print("建议:")
|
||||
for s in risk['suggestions']:
|
||||
print(f" - {s}")
|
||||
print()
|
||||
|
||||
|
||||
def main():
|
||||
"""运行所有测试"""
|
||||
print("\n")
|
||||
print("╔" + "="*58 + "╗")
|
||||
print("║" + " "*15 + "Anti-AI Detector 测试" + " "*20 + "║")
|
||||
print("╚" + "="*58 + "╝")
|
||||
print("\n")
|
||||
|
||||
test_chinese()
|
||||
test_english()
|
||||
test_xiaohongshu()
|
||||
test_it_integration()
|
||||
test_analysis()
|
||||
|
||||
print("="*60)
|
||||
print("所有测试完成!")
|
||||
print("="*60)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user