DeepFal/graylog-ops
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

初始提交：恶意 IP 特征+行为融合检测系统

Browse Source 
- 特征检测（规则/签名）与行为检测（24h 统计）单进程运行
- 融合名单按首次发现时间倒序，展示位置/归属（GeoIP + ASN）
- 钉钉融合推送（加签、24h 节流、去重）
- 页面状态灯、CSV 导出、Docker 镜像构建文件
This commit is contained in:
DeepFal committed 2026-10-18 05:03:05 +08:00
commit 2d76ed6672
42 files changed
+10645
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitignore
+24
View File
@@ -0,0 +1,24 @@
# 凭据
graylog-ops.json
notify.json
config.local.json
*.token
# 运行时数据/采集产物
data/
collect-*/
graylog-snapshot-*/
graylog-collect-*/
snapshots/
*.tar.gz
*.log
.DS_Store
# Python
__pycache__/
*.pyc
.venv/
# Docker 构建临时目录（由 Dockerfile 生成 staging）
docker-build/app/
docker-build/seed/
README.md
+672
View File
@@ -0,0 +1,672 @@
# graylog-ops
Graylog **4.0.x** 只读信息采集 + 封禁判断工具，服务于"用业务访问日志自动封 IP"这条链路。
## 边界（先说清楚）
Graylog 是**旁路日志系统**：它能看到、能聚合、能告警，但**不在业务流量链路上，不能自己封 IP**。
本工具同样只读：不建流、不改索引、不建事件、不下发任何封禁。
```
外部客户端
│
▼
深信服 AD 7.0.8R5（应用交付 / 负载均衡 / 反代）
│ ▲
│ └── 将来封禁的执行点大概率在这里（黑名单 / 访问控制）
▼
业务系统(nginx 等) ──日志──► Graylog ──► graylog-ops（本工具：采集 + 判断）
│
└──► {ip, action, score, reasons}
```
## 深信服 AD 在链路里的影响（关键）
AD 挡在业务系统前面，会直接改变"日志里的客户端 IP"的含义，排查顺序：
1. **AD 是否透传真实客户端 IP**：HTTP 类型的业务看 AD 的"真实客户端 IP / X-Forwarded-For"配置；
四层转发通常保留源地址，七层反代不配置就会变成 AD 自己的地址。
2. **后端 nginx 的 `$remote_addr` 是不是 AD 地址**。若是，日志里必须加
`$http_x_forwarded_for`（或 AD 提供的专用头），并在采集端落成独立字段。
3. 用 `discover` 验证：如果候选字段的"私有 IP 占比"很高、且 `source` 只有一个取值，
基本可以判定拿到的是 AD/反代地址，而不是真实客户端。
4. 把 AD 的管理地址、业务 VIP、回源地址写进白名单，避免把自己的设备封掉。
## 为什么先做"信息获取 + 判断"
封禁规则依赖四个事实，猜错任何一个都会封错人：
1. **真实客户端 IP 落在哪个字段**（`source` 往往是采集器/反代地址，不是访问者）
2. 日志里有哪些可做判据的字段（`status` / `request_uri` / `user_agent` / `response_time` …）
3. 量级（每秒多少条，决定窗口和阈值）
4. 现有 stream / 索引 / 事件定义是怎么组织的
`discover` 回答 1、2；`histogram` / `top-ips` 回答 3；`snapshot` / `streams` / `alerts` 回答 4；
`judge` 把这些聚合成"封谁、为什么、怎么处理"。
## 快速开始
零第三方依赖，只用 Python 3.7+ 标准库。
```bash
cd graylog-ops
# 方式 A：环境变量
export GRAYLOG_URL=http://edu-master:9002
export GRAYLOG_TOKEN='从 Graylog UI 创建的 API token'
bin/graylog-ops probe
# 方式 B：配置文件（推荐，token 不落 shell history）
cp config.example.json graylog-ops.json
# 编辑 graylog-ops.json 填 url / token
bin/graylog-ops --config graylog-ops.json probe
# 方式 C：临时参数
bin/graylog-ops --url http://edu-master:9002 --token "$TOKEN" probe
```
> 认证方式：**API token 当用户名，字面量密码 `token`**（4.0 文档的写法）。
> 工具内部已按此拼 `Authorization: Basic base64(<token>:token)`，你只需要给 `--token`。
> 非 GET 请求会自动带 `X-Requested-By`（Graylog 的 CSRF 保护要求）。
>
> **token 放哪**：写进 `graylog-ops.json`（已在 `.gitignore` 里）并 `chmod 600`。
> 只需要读权限的 token，**不需要管理员账号密码**；用完在 UI 里一键吊销。
> 不要把 token 贴到聊天、工单或截图里——`--config` 读文件时它不会出现在命令行或输出中。
## 命令
| 命令 | 作用 |
| --- | --- |
| `probe` | 连通性 / 认证 / 16 个接口的可用性矩阵，**第一步跑这个** |
| `system` | 版本、节点、索引器健康、deflector、license |
| `streams [--rules]` | stream 列表（分流与权限边界），`--rules` 额外拉规则 |
| `inputs` | input 类型、监听地址、端口 |
| `fields` | Graylog 已注册字段及类型 |
| `indices` | index set 轮转/保留策略、索引大小 |
| `alerts` | 现有事件定义与通知方式（改造前先看，别重复建） |
| `discover` | **采样日志，推断真实客户端 IP 字段**（核心） |
| `judge` | **把聚合信号判成封禁候选**（只读，dry-run 语义） |
| `top-ips` | `terms` 聚合，默认 `--field client_ip`，看 top IP 与计数 |
| `terms` | 任意字段的 terms 聚合 |
| `messages` | 拉原始消息（表格 / `--jsonl`） |
| `histogram` | 消息量时间分布，估算阈值与容量 |
| `snapshot` | 全量环境信息落盘（每个资源一个 JSON + manifest） |
| `raw` | 直接调用任意 API 路径（支持 `--method`/`--body`），应对版本差异 |
| `collect` | 一条命令产出 probe/discover/judge/snapshot 整套信息 |
| `watch` | 持续检测恶意公网 IP + SQLite 清单 + 本地 Web 面板 |
通用参数：`--json`（机器可读）、`--out FILE`（写 JSON）、`-v`（打印每个 HTTP 请求）。
## 五步摸底
```bash
# 1) 能不能通、这个版本到底有哪些接口
bin/graylog-ops probe
# 2) 客户端 IP 在哪个字段（最关键的结论），同时生成 profile
bin/graylog-ops discover --range 3600 --stream 业务访问日志 \
--save-profile profiles/edu.json
# 3) 用真实数据看阈值是否合理
bin/graylog-ops top-ips --field client_ip --query 'status:401' --range 300 --size 30
bin/graylog-ops histogram --range 3600 --interval 5m
# 4) 判断：谁该封、为什么
bin/graylog-ops judge --profile profiles/edu.json --range 300 \
--allow allowlist.example.txt --explain 5
# 5) 整体快照存档，方便离线分析和复盘
bin/graylog-ops snapshot --dir snapshots/$(date +%F) --sample 500
```
> 机器上要能同时访问 `10.180.x`（Graylog）和 `10.230.x`（AD），否则先在跳板机跑。
`discover` 的输出（示例数据）：
```
采样 8 条消息 query=* range=3600s stream=(all)
分数 字段 出现率 IP占比 不同IP 私有IP IP版本 样例
---- --------- ------ ------ ------ ------ ------ ----------------------------
95.0 client_ip 100% 100% 4 12% v4 203.0.113.5, 198.51.100.7
45.0 source 100% 100% 1 100% v4 10.180.254.17
推荐字段: client_ip (confidence=high)
- 字段 `client_ip` 的 IP 占比 100%，样例 203.0.113.5, 198.51.100.7, 192.0.2.44
! `source` 只有 1 个不同取值，而 `client_ip` 有 4 个：`source` 是采集器/反代，封禁必须用 `client_ip`
```
## 判断规则（judge）
固定的 5 个 `terms` 查询 + 3 个统计量，不随 IP 数量增加查询次数：
| 信号 | 查询（字段名来自 profile） | 权重 | 饱和点 | 说明 |
| --- | --- | --- | --- | --- |
| `auth_fail` | `status:(401 OR 403)` | 45 | 10 次 | 认证失败 |
| `scanner_ua` | `http_user_agent:(sqlmap OR nikto OR nuclei OR nmap OR …)` | 50 | 2 次 | 扫描器特征 |
| `sensitive_path` | `request_uri:(wp-login.php OR phpmyadmin OR actuator OR …)` | 25 | 5 次 | 敏感路径探测 |
| `not_found` | `status:404` | 20 | 20 次 | 目录/文件探测 |
| `bad_method` | `request_method:(PUT OR DELETE OR TRACE OR CONNECT)` | 10 | 5 次 | 异常方法 |
| `rate` | 请求数 / 200 | 25 | 200 次 | 绝对量 |
| `spike` | 请求量 ≥ 候选中位数 4 倍 | 15 | 64 倍 | 相对异常 |
| `auth_ratio` | 失败占比 ≥ 50% 且 ≥ 10 次 | 25 | — | 几乎全是失败 |
动作映射：
| 条件 | 动作 |
| --- | --- |
| 有攻击证据 且 score ≥ 75 | `ban_1h` |
| 有攻击证据 且 score ≥ 55 | `ban_30m` |
| score ≥ 40 | `observe`（只记录，不封） |
| 其余 | `ignore` |
三条安全约束（比阈值本身更重要）：
1. **内网地址默认硬排除**（RFC1918 / 回环 / 链路本地 / CGNAT），避免把 AD、LB、NAT 出口封掉；
`--include-internal` 可关闭，但一般不该关。
2. **白名单硬排除**：`--allow` 支持多个文件，一行一个 IP/CIDR，`#` 注释。
把 AD 管理地址、业务 VIP、回源地址、办公出口、监控探针都写进去。
3. **纯流量不封**（`require_evidence`）：没有任何攻击特征、只是请求量大的客户端，
最多判 `observe`，不会因为"量大"被自动拉黑。
`judge` 输出示例：
```
窗口 300s query=* field=client_ip stream=(all) 候选池 4（中位请求数 35.0）
得分 动作 客户端 IP 请求数 依据
---- ------ ------------ ------ -----------------------------------------------
98.1 ban_1h 192.0.2.99 25 扫描器 UA 25 次; 敏感路径 25 次; 404 探测 25 次
75.0 ban_1h 203.0.113.66 40 认证失败 40 次; 失败请求占 100%
已排除（不参与判分）:
10.180.5.5 30 内部地址（RFC1918/回环/CGNAT）
```
**信号命中 0 个 IP 时要注意**：可能是该窗口确实没有这类流量，也可能是查询模板跟字段分词不匹配
（比如 `request_uri` 在 ES 里的分词方式和你以为的不一样）。用 profile 覆盖 `query` 后再跑，
`judge` 会把每个信号的实际查询和命中数都打出来。
## 持续检测服务（watch）
```bash
# 持续运行（面板 + 检测循环）
bin/graylog-ops --config graylog-ops.json watch \
--stream F5_Accesslog --profile profiles/f5.json \
--interval 60 --window 900 --db data/ips.db \
--port 8088 --allow allowlist.example.txt
# 只跑一轮（离线分析，比如 8 小时）
bin/graylog-ops --config graylog-ops.json watch --once \
--stream F5_Accesslog --profile profiles/f5.json \
--window 28800 --db data/ips-8h.db
# 只提供面板，浏览已有数据库
bin/graylog-ops --config graylog-ops.json watch --serve-only \
--db data/ips-8h.db --port 8089
```
- 面板：`http://127.0.0.1:8088`（30 秒自动刷新）
- JSON API：`/api/ips?tier=malicious&limit=100`、`/api/stats`、`/api/ip/<ip>`
- 存储：`data/ips.db`（SQLite）。IP 连续 `--forget` 秒（默认 7200）不再出现就自动移除
- 名单：`--export PATH` 每轮把高置信恶意 IP 写成纯文本（默认剔除 `shared_egress`）
### 面板（给同事看的）
**只有一个地址** `http://<本机IP>:8088`（只读，无鉴权，仅限内网使用）。
历史（17–18 日 25 小时）与实时检测共用同一个 SQLite（`data/ips.db`），
所以打开就是一个完整的大列表：
- **核心名单**（默认视图）：只列高置信恶意 IP，默认按首次攻击倒序（最新在前），点任意一行进详情
- 所有判定为恶意的 IP（含此前标为「疑似共享出口」的）统一归入核心名单，不再有「待确认」档
- **详情页**：评分环、请求总量、平均频率、峰值时段、活跃时段数、命中规则与次数、
请求频率分布图、状态码分布、请求方法分布、访问域名分布，以及**按判定类型抽样的证据**
（优先抽攻击类请求；高流量 IP 不会把日志刷屏），附处置建议
- 搜索框可按 IP 或判定原因过滤；**排序字段与方向是两个独立下拉**（首次攻击 / 最近攻击 / 评分 / 请求数 × 升序 / 降序）
- **单击任意 IP 地址即可复制**（列表和详情页标题都支持，带 toast 提示）
- 列表 30 秒自动刷新（可关闭）
界面截图：[列表页](docs/screenshots/list.png) · [详情页](docs/screenshots/detail.png)
- **一键导出**（右上角两个下拉，都作用于当前筛选/排序结果）：
- 「复制 IP ▾」→ 逗号分隔 / 换行分隔（换行版可直接粘贴给 ipset、防火墙批量导入）
- 「导出 ▾」→ CSV（完整字段） / 仅 IP 列表（每行一个）
- 只读 API：`/api/ips?scope=core|all`、`/api/ip/<ip>`、`/api/stats`
- 导出接口（脚本可直接拉）：
- `GET /api/export.csv?scope=core|all&sort=&order=` → CSV（含等级/评分/首次攻击/原因）
- `GET /api/export.txt?scope=...` → 纯 IP 列表，每行一个
证据**完全按需生成**，本地库只存判定结果（56 个 IP 约 36 KB），点开详情时才连 Graylog：
- **9 个查询并发**（URI/UA/状态码/方法/域名/服务端 IP/VIP + 时间分布 + 样本），首次打开约 0.3–0.5 秒，
结果在内存缓存 10 分钟，再打开约 1 毫秒；进程重启后重新拉取
- 样本走 views 的 `messages` 搜索：旧版 `/search/universal/relative` 在 25 小时窗口 + 复合查询下
会被 Graylog 断连，表现就是"只有分布没有样本"（WebShell 这类 IP 尤其明显）
- 样本按**判定类型**分组：每种类型各保留最近 3 条（两种类型 6 条、三种类型 9 条，以此类推），
同一秒内的重复请求会合并；没有命中规则时退化为最近 5 条通用请求
- 每条样本展示源日志的**完整字段**：客户端/服务端 `IP:端口`、Host、VIP(bigip)、VS、
地区、响应时间 `rt`、请求长度、Referer、采集来源，以及原始日志行
- **长字段可点击弹窗**查看完整内容（URI / UA / Referer / Host / 原始日志行），Esc 或点遮罩关闭
- **轻量快照兜底**：每次查看详情时顺手存一份 1–2 KB 的摘要（URI/状态/方法/VIP/时间分布），
Graylog 索引轮转（约 10 天）后自动回退到快照，并明确标注"日志已过期，以下为判定时的快照"
- 负载：每轮固定 6 个 terms 查询 + 最多 `--verify` 次核验，**与 IP 数量无关**
（本环境 15 分钟窗口约 3.5–5 秒/轮）
规则（速率类信号按窗口归一化到 15 分钟基准，行为类信号不缩放）：
| 规则 | 权重 | 饱和 | 说明 |
| --- | --- | --- | --- |
| 敏感路径探测 | 65 | 2 次 | `/actuator`、`/.env`、`/.git`、`/wp-login`、`/druid`、`/nacos`、`/swagger`、`/console`、`/ISAPI`… |
| 扫描器 UA | 55 | 1 次 | sqlmap / nikto / nuclei / nmap / zgrab … |
| 登录失败（401/403） | 55 | 8 次 | 登录类 URI 上的认证失败 |
| 爬虫 UA | 50 | 20 次 | DotBot 等 |
| 登录接口高频 | 15 | 2000 次 | 只作上下文，单独不足以定性（学生也会反复登录） |
| 404 占比 | 25 | 比例 ≥50% 且 ≥20 次 | 目录/文件探测 |
分级：**≥60 恶意 / ≥30 可疑 / 其余观察**（观察级分数 <`--min-observe-score`(10) 不入库）。
安全约束：内网与白名单硬排除；单一接口高频的自动化客户端自动降级为 `automation`；
候选池取「所有规则命中的 IP ∪ 访问量 top」，因此**低流量扫描器不会被漏掉**。
### 地理信息（离线 GeoIP）
- 列表新增「国家/地区」列（国旗 + 中文国名 + 省/州），详情页也显示
- 数据来自 **DB-IP Lite**（免费、CC BY 4.0），已构建为本地 SQLite：`data/geo/geo.db`（144MB，360 万条 IPv4 段）
- 检测时自动补全；已有 IP 在启动时自动回填
- 更新地理库（每月）：
```bash
cd data/geo
curl -O https://download.db-ip.com/free/dbip-city-lite-2026-10.csv.gz
gunzip -kf dbip-city-lite-2026-10.csv.gz
cd ../.. && python3 -m graylog_ops.geo \
--city-csv data/geo/dbip-city-lite-2026-10.csv --out data/geo/geo.db
```
- 关闭地理功能：`--geo-db ""`
### 新增 IP 推送通知
- 只有 **新增恶意 IP** 或 **观察/可疑 → 恶意升级** 才会推送；同一 IP 不会重复推
- 支持企业微信 / 钉钉 / 飞书 / Slack / 通用 JSON（`type` 字段选择）
- 推送失败只打印日志，不影响检测循环
- 配置：复制 `notify.example.json` 为 `notify.json`（已在 .gitignore 中），填 webhook：
```json
{
"type": "dingtalk",
"url": "https://oapi.dingtalk.com/robot/send?access_token=你的TOKEN",
"secret": "SEC你的加签密钥",
"link_prefix": "http://179.253.234.124:12139/s/301b.../",
"title": "新增恶意 IP"
}
```
| type | 说明 |
| --- | --- |
| `wecom` | 企业微信群机器人（markdown） |
| `dingtalk` | 钉钉自定义机器人（markdown，支持 `secret` 加签） |
| `feishu` | 飞书自定义机器人（text） |
| `slack` | Slack Incoming Webhook |
| `generic` | POST `{"title","text"}`，可接自己的网关 |
测试与自检：
```bash
# 发一条测试消息（验证 webhook 是否通）
./bin/graylog-ops --config graylog-ops.json watch --notify-config notify.json --notify-test \
--stream F5_Accesslog --profile profiles/f5.json --db data/ips.db
```
推送内容示例：
```
**新增恶意 IP**
**112.32.1.211**（中国 · 广东）
评分 100.0 · 请求 108 次
首次攻击 2026-10-17T18:24:09Z
命中：Spring/中间件探测 40 次；配置/凭证探测 8 次
[查看详情](http://179.253.234.124:12139/s/301b.../#/ip/112.32.1.211)
```
### 状态指示灯
页面右上角有一个圆点，表示后端与 Graylog 的连接状态：
- 绿色：最近一轮检测成功
- 红色：最近一轮检测失败
- 黄色：状态过期
- 灰色：尚未开始检测
不展示文字，悬停可看提示。
### 行为检测（已并入主服务，单进程）
特征检测和行为检测现在跑在**同一个进程**里：
- 特征检测线程：每 60 秒一轮（原有逻辑）
- 行为检测线程：每 15 分钟一轮（24h 窗口，纯行为评分，不依赖规则和名单）
- 同一个 Web 前端：融合首页 + 融合详情
**融合名单规则**：
- 只有「建议封禁」才展示和推送：特征命中攻击证据，或行为判定为建议封禁（非共享出口）
- 共享出口/NAT、人工确认、数据不足等行为结果**不展示、不推送**
- 一个 IP 只出现一行，来源标「特征 / 行为 / 双来源」
- 首页按 `first_banned_at`（首次进入建议封禁的时间）倒序，**每个 IP 只置顶一次**，后续再命中不改变位置
- 历史封禁 IP 永久保留在列表里（不参与 24h 再推）
**推送**：
- 融合成一条消息，同一 IP 永远不重复推
- 同一 IP 24 小时内最多推一次；24 小时后仍在活动（最近 24h 有流量）的 IP 会再推一条
- 首次部署时历史 IP 静默（不补推）
**详情页**：同一 IP 融合展示「行为检测画像」和「特征检测证据」。
相关参数：`--behavior-interval 900`（行为刷新间隔）、`--no-behavior`（关闭行为检测）。
### ASN / IP 归属
- 列表「位置 / 归属」合并为一列、上下两行：第一行国家/地区，第二行图标 + 厂商/机构（如 ☁️ 阿里云、📱 中国移动）
- 悬停显示 ASN 号、完整组织名和网络类型
- 详情页显示 ASN 号 + 厂商 + 组织全名 + 类型
- CSV 导出包含 `asn, asn_org, asn_brand, asn_type`
- 品牌映射在 `asn_brand_map.json`（可继续加厂商），数据库 `data/geo/asn.db`
- 更新方式（DB-IP ASN Lite，每月）：
```bash
curl -O https://download.db-ip.com/free/dbip-asn-lite-2026-10.csv.gz
gunzip -kf dbip-asn-lite-2026-10.csv.gz
python3 -m graylog_ops.asn --csv data/geo/dbip-asn-lite-2026-10.csv --out data/geo/asn.db
```
关闭 ASN：`--asn-db ""`
### 名单语义（重要）
- **永久保留**：一旦某个 IP 被判定为恶意（`ever_malicious=1` + `last_malicious_at`），
就永远不会被 `--forget` 清理，即使它后来降级为可疑/观察，也仍留在核心名单和导出名单里。
只有"从未判过恶意"的观察级 IP 才会按 `--forget` 过期删除。
- **时间字段全部来自日志，不是检测时间**：
- 首次攻击 = 日志中最早命中攻击规则的时间（最多回溯 36 小时）
- 最近攻击 = 日志中最近一次命中攻击规则的时间
- 最近出现 = 该 IP 最后一条日志的时间
- `hits` 只是"被检测到的轮次"，不是攻击次数
- **只有窗口内计数发生变化时才回查并更新时间**，避免滚动窗口把"最近攻击"不断刷成当前时间
- 历史数据用 `watch --fix-times`（别名 `--fix-first-seen`）修正
- **默认排序 = 首次攻击倒序（最新攻击的在最上面）**，同一发现时间下最近还在攻击的排前面；
面板右上角可切换「最近攻击（最新优先）」「评分」「请求数」。
- `blocklist.txt` 导出的是**历史累计**的全部恶意 IP，不是当前窗口快照。
- 乱码处理：源日志里非 UTF-8 的字段会被清洗，识别为二进制时显示 `[非 UTF-8 数据]`，
URI/方法/状态码等字段不会再把乱码带进页面。
### 启动方式（按需，不挂后台服务）
按当前约定**不安装 launchd/systemd**，需要看的时候手动起（DSH/终端重启后进程会消失，重新执行即可）：
```bash
cd "/Users/deepfal/Code/Work/ip sec/graylog-ops"
./bin/graylog-ops --config graylog-ops.json watch \
--stream F5_Accesslog --profile profiles/f5.json \
--interval 60 --window 3600 --forget 604800 \
--db data/ips.db --host 0.0.0.0 --port 8088 \
--allow allowlist.example.txt --export data/blocklist.txt --warm-limit 5
```
停止：Ctrl+C（或用 `pkill -f "graylog_ops watch"`）。
维护命令：
```bash
# 用日志重新校准历史 IP 的首次攻击 / 最近攻击 / 最近出现时间（幂等）
./bin/graylog-ops --config graylog-ops.json watch --fix-times \
--stream F5_Accesslog --profile profiles/f5.json --db data/ips.db
# 为名单里的恶意 IP 预生成证据快照
./bin/graylog-ops --config graylog-ops.json watch --warm-only --warm-limit 200 \
--stream F5_Accesslog --profile profiles/f5.json --db data/ips.db
```
## 两日实测结果（本地 10-17 00:00 – 10-18 01:00）
窗口 89,600 秒（约 24.9 小时），12 条规则，`--size 600`：
**55 个候选 → 48 个恶意 = 44 个高置信 + 4 个疑似共享出口；4 个可疑。**
| 探测类型 | 典型路径 | 规则 |
| --- | --- | --- |
| Spring/中间件 | `/actuator*`、`/druid*`、`/nacos*`、`/swagger*`、`/jmx*`、`/heapdump*`、`/console*`、`/manager/html*`、`/solr*` | probe_java |
| 配置/凭证 | `/.env*`、`/.git*`、`/.svn*`、`/.aws*`、`/.ssh*`、`/appsettings*`、`/WEB-INF*`、`/backup*` | probe_config |
| CMS | `/wp-*`、`/xmlrpc.php`、`/phpmyadmin*`、`/administrator*`、`/joomla*`、`/thinkphp*` | probe_cms |
| IoT/路由器 | `/ISAPI*`、`/LAPI*`、`/HNAP1*`、`/boaform*`、`/GponForm*`、`/cgi-bin/luci*` | probe_iot |
| DevOps/监控 | `/jenkins*`、`/grafana*`、`/kibana*`、`/zabbix*`、`/prometheus*`、`/consul*`、`/harbor*` | probe_devops |
| WebShell/上传 | `/shell.php`、`/cmd.php`、`/eval-stdin.php`、`/phpinfo*`、`/vendor/phpunit*` | probe_webshell |
| 注入/穿越 | `etc/passwd`、`jndi:`、`union select`、`%2e%2e`、`%252f`、`base64_decode` | exploit_payload |
| 登录撞库 | `/login*`、`/passport*`、`/captcha*` + 401/403 | login_attack / login_burst |
| 扫描器/爬虫 | sqlmap/nikto/nuclei/ffuf…、DotBot | bad_ua / bot_ua |
高置信名单（截至本轮回填，已写入 `data/blocklist.txt`）：
```
117.136.27.147 113.31.155.142 113.31.155.219 116.176.186.63
39.144.210.4 120.232.115.57 39.144.210.0 186.241.107.91
91.92.242.37 36.159.247.84 113.201.250.120 42.91.116.131
124.152.12.48 120.83.247.23 223.198.163.145 217.60.242.58
112.32.8.112 119.42.146.230 120.214.218.174 213.165.242.55
216.219.95.217 190.122.240.80 91.194.91.202 182.16.82.226
109.105.209.20 42.94.151.105 5.188.86.234 209.99.189.217
165.154.218.226 115.190.159.225 51.158.145.102 111.229.129.233
223.123.92.48 222.83.114.16 109.105.209.18 43.136.79.172
91.92.243.72 35.209.176.178 47.121.125.205 109.205.176.10
130.12.180.171 130.12.180.168
```
疑似共享出口 / NAT（4 个，**排除在名单外**）：42.92.144.29(653344)、27.185.218.213(125241)、118.181.187.248(30978)、42.94.148.133(22757)
判定公式：`(请求量 >= 50000 且 探测占比 < 2%) 或 (请求量 >= 20000 且 探测占比 < 0.5%)`
→ 标记 `risk=shared_egress`，导出名单时自动剔除。
> 规则收紧记录：`/uploadfiles/*`（正常文件目录）和 `/database/*`（正常业务路径）曾被
> 误判为 WebShell/配置探测，已从前缀里移除；`/config.json`、`/settings.json` 同理。
## profile 格式
`discover --save-profile` 生成的骨架，`judge --profile` 读取：
```json
{
"fields": {
"client_ip": "client_ip",
"status": "status",
"uri": "request_uri",
"ua": "http_user_agent",
"method": "request_method"
},
"signals": {
"sensitive_path": {"query": "request_uri:wp-login", "weight": 30, "saturate": 3}
},
"thresholds": {
"min_requests": 30,
"ban_score": 70
}
}
```
`fields` 可被命令行覆盖（`--field` / `--status-field` / `--uri-field` / `--ua-field` / `--method-field`），
`signals` 里 `enabled: false` 可以关掉某个信号。
## Graylog 4.0.8 接口笔记
| 用途 | 路径 |
| --- | --- |
| 版本/集群 | `GET /api/system` |
| 节点 | `GET /api/system/cluster/nodes` |
| 索引器健康 | `GET /api/system/indexer/cluster/health` |
| 字段类型 | `GET /api/system/fields` |
| stream | `GET /api/streams` |
| input | `GET /api/system/inputs` |
| index set | `GET /api/system/indices/index_sets?skip=0&limit=100` |
| 索引 | `GET /api/system/indexer/indices` |
| 事件定义 | `GET /api/events/definitions?page=1&per_page=50` |
| 通知 | `GET /api/events/notifications` |
| 相对时间搜索 | `GET /api/search/universal/relative?query=*&range=300&limit=100&sort=timestamp:desc` |
| 绝对时间搜索 | `GET /api/search/universal/absolute?query=*&from=...&to=...` |
| terms 聚合 | `GET /api/search/universal/terms?field=client_ip&range=300&size=50` |
| 直方图 | `GET /api/search/universal/histogram?query=*&range=3600&interval=1m` |
| 统计 | `GET /api/search/universal/stats?field=...&stats=cardinality,count` |
| stream 过滤 | 追加 `filter=streams:<streamId>` |
4.0 的关键点：
- **legacy 搜索 API 是主路径**。`/api/views/search`（pivot）在 4.0 已经存在，但 DTO 与 5.x/6.x 差异大，
本工具不用它，避免被版本升级打断。需要时可用 `raw` 直接调。
- `size` / `limit` 在 terms 端点上的参数名历史上不一致，客户端遇到 400 会自动换另一个再试，`probe` 会显示实际生效的那个。
- 时间一律 **UTC ISO8601**，返回形如 `2026-10-17T15:00:00.000Z`。
- 搜索返回是 `{"messages": [{"message": {...}}, ...]}`，工具已摊平。
- 4.0.8 跑在 Java 8 / CentOS 7 上，`/api/system` 不返回 ES 版本；后端是 ES 6.8 或 7.10。
## 4.0.8 实测踩坑（本环境已验证）
这几条是拿着真实 token 打 edu-master 得出的，不是文档抄的：
1. **`terms` / `histogram` / `stats` 在 4.0 已移除**（返回 404）。
聚合必须走 views API：`POST /api/views/search/sync`（pivot）。
工具会自动探测：views 可用就用 views，3.x 上自动回退到 legacy 端点。
pivot 的行组是 `{"type":"values","field":"uri","limit":50}`（注意是 `field` 不是 `fields`）。
2. **字段是 keyword 语义**，查询有三条铁律：
- `uri:ketang` → 0 条；`uri:/ketang-studyapi/apis*` → 46 万条（只有**尾部通配符**有效）
- **未转义的 `/` 会被 Lucene 当成正则分隔符**：`uri:/manager/html*` 会误命中所有含
`html` token 的文档（实测 9079 条，全是 UEditor 的 JS 静态文件）。必须写成
`uri:\/manager\/html*`，或引号精确匹配 `uri:"/wp-login.php"`
- 前导通配符 `uri:*admin*` 被禁用（直接 500/400）
judge 的默认信号查询已按此修正（`ua:sqlmap*`、`uri:\/actuator*`）。
3. **部分 F5 VS 的 HTTP 字段为空串**：1 小时样本里 `uri` 约 98% 有值、`status` 约 48%、
`ua` 约 48%。四层 VS 只有 `client_ip/server_ip/method` 等基础字段。
4. `status` 是混格式字符串（`200 OK` / `404` / `502 Bad Gateway`），`status:404` 仍能按
token 命中；`status_code` 字段全空，不要用。
5. `event_type` 只有 `req`，不存在 req/resp 双份，不需要去重。
## 本环境实测结论（2026-10-18）
| 项 | 结论 |
| --- | --- |
| 集群 | 4 节点：`edu-master`(master) + `edu-slave1/3/4`，2354 个 active shard，deflector `graylog_457` |
| 数据链路 | **F5**(`10.180.241.2/3`) --syslog--> Graylog。深信服 AD 不在 F5 访问日志链路上（它对应的是 DNS 类业务） |
| 活跃 stream | `F5_Accesslog`（约 11 万条/5 分钟）、`F5 Attack Logs`、`有源业务数据-Nat-Stream`（NAT 会话）、`syslog_vpn_stream`（VPN）；`站群`、`全量站点数据` **24h 内无数据** |
| 量级 | `F5_Accesslog` 约 130 万条/小时（≈360 条/秒） |
| 字段 | `client_ip` / `server_ip` / `uri` / `host` / `ua` / `status` / `method` / `bigip` / `vs`；**没有 XFF 字段** |
| 关键事实 | `10.185.9.41`(46%) 和 `10.185.9.15`(45%) 两个**内网 IP 占 91% 流量**，是内部 API 轮询（`/ketang-studyapi/apis/*`，一半是 404），必须白名单/内网排除；外网用户 F5 直见真实 IP，所以 `client_ip` 可用于封禁 |
真实判定样例（1 小时窗口，`--verify 5`）：
```
100.0 ban_1h 186.241.107.91 1164 认证失败 24; 敏感路径 36; 404 探测 156
68.5 ban_30m 113.201.250.120 612 认证失败 3; 敏感路径 6; 404 探测 5
54.0 observe 124.70.2.111 666 认证失败 333（集中在单接口，已降级）
```
- `186.241.107.91` 挖到 `/actuator/env`、`/actuator/xyzabc`（fuzz）、`/actuator;/health`（分号绕过尝试）
- `113.201.250.120` 同样在打 `/actuator/env`
- `124.70.2.111` 每 **10.6 秒** POST 同一个业务 API、稳定 403 —— 自动化客户端 token 过期，
被 `--verify` 识别并降级为 `observe`，不误封
`--verify N` 会对前 N 个 ban 候选多做一次 URI 聚合：若流量 90% 以上集中在单一非登录接口，
判定为"疑似业务客户端/自动化任务"并降级。这是本环境实测后加的必要规则。
## 权限
只读账号给这些权限就够（缺哪个会报 403，`probe` 会逐条标出来）：
```
messages:read searches:relative searches:absolute
searches:keyword streams:read inputs:read
indexsets:read eventdefinitions:read eventnotifications:read
```
首次摸底建议直接用管理员 token，确认链路通了再收窄到只读 token。
## 已知限制
- 纯只读，不提供写入/告警创建（下一阶段再做，会单独走 `--apply` 路径）。
- `judge` 是单字段聚合：只按 `client_ip` 分组，不做多字段联合（如 IP×URI）。
这是刻意的取舍——固定查询次数，代价是拿不到"同一 IP 命中几种不同敏感路径"这类信号。
需要时用 `--explain` 抽样人工看。
- 每类聚合最多看 top `--size`（默认 200）个 IP；超出部分不参与判分。
- `discover` 用 offset 分页采样，适合几千条量级；十万级请改用 `terms` / `histogram`。
- 大时间窗 + 高基数 `terms` 会打 ES，`--range` 别一上来就 7d；先 `histogram` 看量。
## 与执行侧衔接（下一步）
判断层已经产出 `{ip, action, score, reasons, signals}`。执行侧有两个候选：
1. **Graylog 事件 + HTTP 通知**：事件定义按 `client_ip` 分组 count 超阈值 → 通知到 ban-service。
优点是近实时；缺点是阈值表达力有限，复杂判定还得在服务里做。
2. **定时跑 `judge`**（如每 1 分钟）+ 直接调 AD 接口：
```
cron: graylog-ops judge --range 120 --json --out /var/lib/ban/candidates.json
→ ban-service 读 candidates
→ 白名单/冷却/幂等校验
→ 调 AD 黑名单 API 下发
→ TTL 到期自动删除 / 复核
```
**AD 接口这块先不猜**。截图里管理入口是
`https://10.230.13.25:4433/index?x_id=<...>#mod=/mod_runtime/homepage/index`，
`x_id` 看着像会话/鉴权参数，前端是 SPA 路由，真正的 XHR 端点得抓包确认。需要你做的只有一件事：
> 在能访问 AD 的机器上打开 Chrome DevTools → Network → 勾选 Fetch/XHR →
> 手工做一遍"添加/删除一条 IP 黑名单" → 右键 `Copy as cURL`（或导出 HAR）发我。
需要覆盖的请求：登录、会话刷新、业务/节点列表、黑名单增删查、下发后的生效确认。
拿到之后我补一个 `sangfor` 模块（登录态维持、黑名单 CRUD、幂等、限流），复用现有的 HTTP 重试/错误归一。
## 环境备忘
| 项 | 值 |
| --- | --- |
| Graylog | 4.0.8+6b8c55d @ `edu-master:9002`（4 节点，Java 8 / CentOS 7） |
| 深信服 AD | 7.0.8R5，UI `https://10.230.13.25:4433`，HA（主）`10.1.1.1`，业务 5 / 节点 3 |
| 网络 | 开发机可访问 `10.180.254.17:9002` 与 `10.230.13.25:4433` |
| 凭据 | `graylog-ops.json`（chmod 600，已 gitignore），只读 token |
## 开发
```bash
python3 tests/demo.py # 用假服务预览各命令输出，不需要真 Graylog
python3 -m unittest discover -s tests # 41 项单测 + 端到端测试
```
测试不需要真实 Graylog：`tests/fake_graylog.py` 起一个本地 HTTP 服务，按 4.0 的响应结构回包，
并实现了一个很小的 Lucene 子集（`field:value`、`field:(a OR b)`）来验证 `judge` 的查询构造。
要加新接口，先在这个假服务里加路由，再补一条 e2e。
```
graylog_ops/
http.py urllib 封装：Basic auth、X-Requested-By、重试、错误归一
config.py CLI / 环境变量 / JSON 文件三级配置
client.py 4.0 REST 接口 -> 普通 dict/list，容忍响应形状差异
detect.py 客户端 IP 字段推断（IP 解析、内部地址判定、评分）
judge.py 判断层：信号采集、基线、评分、动作映射、白名单
commands.py 各子命令（构建 JSON + 人读渲染）
render.py CJK 宽度对齐的表格
tests/
fake_graylog.py 假 Graylog 4.0 服务（含查询子集）
test_detect.py 字段推断单测
test_judge.py 判断层测试（暴力破解/扫描/正常大流量/内网/白名单）
test_e2e.py CLI 端到端
```
## 排错
| 现象 | 原因 / 处理 |
| --- | --- |
| 连接超时 | 服务在 **9002** 端口（9000 是空等）。9002 也不通时走 VPN / 内网跳板机 |
| 401 | token 错或已吊销；确认是"API token"不是 session token |
| 403 | 账号缺权限，看 `probe` 里哪项 FAIL |
| 404 | 该版本没这个接口，用 `raw /api/...` 验证，或 `probe` 看矩阵 |
| 400 / 422 | 参数不被接受，加 `-v` 看实际请求 URL |
| `discover` 没结果 | 时间窗内无数据 / stream 过滤错 / 查询语法错 / 无权限（静默返回空） |
| `discover` 推荐的还是代理 IP | 采集端没解析 XFF；需要 AD 透传 + nginx `log_format` 落真实 IP |
| `judge` 没候选 | 窗口内 IP 数低于 `--min-requests`，或分数低于 `--min-score`；先 `--all` 看全量 |
| `judge` 信号命中 0 | 查询模板与字段分词不匹配（见"判断规则"末尾），用 profile 覆盖 query |
| 封禁候选里出现自己的设备 | 白名单没写全；把 AD 地址/VIP/回源/监控补进 `--allow` 文件 |
allowlist.example.txt
+22
View File
@@ -0,0 +1,22 @@
# 封禁白名单（judge --allow 使用）
# 一行一个 IP 或 CIDR，# 之后是注释。
# 这里列的东西永远不会进入封禁候选，请按实际环境补全。
# --- 基础设施 ---
10.180.254.17 # Graylog 主机 edu-master
10.230.13.25 # 深信服 AD 管理地址（UI :4433）
# --- 深信服 AD ---
10.1.1.1 # AD HA/设备地址（截图显示，按实际确认）
# 10.230.13.0/24 # AD 所在管理网段，酌情放开
# --- 业务 VIP / 反代回源 ---
# 填 AD 上配置的业务 VIP 与后端真实服务器地址
# 10.1.1.0/24
# --- 办公出口 / 集中 NAT ---
# 注意：整个出口网段拉黑会中断整栋楼的上网，务必确认
# 10.180.0.0/16
# --- 监控 / 探测 ---
# 监控探针、拨测、健康检查来源
analysis/behavior_prototype.py
+951
View File
@@ -0,0 +1,951 @@
#!/usr/bin/env python3
"""Behaviour analysis prototype v2 (read-only).
Computes 24h traffic features per client IP from Graylog pivot aggregations,
derives a behaviour profile, combines it with the existing rule score into a
single risk score with an action recommendation, and serves the result on a
separate port. It never writes to the production sqlite database.
"""
import argparse
import csv
import ipaddress
import json
import os
import re
import statistics
import sys
import sqlite3
import time
from concurrent.futures import ThreadPoolExecutor
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
from graylog_ops.http import HttpClient # noqa: E402
from graylog_ops.client import GraylogClient, normalize_histogram # noqa: E402
from graylog_ops import watch # noqa: E402
STREAM = "streams:6ac83059a6878e26256fcacc"
WINDOW = 86400
DATA_FILE = os.path.join(ROOT, "data", "behavior_prototype.json")
DB_FILE = os.path.join(ROOT, "data", "ips.db")
INTERNAL = ("10.", "192.168.", "172.16.", "172.17.", "172.18.", "172.19.",
"172.2", "172.3")
UA_SDK = ("curl", "wget", "python-requests", "python-urllib", "go-http-client",
"okhttp", "java/", "apache-httpclient", "httpcomponents",
"libredtail", "axios", "node-fetch", "aiohttp", "scrapy",
"postmanruntime", "insomnia", "unityplayer", "dart/", "httpie",
"guzzle", "faraday", "restsharp", "libwww", "eeo/", "academy_widget")
UA_BROWSER = ("chrome", "firefox", "safari", "edg/", "opr/", "gecko/20100101")
ASN_CSV = os.path.join(ROOT, "data", "geo", "dbip-asn-lite-2026-10.csv")
_ASN_CACHE = {}
def _load_asn():
if _ASN_CACHE:
return _ASN_CACHE
import bisect
starts, ranges = [], []
try:
with open(ASN_CSV, encoding="utf-8", errors="replace") as fh:
for row in csv.reader(fh):
if len(row) < 4 or ":" in row[0]:
continue
try:
first = int(ipaddress.IPv4Address(row[0]))
last = int(ipaddress.IPv4Address(row[1]))
except ValueError:
continue
starts.append(first)
ranges.append((last, row[2], row[3]))
except FileNotFoundError:
pass
order = sorted(range(len(starts)), key=lambda i: starts[i])
starts = [starts[i] for i in order]
ranges = [ranges[i] for i in order]
_ASN_CACHE.update({"starts": starts, "ranges": ranges, "bisect": bisect})
return _ASN_CACHE
def asn_info(ip):
cache = _load_asn()
starts = cache.get("starts") or []
if not starts:
return None
try:
value = int(ipaddress.IPv4Address(ip))
except ValueError:
return None
index = cache["bisect"].bisect_right(starts, value) - 1
if index >= 0:
last, asn, org = cache["ranges"][index]
if last >= value:
return {"asn": asn, "org": org, "shared": shared_type(org)}
return None
def shared_type(org):
text = (org or "").lower()
cloud = ("alibaba", "aliyun", "tencent", "huawei cloud", "aws", "amazon",
"google", "microsoft", "digitalocean", "vultr", "linode", "hetzner",
"ovh", "wowrack", "idc", "data center", "datacenter", "cloud",
"hosting", "colo")
if any(token in text for token in cloud):
return "hosting"
if "china mobile" in text or "中国移动" in text or "cmnet" in text:
return "mobile"
if ("china unicom" in text or "china telecom" in text
or "chinanet" in text or "联通" in text or "电信" in text
or "broadband" in text):
return "broadband"
if "education" in text or "cernet" in text or "教育" in text:
return "education"
return "unknown"
ORG_CN = (("china mobile", "中国移动"), ("china unicom", "中国联通"),
("china telecom", "中国电信"), ("chinanet", "中国电信"),
("cernet", "中国教育网"), ("education", "教育网"),
("alibaba", "阿里云"), ("aliyun", "阿里云"),
("huawei cloud", "华为云"), ("tencent", "腾讯云"),
("wowrack", "Wowrack 机房"), ("vmiss", "VMISS 机房"),
("idc", "IDC 机房"))
def org_cn(org):
text = (org or "").lower()
for token, name in ORG_CN:
if token in text:
return name
return org or "未知"
SHARE_LABEL = {"mobile": "移动网络", "broadband": "宽带 ISP",
"education": "教育网", "hosting": "云主机/机房",
"unknown": "未知网络"}
def humanize(item):
total = item["total"]
parts = []
if item["uri_card"] >= 20:
parts.append("访问了 %s 个不同路径" % format(item["uri_card"], ","))
if item["four_xx"] >= 20:
parts.append("产生 %s 次失败请求（失败率 %.0f%%）"
% (format(int(item["four_xx"]), ","), item["pct4"]))
elif item["pct4"] >= 20:
parts.append("失败率 %.0f%%" % item["pct4"])
if item["coverage"] <= 30 and item["peak_share"] >= 25:
parts.append("流量集中在很短的时间内")
if item["login_ratio"] >= 20:
parts.append("大量请求集中在登录接口")
if item["host_card"] >= 2:
parts.append("同时访问了 %d 个站点" % item["host_card"])
if item["ua_family"] in ("sdk", "fake-browser", "other"):
parts.append("使用脚本/非浏览器客户端")
behavior = item.get("behavior_type") or "异常自动化"
tail = {
"目录扫描": "符合目录扫描特征",
"短时扫描": "符合短时扫描特征",
"登录爆破": "符合登录爆破特征",
"伪造爬虫 UA": "伪装成搜索引擎爬虫，且请求大量失败",
"采集爬虫": "是采集爬虫",
"异常自动化": "行为高度自动化，不像正常用户",
}.get(behavior, "行为异常")
if parts:
return "24 小时内" + "，".join(parts) + "——" + tail + "。"
return "24 小时内请求 %d 次，%s。" % (total, tail)
def load_client():
cfg = json.load(open(os.path.join(ROOT, "graylog-ops.json")))
return GraylogClient(HttpClient(cfg["url"], cfg["token"], "token"))
def pivot(client, series, row_groups, column_groups=None, query="*",
limit=300, timeout=180000):
body = {"queries": [{
"id": "q1",
"timerange": {"type": "relative", "range": WINDOW},
"query": {"type": "elasticsearch", "query_string": query},
"filter": {"type": "or", "filters": [{"type": "stream",
"id": STREAM.split(":")[-1]}]},
"search_types": [{
"type": "pivot", "id": "st1", "series": series,
"row_groups": [{"type": "values", "field": f, "limit": limit}
for f in row_groups],
"column_groups": column_groups or [], "sort": [], "rollup": True,
}],
}], "parameters": []}
data = client.http.request("POST", "/api/views/search/sync",
params={"timeout": timeout}, body=body)
return data["results"]["q1"]["search_types"]["st1"]
def rows_to_dict(search_type, series_id):
out = {}
for row in search_type.get("rows") or []:
key = row.get("key") or []
if not key:
continue
ip = str(key[0])
for value in row.get("values") or []:
if (value.get("key") or [None])[0] == series_id:
try:
out[ip] = float(value.get("value") or 0)
except (TypeError, ValueError):
out[ip] = 0.0
return out
def hourly_matrix(search_type):
out = {}
for row in search_type.get("rows") or []:
key = row.get("key") or []
if not key:
continue
counts = []
for value in row.get("values") or []:
stamp = (value.get("key") or [None])[0]
if not stamp or "cnt" not in (value.get("key") or []):
continue
try:
counts.append((stamp, float(value.get("value") or 0)))
except (TypeError, ValueError):
pass
out[str(key[0])] = counts
return out
def bucket_stats(counts):
active = [c for c in counts if c > 0]
if not counts:
return {"coverage": 0.0, "cv": None, "peak_share": 0.0}
cv = (statistics.pstdev(active) / statistics.fmean(active)
if len(active) > 2 and statistics.fmean(active) else None)
peak = max(counts) if counts else 0
return {"coverage": len(active) / len(counts) * 100,
"cv": cv,
"peak_share": (peak / sum(counts) * 100) if sum(counts) else 0.0}
def uri_template(uri):
path = str(uri or "").split("?")[0]
path = re.sub(r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-"
r"[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "{uuid}", path)
path = re.sub(r"[0-9a-fA-F]{16,}", "{hex}", path)
path = re.sub(r"\d+", "{n}", path)
return path
def ua_family(ua):
text = str(ua or "").strip()
if not text:
return "empty"
lower = text.lower()
if any(token in lower for token in UA_SDK):
return "sdk"
if lower.startswith("mozilla/") and any(t in lower for t in UA_BROWSER):
return "browser"
if lower.startswith("mozilla/5.0"):
return "fake-browser"
return "other"
def ua_garbage(ua_card, top_ua):
"""The F5 UA field carries binary garbage on some VS: detect it."""
if ua_card > 50:
return True
text = str(top_ua or "")
if not text:
return False
printable = sum(1 for char in text if 32 <= ord(char) < 127 or ord(char) > 127)
return printable / max(1, len(text)) < 0.85
def rule_data(client, fields):
rules = {r["name"]: {"title": r["title"], "query": r["query"](fields),
"weight": r["weight"]}
for r in watch.RULE_DEFS}
hits = {}
def run(item):
name, meta = item
try:
st = pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query=meta["query"], limit=300)
return name, rows_to_dict(st, "cnt")
except Exception:
return name, {}
with ThreadPoolExecutor(max_workers=6) as pool:
for name, counts in pool.map(run, rules.items()):
for ip, count in counts.items():
if count > 0:
hits.setdefault(ip, {})[name] = int(count)
return rules, hits
def detail_for(client, ip):
q = 'client_ip:"%s"' % ip
detail = {"uri": [], "ua": [], "host": []}
try:
st = pivot(client, [{"type": "count", "id": "cnt"}], ["uri"],
query=q, limit=40)
for row in st.get("rows") or []:
key = row.get("key") or []
value = (row.get("values") or [{}])[0].get("value")
if key:
detail["uri"].append((str(key[0]), int(float(value or 0))))
except Exception:
pass
for label, field, size in (("ua", "ua", 3), ("host", "host", 2)):
try:
st = pivot(client, [{"type": "count", "id": "cnt"}], [field],
query=q, limit=size)
for row in st.get("rows") or []:
key = row.get("key") or []
value = (row.get("values") or [{}])[0].get("value")
if key:
detail[label].append((str(key[0]), int(float(value or 0))))
except Exception:
pass
try:
hist = client.histogram(query=q, range_seconds=WINDOW, interval="5m",
filter_str=STREAM)
detail["buckets5m"] = [b["count"] for b in normalize_histogram(hist)]
except Exception:
detail["buckets5m"] = []
return ip, detail
def gather_details(client, ips):
out = {}
with ThreadPoolExecutor(max_workers=10) as pool:
for ip, detail in pool.map(lambda value: detail_for(client, value), ips):
out[ip] = detail
return out
def rule_score_for(total, hits):
tier, score, reasons = watch.classify(total, hits)
return tier, score, reasons
def behavior_profile(feat):
"""Return (category, adjustment, confidence, notes) from behaviour only."""
notes = []
total = feat["total"]
coverage = feat["coverage"]
cv = feat["cv"]
pct2, pct4 = feat["pct2"], feat["pct4"]
status_ok = feat["status_known"] >= 10
ua_ok = not feat["ua_garbage"]
ua_card = feat["ua_card"]
if not ua_ok:
notes.append("UA 字段异常（%d 种）" % ua_card)
if not status_ok:
notes.append("status 字段缺失（仅 %.0f%%）" % feat["status_known"])
if total < 100:
notes.append("24h 仅 %d 次请求，行为样本不足" % total)
return "insufficient", -10, "低", notes
if coverage >= 80 and cv is not None and cv <= 1.2 and ua_ok and ua_card <= 2:
if status_ok and pct4 >= 50:
notes.append("长期稳定轮询、%.0f%% 失败" % pct4)
return "integration_error", -35, "中", notes
notes.append("覆盖率 %.0f%%、小时CV %.2f、UA %d 个%s"
% (coverage, cv, ua_card,
("、2xx %.0f%%" % pct2) if status_ok else ""))
return "server", -35, "高" if status_ok else "中", notes
if coverage >= 80 and cv is not None and cv > 1.2 and ua_ok and ua_card <= 2:
if (status_ok and pct2 >= 85) or not status_ok:
notes.append("覆盖率 %.0f%%、小时CV %.2f，周期性批量任务" % (coverage, cv))
return "batch_job", -30, "中", notes
if ua_ok and ua_card >= 5 and feat["total"] >= 300:
notes.append("%d 种 UA、请求 %d 次，疑似共享出口/NAT"
% (ua_card, total))
return "shared_nat", -40, "中", notes
if ((feat["uri_templates"] >= 200 and total >= 500 and ua_ok
and ua_card <= 3 and (not status_ok or pct2 >= 80))
or (feat["get_ratio"] >= 85 and feat["uri_templates"] >= 100
and total >= 1000 and ua_ok)):
notes.append("访问 %d 个 URI 模板，UA 稳定%s"
% (feat["uri_templates"],
("、2xx %.0f%%" % pct2) if status_ok else ""))
return "crawler", -15, "中", notes
if status_ok and pct4 >= 50 and total >= 1000 and ua_ok and ua_card <= 3:
notes.append("请求 %d 次、%.0f%% 失败，疑似鉴权/配置异常" % (total, pct4))
return "integration_error", -35, "中", notes
if coverage <= 30 and ua_card >= 3 and (not status_ok or pct4 >= 20):
notes.append("短时活跃、%d 种 UA、%.0f%% 4xx" % (ua_card, pct4))
return "script_candidate", 10, "低", notes
if coverage <= 30 and (not status_ok or pct2 >= 90):
notes.append("短时活跃、成功率 %.0f%%" % pct2)
return "human", -25, "低", notes
notes.append("覆盖率 %.0f%%、CV %s、4xx %.0f%%"
% (coverage, ("%.2f" % cv) if cv is not None else "-", pct4))
return "uncertain", 0, "低", notes
def combine(rule_tier, rule_score, behavior_cat, behavior_adj):
"""Hybrid risk score + action recommendation."""
score = rule_score + behavior_adj
server_like = behavior_cat in ("server", "batch_job", "shared_nat",
"integration_error")
if behavior_cat == "insufficient":
server_like = False
risk = max(0.0, min(100.0, score))
if rule_score >= 60:
action = "人工确认" if server_like else "封禁"
elif risk >= 45:
action = "人工确认"
elif risk >= 25:
action = "观察"
else:
action = "正常"
if server_like and action == "封禁":
action = "人工确认"
return round(risk, 1), action, server_like
def behavior_score(feat):
"""Behaviour-only risk score. Uses no rule hits and no known list."""
total = feat["total"]
if total < 10:
return 0, ["请求过少"]
score = 0
signals = []
pct4 = feat["pct4"]
if pct4 >= 50:
score += 30
signals.append("高失败率 %.0f%%" % pct4)
elif pct4 >= 30:
score += 15
signals.append("失败率 %.0f%%" % pct4)
if feat["uri_card"] >= 20:
score += 20
signals.append("路径分散（%d 个 URI）" % feat["uri_card"])
elif feat["uri_card"] >= 8:
score += 8
if feat["coverage"] <= 30 and feat["peak_share"] >= 25:
score += 20
signals.append("短时爆发（覆盖 %.0f%%、峰值 %.0f%%）"
% (feat["coverage"], feat["peak_share"]))
if feat["login_ratio"] >= 20:
score += 15
signals.append("登录接口集中 %.0f%%" % feat["login_ratio"])
if feat["host_card"] >= 2:
score += 5
signals.append("多站点（%d 个 Host）" % feat["host_card"])
if feat["ua_card"] == 1:
score += 5
if feat["ua_family"] in ("sdk", "fake-browser", "other"):
score += 10
signals.append("自动化 UA（%s）" % feat["ua_family"])
elif feat["ua_family"] == "browser":
score -= 5
if feat.get("four_xx", 0) >= 100 and pct4 >= 20:
score += 10
signals.append("失败请求 %d 次" % feat["four_xx"])
# business/integration client: mostly POST or a few steady endpoints
if feat["get_ratio"] < 30:
score -= 25
signals.append("非 GET 为主（业务接口）")
if pct4 < 15 and feat["pct2"] >= 80:
score -= 20
signals.append("低失败率（采集/浏览）")
if (feat["coverage"] >= 80 and feat["cv"] is not None
and feat["cv"] <= 1.2 and feat["uri_templates"] <= 8):
score -= 20
signals.append("长期稳定单接口")
if feat["uri_templates"] >= 15:
score += 10
signals.append("路径模板分散（%d 个）" % feat["uri_templates"])
if feat["status_known"] < 10:
score -= 10
signals.append("状态字段缺失")
return max(0, min(100, score)), signals
def empty_feat(ip):
return {
"ip": ip, "total": 0, "uri_card": 0, "ua_card": 0, "host_card": 0,
"method_card": 0, "get_ratio": 0.0, "login_ratio": 0.0,
"pct2": 0.0, "pct3": 0.0, "pct4": 0.0, "pct5": 0.0,
"status_known": 0.0, "coverage": 0.0, "cv": None, "peak_share": 0.0,
"coverage5m": 0.0, "cv5m": None, "peak_share5m": 0.0,
"uri_templates": 0, "ua_family": "", "ua_garbage": False,
"top_uri": "", "top_ua": "",
}
def finalize_feat(feat, status_map, get_map, login_map, hourly_map):
ip = feat["ip"]
buckets = [c for _, c in hourly_map.get(ip, [])]
hour = bucket_stats(buckets)
sum_status = sum(status_map[cls].get(ip, 0) for cls in status_map)
total = feat["total"]
feat.update({
"get_ratio": get_map.get(ip, 0) / total * 100 if total else 0.0,
"login_ratio": login_map.get(ip, 0) / total * 100 if total else 0.0,
"pct2": status_map["2*"].get(ip, 0) / max(1, sum_status) * 100,
"pct3": status_map["3*"].get(ip, 0) / max(1, sum_status) * 100,
"pct4": status_map["4*"].get(ip, 0) / max(1, sum_status) * 100,
"pct5": status_map["5*"].get(ip, 0) / max(1, sum_status) * 100,
"status_known": (sum_status / total * 100) if total else 0.0,
"coverage": hour["coverage"],
"cv": hour["cv"],
"peak_share": hour["peak_share"],
"four_xx": status_map["4*"].get(ip, 0),
})
return feat
def analyze():
client = load_client()
profile = json.load(open(os.path.join(ROOT, "profiles", "f5.json")))
fields = profile.get("fields", profile)
known = {}
try:
db = sqlite3.connect(DB_FILE)
db.row_factory = sqlite3.Row
for row in db.execute("select ip, ever_malicious from ip_stats"):
known[row["ip"]] = bool(row["ever_malicious"])
except sqlite3.Error:
pass
print("拉取 24h 全量聚合 ...", flush=True)
st = pivot(client,
[{"type": "count", "id": "cnt"},
{"type": "card", "field": "uri", "id": "uri_card"},
{"type": "card", "field": "ua", "id": "ua_card"},
{"type": "card", "field": "host", "id": "host_card"},
{"type": "card", "field": "method", "id": "method_card"}],
["client_ip"], limit=600)
total = rows_to_dict(st, "cnt")
uri_card = rows_to_dict(st, "uri_card")
ua_card = rows_to_dict(st, "ua_card")
host_card = rows_to_dict(st, "host_card")
method_card = rows_to_dict(st, "method_card")
status = {cls: {} for cls in ("2*", "3*", "4*", "5*")}
for cls in status:
st_cls = pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query="status:%s" % cls, limit=600)
status[cls] = rows_to_dict(st_cls, "cnt")
get_count = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query="method:GET", limit=600), "cnt")
LOGIN_QUERY = ("uri:(\\/login* OR \\/passport* OR \\/captcha* "
"OR \\/jsxy\\/api\\/login*)")
login_count = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query=LOGIN_QUERY, limit=600), "cnt")
st_hour = pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
column_groups=[{"type": "time", "field": "timestamp",
"interval": "1h"}], limit=600)
hourly = hourly_matrix(st_hour)
# IPs with the most 4xx are the ones a behaviour model should look at;
# they are usually outside the volume top-600.
st_4xx = pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query="status:4*", limit=800)
four_xx = rows_to_dict(st_4xx, "cnt")
base = {}
for ip, count in total.items():
if count <= 0:
continue
feat = empty_feat(ip)
feat["total"] = int(count)
feat["uri_card"] = int(uri_card.get(ip, 0))
feat["ua_card"] = int(ua_card.get(ip, 0))
feat["host_card"] = int(host_card.get(ip, 0))
feat["method_card"] = int(method_card.get(ip, 0))
base[ip] = finalize_feat(feat, status, get_count, login_count, hourly)
extra = sorted(set(four_xx) - set(total), key=lambda ip: -four_xx[ip])
extra = [ip for ip in extra if not ip.startswith(INTERNAL)][:600]
print("补充 %d 个高 4xx IP 的计数 ..." % len(extra), flush=True)
for offset in range(0, len(extra), 80):
chunk = extra[offset:offset + 80]
chunk_query = " OR ".join('client_ip:"%s"' % ip for ip in chunk)
st = pivot(client,
[{"type": "count", "id": "cnt"},
{"type": "card", "field": "uri", "id": "uri_card"},
{"type": "card", "field": "ua", "id": "ua_card"},
{"type": "card", "field": "host", "id": "host_card"},
{"type": "card", "field": "method", "id": "method_card"}],
["client_ip"], query=chunk_query, limit=200)
totals = rows_to_dict(st, "cnt")
uris = rows_to_dict(st, "uri_card")
uas = rows_to_dict(st, "ua_card")
hosts = rows_to_dict(st, "host_card")
methods = rows_to_dict(st, "method_card")
chunk_status = {}
for cls in ("2*", "3*", "4*", "5*"):
chunk_status[cls] = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query='status:%s AND (%s)' % (cls, chunk_query), limit=200),
"cnt")
chunk_get = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query='method:GET AND (%s)' % chunk_query, limit=200), "cnt")
chunk_login = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query='%s AND (%s)' % (LOGIN_QUERY, chunk_query), limit=200),
"cnt")
chunk_hour = hourly_matrix(pivot(
client, [{"type": "count", "id": "cnt"}], ["client_ip"],
column_groups=[{"type": "time", "field": "timestamp",
"interval": "1h"}], query=chunk_query, limit=200))
for ip in chunk:
if ip not in totals or totals[ip] <= 0:
continue
for cls in chunk_status:
status[cls].update(chunk_status[cls])
ctotal = totals[ip]
feat = empty_feat(ip)
feat["total"] = int(ctotal)
feat["uri_card"] = int(uris.get(ip, 0))
feat["ua_card"] = int(uas.get(ip, 0))
feat["host_card"] = int(hosts.get(ip, 0))
feat["method_card"] = int(methods.get(ip, 0))
base[ip] = finalize_feat(feat, status, chunk_get, chunk_login, chunk_hour)
get_count.update(chunk_get)
login_count.update(chunk_login)
print("候选 %d 个，预筛细节 ..." % len(base), flush=True)
shortlist = []
for ip, feat in base.items():
if ip.startswith(INTERNAL):
continue
if feat["total"] >= 10 and (feat["pct4"] >= 20 or feat["four_xx"] >= 20
or feat["uri_card"] >= 12):
shortlist.append(ip)
shortlist.sort(key=lambda ip: -base[ip]["four_xx"])
shortlist = shortlist[:260]
details = gather_details(client, shortlist)
geo = None
try:
from graylog_ops.geo import GeoIP
geo = GeoIP(os.path.join(ROOT, "data", "geo", "geo.db"))
except (FileNotFoundError, sqlite3.Error):
geo = None
flagged = []
for ip, feat in base.items():
if ip.startswith(INTERNAL):
continue
detail = details.get(ip) or {}
five = bucket_stats(detail.get("buckets5m") or [])
uris = detail.get("uri") or []
uas = detail.get("ua") or []
feat["coverage5m"] = five["coverage"]
feat["cv5m"] = five["cv"]
feat["peak_share5m"] = five["peak_share"]
feat["uri_templates"] = len({uri_template(uri) for uri, _ in uris})
feat["top_uri"] = uris[0][0] if uris else ""
feat["top_ua"] = uas[0][0] if uas else ""
feat["ua_family"] = ua_family(feat["top_ua"])
feat["ua_garbage"] = ua_garbage(feat["ua_card"], feat["top_ua"])
score, signals = behavior_score(feat)
if score < 60 or len(signals) < 2:
continue
crawler_tokens = ("sogou", "baiduspider", "googlebot", "bingbot",
"yandex", "bytespider", "ahrefs", "semrush", "mj12",
"dotbot", "petalbot")
is_crawler_ua = any(token in feat["top_ua"].lower()
for token in crawler_tokens)
if is_crawler_ua:
behavior_type = ("伪造爬虫 UA" if feat["pct4"] >= 50
else "采集爬虫")
elif feat["login_ratio"] >= 20:
behavior_type = "登录爆破"
elif feat["pct4"] >= 50 and feat["uri_card"] >= 20:
behavior_type = "目录扫描"
elif feat["coverage"] <= 30 and feat["peak_share"] >= 25:
behavior_type = "短时扫描"
elif feat["uri_templates"] >= 100 and feat["pct2"] >= 70:
behavior_type = "内容抓取"
else:
behavior_type = "异常自动化"
geo_info = geo.lookup(ip) if geo else None
feat.update({
"score": score,
"signals": signals,
"behavior_type": behavior_type,
"in_list": known.get(ip, False),
"country_code": (geo_info or {}).get("country_code", ""),
"country_name": (geo_info or {}).get("country_name", ""),
"region": (geo_info or {}).get("region", ""),
"city": (geo_info or {}).get("city", ""),
})
info = asn_info(ip) or {"asn": "", "org": "未知", "shared": "unknown"}
shared = info["shared"]
org = info["org"] or "未知"
multi_device = feat["ua_card"] >= 5 or feat["host_card"] >= 10
if shared in ("mobile", "education"):
verdict, vclass = "仅告警（共享出口）", "v-watch"
share_note = ("出口属于%s（%s），大概率有大量用户共享；"
"整体封禁会误伤，建议只告警或按 UA/路径单独拦截。"
% (org_cn(org), SHARE_LABEL[shared]))
elif shared == "broadband" and multi_device:
verdict, vclass = "仅告警（疑似多用户）", "v-watch"
share_note = ("出口属于%s（宽带 ISP），且该 IP 出现 %d 种 UA / %d 个站点，"
"疑似多用户共享，整体封禁可能误伤。"
% (org_cn(org), feat["ua_card"], feat["host_card"]))
elif shared == "broadband":
if feat["score"] >= 75:
verdict, vclass = "建议封禁", "v-ban"
else:
verdict, vclass = "建议人工确认", "v-review"
share_note = ("出口属于%s（宽带 ISP），但该 IP 只有 %d 种 UA，"
"更像单台设备/家庭网络，封禁影响面小。"
% (org_cn(org), feat["ua_card"]))
elif shared == "hosting":
verdict, vclass = "建议封禁", "v-ban"
share_note = "出口属于%s（云主机/机房），不是家庭或移动网络，封禁影响面小。" % org_cn(org)
else:
verdict, vclass = "建议人工确认", "v-review"
share_note = "出口属于%s，归属不明确，建议人工核实后再决定。" % org_cn(org)
feat.update({
"verdict": verdict, "vclass": vclass,
"reason_text": humanize(feat),
"share_note": share_note,
"asn": info["asn"], "asn_org": org, "shared_type": shared,
})
flagged.append(feat)
flagged.sort(key=lambda item: (-item["score"], -item["total"]))
metrics = {
"flagged": len(flagged),
"new": sum(1 for item in flagged if not item["in_list"]),
"known": sum(1 for item in flagged if item["in_list"]),
"high": sum(1 for item in flagged if item["score"] >= 80),
"ban": sum(1 for item in flagged if item["verdict"] == "建议封禁"),
"watch": sum(1 for item in flagged if item["verdict"].startswith("仅告警")),
"review": sum(1 for item in flagged if item["verdict"] == "建议人工确认"),
"candidates": len(base),
}
payload = {
"generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"window_seconds": WINDOW,
"items": flagged,
"metrics": metrics,
}
os.makedirs(os.path.dirname(DATA_FILE), exist_ok=True)
with open(DATA_FILE, "w", encoding="utf-8") as fh:
json.dump(payload, fh, ensure_ascii=False)
if geo is not None:
geo.close()
return payload
def serve(port):
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def _send(self, code, body, ctype="application/json; charset=utf-8"):
raw = body.encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def do_GET(self):
if self.path.startswith("/data.json"):
try:
with open(DATA_FILE, encoding="utf-8") as fh:
return self._send(200, fh.read())
except FileNotFoundError:
return self._send(404, '{"error":"no data"}')
return self._send(200, PAGE, "text/html; charset=utf-8")
server = ThreadingHTTPServer(("0.0.0.0", port), Handler)
print("行为分析原型: http://0.0.0.0:%d/ (Ctrl+C 退出)" % port, flush=True)
server.serve_forever()
PAGE = r"""<!doctype html>
<html lang="zh"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>行为检测 · 独立判定</title>
<style>
:root{--bg:#F6F4EE;--surface:#FFF;--surface-2:#FBFAF6;--ink:#242119;--ink-2:#4B463B;
--muted:#8B8577;--line:#E8E3D9;--accent:#C15F3C;--accent-soft:#F4E6DE;--green:#2E6B4F;
--green-soft:#E3EEE7;--amber:#8F6418;--amber-soft:#F6EBD2;--red:#A63A28;--red-soft:#F6E1DB;
--blue:#34557F;--blue-soft:#E6ECF5;--radius:14px;--shadow:0 1px 2px rgba(36,33,25,.05),0 10px 30px rgba(36,33,25,.045)}
*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--ink);
font:14px/1.65 -apple-system,BlinkMacSystemFont,"PingFang SC",sans-serif}
header{padding:18px 28px;border-bottom:1px solid var(--line);background:rgba(246,244,238,.9);
backdrop-filter:blur(10px);position:sticky;top:0;z-index:20;display:flex;justify-content:space-between;align-items:center}
h1{font-family:Georgia,"Songti SC",serif;font-size:19px;margin:0;font-weight:600}
.sub{color:var(--muted);font-size:12px;margin-top:2px}
main{padding:24px 28px 80px;max-width:1200px;margin:0 auto}
.summary{font-size:15px;line-height:1.9;margin-bottom:16px}
.summary b{color:var(--accent-deep)}
.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(140px,1fr));gap:12px;margin-bottom:18px}
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:14px 16px;box-shadow:var(--shadow)}
.card .n{font-family:Georgia,serif;font-size:25px;font-weight:600}
.card .k{color:var(--muted);font-size:12px;margin-top:2px}
.toolbar{display:flex;gap:8px;flex-wrap:wrap;margin-bottom:14px;align-items:center}
input[type=search]{flex:1;min-width:240px;padding:10px 14px;border:1px solid var(--line);
border-radius:10px;background:var(--surface);font-size:13px;outline:none}
.chips{display:flex;gap:2px;background:#EFEAE0;border:1px solid var(--line);padding:3px;border-radius:10px;flex-wrap:wrap}
.chip{border:0;background:transparent;padding:6px 11px;border-radius:7px;cursor:pointer;font-size:12px;color:var(--ink-2)}
.chip.active{background:var(--surface);color:var(--accent);font-weight:600;box-shadow:0 1px 2px rgba(0,0,0,.08)}
.item{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);
padding:16px 20px;margin-bottom:10px;box-shadow:var(--shadow)}
.line1{display:flex;align-items:center;gap:10px;flex-wrap:wrap}
.ip{font-family:ui-monospace,Menlo,monospace;font-size:15px;font-weight:600}
.geo{color:var(--ink-2);font-size:13px}
.badge{display:inline-block;padding:2px 10px;border-radius:20px;font-size:12px;white-space:nowrap;font-weight:600}
.v-ban{background:var(--red-soft);color:var(--red)}
.v-watch{background:var(--blue-soft);color:var(--blue)}
.v-review{background:var(--amber-soft);color:var(--amber)}
.b-new{background:var(--red-soft);color:var(--red);font-weight:500}
.b-known{background:#EFEBE2;color:var(--muted);font-weight:500}
.score{margin-left:auto;color:var(--muted);font-size:12px}
.reason{font-size:14px;color:var(--ink);margin:9px 0 4px;line-height:1.75}
.note{font-size:13px;color:var(--muted);margin-bottom:2px}
details{margin-top:8px}
summary{cursor:pointer;color:var(--accent-deep);font-size:12.5px}
.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));
gap:6px 18px;margin-top:8px;font-size:12.5px;color:var(--ink-2)}
.grid b{color:var(--muted);font-weight:500;margin-right:6px}
.grid .wide{grid-column:1/-1;word-break:break-all}
.empty{padding:48px;text-align:center;color:var(--muted)}
.algo{margin-top:22px;background:var(--surface);border:1px solid var(--line);
border-radius:var(--radius);padding:14px 18px;box-shadow:var(--shadow)}
.algo summary{cursor:pointer;color:var(--accent-deep);font-weight:600;font-size:13px}
.algo ul{margin:10px 0 0;padding-left:18px}
.algo li{font-size:12.5px;color:var(--ink-2);margin:5px 0;line-height:1.6}
@media (max-width:900px){.cards{grid-template-columns:repeat(2,1fr)}}
</style></head><body>
<header><div><h1>行为检测 · 独立判定</h1>
<div class="sub" id="sub">不用规则、不看名单，纯行为特征算出的恶意 IP · 只读</div></div>
<div class="muted" id="meta">加载中…</div></header>
<main>
<div class="summary" id="summary"></div>
<div class="cards" id="cards"></div>
<div class="toolbar">
<input type="search" id="q" placeholder="搜索 IP / 国家 / 原因">
<div class="chips" id="chips"></div>
</div>
<div id="list"></div>
<details class="algo">
<summary>算法说明（纯行为评分，不依赖规则和名单）</summary>
<ul>
<li><b>范围</b>：24h 内全部公网 IP（约 1000+ 个候选），先按失败率/路径分散预筛，再抽样细节。</li>
<li><b>评分信号</b>：高失败率、路径分散、短时爆发、登录接口集中、多站点、自动化 UA、
路径模板分散、失败请求量大；业务接口（非 GET 为主）和长期稳定单接口会降权。</li>
<li><b>判定</b>：行为分 ≥60 且至少 2 个信号 → 行为判定恶意。</li>
<li><b>建议动作</b>：出口是移动/宽带/教育网（共享网络）→ 仅告警，避免误伤；
出口是云主机/机房 → 建议封禁；其余 → 人工确认。</li>
<li><b>名单</b>列只用于事后对比：红色「新发现」= 不在现有规则名单。</li>
</ul>
</details>
</main>
<script>
const state = {flt:'all', q:'', items:[], metrics:null};
const $ = s => document.querySelector(s);
const esc = v => String(v ?? '').replace(/[&<>"']/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
const flagOf = code => (!code || code.length !== 2 || code === 'ZZ') ? ''
: String.fromCodePoint(...[...code.toUpperCase()].map(ch => 0x1F1E6 + ch.charCodeAt(0) - 65));
const fmt = n => n >= 10000 ? (n/10000).toFixed(1)+'万' : (n||0).toLocaleString();
function filtered(){
const q = state.q.trim().toLowerCase();
return state.items.filter(it => {
if (state.flt === 'ban' && it.verdict !== '建议封禁') return false;
if (state.flt === 'watch' && !it.verdict.startsWith('仅告警')) return false;
if (state.flt === 'review' && it.verdict !== '建议人工确认') return false;
if (state.flt === 'new' && it.in_list) return false;
if (!q) return true;
const hay = [it.ip, it.country_name, it.region, it.reason_text,
it.share_note, it.asn_org, ...(it.signals||[])].join(' ').toLowerCase();
return hay.includes(q);
});
}
function render(){
const rows = filtered();
$('#list').innerHTML = rows.map(it => `<div class="item">
<div class="line1">
<span class="ip">${esc(it.ip)}</span>
<span class="geo">${flagOf(it.country_code)} ${esc([it.country_name, it.region].filter(Boolean).join(' · ') || '未知地区')}</span>
<span class="badge ${it.vclass}">${esc(it.verdict)}</span>
${it.in_list ? '<span class="badge b-known">名单内</span>' : '<span class="badge b-new">新发现</span>'}
<span class="score">行为分 ${it.score}</span>
</div>
<div class="reason">${esc(it.reason_text)}</div>
<div class="note">${esc(it.share_note)}</div>
<details><summary>查看技术细节</summary>
<div class="grid">
<div><b>请求量</b>${fmt(it.total)} 次</div>
<div><b>失败率</b>${it.pct4.toFixed(0)}%</div>
<div><b>不同路径</b>${it.uri_card}</div>
<div><b>路径模板</b>${it.uri_templates}</div>
<div><b>活跃时段</b>${it.coverage.toFixed(0)}%</div>
<div><b>峰值占比</b>${it.peak_share.toFixed(0)}%</div>
<div><b>站点数</b>${it.host_card}</div>
<div><b>UA 数</b>${it.ua_card}（${esc(it.ua_family || '-')})</div>
<div><b>ASN</b>${esc(it.asn_org)}</div>
<div class="wide"><b>Top URI</b>${esc(it.top_uri || '-')}</div>
<div class="wide"><b>Top UA</b>${esc(it.top_ua || '-')}</div>
<div class="wide"><b>信号</b>${(it.signals||[]).map(esc).join('；')}</div>
</div>
</details>
</div>`).join('') || '<div class="empty">没有匹配的记录</div>';
$('#meta').textContent = `共 ${state.items.length} 个（显示 ${rows.length}）`;
}
function cards(){
const m = state.metrics || {};
const items = [
['判出恶意', m.flagged || 0], ['建议封禁', m.ban || 0],
['仅告警（共享出口）', m.watch || 0], ['建议人工确认', m.review || 0],
['新发现', m.new || 0],
];
$('#cards').innerHTML = items.map(([label, value]) =>
`<div class="card"><div class="n">${value}</div><div class="k">${label}</div></div>`).join('');
$('#summary').innerHTML = `行为分析在 <b>${m.candidates||0}</b> 个候选 IP 中判出 <b>${m.flagged||0}</b> 个恶意，`
+ `其中 <b>${m.new||0}</b> 个不在现有规则名单里；建议封禁 <b>${m.ban||0}</b> 个，`
+ `共享出口仅告警 <b>${m.watch||0}</b> 个，人工确认 <b>${m.review||0}</b> 个。`;
const chips = [['all','全部'],['ban','建议封禁'],['watch','仅告警'],['review','人工确认'],['new','新发现']];
$('#chips').innerHTML = chips.map(([key,label]) =>
`<button class="chip ${state.flt===key?'active':''}" data-flt="${key}">${label}</button>`).join('');
$('#chips').querySelectorAll('.chip').forEach(el => el.addEventListener('click', () => {
state.flt = el.dataset.flt; cards(); render();
}));
}
fetch('/data.json').then(r=>r.json()).then(data => {
state.items = data.items || []; state.metrics = data.metrics || {};
$('#sub').textContent = `不用规则、不看名单，纯行为特征算出的恶意 IP · 生成于 ${data.generated_at} · 只读`;
cards(); render();
}).catch(err => { $('#meta').textContent = '加载失败: ' + err; });
$('#q').addEventListener('input', e => { state.q = e.target.value; render(); });
</script></body></html>
"""
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--analyze", action="store_true")
parser.add_argument("--serve", action="store_true")
parser.add_argument("--port", type=int, default=8090)
args = parser.parse_args()
if args.analyze or not os.path.exists(DATA_FILE):
payload = analyze()
print("分析完成: %d 个 IP" % len(payload["items"]), flush=True)
if args.serve:
serve(args.port)
asn_brand_map.json
+43
View File
@@ -0,0 +1,43 @@
{
"items": [
{"keywords": ["alibaba", "aliyun"], "brand": "阿里云", "type": "cloud", "icon": "☁️"},
{"keywords": ["tencent"], "brand": "腾讯云", "type": "cloud", "icon": "☁️"},
{"keywords": ["huawei cloud", "huaweicloud"], "brand": "华为云", "type": "cloud", "icon": "☁️"},
{"keywords": ["baidu"], "brand": "百度云", "type": "cloud", "icon": "☁️"},
{"keywords": ["kingsoft", "ksyun"], "brand": "金山云", "type": "cloud", "icon": "☁️"},
{"keywords": ["jd cloud", "jingdong"], "brand": "京东云", "type": "cloud", "icon": "☁️"},
{"keywords": ["ucloud"], "brand": "UCloud", "type": "cloud", "icon": "☁️"},
{"keywords": ["ctyun", "china telecom cloud"], "brand": "天翼云", "type": "cloud", "icon": "☁️"},
{"keywords": ["china mobile cloud", "cmcloud"], "brand": "移动云", "type": "cloud", "icon": "☁️"},
{"keywords": ["cucloud", "china unicom cloud"], "brand": "联通云", "type": "cloud", "icon": "☁️"},
{"keywords": ["amazon", "aws"], "brand": "AWS", "type": "cloud", "icon": "☁️"},
{"keywords": ["microsoft"], "brand": "Azure", "type": "cloud", "icon": "☁️"},
{"keywords": ["google"], "brand": "Google Cloud", "type": "cloud", "icon": "☁️"},
{"keywords": ["oracle"], "brand": "Oracle Cloud", "type": "cloud", "icon": "☁️"},
{"keywords": ["ibm"], "brand": "IBM Cloud", "type": "cloud", "icon": "☁️"},
{"keywords": ["digitalocean", "digital ocean"], "brand": "DigitalOcean", "type": "cloud", "icon": "☁️"},
{"keywords": ["vultr", "constant company"], "brand": "Vultr", "type": "cloud", "icon": "☁️"},
{"keywords": ["linode", "akamai"], "brand": "Akamai/Linode", "type": "cloud", "icon": "☁️"},
{"keywords": ["hetzner"], "brand": "Hetzner", "type": "cloud", "icon": "☁️"},
{"keywords": ["ovh"], "brand": "OVH", "type": "cloud", "icon": "☁️"},
{"keywords": ["contabo"], "brand": "Contabo", "type": "cloud", "icon": "☁️"},
{"keywords": ["scaleway"], "brand": "Scaleway", "type": "cloud", "icon": "☁️"},
{"keywords": ["rackspace"], "brand": "Rackspace", "type": "cloud", "icon": "☁️"},
{"keywords": ["upcloud"], "brand": "UpCloud", "type": "cloud", "icon": "☁️"},
{"keywords": ["equinix"], "brand": "Equinix", "type": "idc", "icon": "🏢"},
{"keywords": ["digital realty"], "brand": "Digital Realty", "type": "idc", "icon": "🏢"},
{"keywords": ["21vianet", "vianet", "世纪互联"], "brand": "世纪互联", "type": "idc", "icon": "🏢"},
{"keywords": ["chinanet idc", "idc network"], "brand": "IDC 机房", "type": "idc", "icon": "🏢"},
{"keywords": ["wowrack", "colocation", "colo"], "brand": "IDC 机房", "type": "idc", "icon": "🏢"},
{"keywords": ["data center", "datacenter", "hosting"], "brand": "IDC 机房", "type": "idc", "icon": "🏢"},
{"keywords": ["china mobile", "cmnet"], "brand": "中国移动", "type": "mobile", "icon": "📱"},
{"keywords": ["china unicom", "china169"], "brand": "中国联通", "type": "broadband", "icon": "🏠"},
{"keywords": ["chinanet", "china telecom", "chinatelecom"], "brand": "中国电信", "type": "broadband", "icon": "🏠"},
{"keywords": ["china broadcasting", "cbnet", "广电"], "brand": "中国广电", "type": "broadband", "icon": "🏠"},
{"keywords": ["great wall broadband", "gwbn"], "brand": "长城宽带", "type": "broadband", "icon": "🏠"},
{"keywords": ["dr.peng", "pengbo", "鹏博士"], "brand": "鹏博士", "type": "broadband", "icon": "🏠"},
{"keywords": ["cernet", "education network", "education and research", "university", "college", "教育"], "brand": "教育网", "type": "education", "icon": "🎓"},
{"keywords": ["government", "gov.cn"], "brand": "政府机构", "type": "enterprise", "icon": "🏛️"}
],
"defaults": {"brand": "未知", "type": "unknown", "icon": "❓"}
}
bin/graylog-ops
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Wrapper so the tool can be called as `bin/graylog-ops ...` from anywhere.
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
export PYTHONPATH="${here}${PYTHONPATH:+:${PYTHONPATH}}"
exec python3 -m graylog_ops "$@"
config.example.json
+7
View File
@@ -0,0 +1,7 @@
{
"url": "http://edu-master:9002",
"token": "REPLACE_WITH_API_TOKEN",
"insecure": false,
"timeout": 30,
"retries": 2
}
docker-build/Dockerfile
+30
View File
@@ -0,0 +1,30 @@
FROM python:3.12-slim
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
DATA_DIR=/data
WORKDIR /app
# application code (no credentials, no config files)
COPY app/graylog_ops /app/graylog_ops
COPY app/bin /app/bin
COPY app/profiles /app/profiles
COPY app/allowlist.example.txt /app/allowlist.example.txt
COPY app/asn_brand_map.json /app/asn_brand_map.json
# seed data: current production snapshot (sqlite + geo + asn + blocklist)
COPY seed /app/seed
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
VOLUME ["/data"]
EXPOSE 8088
HEALTHCHECK --interval=60s --timeout=5s --start-period=30s \
CMD python3 -c "import os,urllib.request,sys; \
url='http://127.0.0.1:%s/healthz' % os.environ.get('APP_PORT','8088'); \
sys.exit(0 if urllib.request.urlopen(url, timeout=4).status == 200 else 1)"
ENTRYPOINT ["/app/entrypoint.sh"]
docker-build/entrypoint.sh
Executable
+30
View File
@@ -0,0 +1,30 @@
#!/bin/sh
set -eu
DATA_DIR="${DATA_DIR:-/data}"
if [ ! -f "$DATA_DIR/ips.db" ]; then
echo "[init] 初始化数据目录: $DATA_DIR"
mkdir -p "$DATA_DIR"
cp -a /app/seed/. "$DATA_DIR"/
fi
set -- watch \
--stream "${GRAYLOG_STREAM:-F5_Accesslog}" \
--profile "${PROFILE:-/app/profiles/f5.json}" \
--interval "${CHECK_INTERVAL:-60}" \
--window "${WINDOW:-3600}" \
--forget "${FORGET:-604800}" \
--db "$DATA_DIR/ips.db" \
--host 0.0.0.0 \
--port "${APP_PORT:-8088}" \
--path-prefix "${APP_PATH_PREFIX:-}" \
--geo-db "$DATA_DIR/geo/geo.db" \
--asn-db "$DATA_DIR/geo/asn.db" \
--allow /app/allowlist.example.txt \
--export "$DATA_DIR/blocklist.txt" \
--behavior-interval "${BEHAVIOR_INTERVAL:-900}" \
--warm-limit "${WARM_LIMIT:-5}"
if [ "${NO_BEHAVIOR:-0}" = "1" ]; then
set -- "$@" --no-behavior
fi
exec python3 -m graylog_ops "$@"
docs/screenshots/behavior-prototype.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 331 KiB

docs/screenshots/detail-asn.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 149 KiB

docs/screenshots/detail.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 209 KiB

docs/screenshots/list-fused.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 179 KiB

docs/screenshots/list-geo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 215 KiB

docs/screenshots/list.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 196 KiB

graylog_ops/__init__.py
+8
View File
@@ -0,0 +1,8 @@
"""Read-only Graylog inspection toolkit.
Target API: Graylog 4.0.x (legacy ``/api/search/universal`` surface).
No third-party dependencies: stdlib only, so it also runs on old CentOS 7
python3 installations.
"""
__version__ = "0.2.0"
graylog_ops/__main__.py
+6
View File
@@ -0,0 +1,6 @@
import sys
from .cli import main
if __name__ == "__main__":
sys.exit(main())
graylog_ops/asn.py
+107
View File
@@ -0,0 +1,107 @@
"""OFFLINE ASN lookup + provider branding (DB-IP ASN Lite)."""
import bisect
import csv
import ipaddress
import json
import os
import sqlite3
DEFAULT_MAP = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"asn_brand_map.json")
def load_brand_map(path=None):
try:
with open(path or DEFAULT_MAP, encoding="utf-8") as handle:
data = json.load(handle)
except (OSError, ValueError):
data = {"items": [], "defaults": {"brand": "未知", "type": "unknown",
"icon": "❓"}}
return data
def classify_org(org, brand_map=None):
brand_map = brand_map or load_brand_map()
text = (org or "").lower()
for item in brand_map.get("items", []):
if any(keyword in text for keyword in item.get("keywords", [])):
return item["brand"], item["type"], item["icon"]
fallback = brand_map.get("defaults") or {}
return (fallback.get("brand", "未知"), fallback.get("type", "unknown"),
fallback.get("icon", "❓"))
def build_asn_db(csv_path, out_path):
os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
conn = sqlite3.connect(out_path)
conn.execute("DROP TABLE IF EXISTS asn")
conn.execute("CREATE TABLE asn (start INTEGER PRIMARY KEY, end INTEGER, "
"asn TEXT, org TEXT)")
batch, total = [], 0
def flush():
nonlocal batch
if batch:
conn.executemany("INSERT OR REPLACE INTO asn VALUES (?,?,?,?)", batch)
batch = []
with open(csv_path, encoding="utf-8", errors="replace") as handle:
for row in csv.reader(handle):
if len(row) < 4 or ":" in row[0]:
continue
try:
first = int(ipaddress.IPv4Address(row[0]))
last = int(ipaddress.IPv4Address(row[1]))
except ValueError:
continue
batch.append((first, last, row[2], row[3]))
total += 1
if len(batch) >= 100000:
flush()
conn.commit()
flush()
conn.commit()
conn.execute("PRAGMA journal_mode=WAL")
conn.close()
return total
class AsnDB:
def __init__(self, path, brand_map_path=None):
self.conn = sqlite3.connect(path, check_same_thread=False)
self.brand_map = load_brand_map(brand_map_path)
self._prefix = None
with self.conn:
try:
self.conn.execute("PRAGMA query_only=ON")
except sqlite3.Error:
pass
def lookup(self, ip):
try:
value = int(ipaddress.IPv4Address(ip))
except ValueError:
return None
row = self.conn.execute(
"SELECT end, asn, org FROM asn WHERE start <= ? "
"ORDER BY start DESC LIMIT 1", (value,)).fetchone()
if not row or row[0] < value:
return None
brand, kind, icon = classify_org(row[2], self.brand_map)
return {"asn": row[1], "org": row[2], "brand": brand,
"type": kind, "icon": icon}
def close(self):
try:
self.conn.close()
except sqlite3.Error:
pass
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--csv", default="data/geo/dbip-asn-lite-2026-10.csv")
parser.add_argument("--out", default="data/geo/asn.db")
args = parser.parse_args()
print("imported %d ranges -> %s" % (build_asn_db(args.csv, args.out), args.out))
graylog_ops/behavior.py
+695
View File
@@ -0,0 +1,695 @@
"""Behaviour detection running inside the main service.
It scans 24h of F5 traffic with Graylog pivot aggregations, scores each IP by
behaviour only (no rule hits, no known list), and stores the result in SQLite.
A background thread keeps it refreshed; the web UI reads the stored result.
"""
import csv
import ipaddress
import json
import os
import statistics
import threading
import time
from .http import GraylogError
from .asn import AsnDB
WINDOW = 86400
INTERNAL = ("10.", "192.168.", "172.16.", "172.17.", "172.18.", "172.19.",
"172.2", "172.3")
UA_SDK = ("curl", "wget", "python-requests", "python-urllib", "go-http-client",
"okhttp", "java/", "apache-httpclient", "httpcomponents",
"libredtail", "axios", "node-fetch", "aiohttp", "scrapy",
"postmanruntime", "insomnia", "unityplayer", "dart/", "httpie",
"guzzle", "faraday", "restsharp", "libwww", "eeo/", "academy_widget")
UA_BROWSER = ("chrome", "firefox", "safari", "edg/", "opr/", "gecko/20100101")
CRAWLER_TOKENS = ("sogou", "baiduspider", "googlebot", "bingbot", "yandex",
"bytespider", "ahrefs", "semrush", "mj12", "dotbot",
"petalbot")
RULE_TITLES = {
"probe_java": "Spring/中间件探测", "probe_config": "配置/凭证探测",
"probe_cms": "CMS 探测", "probe_iot": "IoT/路由器探测",
"probe_devops": "DevOps/监控探测", "probe_webshell": "WebShell/上传探测",
"exploit_payload": "漏洞利用 payload", "login_attack": "登录失败尝试",
"login_burst": "登录接口高频", "bad_ua": "扫描器 UA",
"bot_ua": "僵尸网络 UA", "not_found": "404 探测",
}
ASN_CACHE = {}
def _asn_csv(root):
geo_dir = os.path.join(root, "data", "geo")
try:
names = sorted(name for name in os.listdir(geo_dir)
if name.startswith("dbip-asn-lite-") and name.endswith(".csv"))
except FileNotFoundError:
return None
return os.path.join(geo_dir, names[-1]) if names else None
def _load_asn(path):
if ASN_CACHE:
return ASN_CACHE
starts, ranges = [], []
try:
with open(path, encoding="utf-8", errors="replace") as fh:
for row in csv.reader(fh):
if len(row) < 4 or ":" in row[0]:
continue
try:
first = int(ipaddress.IPv4Address(row[0]))
last = int(ipaddress.IPv4Address(row[1]))
except ValueError:
continue
starts.append(first)
ranges.append((last, row[2], row[3]))
except (FileNotFoundError, TypeError):
pass
order = sorted(range(len(starts)), key=lambda i: starts[i])
ASN_CACHE.update({"starts": [starts[i] for i in order],
"ranges": [ranges[i] for i in order]})
return ASN_CACHE
def shared_type(org):
text = (org or "").lower()
cloud = ("alibaba", "aliyun", "tencent", "huawei cloud", "aws", "amazon",
"google", "microsoft", "digitalocean", "vultr", "linode", "hetzner",
"ovh", "wowrack", "idc", "data center", "datacenter", "cloud",
"hosting", "colo")
if any(token in text for token in cloud):
return "hosting"
if "china mobile" in text or "中国移动" in text or "cmnet" in text:
return "mobile"
if ("china unicom" in text or "china telecom" in text or "chinanet" in text
or "联通" in text or "电信" in text or "broadband" in text):
return "broadband"
if "education" in text or "cernet" in text or "教育" in text:
return "education"
return "unknown"
ORG_CN = (("china mobile", "中国移动"), ("china unicom", "中国联通"),
("china telecom", "中国电信"), ("chinanet", "中国电信"),
("cernet", "中国教育网"), ("education", "教育网"),
("alibaba", "阿里云"), ("aliyun", "阿里云"), ("huawei cloud", "华为云"),
("tencent", "腾讯云"), ("wowrack", "Wowrack 机房"),
("vmiss", "VMISS 机房"), ("idc", "IDC 机房"))
TYPE_TO_SHARED = {"mobile": "mobile", "broadband": "broadband",
"education": "education", "cloud": "hosting", "idc": "hosting"}
SHARE_LABEL = {"mobile": "移动网络", "broadband": "宽带 ISP",
"education": "教育网", "hosting": "云主机/机房", "unknown": "未知网络"}
def org_cn(org):
text = (org or "").lower()
for token, name in ORG_CN:
if token in text:
return name
return org or "未知"
def asn_info(ip, root):
cache = _load_asn(_asn_csv(root))
starts = cache.get("starts") or []
if not starts:
return {"asn": "", "org": "未知", "shared": "unknown"}
try:
value = int(ipaddress.IPv4Address(ip))
except ValueError:
return {"asn": "", "org": "未知", "shared": "unknown"}
import bisect
index = bisect.bisect_right(starts, value) - 1
if index >= 0:
last, asn, org = cache["ranges"][index]
if last >= value:
return {"asn": asn, "org": org, "shared": shared_type(org)}
return {"asn": "", "org": "未知", "shared": "unknown"}
def pivot(client, series, row_groups, column_groups=None, query="*",
limit=300, timeout=180000):
body = {"queries": [{
"id": "q1",
"timerange": {"type": "relative", "range": WINDOW},
"query": {"type": "elasticsearch", "query_string": query},
"filter": {"type": "or", "filters": [{"type": "stream",
"id": "6ac83059a6878e26256fcacc"}]},
"search_types": [{
"type": "pivot", "id": "st1", "series": series,
"row_groups": [{"type": "values", "field": f, "limit": limit}
for f in row_groups],
"column_groups": column_groups or [], "sort": [], "rollup": True,
}],
}], "parameters": []}
data = client.http.request("POST", "/api/views/search/sync",
params={"timeout": timeout}, body=body)
return data["results"]["q1"]["search_types"]["st1"]
def rows_to_dict(search_type, series_id):
out = {}
for row in search_type.get("rows") or []:
key = row.get("key") or []
if not key:
continue
ip = str(key[0])
for value in row.get("values") or []:
if (value.get("key") or [None])[0] == series_id:
try:
out[ip] = float(value.get("value") or 0)
except (TypeError, ValueError):
out[ip] = 0.0
return out
def hourly_matrix(search_type):
out = {}
for row in search_type.get("rows") or []:
key = row.get("key") or []
if not key:
continue
counts = []
for value in row.get("values") or []:
stamp = (value.get("key") or [None])[0]
if not stamp or "cnt" not in (value.get("key") or []):
continue
try:
counts.append(float(value.get("value") or 0))
except (TypeError, ValueError):
pass
out[str(key[0])] = counts
return out
def bucket_stats(counts):
active = [c for c in counts if c > 0]
if not counts:
return {"coverage": 0.0, "cv": None, "peak_share": 0.0}
cv = (statistics.pstdev(active) / statistics.fmean(active)
if len(active) > 2 and statistics.fmean(active) else None)
total = sum(counts)
return {"coverage": len(active) / len(counts) * 100,
"cv": cv,
"peak_share": (max(counts) / total * 100) if total else 0.0}
def uri_template(uri):
import re
path = str(uri or "").split("?")[0]
path = re.sub(
r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-"
r"[0-9a-fA-F]{12}", "{uuid}", path)
path = re.sub(r"[0-9a-fA-F]{16,}", "{hex}", path)
path = re.sub(r"\d+", "{n}", path)
return path
def ua_family(ua):
text = str(ua or "").strip()
if not text:
return "empty"
lower = text.lower()
if any(token in lower for token in UA_SDK):
return "sdk"
if lower.startswith("mozilla/") and any(t in lower for t in UA_BROWSER):
return "browser"
if lower.startswith("mozilla/5.0"):
return "fake-browser"
return "other"
def ua_garbage(ua_card, top_ua):
if ua_card > 50:
return True
text = str(top_ua or "")
if not text:
return False
printable = sum(1 for char in text if 32 <= ord(char) < 127 or ord(char) > 127)
return printable / max(1, len(text)) < 0.85
def behavior_score(feat):
total = feat["total"]
if total < 10:
return 0, ["请求过少"]
score = 0
signals = []
pct4 = feat["pct4"]
if pct4 >= 50:
score += 30
signals.append("高失败率 %.0f%%" % pct4)
elif pct4 >= 30:
score += 15
signals.append("失败率 %.0f%%" % pct4)
if feat["uri_card"] >= 20:
score += 20
signals.append("路径分散（%d 个 URI）" % feat["uri_card"])
elif feat["uri_card"] >= 8:
score += 8
if feat["coverage"] <= 30 and feat["peak_share"] >= 25:
score += 20
signals.append("短时爆发（覆盖 %.0f%%、峰值 %.0f%%）"
% (feat["coverage"], feat["peak_share"]))
if feat["login_ratio"] >= 20:
score += 15
signals.append("登录接口集中 %.0f%%" % feat["login_ratio"])
if feat["host_card"] >= 2:
score += 5
signals.append("多站点（%d 个 Host）" % feat["host_card"])
if feat["ua_card"] == 1:
score += 5
if feat["ua_family"] in ("sdk", "fake-browser", "other"):
score += 10
signals.append("自动化 UA（%s）" % feat["ua_family"])
elif feat["ua_family"] == "browser":
score -= 5
if feat.get("four_xx", 0) >= 100 and pct4 >= 20:
score += 10
signals.append("失败请求 %d 次" % int(feat["four_xx"]))
if feat["get_ratio"] < 30:
score -= 25
signals.append("非 GET 为主（业务接口）")
if pct4 < 15 and feat["pct2"] >= 80:
score -= 20
signals.append("低失败率（采集/浏览）")
if (feat["coverage"] >= 80 and feat["cv"] is not None
and feat["cv"] <= 1.2 and feat["uri_templates"] <= 8):
score -= 20
signals.append("长期稳定单接口")
if feat["uri_templates"] >= 15:
score += 10
signals.append("路径模板分散（%d 个）" % feat["uri_templates"])
if feat["status_known"] < 10:
score -= 10
signals.append("状态字段缺失")
return max(0, min(100, score)), signals
def humanize(item):
parts = []
if item["uri_card"] >= 20:
parts.append("访问了 %s 个不同路径" % format(item["uri_card"], ","))
if item["four_xx"] >= 20:
parts.append("产生 %s 次失败请求（失败率 %.0f%%）"
% (format(int(item["four_xx"]), ","), item["pct4"]))
elif item["pct4"] >= 20:
parts.append("失败率 %.0f%%" % item["pct4"])
if item["coverage"] <= 30 and item["peak_share"] >= 25:
parts.append("流量集中在很短的时间内")
if item["login_ratio"] >= 20:
parts.append("大量请求集中在登录接口")
if item["host_card"] >= 2:
parts.append("同时访问了 %d 个站点" % item["host_card"])
if item["ua_family"] in ("sdk", "fake-browser", "other"):
parts.append("使用脚本/非浏览器客户端")
behavior = item.get("behavior_type") or "异常自动化"
tail = {"目录扫描": "符合目录扫描特征", "短时扫描": "符合短时扫描特征",
"登录爆破": "符合登录爆破特征",
"伪造爬虫 UA": "伪装成搜索引擎爬虫，且请求大量失败",
"采集爬虫": "是采集爬虫", "异常自动化": "行为高度自动化，不像正常用户"}.get(
behavior, "行为异常")
if parts:
return "24 小时内" + "，".join(parts) + "——" + tail + "。"
return "24 小时内请求 %d 次，%s。" % (item["total"], tail)
def behavior_type(feat, top_ua):
tokens = CRAWLER_TOKENS
if any(token in (top_ua or "").lower() for token in tokens):
return "伪造爬虫 UA" if feat["pct4"] >= 50 else "采集爬虫"
if feat["login_ratio"] >= 20:
return "登录爆破"
if feat["pct4"] >= 50 and feat["uri_card"] >= 20:
return "目录扫描"
if feat["coverage"] <= 30 and feat["peak_share"] >= 25:
return "短时扫描"
if feat["uri_templates"] >= 100 and feat["pct2"] >= 70:
return "内容抓取"
return "异常自动化"
def empty_feat(ip):
return {"ip": ip, "total": 0, "uri_card": 0, "ua_card": 0, "host_card": 0,
"method_card": 0, "get_ratio": 0.0, "login_ratio": 0.0,
"pct2": 0.0, "pct3": 0.0, "pct4": 0.0, "pct5": 0.0,
"status_known": 0.0, "coverage": 0.0, "cv": None, "peak_share": 0.0,
"uri_templates": 0, "ua_family": "", "ua_garbage": False,
"top_uri": "", "top_ua": "", "four_xx": 0}
class BehaviorAnalyzer:
def __init__(self, client, store, fields, filter_str=None, root=None,
interval=900, window=WINDOW, notifier=None):
self.client = client
self.store = store
self.fields = fields
self.filter_str = filter_str
self.root = root or os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
self.interval = max(120, int(interval or 900))
self.window = int(window or WINDOW)
self.notifier = notifier
self.last_run = None
self.last_error = None
self.runs = 0
self.asn = None
try:
asn_path = os.path.join(self.root, "data", "geo", "asn.db")
if os.path.exists(asn_path):
self.asn = AsnDB(asn_path)
except Exception:
self.asn = None
self.geo = None
try:
from .geo import GeoIP
geo_path = os.path.join(self.root, "data", "geo", "geo.db")
if os.path.exists(geo_path):
self.geo = GeoIP(geo_path)
except Exception:
self.geo = None
# ---------- data collection ----------
def _details(self, ips):
def run(ip):
q = 'client_ip:"%s"' % ip
detail = {"uri": [], "ua": [], "buckets5m": []}
try:
st = pivot(self.client, [{"type": "count", "id": "cnt"}],
["uri"], query=q, limit=40)
for row in st.get("rows") or []:
key = row.get("key") or []
value = (row.get("values") or [{}])[0].get("value")
if key:
detail["uri"].append((str(key[0]), int(float(value or 0))))
except GraylogError:
pass
try:
st = pivot(self.client, [{"type": "count", "id": "cnt"}],
["ua"], query=q, limit=3)
for row in st.get("rows") or []:
key = row.get("key") or []
value = (row.get("values") or [{}])[0].get("value")
if key:
detail["ua"].append((str(key[0]), int(float(value or 0))))
except GraylogError:
pass
try:
hist = self.client.histogram(query=q, range_seconds=self.window,
interval="5m",
filter_str=self.filter_str)
from .client import normalize_histogram
detail["buckets5m"] = [b["count"]
for b in normalize_histogram(hist)]
except GraylogError:
pass
return ip, detail
from concurrent.futures import ThreadPoolExecutor
out = {}
with ThreadPoolExecutor(max_workers=8) as pool:
for ip, detail in pool.map(run, ips):
out[ip] = detail
return out
def analyze(self):
from concurrent.futures import ThreadPoolExecutor
fields = self.fields
client = self.client
st = pivot(client,
[{"type": "count", "id": "cnt"},
{"type": "card", "field": "uri", "id": "uri_card"},
{"type": "card", "field": "ua", "id": "ua_card"},
{"type": "card", "field": "host", "id": "host_card"},
{"type": "card", "field": "method", "id": "method_card"}],
["client_ip"], limit=600)
total = rows_to_dict(st, "cnt")
uri_card = rows_to_dict(st, "uri_card")
ua_card = rows_to_dict(st, "ua_card")
host_card = rows_to_dict(st, "host_card")
method_card = rows_to_dict(st, "method_card")
status = {}
for cls in ("2*", "3*", "4*", "5*"):
status[cls] = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query="status:%s" % cls, limit=600), "cnt")
get_count = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query="method:GET", limit=600), "cnt")
LOGIN_QUERY = ("uri:(\\\\/login* OR \\\\/passport* OR \\\\/captcha* "
"OR \\\\/jsxy\\\\/api\\\\/login*)")
login_count = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query=LOGIN_QUERY, limit=600), "cnt")
hourly = hourly_matrix(pivot(
client, [{"type": "count", "id": "cnt"}], ["client_ip"],
column_groups=[{"type": "time", "field": "timestamp", "interval": "1h"}],
limit=600))
four_xx = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query="status:4*", limit=800), "cnt")
def build(ip, count, uris, uas, hosts, methods, status_map, get_map,
login_map, hourly_map):
hour = bucket_stats(hourly_map.get(ip, []))
sum_status = sum(status_map[cls].get(ip, 0) for cls in status_map)
feat = empty_feat(ip)
feat.update({
"total": int(count),
"uri_card": int(uris.get(ip, 0)),
"ua_card": int(uas.get(ip, 0)),
"host_card": int(hosts.get(ip, 0)),
"method_card": int(methods.get(ip, 0)),
"get_ratio": get_map.get(ip, 0) / count * 100 if count else 0.0,
"login_ratio": login_map.get(ip, 0) / count * 100 if count else 0.0,
"pct2": status_map["2*"].get(ip, 0) / max(1, sum_status) * 100,
"pct3": status_map["3*"].get(ip, 0) / max(1, sum_status) * 100,
"pct4": status_map["4*"].get(ip, 0) / max(1, sum_status) * 100,
"pct5": status_map["5*"].get(ip, 0) / max(1, sum_status) * 100,
"status_known": (sum_status / count * 100) if count else 0.0,
"coverage": hour["coverage"], "cv": hour["cv"],
"peak_share": hour["peak_share"],
"four_xx": status_map["4*"].get(ip, 0),
})
return feat
base = {}
for ip, count in total.items():
if count <= 0:
continue
base[ip] = build(ip, count, uri_card, ua_card, host_card,
method_card, status, get_count, login_count, hourly)
extra = sorted(set(four_xx) - set(total), key=lambda ip: -four_xx[ip])
extra = [ip for ip in extra if not ip.startswith(INTERNAL)][:600]
for offset in range(0, len(extra), 80):
chunk = extra[offset:offset + 80]
chunk_query = " OR ".join('client_ip:"%s"' % ip for ip in chunk)
st = pivot(client,
[{"type": "count", "id": "cnt"},
{"type": "card", "field": "uri", "id": "uri_card"},
{"type": "card", "field": "ua", "id": "ua_card"},
{"type": "card", "field": "host", "id": "host_card"},
{"type": "card", "field": "method", "id": "method_card"}],
["client_ip"], query=chunk_query, limit=200)
totals = rows_to_dict(st, "cnt")
uris = rows_to_dict(st, "uri_card")
uas = rows_to_dict(st, "ua_card")
hosts = rows_to_dict(st, "host_card")
methods = rows_to_dict(st, "method_card")
chunk_status = {}
for cls in ("2*", "3*", "4*", "5*"):
chunk_status[cls] = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query='status:%s AND (%s)' % (cls, chunk_query),
limit=200), "cnt")
chunk_get = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query='method:GET AND (%s)' % chunk_query, limit=200), "cnt")
chunk_login = rows_to_dict(
pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"],
query='%s AND (%s)' % (LOGIN_QUERY, chunk_query),
limit=200), "cnt")
chunk_hour = hourly_matrix(pivot(
client, [{"type": "count", "id": "cnt"}], ["client_ip"],
column_groups=[{"type": "time", "field": "timestamp",
"interval": "1h"}], query=chunk_query, limit=200))
for cls in chunk_status:
status[cls].update(chunk_status[cls])
for ip in chunk:
if ip not in totals or totals[ip] <= 0:
continue
get_count[ip] = chunk_get.get(ip, 0)
login_count[ip] = chunk_login.get(ip, 0)
base[ip] = build(ip, totals[ip], uris, uas, hosts, methods,
status, get_count, login_count, chunk_hour)
shortlist = [ip for ip, feat in base.items()
if not ip.startswith(INTERNAL) and feat["total"] >= 10
and (feat["pct4"] >= 20 or feat["four_xx"] >= 20
or feat["uri_card"] >= 12)]
shortlist.sort(key=lambda ip: -base[ip]["four_xx"])
shortlist = shortlist[:260]
details = self._details(shortlist)
items = []
for ip, feat in base.items():
if ip.startswith(INTERNAL):
continue
detail = details.get(ip) or {}
five = bucket_stats(detail.get("buckets5m") or [])
uris = detail.get("uri") or []
uas = detail.get("ua") or []
feat["uri_templates"] = len({uri_template(uri) for uri, _ in uris})
feat["top_uri"] = uris[0][0] if uris else ""
feat["top_ua"] = uas[0][0] if uas else ""
feat["ua_family"] = ua_family(feat["top_ua"])
feat["ua_garbage"] = ua_garbage(feat["ua_card"], feat["top_ua"])
feat["coverage5m"] = five["coverage"]
score, signals = behavior_score(feat)
if score < 60 or len(signals) < 2:
continue
feat["score"] = score
feat["signals"] = signals
feat["behavior_type"] = behavior_type(feat, feat["top_ua"])
info = (self.asn.lookup(ip) if self.asn is not None else None) or {
"asn": "", "org": "未知", "brand": "未知",
"type": "unknown", "icon": "❓"}
shared = TYPE_TO_SHARED.get(info["type"], "unknown")
multi = feat["ua_card"] >= 5 or feat["host_card"] >= 10
if shared in ("mobile", "education"):
verdict = "仅告警（共享出口）"
note = ("出口属于%s（%s），大概率有大量用户共享。"
% (info["brand"], SHARE_LABEL[shared]))
elif shared == "broadband" and multi:
verdict = "仅告警（疑似多用户）"
note = ("出口属于%s（宽带 ISP），但该 IP 出现 %d 种 UA / %d 个站点，"
"疑似多用户共享。" % (info["brand"], feat["ua_card"],
feat["host_card"]))
elif shared == "broadband":
verdict = "建议封禁" if score >= 75 else "建议人工确认"
note = ("出口属于%s（宽带 ISP），但只有 %d 种 UA，更像单台设备。"
% (info["brand"], feat["ua_card"]))
elif shared == "hosting":
verdict = "建议封禁"
note = "出口属于%s（云主机/机房）。" % info["brand"]
else:
verdict = "建议人工确认"
note = "出口属于%s，归属不明确。" % info["brand"]
geo = None
if self.geo is not None:
geo = self.geo.lookup(ip)
feat.update({
"country_code": (geo or {}).get("country_code", ""),
"country_name": (geo or {}).get("country_name", ""),
"region": (geo or {}).get("region", ""),
"city": (geo or {}).get("city", ""),
"verdict": verdict, "share_note": note,
"reason_text": humanize(feat),
"asn": info["asn"], "asn_org": info["org"],
"asn_brand": info["brand"], "asn_type": info["type"],
"asn_icon": info["icon"], "shared_type": shared,
})
items.append(feat)
items.sort(key=lambda item: (-item["score"], -item["total"]))
return items
def run_once(self):
started = time.time()
try:
items = self.analyze()
self.store.replace_behavior_profiles(items)
self.store.sync_fused()
self.last_run = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
self.last_error = None
self.runs += 1
print("[%s] 行为分析: %d 个违规 IP，%d 个建议封禁，耗时 %.1fs"
% (self.last_run, len(items),
sum(1 for item in items if item["verdict"] == "建议封禁"),
time.time() - started), flush=True)
if self.notifier is not None:
notify_fused(self.store, self.notifier, self.root)
return items
except Exception as exc: # keep the service alive
self.last_error = str(exc)
print("[行为分析失败] %s" % exc, flush=True)
return []
def run_forever(self):
time.sleep(8)
while True:
self.run_once()
time.sleep(self.interval)
TYPE_LABEL = {"cloud": "云服务商", "idc": "IDC/机房", "mobile": "移动网络",
"broadband": "宽带", "education": "教育网",
"enterprise": "企业/政府", "unknown": "未知"}
def _local_time(value):
import datetime as _dt
try:
stamp = _dt.datetime.fromisoformat(str(value).replace("Z", "+00:00"))
return stamp.astimezone().strftime("%m-%d %H:%M")
except (TypeError, ValueError):
return str(value or "-")
_PUSH_ERROR_AT = 0
def notify_fused(store, notifier, root=None):
"""Push a single merged message for IPs due for a reminder (24h)."""
due = store.due_pushes(active_hours=24, cooldown_hours=24)
if not due:
return 0
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
lines = ["**建议封禁 · 融合检测**", ""]
for row in due[:20]:
geo = " · ".join(part for part in (row.get("country_name"),
row.get("region")) if part)
lines.append("**%s**%s" % (row["ip"], ("（%s）" % geo) if geo else ""))
source = ("双来源" if row.get("sources") == "both"
else ("行为检测" if row.get("sources") == "behavior"
else "特征检测"))
brand = row.get("asn_brand") or ""
if brand or row.get("asn"):
lines.append("归属：%s %s%s"
% (row.get("asn_icon") or "", brand or "未知",
(" · AS" + str(row["asn"])) if row.get("asn") else ""))
score = max(float(row.get("feature_score") or 0),
float(row.get("behavior_score") or 0))
lines.append("来源：%s · 发现：%s"
% (source, _local_time(row.get("first_banned_at"))))
lines.append("评分 %.0f · 请求 %s 次"
% (score, format(int(row.get("total") or 0), ",")))
if int(row.get("push_count") or 0) > 0:
lines.append("⏰ 该 IP 仍在活动（第 %d 次提醒）"
% (int(row["push_count"]) + 1))
if row.get("fused_reason"):
lines.append("依据：" + row["fused_reason"])
link = notifier.link_for(row["ip"])
if link:
lines.append("[查看详情](%s)" % link)
lines.append("")
try:
notifier.post("\n".join(lines).strip())
except Exception as exc:
# 失败时不刷屏：每 5 分钟最多提示一次；下一轮会自动重试
global _PUSH_ERROR_AT
now = time.time()
if now - _PUSH_ERROR_AT > 300:
print("推送失败（每 5 分钟最多提示一次）: %s" % exc, flush=True)
_PUSH_ERROR_AT = now
return 0
store.mark_pushed([row["ip"] for row in due[:20]], now)
return len(due[:20])
graylog_ops/cli.py
+289
View File
@@ -0,0 +1,289 @@
"""CLI entry point."""
import argparse
import sys
from . import __version__, commands
from .client import GraylogClient
from .config import ConfigError, load_config
from .http import GraylogError, HttpClient
EPILOG = """\
示例:
# 1) 先确认连通性、权限和这个版本支持哪些接口
graylog-ops --url http://edu-master:9000 --token $GRAYLOG_TOKEN probe
# 2) 摸清"客户端 IP 到底在哪个字段"（最关键的一步）
graylog-ops discover --range 3600 --stream 业务访问日志
# 3) 按条件看 top IP，判断阈值是否合理
graylog-ops top-ips --field client_ip --query 'status:401' --range 300 --size 30
# 3.5) 生成 profile，然后做封禁判断（只读，dry-run 语义）
graylog-ops discover --range 3600 --save-profile profiles/edu.json
graylog-ops judge --profile profiles/edu.json --range 300 --allow allowlist.txt --explain 5
# 4) 把环境信息整体落盘，便于离线分析/复盘
graylog-ops snapshot --dir snapshots/$(date +%F) --sample 500
# 5) 探测未知接口（版本差异时用）
graylog-ops raw /api/system/inputs
graylog-ops raw /api/search/universal/terms --param field=source --param range=60
# 6) 一次性产出整套信息（probe/discover/judge/snapshot），便于离线分析
graylog-ops collect --dir collect-2026-10-18 --range 3600 --allow allowlist.example.txt
环境变量: GRAYLOG_URL GRAYLOG_TOKEN GRAYLOG_USER GRAYLOG_PASSWORD
GRAYLOG_INSECURE GRAYLOG_TIMEOUT GRAYLOG_RETRIES
配置文件: ~/.config/graylog-ops/config.json 或 ./graylog-ops.json
"""
def _add_common(parser, suppress):
"""Add connection/output flags.
With ``suppress=True`` the defaults are SUPPRESS so a sub-parser never
clobbers a value that was already parsed by the root parser.
"""
def default(value):
return argparse.SUPPRESS if suppress else value
parser.add_argument("--url", metavar="URL", default=default(None),
help="Graylog 地址，如 http://edu-master:9000")
parser.add_argument("--token", metavar="TOKEN", default=default(None),
help="API token（推荐；设置后忽略 --user/--password）")
parser.add_argument("--user", metavar="USER", default=default(None),
help="用户名（配合 --password）")
parser.add_argument("--password", metavar="PASSWORD", default=default(None),
help="密码")
parser.add_argument("--insecure", action="store_true",
default=argparse.SUPPRESS if suppress else False,
help="跳过 HTTPS 证书校验")
parser.add_argument("--timeout", type=float, metavar="SECONDS", default=default(None),
help="单请求超时，默认 30")
parser.add_argument("--retries", type=int, metavar="N", default=default(None),
help="失败重试次数，默认 2")
parser.add_argument("--config", metavar="PATH", default=default(None),
help="JSON 配置文件路径")
parser.add_argument("--json", action="store_true",
default=argparse.SUPPRESS if suppress else False,
help="输出 JSON（便于管道/脚本消费）")
parser.add_argument("--out", metavar="PATH", default=default(None),
help="把结果 JSON 写入文件")
parser.add_argument("-v", "--verbose", action="store_true",
default=argparse.SUPPRESS if suppress else False,
help="打印每个 HTTP 请求")
def build_parser():
common = argparse.ArgumentParser(add_help=False)
_add_common(common, suppress=True)
parser = argparse.ArgumentParser(
prog="graylog-ops",
description="Graylog 4.0 只读信息采集工具（为 IP 封禁规则摸底）",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=EPILOG,
)
_add_common(parser, suppress=False)
parser.add_argument("--version", action="version", version="graylog-ops %s" % __version__)
sub = parser.add_subparsers(dest="command", metavar="<command>")
def add(name, handler, help_text):
cmd = sub.add_parser(name, parents=[common], help=help_text, description=help_text)
cmd.set_defaults(func=handler)
return cmd
add("probe", commands.cmd_probe, "检查连通性、认证、接口可用性矩阵")
add("system", commands.cmd_system, "节点/索引器/版本/license 概要")
add("streams", commands.cmd_streams, "列出 stream（分流与权限边界）").add_argument(
"--rules", action="store_true", help="同时拉取每个 stream 的规则（N+1 请求）")
add("inputs", commands.cmd_inputs, "列出 input 及监听地址")
add("fields", commands.cmd_fields, "列出 Graylog 已注册字段与类型")
add("indices", commands.cmd_indices, "index set 与索引大小/保留策略")
add("alerts", commands.cmd_alerts, "现有事件定义与通知方式（改造前先看）").add_argument(
"--limit", type=int, default=None, help="最多返回条数")
discover = add("discover", commands.cmd_discover, "采样日志，推断真实客户端 IP 字段")
discover.add_argument("--stream", help="stream id 或标题子串")
discover.add_argument("--query", default="*", help="查询条件，默认 *")
discover.add_argument("--range", type=int, default=3600, help="时间窗（秒），默认 3600")
discover.add_argument("--sample", type=int, default=500, help="采样消息数，默认 500")
discover.add_argument("--save-profile", metavar="PATH",
help="把推断出的字段映射写成 profile，供 judge 使用")
judge = add("judge", commands.cmd_judge,
"把聚合信号判成封禁候选（只读，不执行封禁）")
judge.add_argument("--profile", metavar="PATH", help="discover --save-profile 生成的文件")
judge.add_argument("--range", type=int, default=300, help="时间窗（秒），默认 300")
judge.add_argument("--query", default="*", help="限定查询范围，默认 *")
judge.add_argument("--stream", help="stream id 或标题子串")
judge.add_argument("--field", help="客户端 IP 字段（覆盖 profile）")
judge.add_argument("--status-field", help="状态码字段")
judge.add_argument("--uri-field", help="URI 字段")
judge.add_argument("--ua-field", help="User-Agent 字段")
judge.add_argument("--method-field", help="HTTP 方法字段")
judge.add_argument("--allow", action="append", metavar="FILE",
help="白名单文件（IP/CIDR，可重复）")
judge.add_argument("--include-internal", action="store_true",
help="允许内网地址进入候选（默认排除）")
judge.add_argument("--min-requests", type=int, help="低于该请求数的 IP 不参与判分")
judge.add_argument("--min-score", type=float, help="低于该分数的候选不输出")
judge.add_argument("--top", type=int, default=30, help="最多输出条数，默认 30")
judge.add_argument("--size", type=int, default=200, help="每类聚合返回的 IP 数上限，默认 200")
judge.add_argument("--verify", type=int, default=0, metavar="N",
help="对前 N 个候选反查 URI 集中度（识别自动化客户端，每候选多 1 次查询）")
judge.add_argument("--explain", type=int, default=0, metavar="N",
help="为前 N 个候选附带样例消息")
judge.add_argument("--all", action="store_true", help="输出全部候选（含 ignore）")
watch = add("watch", commands.cmd_watch,
"持续检测恶意公网 IP + 本地 Web 面板（SQLite 持久化）")
watch.add_argument("--profile", metavar="PATH", help="字段映射 profile")
watch.add_argument("--stream", help="只检测某个 stream")
watch.add_argument("--interval", type=int, default=60, help="检测间隔（秒），默认 60")
watch.add_argument("--window", type=int, default=900, help="检测时间窗（秒），默认 900")
watch.add_argument("--host", default="127.0.0.1", help="面板监听地址")
watch.add_argument("--port", type=int, default=8088, help="面板端口，默认 8088")
watch.add_argument("--db", default="data/ips.db", help="SQLite 路径")
watch.add_argument("--behavior-interval", type=int, default=900,
help="行为检测刷新间隔（秒，默认 900）")
watch.add_argument("--no-behavior", action="store_true",
help="关闭行为检测线程，只跑特征检测")
watch.add_argument("--notify-config", default="notify.json",
help="新增恶意 IP 的 webhook 配置（不存在则不推送）")
watch.add_argument("--notify-test", action="store_true",
help="发送一条测试推送后退出")
watch.add_argument("--asn-db", default="data/geo/asn.db",
help="ASN 库路径（空字符串禁用）")
watch.add_argument("--geo-db", default="data/geo/geo.db",
help="DB-IP Lite 生成的离线地理库路径（空字符串禁用）")
watch.add_argument("--path-prefix", metavar="PATH", default="",
help="只在该路径前缀下提供服务，根路径返回 404（用于对外分享）")
watch.add_argument("--forget", type=int, default=7200, help="多久没再出现就从列表移除（秒）")
watch.add_argument("--allow", action="append", metavar="FILE", help="白名单文件，可重复")
watch.add_argument("--field", help="客户端 IP 字段")
watch.add_argument("--status-field", help="状态码字段")
watch.add_argument("--uri-field", help="URI 字段")
watch.add_argument("--ua-field", help="User-Agent 字段")
watch.add_argument("--min-requests", type=int, default=3, help="低于该请求数不参与判定")
watch.add_argument("--size", type=int, default=200, help="每类聚合返回的 IP 数上限")
watch.add_argument("--verify", type=int, default=8, help="对前 N 个候选做 URI 核验")
watch.add_argument("--min-observe-score", type=float, default=10.0,
help="观察级候选低于该分数不入库，默认 10")
watch.add_argument("--top", type=int, default=50, help="--once 模式下打印的条数")
watch.add_argument("--export", metavar="PATH",
help="把高置信恶意 IP 导出为纯文本名单（排除疑似 NAT）")
watch.add_argument("--once", action="store_true", help="只跑一轮后退出（用于离线分析）")
watch.add_argument("--warm-limit", type=int, default=5,
help="每轮为多少个恶意 IP 预生成证据快照，默认 5")
watch.add_argument("--fix-times", "--fix-first-seen", dest="fix_first_seen",
action="store_true",
help="回查日志，修正首次攻击 / 最近攻击 / 最近出现时间")
watch.add_argument("--warm-only", action="store_true",
help="只预热证据快照后退出（配合 --warm-limit）")
watch.add_argument("--serve-only", action="store_true",
help="只提供面板，不做检测（用于浏览已有数据库）")
collect = add("collect", commands.cmd_collect,
"一条命令产出 probe/discover/judge/snapshot 整套信息")
collect.add_argument("--dir", help="输出目录，默认 graylog-collect-<时间戳>")
collect.add_argument("--range", type=int, default=3600, help="discover 的时间窗（秒）")
collect.add_argument("--judge-range", type=int, default=300, help="judge 的时间窗（秒）")
collect.add_argument("--stream", help="只针对某个 stream（传给 discover/judge）")
collect.add_argument("--allow", action="append", metavar="FILE", help="白名单文件，可重复")
for name, default_field in (("terms", None), ("top-ips", "client_ip")):
cmd = add(name, commands.cmd_terms,
"按字段做 terms 聚合（top-ips 是 field=client_ip 的快捷方式）")
cmd.add_argument("--field", required=(default_field is None), default=default_field,
help="聚合字段")
cmd.add_argument("--query", default="*", help="查询条件，默认 *")
cmd.add_argument("--range", type=int, default=300, help="时间窗（秒），默认 300")
cmd.add_argument("--size", type=int, default=50, help="返回条数，默认 50")
cmd.add_argument("--min-count", type=int, default=None, help="过滤低于该计数的项")
cmd.add_argument("--stream", help="stream id 或标题子串")
cmd.add_argument("--from", dest="from_ts", metavar="ISO8601", help="绝对时间起（UTC）")
cmd.add_argument("--to", dest="to_ts", metavar="ISO8601", help="绝对时间止（UTC）")
messages = add("messages", commands.cmd_messages, "拉取原始消息（JSON/表格/JSONL）")
messages.add_argument("--query", default="*", help="查询条件，默认 *")
messages.add_argument("--range", type=int, default=300, help="时间窗（秒），默认 300")
messages.add_argument("--limit", type=int, default=50, help="条数，默认 50")
messages.add_argument("--offset", type=int, default=0, help="偏移")
messages.add_argument("--fields", help="只取这些字段，逗号分隔")
messages.add_argument("--stream", help="stream id 或标题子串")
messages.add_argument("--jsonl", action="store_true", help="每行一个 JSON 消息")
histogram = add("histogram", commands.cmd_histogram, "消息量时间分布（估算阈值/容量）")
histogram.add_argument("--query", default="*", help="查询条件，默认 *")
histogram.add_argument("--range", type=int, default=3600, help="时间窗（秒），默认 3600")
histogram.add_argument("--interval", default="1m", help="时间桶，如 1m/5m/1h")
histogram.add_argument("--stream", help="stream id 或标题子串")
snapshot = add("snapshot", commands.cmd_snapshot, "整体环境快照落到目录（每个资源一个 json）")
snapshot.add_argument("--dir", help="输出目录，默认 graylog-snapshot-<时间戳>")
snapshot.add_argument("--sample", type=int, default=0, help="附带采样消息条数，默认 0")
raw = add("raw", commands.cmd_raw, "直接调用任意 API 路径（探索未封装的接口）")
raw.add_argument("path", help="如 /api/system/inputs")
raw.add_argument("--method", default="GET", help="HTTP 方法，默认 GET")
raw.add_argument("--body", metavar="JSON", help="JSON 请求体，或 @文件路径")
raw.add_argument("--param", action="append", metavar="K=V", help="查询参数，可重复")
raw.add_argument("--text", action="store_true", help="按文本返回（非 JSON）")
return parser
def main(argv=None):
parser = build_parser()
args = parser.parse_args(argv)
if not getattr(args, "command", None):
parser.print_help()
return 2
try:
config = load_config(args)
except ConfigError as exc:
print("配置错误: %s" % exc, file=sys.stderr)
return 2
http_client = HttpClient(
config.url,
username=config.credentials[0],
password=config.credentials[1],
timeout=config.timeout,
retries=config.retries,
insecure=config.insecure,
verbose=bool(getattr(args, "verbose", False)),
)
ctx = commands.Context(args, config, GraylogClient(http_client))
try:
return args.func(ctx, args) or 0
except GraylogError as exc:
print("Graylog API 错误: %s" % exc, file=sys.stderr)
if exc.status == 401:
print(
"提示: token 认证是把 token 当用户名、字面量 token 当密码；"
"确认 token 未吊销且账号有对应权限",
file=sys.stderr,
)
elif exc.status == 403:
print("提示: 账号缺少该资源权限（如 messages:read / searches:*）", file=sys.stderr)
elif exc.status == 404:
print(
"提示: 该版本可能没有这个接口。先跑 `probe` 看可用性，"
"或用 `raw /api/...` 直接验证",
file=sys.stderr,
)
elif exc.status in (400, 422):
print("提示: 参数不被该版本接受，加 -v 查看实际请求的 URL", file=sys.stderr)
return 1
except BrokenPipeError:
return 0
except KeyboardInterrupt:
print("", file=sys.stderr)
return 130
graylog_ops/client.py
+463
View File
@@ -0,0 +1,463 @@
"""Thin, defensive wrappers around the Graylog 4.0 REST API.
Every method returns plain Python data (dict/list) that has been normalised
just enough for callers not to care about cosmetic response differences.
Anything that is version specific and ambiguous is passed through untouched
and documented as such.
Endpoint reference for 4.0 lives in the server itself:
``http://<graylog>:9002/api/api-browser/``
"""
import json
from .http import GraylogError
def as_list(data, *keys):
"""Best effort extraction of a list from a Graylog response."""
if isinstance(data, list):
return data
if isinstance(data, dict):
for key in keys:
value = data.get(key)
if isinstance(value, list):
return value
for value in data.values():
if isinstance(value, list):
return value
return []
def dig(data, *keys, **kwargs):
default = kwargs.get("default")
current = data
for key in keys:
if not isinstance(current, dict):
return default
current = current.get(key)
if current is None:
return default
return current
def normalize_terms(data):
"""Turn a terms aggregation response into ``[{"term", "count"}, ...]``.
Graylog has shipped a couple of shapes for this over the years, so accept
the common ones instead of assuming a single layout.
"""
items = as_list(data, "terms", "values", "results")
rows = []
for item in items:
if isinstance(item, dict):
term = item.get("term", item.get("value", item.get("name")))
count = item.get("count", item.get("doc_count", item.get("hits", 0)))
rows.append({"term": term, "count": count})
elif isinstance(item, (list, tuple)) and len(item) >= 2:
rows.append({"term": item[0], "count": item[1]})
return rows
def normalize_histogram(data):
"""Turn a histogram response into ``[{"timestamp", "count"}, ...]``."""
if not isinstance(data, dict):
return []
results = data.get("results")
rows = []
if isinstance(results, dict):
for timestamp, value in results.items():
if isinstance(value, dict):
count = value.get("count", value.get("doc_count", 0))
else:
count = value
rows.append({"timestamp": timestamp, "count": count or 0})
elif isinstance(results, list):
for item in results:
if isinstance(item, dict):
rows.append({
"timestamp": item.get("timestamp", item.get("time")),
"count": item.get("count", 0),
})
rows.sort(key=lambda row: str(row["timestamp"]))
return rows
class GraylogClient:
def __init__(self, http):
self.http = http
# Which aggregation backend to use. Graylog 4.0 removed the legacy
# /api/search/universal/{terms,histogram,stats} endpoints, so 4.x must
# use the views API (POST /api/views/search/sync). 3.x has no views API.
self.agg_backend = None
# Legacy fallback only: some builds only accept "size" on terms,
# others tolerate/require "limit".
self.terms_size_param = "size"
# ---------------------------------------------------------------- system
def system_info(self):
return self.http.get("/api/system") or {}
def cluster_nodes(self):
return as_list(self.http.get("/api/system/cluster/nodes"), "nodes")
def indexer_health(self):
return self.http.get("/api/system/indexer/cluster/health") or {}
def indexer_cluster_name(self):
return self.http.get("/api/system/indexer/cluster/name") or ""
def deflector(self):
return self.http.get("/api/system/deflector") or {}
def field_types(self):
data = self.http.get("/api/system/fields")
if isinstance(data, dict):
fields = data.get("fields")
if isinstance(fields, list):
return fields
if isinstance(fields, dict):
return [{"name": name, "type": kind} for name, kind in fields.items()]
return as_list(data, "fields")
def license_info(self):
return self.http.get("/api/system/license") or {}
def index_sets(self):
data = self.http.get("/api/system/indices/index_sets", {"skip": 0, "limit": 100})
return as_list(data, "index_sets")
def indices(self):
data = self.http.get("/api/system/indexer/indices")
if isinstance(data, dict):
# Strict per-key access: as_list()'s "first list in the dict"
# fallback would merge the same list three times here.
merged = []
for key in ("indices", "closed_indices", "reopened_indices"):
value = data.get(key)
if isinstance(value, list):
merged.extend(value)
return merged
return as_list(data)
# --------------------------------------------------------------- streams
def streams(self):
return as_list(self.http.get("/api/streams"), "streams")
def stream(self, stream_id):
return self.http.get("/api/streams/%s" % stream_id) or {}
# ---------------------------------------------------------------- inputs
def inputs(self):
return as_list(self.http.get("/api/system/inputs"), "inputs")
def extractors(self, input_id):
data = self.http.get("/api/system/inputs/%s/extractors" % input_id)
return as_list(data, "extractors")
# ------------------------------------------------------------- pipelines
def pipelines(self):
return as_list(self.http.get("/api/system/pipelines/pipeline"), "pipelines")
def pipeline_rules(self):
return as_list(self.http.get("/api/system/pipelines/rule"), "rules")
# ---------------------------------------------------------------- events
def _paged(self, path, key, limit=None, per_page=50, params=None):
items = []
page = 1
while True:
query = dict(params or {})
query.update({"page": page, "per_page": per_page})
data = self.http.get(path, query)
batch = as_list(data, key)
if not batch:
return items
items.extend(batch)
if limit is not None and len(items) >= limit:
return items[:limit]
# Trust the server's total when it tells us there is nothing left;
# this also stops runaway pagination on servers that ignore `page`.
total = dig(data, "total")
if isinstance(total, int) and len(items) >= total:
return items
if len(batch) < per_page:
return items
page += 1
if page > 500: # defensive stop
return items
def event_definitions(self, limit=None, per_page=50):
return self._paged(
"/api/events/definitions",
"event_definitions",
limit=limit,
per_page=per_page,
params={"sort": "title", "order": "asc"},
)
def notifications(self, limit=None, per_page=50):
return self._paged(
"/api/events/notifications", "notifications", limit=limit, per_page=per_page
)
def events(self, limit=None, per_page=50):
data = self.http.get("/api/events", {"page": 1, "per_page": per_page})
items = as_list(data, "events")
return items[:limit] if limit else items
# --------------------------------------------------------------- sidecar
def sidecars(self):
return as_list(self.http.get("/api/sidecars"), "sidecars")
# ---------------------------------------------------------------- search
@staticmethod
def _time_params(range_seconds=None, from_ts=None, to_ts=None):
if from_ts and to_ts:
return {"from": from_ts, "to": to_ts}
return {"range": int(range_seconds if range_seconds is not None else 300)}
def search_relative(self, query="*", range_seconds=300, limit=100, offset=0,
sort="timestamp:desc", fields=None, filter_str=None):
params = {"query": query or "*", "limit": int(limit), "offset": int(offset), "sort": sort}
params.update(self._time_params(range_seconds=range_seconds))
if fields:
params["fields"] = ",".join(fields)
if filter_str:
params["filter"] = filter_str
data = self.http.get("/api/search/universal/relative", params)
return self._messages(data)
def search_absolute(self, query="*", from_ts=None, to_ts=None, limit=100, offset=0,
sort="timestamp:desc", fields=None, filter_str=None):
params = {"query": query or "*", "limit": int(limit), "offset": int(offset), "sort": sort}
params.update(self._time_params(from_ts=from_ts, to_ts=to_ts))
if fields:
params["fields"] = ",".join(fields)
if filter_str:
params["filter"] = filter_str
data = self.http.get("/api/search/universal/absolute", params)
return self._messages(data)
@staticmethod
def _messages(data):
if not isinstance(data, dict):
return []
messages = data.get("messages")
if not isinstance(messages, list):
return []
result = []
for item in messages:
if isinstance(item, dict) and isinstance(item.get("message"), dict):
message = dict(item["message"])
if item.get("index") and "_index" not in message:
message["_index"] = item["index"]
result.append(message)
elif isinstance(item, dict):
result.append(item)
return result
# ------------------------------------------------------- views aggregation
BASE_PATH = "/api/views/search"
@staticmethod
def _timerange(range_seconds=None, from_ts=None, to_ts=None):
if from_ts and to_ts:
return {"type": "absolute", "from": from_ts, "to": to_ts}
return {"type": "relative", "range": int(range_seconds or 300)}
@staticmethod
def _stream_filter(filter_str):
"""Translate the legacy ``streams:<id>[,<id>]`` filter to a views filter."""
if not filter_str or not filter_str.startswith("streams:"):
return None
ids = [part.strip() for part in filter_str[len("streams:"):].split(",") if part.strip()]
if not ids:
return None
return {"type": "or", "filters": [{"type": "stream", "id": sid} for sid in ids]}
def _views_pivot(self, query, timerange, row_groups, column_groups, series,
sort, filter_str=None, timeout_ms=60000):
search_type = {
"type": "pivot",
"id": "st1",
"series": series,
"row_groups": row_groups,
"column_groups": column_groups,
"sort": sort,
"rollup": True,
}
query_object = {
"id": "q1",
"timerange": timerange,
"query": {"type": "elasticsearch", "query_string": query or "*"},
"search_types": [search_type],
}
stream_filter = self._stream_filter(filter_str)
if stream_filter:
query_object["filter"] = stream_filter
body = {"queries": [query_object], "parameters": []}
data = self.http.request(
"POST", self.BASE_PATH + "/sync",
params={"timeout": int(timeout_ms)}, body=body,
)
results = ((data or {}).get("results") or {}).get("q1") or {}
errors = results.get("errors") or []
if errors:
raise GraylogError(
"views search error: %s" % json.dumps(errors, ensure_ascii=False)[:400],
path=self.BASE_PATH + "/sync", method="POST",
)
search_types = results.get("search_types") or {}
if isinstance(search_types, list):
search_types = {item.get("id"): item for item in search_types
if isinstance(item, dict)}
return search_types.get("st1") or {}
def _terms_views(self, field, query, size, range_seconds, from_ts, to_ts, filter_str):
search_type = self._views_pivot(
query=query,
timerange=self._timerange(range_seconds, from_ts, to_ts),
row_groups=[{"type": "values", "field": field, "limit": int(size)}],
column_groups=[],
series=[{"type": "count", "id": "c1"}],
sort=[{"type": "pivot", "field": "c1", "direction": "Descending"}],
filter_str=filter_str,
)
terms = []
for row in search_type.get("rows") or []:
key = row.get("key") or []
if not key:
continue # rollup row (total)
values = row.get("values") or []
count = values[0].get("value") if values else 0
terms.append({"term": key[0], "count": count})
return {
"field": field,
"terms": terms,
"total": search_type.get("total", sum(row["count"] or 0 for row in terms)),
}
def _histogram_views(self, query, range_seconds, interval, filter_str, from_ts, to_ts):
search_type = self._views_pivot(
query=query,
timerange=self._timerange(range_seconds, from_ts, to_ts),
row_groups=[],
column_groups=[{"type": "time", "field": "timestamp", "interval": interval}],
series=[{"type": "count", "id": "c1"}],
sort=[],
filter_str=filter_str,
)
results = {}
for row in search_type.get("rows") or []:
for item in row.get("values") or []:
key = item.get("key") or []
# Column keys are [timestamp, series_id]; the pivot rollup row
# only carries [series_id] and must not become a time bucket.
if len(key) < 2:
continue
results[key[0]] = {"count": item.get("value", 0)}
return {"results": results, "interval": interval}
# ------------------------------------------------------------- aggregation
def search_views_messages(self, query="*", range_seconds=300, limit=5,
offset=0, filter_str=None, from_ts=None, to_ts=None,
timeout_ms=60000, sort_order="DESC"):
"""Message search through the views API (the legacy endpoint is flaky
on long windows with complex queries and drops the connection)."""
search_type = {
"type": "messages",
"id": "st1",
"limit": int(limit),
"offset": int(offset),
"sort": [{"field": "timestamp",
"order": "ASC" if str(sort_order).upper() == "ASC" else "DESC"}],
}
query_object = {
"id": "q1",
"timerange": self._timerange(range_seconds, from_ts, to_ts),
"query": {"type": "elasticsearch", "query_string": query or "*"},
"search_types": [search_type],
}
stream_filter = self._stream_filter(filter_str)
if stream_filter:
query_object["filter"] = stream_filter
body = {"queries": [query_object], "parameters": []}
data = self.http.request(
"POST", self.BASE_PATH + "/sync",
params={"timeout": int(timeout_ms)}, body=body,
)
results = ((data or {}).get("results") or {}).get("q1") or {}
errors = results.get("errors") or []
if errors:
raise GraylogError(
"views message search error: %s" % json.dumps(errors, ensure_ascii=False)[:300],
path=self.BASE_PATH + "/sync", method="POST",
)
search_types = results.get("search_types") or {}
if isinstance(search_types, list):
search_types = {item.get("id"): item for item in search_types
if isinstance(item, dict)}
search_type_result = search_types.get("st1") or {}
return self._messages({"messages": search_type_result.get("messages") or []})
def terms(self, field, query="*", size=50, range_seconds=300,
from_ts=None, to_ts=None, filter_str=None):
if self.agg_backend in (None, "views"):
try:
data = self._terms_views(field, query, size, range_seconds,
from_ts, to_ts, filter_str)
self.agg_backend = "views"
return data
except GraylogError as exc:
if exc.status != 404 or self.agg_backend == "views":
raise
self.agg_backend = "legacy"
return self._terms_legacy(field, query, size, range_seconds,
from_ts, to_ts, filter_str)
def _terms_legacy(self, field, query, size, range_seconds, from_ts, to_ts, filter_str):
params = {"query": query or "*", "field": field}
params.update(self._time_params(range_seconds, from_ts, to_ts))
if filter_str:
params["filter"] = filter_str
params[self.terms_size_param] = int(size)
try:
return self.http.get("/api/search/universal/terms", params)
except GraylogError as exc:
# Older/newer builds disagree on the parameter name; retry once.
if exc.status in (400, 422) and self.terms_size_param == "size":
params.pop("size", None)
params["limit"] = int(size)
data = self.http.get("/api/search/universal/terms", params)
self.terms_size_param = "limit"
return data
raise
def histogram(self, query="*", range_seconds=3600, interval="1m",
filter_str=None, from_ts=None, to_ts=None):
if self.agg_backend in (None, "views"):
try:
data = self._histogram_views(query, range_seconds, interval,
filter_str, from_ts, to_ts)
self.agg_backend = "views"
return data
except GraylogError as exc:
if exc.status != 404 or self.agg_backend == "views":
raise
self.agg_backend = "legacy"
params = {"query": query or "*", "interval": interval}
params.update(self._time_params(range_seconds, from_ts, to_ts))
if filter_str:
params["filter"] = filter_str
return self.http.get("/api/search/universal/histogram", params)
def stats(self, field, query="*", range_seconds=300, stats=None,
filter_str=None, from_ts=None, to_ts=None):
params = {"query": query or "*", "field": field}
params.update(self._time_params(range_seconds, from_ts, to_ts))
if stats:
params["stats"] = ",".join(stats)
if filter_str:
params["filter"] = filter_str
return self.http.get("/api/search/universal/stats", params)
graylog_ops/commands.py
+1174
View File
File diff suppressed because it is too large. Load diff
graylog_ops/config.py
+161
View File
@@ -0,0 +1,161 @@
"""Configuration resolution.
Precedence (highest wins):
1. command line flags
2. environment variables (GRAYLOG_URL, GRAYLOG_TOKEN, ...)
3. JSON config file (--config, ./graylog-ops.json,
~/.config/graylog-ops/config.json)
File format::
{
"url": "http://edu-master:9000",
"token": "xxxxxxxx",
"insecure": false,
"timeout": 30
}
"""
import json
import os
ENV_KEYS = {
"url": "GRAYLOG_URL",
"token": "GRAYLOG_TOKEN",
"user": "GRAYLOG_USER",
"password": "GRAYLOG_PASSWORD",
"insecure": "GRAYLOG_INSECURE",
"timeout": "GRAYLOG_TIMEOUT",
"retries": "GRAYLOG_RETRIES",
}
DEFAULT_CONFIG_PATHS = (
os.path.expanduser("~/.config/graylog-ops/config.json"),
"graylog-ops.json",
)
class ConfigError(Exception):
"""User facing configuration problem."""
def _truthy(value):
return str(value).strip().lower() in ("1", "true", "yes", "on", "y")
def _read_file(path):
try:
with open(path, "r", encoding="utf-8") as handle:
data = json.load(handle)
except OSError as exc:
raise ConfigError("cannot read config %s: %s" % (path, exc))
except ValueError as exc:
raise ConfigError("config %s is not valid JSON: %s" % (path, exc))
if not isinstance(data, dict):
raise ConfigError("config %s must contain a JSON object" % path)
return data
class Config:
def __init__(self, url, token=None, user=None, password=None,
insecure=False, timeout=30.0, retries=2, source=None):
self.url = url
self.token = token
self.user = user
self.password = password
self.insecure = bool(insecure)
self.timeout = float(timeout)
self.retries = int(retries)
self.source = source
@property
def credentials(self):
"""(username, password) pair for HTTP Basic auth."""
if self.token:
# Graylog token auth: username is the token, password is "token".
return self.token, "token"
return self.user, self.password
@property
def auth_kind(self):
return "token" if self.token else "user/password"
def _pick(values, key, default=None):
value = values.get(key, default)
return value
def load_config(args):
"""Build a Config from CLI args, env vars and the config file."""
values = {}
chosen = None
explicit = getattr(args, "config", None)
if explicit:
if not os.path.exists(explicit):
raise ConfigError("config file not found: %s" % explicit)
values.update(_read_file(explicit))
chosen = explicit
else:
for candidate in DEFAULT_CONFIG_PATHS:
if os.path.exists(candidate):
values.update(_read_file(candidate))
chosen = candidate
break
for key, env_name in ENV_KEYS.items():
env_value = os.environ.get(env_name)
if env_value is not None and env_value.strip() != "":
values[key] = env_value
for key in ("url", "token", "user", "password", "timeout", "retries"):
flag_value = getattr(args, key, None)
if flag_value is not None and flag_value != "":
values[key] = flag_value
if getattr(args, "insecure", False):
values["insecure"] = True
url = values.get("url")
if not url:
raise ConfigError(
"no Graylog URL: pass --url, set GRAYLOG_URL, or add \"url\" to the config file"
)
url = str(url).strip()
if "://" not in url:
url = "http://" + url
token = (values.get("token") or "").strip() or None
user = (values.get("user") or "").strip() or None
password = values.get("password")
if not token and not (user and password):
raise ConfigError(
"no credentials: pass --token or --user/--password "
"(env GRAYLOG_TOKEN / GRAYLOG_USER + GRAYLOG_PASSWORD)"
)
timeout = values.get("timeout", 30)
retries = values.get("retries", 2)
try:
timeout = float(timeout)
except (TypeError, ValueError):
raise ConfigError("timeout must be a number, got %r" % (timeout,))
try:
retries = int(retries)
except (TypeError, ValueError):
raise ConfigError("retries must be an integer, got %r" % (retries,))
insecure = values.get("insecure", False)
if isinstance(insecure, str):
insecure = _truthy(insecure)
return Config(
url=url,
token=token,
user=user,
password=password,
insecure=bool(insecure),
timeout=timeout,
retries=retries,
source=chosen,
)
graylog_ops/detect.py
+300
View File
@@ -0,0 +1,300 @@
"""Field heuristics: find the field that actually holds the client IP.
The ban pipeline is only as good as the field it keys on. In a typical
deployment behind a reverse proxy / CDN, the field you *think* is the client
IP is the proxy's address:
* ``source`` -- the host that shipped the log line to Graylog
* ``gl2_remote_ip`` -- the host that opened the GELF/Beats connection
* ``remote_addr`` -- could be the proxy if XFF is not parsed
So instead of assuming, sample real messages and score every field by
"how often does this field hold an IP address" plus "does the field name look
like a client address".
"""
import ipaddress
import re
# Field names that strongly suggest a client address.
STRONG_HINT = re.compile(
r"(?i)client[_. -]?ip|remote[_. -]?addr|real[_. -]?ip|"
r"(^|[_. -])ip([_. -]|$)|x[_. -]?forwarded[_. -]?for|"
r"(^|[_. -])xff([_. -]|$)|src[_. -]?ip|c[_. -]?ip"
)
WEAK_HINT = re.compile(
r"(?i)addr|host|client|proxy|peer|origin|caller|visitor|remote|forward|endpoint"
)
# Graylog internals that are never the client IP (gl2_remote_ip is the shipper).
INTERNAL_PREFIXES = ("gl2_", "_")
INTERNAL_NAMES = ("timestamp", "streams", "message")
HTTP_HINTS = {
"status": re.compile(r"(?i)^(http[_. ]?)?status([_. ]?code)?$|^response([_. ]?(code|status))?$|^code$"),
"uri": re.compile(r"(?i)(request[_. ]?)?uri$|^url$|^path$|^request$|^endpoint$|^target$"),
"method": re.compile(r"(?i)^(http[_. ]?)?method$|^verb$"),
"user_agent": re.compile(r"(?i)user[_. -]?agent|^ua$|^agent$"),
"host": re.compile(r"(?i)^(http[_. ]?)?host$|^vhost$|^domain$|^server_name$"),
"bytes": re.compile(r"(?i)bytes|^size$|^body_bytes_sent$"),
"referer": re.compile(r"(?i)refer(r)?er$"),
"duration": re.compile(r"(?i)request[_. ]?time|response[_. ]?time|duration|latency|elapsed"),
}
# Deliberately not using ``ipaddress.is_private``: that property also covers
# documentation ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24) and
# other special-use blocks, which would make every public test address look
# like an internal proxy. For ban decisions "internal" means RFC1918 plus
# loopback, link-local and CGNAT.
_INTERNAL_V4 = tuple(ipaddress.ip_network(cidr) for cidr in (
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16",
"100.64.0.0/10", # CGNAT
"127.0.0.0/8",
"169.254.0.0/16",
))
_INTERNAL_V6 = tuple(ipaddress.ip_network(cidr) for cidr in (
"::1/128",
"fc00::/7", # unique local
"fe80::/10", # link local
))
def is_internal(ip):
networks = _INTERNAL_V4 if ip.version == 4 else _INTERNAL_V6
return any(ip in network for network in networks)
def as_ip(value):
"""Return an ``ipaddress`` object if *value* looks like a bare IP.
Accepts ``1.2.3.4``, ``2001:db8::1``, ``[2001:db8::1]:443`` and
``1.2.3.4:5678``. Rejects CIDRs, hostnames and arbitrary integers.
"""
if not isinstance(value, str):
return None
text = value.strip().strip('"')
if not text or len(text) > 64:
return None
try:
return ipaddress.ip_address(text)
except ValueError:
pass
if text.startswith("[") and "]" in text:
try:
return ipaddress.ip_address(text[1:text.index("]")])
except ValueError:
return None
if text.count(":") == 1 and "." in text:
host = text.split(":", 1)[0]
try:
return ipaddress.ip_address(host)
except ValueError:
return None
return None
def _empty(name):
return {
"name": name,
"present": 0,
"ip_messages": 0,
"ip_values": 0,
"distinct": set(),
"distinct_ip": set(),
"private": 0,
"versions": set(),
"examples": [],
}
def _analyse(messages):
total = len(messages)
stats = {}
embedded = 0
for message in messages:
embedded_here = False
for name, raw in message.items():
values = raw if isinstance(raw, list) else [raw]
entry = stats.get(name)
if entry is None:
entry = stats[name] = _empty(name)
emitted = False
has_ip = False
for value in values:
if value is None or value == "":
continue
text = str(value)[:160]
entry["distinct"].add(text)
emitted = True
ip = as_ip(value)
if ip is not None:
has_ip = True
entry["ip_values"] += 1
entry["distinct_ip"].add(str(ip))
entry["versions"].add(ip.version)
if is_internal(ip):
entry["private"] += 1
text_ip = str(ip)
if len(entry["examples"]) < 3 and text_ip not in entry["examples"]:
entry["examples"].append(text_ip)
if name == "message":
embedded_here = True
if emitted:
entry["present"] += 1
if has_ip:
entry["ip_messages"] += 1
if embedded_here:
embedded += 1
return stats, total, embedded
def _score(name, entry, total):
present = max(entry["present"], 1)
ip_rate = entry["ip_messages"] / present
presence = entry["present"] / max(total, 1)
score = 60.0 * ip_rate + 5.0 * presence
notes = []
if STRONG_HINT.search(name):
score += 30
elif WEAK_HINT.search(name):
score += 10
if name == "source":
score -= 20
notes.append("`source` 是日志发送方（shipper/反代），通常不是真实客户端")
elif name == "gl2_remote_ip":
score -= 25
notes.append("`gl2_remote_ip` 是向 Graylog 发送消息的对端地址，不是访问者")
return score, notes
def _confidence(best):
if not best:
return "none"
if best["score"] >= 90 and best["ip_rate"] >= 0.95:
return "high"
if best["score"] >= 60:
return "medium"
return "low"
def analyse_messages(messages):
"""Build the field report used by ``graylog-ops discover``."""
stats, total, embedded = _analyse(messages)
if total == 0:
return {
"sampled": 0,
"fields": [],
"candidates": [],
"http_fields": {},
"embedded_ip_in_message_rate": 0.0,
"recommendation": {
"field": None,
"confidence": "none",
"notes": ["没有采到消息：检查时间范围、查询条件、stream 过滤或权限"],
"warnings": [],
},
"sample_message": None,
}
fields = []
candidates = []
for name, entry in stats.items():
if entry["present"] == 0:
continue
ip_rate = entry["ip_messages"] / entry["present"]
row = {
"name": name,
"present": entry["present"],
"presence": round(entry["present"] / total, 4),
"distinct": len(entry["distinct"]),
"distinct_ips": len(entry["distinct_ip"]),
"ip_rate": round(ip_rate, 4),
"examples": entry["examples"],
}
fields.append(row)
if name in INTERNAL_NAMES or name.startswith(INTERNAL_PREFIXES):
continue
if ip_rate <= 0:
continue
score, notes = _score(name, entry, total)
private_ratio = (entry["private"] / entry["ip_values"]) if entry["ip_values"] else 0.0
candidates.append({
"field": name,
"score": round(score, 1),
"ip_rate": round(ip_rate, 4),
"presence": round(entry["present"] / total, 4),
"distinct_ips": len(entry["distinct_ip"]),
"private_ratio": round(private_ratio, 4),
"versions": sorted(entry["versions"]),
"examples": entry["examples"],
"notes": notes,
})
fields.sort(key=lambda row: (-row["present"], row["name"]))
candidates.sort(key=lambda row: (-row["score"], -row["distinct_ips"]))
http_fields = {kind: [] for kind in HTTP_HINTS}
for name in stats:
for kind, pattern in HTTP_HINTS.items():
if pattern.search(name):
http_fields[kind].append(name)
http_fields = {kind: names for kind, names in http_fields.items() if names}
best = candidates[0] if candidates else None
source_entry = stats.get("source")
warnings = []
notes = []
if best:
notes.append(
"字段 `%s` 的 IP 占比 %.0f%%，样例 %s"
% (best["field"], best["ip_rate"] * 100, ", ".join(best["examples"][:3]) or "-")
)
if source_entry and len(source_entry["distinct"]) <= 2 and best["distinct_ips"] > 2:
warnings.append(
"`source` 只有 %d 个不同取值，而 `%s` 有 %d 个：`source` 是采集器/反代，"
"封禁必须用 `%s`" % (len(source_entry["distinct"]), best["field"],
best["distinct_ips"], best["field"])
)
if best["private_ratio"] >= 0.5:
warnings.append(
"`%s` 中 %.0f%% 是内网地址（私有/回环），可能仍是 LB 或代理地址，"
"需要确认 X-Forwarded-For 是否已在采集侧解析"
% (best["field"], best["private_ratio"] * 100)
)
else:
notes.append("没有发现任何字段存放纯 IP 值")
# Only worth flagging when there is no usable field: then the IP lives in
# the free-text `message` and has to be extracted server side.
if embedded / max(total, 1) >= 0.5 and not best:
warnings.append(
"%.0f%% 的消息在 `message` 文本里内嵌了 IP。要么在 input 上加 extractor / "
"pipeline 解析成独立字段，要么在采集端（nginx log_format）直接输出 client_ip 字段"
% (100.0 * embedded / total)
)
# HTTP-ish fields help with the next phase (per-URI thresholds), surface them.
if not best and http_fields:
warnings.append("只找到 HTTP 相关字段，没有 IP 字段，见 http_fields")
return {
"sampled": total,
"fields": fields,
"candidates": candidates,
"http_fields": http_fields,
"embedded_ip_in_message_rate": round(embedded / total, 4),
"recommendation": {
"field": best["field"] if best else None,
"confidence": _confidence(best),
"notes": notes,
"warnings": warnings,
},
"sample_message": messages[0] if messages else None,
}
graylog_ops/geo.py
+138
View File
@@ -0,0 +1,138 @@
"""Offline GeoIP lookup backed by the free DB-IP Lite CSV dumps.
The city dump has no country name, so a small code -> name table is shipped
here; anything unknown falls back to the ISO code.
"""
import csv
import ipaddress
import os
import sqlite3
COUNTRY_CN = {
"CN": "中国", "HK": "中国香港", "TW": "中国台湾", "MO": "中国澳门",
"US": "美国", "CA": "加拿大", "MX": "墨西哥", "BR": "巴西", "AR": "阿根廷",
"CL": "智利", "CO": "哥伦比亚", "PE": "秘鲁", "VE": "委内瑞拉",
"GB": "英国", "IE": "爱尔兰", "FR": "法国", "DE": "德国", "NL": "荷兰",
"BE": "比利时", "LU": "卢森堡", "CH": "瑞士", "AT": "奥地利", "IT": "意大利",
"ES": "西班牙", "PT": "葡萄牙", "GR": "希腊", "PL": "波兰", "CZ": "捷克",
"SK": "斯洛伐克", "HU": "匈牙利", "RO": "罗马尼亚", "BG": "保加利亚",
"HR": "克罗地亚", "RS": "塞尔维亚", "SI": "斯洛文尼亚", "LT": "立陶宛",
"LV": "拉脱维亚", "EE": "爱沙尼亚", "FI": "芬兰", "SE": "瑞典",
"NO": "挪威", "DK": "丹麦", "IS": "冰岛", "UA": "乌克兰", "BY": "白俄罗斯",
"RU": "俄罗斯", "TR": "土耳其", "IL": "以色列", "SA": "沙特阿拉伯",
"AE": "阿联酋", "QA": "卡塔尔", "KW": "科威特", "IR": "伊朗", "IQ": "伊拉克",
"EG": "埃及", "ZA": "南非", "NG": "尼日利亚", "KE": "肯尼亚", "MA": "摩洛哥",
"ET": "埃塞俄比亚", "IN": "印度", "PK": "巴基斯坦", "BD": "孟加拉国",
"LK": "斯里兰卡", "NP": "尼泊尔", "MM": "缅甸", "KH": "柬埔寨", "LA": "老挝",
"TH": "泰国", "VN": "越南", "MY": "马来西亚", "SG": "新加坡", "ID": "印度尼西亚",
"PH": "菲律宾", "JP": "日本", "KR": "韩国", "MN": "蒙古", "KZ": "哈萨克斯坦",
"UZ": "乌兹别克斯坦", "GE": "格鲁吉亚", "AM": "亚美尼亚", "AZ": "阿塞拜疆",
"AU": "澳大利亚", "NZ": "新西兰", "ZZ": "未知",
}
CN_REGION = {
"Beijing": "北京", "Tianjin": "天津", "Hebei": "河北", "Shanxi": "山西",
"Inner Mongolia": "内蒙古", "Liaoning": "辽宁", "Jilin": "吉林",
"Heilongjiang": "黑龙江", "Shanghai": "上海", "Jiangsu": "江苏",
"Zhejiang": "浙江", "Anhui": "安徽", "Fujian": "福建", "Jiangxi": "江西",
"Shandong": "山东", "Henan": "河南", "Hubei": "湖北", "Hunan": "湖南",
"Guangdong": "广东", "Guangxi": "广西", "Hainan": "海南", "Chongqing": "重庆",
"Sichuan": "四川", "Guizhou": "贵州", "Yunnan": "云南", "Tibet": "西藏",
"Shaanxi": "陕西", "Gansu": "甘肃", "Qinghai": "青海", "Ningxia": "宁夏",
"Xinjiang": "新疆", "Taiwan": "台湾", "Hong Kong": "香港", "Macau": "澳门",
}
def flag(cc):
if not cc or len(cc) != 2 or not cc.isalpha() or cc == "ZZ":
return ""
return "".join(chr(0x1F1E6 + ord(char) - 65) for char in cc.upper())
def country_name(cc):
return COUNTRY_CN.get(cc, cc or "")
def region_name(cc, region):
if cc == "CN" and region:
return CN_REGION.get(region, region)
return region or ""
def build_geo_db(city_csv, out_db, country_csv=None):
"""Import the DB-IP city CSV (IPv4 only) into a small SQLite database."""
os.makedirs(os.path.dirname(out_db) or ".", exist_ok=True)
conn = sqlite3.connect(out_db)
conn.execute("DROP TABLE IF EXISTS geo")
conn.execute("CREATE TABLE geo (start INTEGER PRIMARY KEY, end INTEGER, "
"cc TEXT, region TEXT, city TEXT)")
batch = []
total = 0
def flush():
nonlocal batch
if batch:
conn.executemany("INSERT OR REPLACE INTO geo VALUES (?,?,?,?,?)", batch)
batch = []
with open(city_csv, newline="", encoding="utf-8", errors="replace") as fh:
for row in csv.reader(fh):
if len(row) < 6:
continue
start, end, cc = row[0], row[1], row[3]
if ":" in start or ":" in end or not cc:
continue
try:
first = int(ipaddress.IPv4Address(start))
last = int(ipaddress.IPv4Address(end))
except ValueError:
continue
batch.append((first, last, cc, row[4], row[5]))
total += 1
if len(batch) >= 100000:
flush()
conn.commit()
flush()
conn.commit()
conn.execute("PRAGMA journal_mode=WAL")
conn.close()
return total
class GeoIP:
"""Read-only lookup over the generated geo.db."""
def __init__(self, path):
if not path or not os.path.exists(path):
raise FileNotFoundError(path)
self.conn = sqlite3.connect("file:%s?mode=ro" % path, uri=True,
check_same_thread=False)
def lookup(self, ip):
try:
value = int(ipaddress.ip_address(ip))
except ValueError:
return None
row = self.conn.execute(
"SELECT end, cc, region, city FROM geo WHERE start <= ? "
"ORDER BY start DESC LIMIT 1", (value,)).fetchone()
if not row or row[0] < value:
return None
return {"country_code": row[1], "country_name": country_name(row[1]),
"region": region_name(row[1], row[2]), "city": row[3] or ""}
def close(self):
try:
self.conn.close()
except sqlite3.Error:
pass
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--city-csv", default="data/geo/dbip-city-lite-2026-10.csv")
parser.add_argument("--out", default="data/geo/geo.db")
args = parser.parse_args()
count = build_geo_db(args.city_csv, args.out)
print("imported %d IPv4 ranges -> %s" % (count, args.out))
graylog_ops/http.py
+214
View File
@@ -0,0 +1,214 @@
"""Minimal HTTP layer for the Graylog REST API.
Deliberately built on urllib so the tool has no third-party dependencies and
can be copied onto old jump hosts. Handles the three things Graylog's API
demands that plain urllib does not do for you:
* HTTP Basic auth (API token used as username with the literal password
``token``, or a real username/password pair),
* the ``X-Requested-By`` header required by Graylog's CSRF protection on all
non-GET requests,
* bounded retries with backoff for 429/5xx and transient connection errors.
"""
import base64
import json
import random
import socket
import ssl
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
USER_AGENT = "graylog-ops/0.1 (read-only recon)"
RETRY_STATUS = frozenset((429, 500, 502, 503, 504))
class GraylogError(Exception):
"""Any failure talking to the Graylog API."""
def __init__(self, message, status=None, body=None, path=None, method=None):
super().__init__(message)
self.message = message
self.status = status
self.body = body
self.path = path
self.method = method
def __str__(self):
parts = []
if self.status is not None:
parts.append("HTTP %s" % self.status)
if self.method and self.path:
parts.append("%s %s" % (self.method, self.path))
parts.append(self.message)
return " | ".join(parts)
def _basic_auth(username, password):
raw = ("%s:%s" % (username, password)).encode("utf-8")
return "Basic " + base64.b64encode(raw).decode("ascii")
def _error_detail(body):
"""Pull a human readable message out of a Graylog/ES error body."""
if not body:
return ""
try:
data = json.loads(body)
except (TypeError, ValueError):
return body.strip()[:300]
if isinstance(data, dict):
for key in ("message", "error", "description"):
value = data.get(key)
if isinstance(value, str) and value.strip():
return value.strip()
if isinstance(data.get("error"), dict):
reason = data["error"].get("reason")
if isinstance(reason, str):
return reason
return body.strip()[:300]
class HttpClient:
def __init__(
self,
base_url,
username=None,
password=None,
timeout=30.0,
retries=2,
insecure=False,
verbose=False,
):
self.base_url = base_url.rstrip("/")
self.timeout = float(timeout)
self.retries = max(0, int(retries))
self.verbose = verbose
self.auth = None
if username is not None:
self.auth = _basic_auth(username, "" if password is None else password)
handlers = []
if insecure:
context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
handlers.append(urllib.request.HTTPSHandler(context=context))
# build_opener keeps the default ProxyHandler, so HTTP(S)_PROXY env vars
# keep working for environments that need a proxy to reach Graylog.
self.opener = urllib.request.build_opener(*handlers)
# -- url helpers -----------------------------------------------------
def url(self, path, params=None):
if not path.startswith("/"):
path = "/" + path
target = self.base_url + path
if params:
clean = {}
for key, value in params.items():
if value is None:
continue
if isinstance(value, bool):
value = "true" if value else "false"
clean[key] = value
if clean:
sep = "&" if "?" in target else "?"
target += sep + urllib.parse.urlencode(clean, doseq=True)
return target
# -- request core ----------------------------------------------------
def request(self, method, path, params=None, body=None, expect_json=True):
method = method.upper()
url = self.url(path, params)
payload = None
headers = {
"Accept": "application/json",
"Accept-Encoding": "identity",
"User-Agent": USER_AGENT,
# Graylog rejects state changing requests without this header.
"X-Requested-By": "graylog-ops",
}
if body is not None:
payload = json.dumps(body).encode("utf-8")
headers["Content-Type"] = "application/json"
if self.auth:
headers["Authorization"] = self.auth
started = time.time()
last_error = None
for attempt in range(self.retries + 1):
request = urllib.request.Request(
url, data=payload, headers=headers, method=method
)
try:
with self.opener.open(request, timeout=self.timeout) as response:
status = response.getcode()
raw = response.read()
self._debug(method, url, status, started)
if not raw.strip():
return None
text = raw.decode("utf-8", "replace")
if not expect_json:
return text
try:
return json.loads(text)
except ValueError:
raise GraylogError(
"response was not JSON",
status=status,
body=text[:500],
path=path,
method=method,
)
except urllib.error.HTTPError as exc:
text = exc.read().decode("utf-8", "replace")
detail = _error_detail(text)
self._debug(method, url, exc.code, started)
last_error = GraylogError(
detail or ("request rejected" if exc.code != 401 else "authentication failed"),
status=exc.code,
body=text[:2000],
path=path,
method=method,
)
if exc.code in RETRY_STATUS and attempt < self.retries:
self._sleep(attempt, exc.headers.get("Retry-After"))
continue
raise last_error
except (urllib.error.URLError, socket.timeout, ConnectionError, OSError) as exc:
reason = getattr(exc, "reason", exc)
self._debug(method, url, "ERR", started)
last_error = GraylogError(
"connection failed: %s" % reason,
path=path,
method=method,
)
if attempt < self.retries:
self._sleep(attempt, None)
continue
raise last_error
raise last_error # pragma: no cover - loop always raises or returns
# -- internals -------------------------------------------------------
def _sleep(self, attempt, retry_after):
delay = (1.5 ** attempt) + random.uniform(0, 0.3)
try:
if retry_after is not None:
delay = max(delay, float(retry_after))
except (TypeError, ValueError):
pass
time.sleep(min(delay, 30.0))
def _debug(self, method, url, status, started):
if not self.verbose:
return
elapsed = int((time.time() - started) * 1000)
sys.stderr.write("[http] %s %s -> %s (%dms)\n" % (method, url, status, elapsed))
sys.stderr.flush()
# -- convenience -----------------------------------------------------
def get(self, path, params=None):
return self.request("GET", path, params=params)
graylog_ops/judge.py
+397
View File
@@ -0,0 +1,397 @@
"""Judgement layer: aggregated Graylog signals -> ranked ban candidates.
This module is **read only**. It answers one question:
"在这个时间窗里，哪些客户端 IP 看起来像在攻击，为什么，建议怎么处理？"
It does not touch the enforcement side (Sangfor AD / WAF / iptables). The
output is a list of ``{ip, score, action, reasons, signals}`` which the next
phase can feed into whatever blacklist API we end up using.
Design notes
------------
* A **fixed number of ``terms`` queries** (one per signal) keeps the load on
Elasticsearch predictable no matter how many IPs show up in the window.
* Signal queries are templates over field names. The report exposes each
signal's query and hit count, so they can be tuned against real data. A
signal that matched 0 IPs usually means the query does not fit the field
mapping (e.g. ``request_uri`` is a keyword-ish analyzed field).
* **Volume alone never bans** (``require_evidence``). A busy but legitimate
client must not be blackholed by a rate threshold alone; at least one
behavioural signal (auth failures / scanner UA / sensitive path / 404
probing) has to be present for a ban recommendation.
"""
import ipaddress
import json
import math
import statistics
from . import detect
from .client import normalize_terms
from .http import GraylogError
DEFAULT_FIELDS = {
"client_ip": "client_ip",
"status": "status",
"uri": "request_uri",
"ua": "http_user_agent",
"method": "request_method",
}
# name, title, required field keys, query template, weight, saturation count
SIGNAL_DEFS = (
{
"name": "auth_fail",
"title": "认证失败",
"requires": ("status",),
"query": lambda fields: "%s:(401 OR 403)" % fields["status"],
"weight": 45.0,
"saturate": 10.0,
},
{
"name": "scanner_ua",
"title": "扫描器 UA",
"requires": ("ua",),
"query": lambda fields: (
"%s:(sqlmap* OR nikto* OR nuclei* OR nmap* OR masscan* OR zgrab* OR "
"wpscan* OR gobuster* OR dirbuster* OR dirsearch* OR acunetix*)"
% fields["ua"]
),
"weight": 50.0,
"saturate": 2.0,
},
{
"name": "sensitive_path",
"title": "敏感路径",
"requires": ("uri",),
# Slashes must be escaped, otherwise Lucene treats /.../ as a regex.
"query": lambda fields: (
"%s:(\\/wp-login* OR \\/phpmyadmin* OR \\/.env* OR \\/actuator* "
"OR \\/cgi-bin* OR \\/manager\\/html*)"
% fields["uri"]
),
"weight": 25.0,
"saturate": 5.0,
},
{
"name": "not_found",
"title": "404 探测",
"requires": ("status",),
"query": lambda fields: "%s:404" % fields["status"],
"weight": 20.0,
"saturate": 20.0,
},
{
"name": "bad_method",
"title": "异常方法",
"requires": ("method",),
"query": lambda fields: "%s:(PUT OR DELETE OR TRACE OR CONNECT)" % fields["method"],
"weight": 10.0,
"saturate": 5.0,
},
)
# URIs containing these hints mean auth failures are likely credential
# attacks rather than an integration with a broken/expired token.
AUTH_PATH_HINTS = ("login", "signin", "auth", "token", "captcha", "oauth", "password")
DEFAULT_THRESHOLDS = {
"min_requests": 20, # ignore IPs with fewer requests in the window
"min_score": 40.0, # below this: ignore
"short_ban_score": 55.0, # >= this: short ban recommendation
"ban_score": 75.0, # >= this: ban recommendation
"rate_weight": 25.0,
"rate_saturate": 200.0, # requests in window for full rate points
"spike_weight": 15.0,
"auth_ratio_weight": 25.0,
"require_evidence": True,
"evidence_signals": ("auth_fail", "scanner_ua", "sensitive_path", "not_found", "bad_method"),
}
# --------------------------------------------------------------------- profile
def load_profile(path):
if not path:
return {}
try:
with open(path, "r", encoding="utf-8") as handle:
data = json.load(handle)
except OSError as exc:
raise SystemExit("无法读取 profile %s: %s" % (path, exc))
except ValueError as exc:
raise SystemExit("profile %s 不是合法 JSON: %s" % (path, exc))
if not isinstance(data, dict):
raise SystemExit("profile %s 必须是 JSON 对象" % path)
return data
def build_fields(profile, args=None):
fields = dict(DEFAULT_FIELDS)
for key, value in (profile.get("fields") or {}).items():
if value:
fields[key] = value
flag_map = {
"client_ip": "field",
"status": "status_field",
"uri": "uri_field",
"ua": "ua_field",
"method": "method_field",
}
for key, attribute in flag_map.items():
value = getattr(args, attribute, None) if args is not None else None
if value:
fields[key] = value
return fields
def build_signals(fields, overrides=None):
overrides = overrides or {}
signals = []
for definition in SIGNAL_DEFS:
if any(not fields.get(key) for key in definition["requires"]):
continue
signal = {
"name": definition["name"],
"title": definition["title"],
"query": definition["query"](fields),
"weight": float(definition["weight"]),
"saturate": float(definition["saturate"]),
}
override = overrides.get(signal["name"]) or {}
if override.get("enabled") is False:
continue
if override.get("query"):
signal["query"] = override["query"]
if override.get("weight") is not None:
signal["weight"] = float(override["weight"])
if override.get("saturate") is not None:
signal["saturate"] = float(override["saturate"])
signals.append(signal)
return signals
# ------------------------------------------------------------------- allowlist
class Allowlist:
"""Never ban infrastructure, internal addresses or explicitly listed nets."""
def __init__(self, entries=None, exclude_internal=True):
self.entries = list(entries or [])
self.exclude_internal = exclude_internal
def reason(self, ip_text):
try:
ip = ipaddress.ip_address(ip_text)
except ValueError:
return None
for network, source in self.entries:
if ip.version == network.version and ip in network:
return "白名单 %s" % source
if self.exclude_internal and detect.is_internal(ip):
return "内部地址（RFC1918/回环/CGNAT）"
return None
def load_allowlist(paths):
entries = []
for path in paths or []:
try:
with open(path, "r", encoding="utf-8") as handle:
lines = handle.readlines()
except OSError as exc:
raise SystemExit("无法读取白名单 %s: %s" % (path, exc))
for number, line in enumerate(lines, 1):
text = line.split("#", 1)[0].strip()
if not text:
continue
try:
network = ipaddress.ip_network(text, strict=False)
except ValueError:
raise SystemExit("白名单 %s:%d 不是合法 IP/CIDR: %r" % (path, number, text))
entries.append((network, "%s:%d" % (path, number)))
return entries
# ----------------------------------------------------------------- collection
def collect_signals(client, field, signals, window_seconds, query="*",
filter_str=None, size=200):
"""Run one terms query per signal plus one for the total request counts."""
term_frame = lambda data: {
row["term"]: int(row["count"] or 0)
for row in normalize_terms(data)
if row["term"] not in (None, "")
}
totals = term_frame(client.terms(
field=field, query=query, size=size,
range_seconds=window_seconds, filter_str=filter_str,
))
per_signal = {}
errors = []
for signal in signals:
try:
data = client.terms(
field=field, query=signal["query"], size=size,
range_seconds=window_seconds, filter_str=filter_str,
)
per_signal[signal["name"]] = term_frame(data)
except GraylogError as exc:
per_signal[signal["name"]] = {}
errors.append({
"signal": signal["name"],
"status": exc.status,
"error": exc.message,
})
return totals, per_signal, errors
# ------------------------------------------------------------------- verify
def verify_candidates(client, candidates, fields, window_seconds,
filter_str=None, limit=5, size=10):
"""Drill into top candidates: is the traffic one automated endpoint poller?
A client that hammers a *single* business API at a steady rate is usually an
integration with a broken/expired token, not an attacker. Banning it breaks
a business system, so such candidates are downgraded to ``observe`` and the
decision is left to a human.
"""
uri_field = fields.get("uri")
client_field = fields.get("client_ip")
if not uri_field or not client_field:
return
for candidate in candidates[:limit]:
if not candidate["action"].startswith("ban"):
continue
query = '%s:"%s"' % (client_field, candidate["ip"])
try:
data = client.terms(
field=uri_field, query=query, size=size,
range_seconds=window_seconds, filter_str=filter_str,
)
except GraylogError as exc:
candidate["verify_error"] = exc.message
continue
rows = [(row["term"], row["count"]) for row in normalize_terms(data)
if row["term"] not in (None, "")]
total = sum(count for _, count in rows)
candidate["verify"] = {
"distinct_uris": len(rows),
"total_requests": total,
"top_uri": rows[0][0] if rows else None,
"top_uri_ratio": round(rows[0][1] / total, 4) if rows and total else 0.0,
}
if not rows or not total:
continue
top_uri, top_count = rows[0]
ratio = top_count / float(total)
login_like = any(hint in str(top_uri).lower() for hint in AUTH_PATH_HINTS)
candidate["verify"]["login_like"] = login_like
if ratio >= 0.9 and len(rows) <= 2 and not login_like:
candidate["verify"]["automation_suspected"] = True
candidate["action"] = "observe"
candidate["score"] = round(min(candidate["score"], 54.0), 1)
candidate["reasons"].append(
"流量集中在单一接口 %s（%.0f%%），疑似业务客户端/自动化任务，已降级为 observe"
% (top_uri, ratio * 100)
)
# -------------------------------------------------------------------- scoring
def score_candidates(totals, per_signal, signals, thresholds, allowlist):
pool = {ip: count for ip, count in totals.items()
if count >= thresholds["min_requests"]}
median = statistics.median(pool.values()) if pool else 0
candidates = []
excluded = []
for ip, requests in pool.items():
reason = allowlist.reason(ip)
if reason:
excluded.append({"ip": ip, "requests": requests, "reason": reason})
continue
points = {}
counts = {}
reasons = []
for signal in signals:
count = per_signal.get(signal["name"], {}).get(ip, 0)
if not count:
continue
counts[signal["name"]] = count
contributed = signal["weight"] * min(1.0, count / signal["saturate"])
points[signal["name"]] = round(contributed, 1)
reasons.append("%s %d 次" % (signal["title"], count))
rate_points = thresholds["rate_weight"] * min(
1.0, requests / float(thresholds["rate_saturate"]))
points["rate"] = round(rate_points, 1)
if median and requests >= max(50, median * 4):
ratio = requests / float(median)
spike = thresholds["spike_weight"] * min(1.0, math.log(ratio, 2) / 6.0)
points["spike"] = round(spike, 1)
reasons.append("请求量是候选中位数的 %.0f 倍" % ratio)
auth = counts.get("auth_fail", 0)
auth_ratio = (auth / float(requests)) if requests else 0.0
if auth >= 10 and auth_ratio >= 0.5:
points["auth_ratio"] = thresholds["auth_ratio_weight"]
reasons.append("失败请求占 %.0f%%" % (auth_ratio * 100))
score = min(100.0, sum(points.values()))
evidence = any(counts.get(name) for name in thresholds["evidence_signals"])
if evidence and score >= thresholds["ban_score"]:
action = "ban_1h"
elif evidence and score >= thresholds["short_ban_score"]:
action = "ban_30m"
elif score >= thresholds["min_score"]:
action = "observe"
else:
action = "ignore"
if not evidence and thresholds["require_evidence"] and action in ("ban_1h", "ban_30m"):
action = "observe"
reasons.append("仅请求量异常、无攻击特征，不自动封禁")
candidates.append({
"ip": ip,
"score": round(score, 1),
"requests": requests,
"action": action,
"evidence": evidence,
"auth_ratio": round(auth_ratio, 4),
"signals": counts,
"points": points,
"reasons": reasons,
})
candidates.sort(key=lambda row: (-row["score"], -row["requests"], row["ip"]))
excluded.sort(key=lambda row: (-row["requests"], row["ip"]))
signal_stats = []
for signal in signals:
matched = per_signal.get(signal["name"], {})
signal_stats.append({
"name": signal["name"],
"title": signal["title"],
"query": signal["query"],
"ips_matched": len(matched),
"requests_matched": sum(matched.values()),
})
return {
"median_requests": median,
"candidate_pool": len(pool),
"candidates": candidates,
"excluded": excluded,
"signal_stats": signal_stats,
}
graylog_ops/notify.py
+167
View File
@@ -0,0 +1,167 @@
"""Webhook notifications for newly convicted malicious IPs.
Supported payload flavours: WeCom (企业微信), DingTalk, Feishu, Slack and a
generic JSON body. Delivery failures are reported but never break detection.
"""
import base64
import hashlib
import re
import hmac
import json
import time
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime
from .geo import flag
DEFAULT_TITLE = "新增恶意 IP"
class Notifier:
def __init__(self, config):
self.kind = (config.get("type") or "generic").lower()
self.url = config.get("url") or ""
if not self.url:
raise ValueError("notify config is missing 'url'")
self.secret = config.get("secret") or ""
self.link_prefix = config.get("link_prefix") or ""
self.timeout = int(config.get("timeout") or 10)
self.title = config.get("title") or DEFAULT_TITLE
def link_for(self, ip):
if not self.link_prefix:
return ""
return self.link_prefix.rstrip("/") + "/#/ip/" + ip
@staticmethod
def local_time(value):
try:
text = str(value).replace("Z", "+00:00")
return datetime.fromisoformat(text).astimezone().strftime("%m-%d %H:%M")
except (ValueError, TypeError):
return str(value)
@staticmethod
def risk_label(score):
if score >= 90:
return "🔴 高危"
if score >= 70:
return "🟠 中危"
return "🟡 关注"
def render(self, entries):
count = len(entries)
heading = self.title if count == 1 else "新增 %d 个恶意 IP" % count
lines = ["### 🚨 %s" % heading, ""]
for index, entry in enumerate(entries, 1):
ip = entry.get("ip", "?")
geo = " · ".join(part for part in
(entry.get("country_name"), entry.get("region")) if part)
flag_text = flag(entry.get("country_code"))
place = ("%s %s" % (flag_text, geo)).strip() if geo else ""
score = float(entry.get("score") or 0)
lead = ("**%d. %s**" % (index, ip)) if count > 1 else ("**%s**" % ip)
lines.append("%s%s" % (lead, (" " + place) if place else ""))
lines.append("评分 **%.1f**（%s）｜请求 **%s** 次"
% (score, self.risk_label(score), entry.get("total") or 0))
if entry.get("first_seen"):
lines.append("首次攻击 **%s**" % self.local_time(entry["first_seen"]))
rules = entry.get("rules_text")
if rules:
lines.append("")
lines.append("**命中类型**")
for part in rules.split("；"):
part = part.strip()
if not part:
continue
part = re.sub(r"\s+(\d+)\s*次$", r" × \1", part)
lines.append("- " + part)
link = self.link_for(ip)
if link:
lines.append("")
lines.append("[🔗 查看详情与证据](%s)" % link)
if index != count:
lines.append("")
lines.append("---")
lines.append("")
return "\n".join(lines)
def signed_url(self, timestamp=None):
"""DingTalk custom robots with 加签 need timestamp + HMAC-SHA256."""
if self.kind != "dingtalk" or not self.secret:
return self.url
stamp = str(timestamp if timestamp is not None
else int(time.time() * 1000))
string_to_sign = stamp + "\n" + self.secret
digest = hmac.new(self.secret.encode("utf-8"),
string_to_sign.encode("utf-8"),
hashlib.sha256).digest()
sign = urllib.parse.quote_plus(base64.b64encode(digest).decode("ascii"))
separator = "&" if "?" in self.url else "?"
return "%s%stimestamp=%s&sign=%s" % (self.url, separator, stamp, sign)
def payload(self, text, title=None):
title = title or self.title
if self.kind == "wecom":
return {"msgtype": "markdown", "markdown": {"content": text}}
if self.kind == "dingtalk":
return {"msgtype": "markdown", "markdown": {"title": title, "text": text}}
if self.kind == "feishu":
return {"msg_type": "text", "content": {"text": text}}
if self.kind == "slack":
return {"text": text}
return {"title": title, "text": text}
def post(self, text, title=None):
body = json.dumps(self.payload(text, title), ensure_ascii=False).encode("utf-8")
request = urllib.request.Request(
self.signed_url(), data=body, method="POST",
headers={"Content-Type": "application/json; charset=utf-8"})
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
reply = response.read().decode("utf-8", "replace")
except urllib.error.HTTPError as exc:
raise RuntimeError("webhook HTTP %s: %s"
% (exc.code, exc.read()[:200].decode("utf-8", "replace")))
except urllib.error.URLError as exc:
raise RuntimeError("webhook unreachable: %s" % exc.reason)
if reply and self.kind in ("wecom", "dingtalk", "feishu"):
try:
parsed = json.loads(reply)
except ValueError:
return reply
code = parsed.get("errcode", parsed.get("code", 0))
if code:
raise RuntimeError("webhook rejected: %s" % reply[:200])
return reply
def send(self, entries):
if not entries:
return ""
return self.post(self.render(entries))
def send_test(self):
return self.post("**%s**\n\n这是一条测试推送，说明通知通道工作正常。"
% self.title, title=self.title + " · 测试")
def load_notifier(path):
with open(path, "r", encoding="utf-8") as handle:
return Notifier(json.load(handle))
def notifier_from_env():
"""Build a notifier from NOTIFY_* environment variables (container mode)."""
import os
url = (os.environ.get("NOTIFY_URL") or "").strip()
if not url:
return None
return Notifier({
"type": os.environ.get("NOTIFY_TYPE", "generic"),
"url": url,
"secret": os.environ.get("NOTIFY_SECRET", ""),
"link_prefix": os.environ.get("NOTIFY_LINK_PREFIX", ""),
"title": os.environ.get("NOTIFY_TITLE", "新增恶意 IP"),
})
graylog_ops/render.py
+77
View File
@@ -0,0 +1,77 @@
"""Terminal rendering helpers (CJK-aware column widths)."""
import unicodedata
def display_width(text):
width = 0
for char in str(text):
if unicodedata.east_asian_width(char) in ("W", "F"):
width += 2
else:
width += 1
return width
def clip(text, limit=60):
text = "" if text is None else str(text)
text = text.replace("\n", " ").replace("\r", " ").replace("\t", " ")
if len(text) <= limit:
return text
return text[: max(0, limit - 1)] + "…"
def table(headers, rows, align=None):
"""Render a plain-text table. *align* is a list of ``l``/``r`` per column."""
rows = [["" if cell is None else str(cell) for cell in row] for row in rows]
if align is None:
align = ["l"] * len(headers)
widths = [display_width(header) for header in headers]
for row in rows:
for index, cell in enumerate(row[: len(widths)]):
widths[index] = max(widths[index], display_width(cell))
def render(cells):
parts = []
for index, cell in enumerate(cells):
padding = widths[index] - display_width(cell)
if align[index] == "r":
parts.append(" " * padding + cell)
else:
parts.append(cell + " " * padding)
return " ".join(parts).rstrip()
lines = [render(headers), " ".join("-" * width for width in widths)]
lines.extend(render(row) for row in rows)
return "\n".join(lines)
def human_bytes(size):
if not isinstance(size, (int, float)):
return str(size)
value = float(size)
for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
if value < 1024 or unit == "TiB":
if unit == "B":
return "%d B" % value
return "%.1f %s" % (value, unit)
value /= 1024.0
return str(size)
def percent(part, total):
if not total:
return "-"
return "%.1f%%" % (100.0 * part / total)
def kv_block(pairs):
"""Render aligned ``key: value`` lines."""
if not pairs:
return ""
width = max(display_width(key) for key, _ in pairs)
lines = []
for key, value in pairs:
padding = " " * (width - display_width(key))
lines.append("%s%s : %s" % (key, padding, "" if value is None else value))
return "\n".join(lines)
graylog_ops/watch.py
+2768
View File
File diff suppressed because it is too large. Load diff
notify.example.json
+7
View File
@@ -0,0 +1,7 @@
{
"type": "dingtalk",
"url": "https://oapi.dingtalk.com/robot/send?access_token=REPLACE_WITH_TOKEN",
"secret": "SEC_REPLACE_WITH_SIGN_SECRET",
"link_prefix": "http://179.253.234.124:12139/s/301b0525d852feacedcbf813a422f6e5/",
"title": "新增恶意 IP"
}
profiles/f5.json
+19
View File
@@ -0,0 +1,19 @@
{
"fields": {
"client_ip": "client_ip",
"status": "status",
"uri": "uri",
"ua": "ua",
"method": "method",
"host": "host"
},
"signals": {
"bad_method": {
"enabled": false
}
},
"thresholds": {
"min_requests": 30,
"min_score": 40.0
}
}
tests/demo.py
+50
View File
@@ -0,0 +1,50 @@
"""Preview the CLI output without a real Graylog instance.
Runs a handful of commands against the bundled fake Graylog 4.0 server so you
can see the exact shape of the tables/JSON before pointing the tool at
edu-master.
python3 tests/demo.py
"""
import io
import os
import sys
from contextlib import redirect_stdout
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.dirname(HERE))
sys.path.insert(0, HERE)
from fake_graylog import JUDGE_MESSAGES, FakeGraylog # noqa: E402
from graylog_ops import cli # noqa: E402
COMMANDS = (
["probe"],
["streams"],
["discover", "--range", "3600"],
["top-ips", "--field", "client_ip", "--query", "status:401", "--size", "5"],
["judge", "--range", "300", "--explain", "3"],
)
def main():
server = FakeGraylog(messages=JUDGE_MESSAGES).start()
base = ["--url", server.url, "--token", "testtoken"]
try:
for command in COMMANDS:
print("=" * 78)
print("$ graylog-ops " + " ".join(command))
print("-" * 78)
captured = io.StringIO()
with redirect_stdout(captured):
cli.main(base + command)
print(captured.getvalue().rstrip())
print()
finally:
server.stop()
return 0
if __name__ == "__main__":
sys.exit(main())
tests/fake_graylog.py
+468
View File
@@ -0,0 +1,468 @@
"""A tiny fake Graylog 4.0 server used by the end-to-end tests.
It intentionally mimics the response envelopes of the real 4.0 API (including
the ``{"messages": [{"message": {...}}]}`` shape of the legacy search API) so
the client code is exercised the same way it will be against edu-master.
"""
import base64
import json
import re
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse
TOKEN = "testtoken"
EXPECTED_AUTH = "Basic " + base64.b64encode(
("%s:token" % TOKEN).encode("utf-8")
).decode("ascii")
VERSION = "4.0.8+6b8c55d"
MESSAGES = [
{
"timestamp": "2026-10-17T15:00:00.000Z",
"source": "10.180.254.17",
"client_ip": "203.0.113.5",
"status": 401,
"request_method": "POST",
"request_uri": "/api/login",
"http_user_agent": "python-requests/2.31",
"response_time": 0.011,
"message": '203.0.113.5 - - "POST /api/login HTTP/1.1" 401 0.011',
},
{
"timestamp": "2026-10-17T15:00:05.000Z",
"source": "10.180.254.17",
"client_ip": "203.0.113.5",
"status": 401,
"request_method": "POST",
"request_uri": "/api/login",
"http_user_agent": "python-requests/2.31",
"response_time": 0.009,
"message": '203.0.113.5 - - "POST /api/login HTTP/1.1" 401 0.009',
},
{
"timestamp": "2026-10-17T15:00:10.000Z",
"source": "10.180.254.17",
"client_ip": "203.0.113.5",
"status": 401,
"request_method": "POST",
"request_uri": "/api/login",
"http_user_agent": "python-requests/2.31",
"response_time": 0.010,
"message": '203.0.113.5 - - "POST /api/login HTTP/1.1" 401 0.010',
},
{
"timestamp": "2026-10-17T15:00:15.000Z",
"source": "10.180.254.17",
"client_ip": "198.51.100.7",
"status": 401,
"request_method": "GET",
"request_uri": "/admin",
"http_user_agent": "Mozilla/5.0",
"response_time": 0.021,
"message": '198.51.100.7 - - "GET /admin HTTP/1.1" 401 0.021',
},
{
"timestamp": "2026-10-17T15:00:20.000Z",
"source": "10.180.254.17",
"client_ip": "198.51.100.7",
"status": 403,
"request_method": "GET",
"request_uri": "/admin",
"http_user_agent": "Mozilla/5.0",
"response_time": 0.018,
"message": '198.51.100.7 - - "GET /admin HTTP/1.1" 403 0.018',
},
{
"timestamp": "2026-10-17T15:00:25.000Z",
"source": "10.180.254.17",
"client_ip": "192.0.2.44",
"status": 404,
"request_method": "GET",
"request_uri": "/wp-login.php",
"http_user_agent": "sqlmap/1.7",
"response_time": 0.006,
"message": '192.0.2.44 - - "GET /wp-login.php HTTP/1.1" 404 0.006',
},
{
"timestamp": "2026-10-17T15:00:30.000Z",
"source": "10.180.254.17",
"client_ip": "192.0.2.44",
"status": 404,
"request_method": "GET",
"request_uri": "/.env",
"http_user_agent": "sqlmap/1.7",
"response_time": 0.005,
"message": '192.0.2.44 - - "GET /.env HTTP/1.1" 404 0.005',
},
{
"timestamp": "2026-10-17T15:00:35.000Z",
"source": "10.180.254.17",
"client_ip": "10.180.2.9",
"status": 200,
"request_method": "GET",
"request_uri": "/health",
"http_user_agent": "kube-probe/1.25",
"response_time": 0.002,
"message": '10.180.2.9 - - "GET /health HTTP/1.1" 200 0.002',
},
]
def _build_judge_messages():
"""Synthetic traffic for the judgement tests (brute force / scanner / legit)."""
rows = []
def add(ip, count, status, uri, agent, method="GET"):
for index in range(count):
rows.append({
"timestamp": "2026-10-17T15:%02d:%02d.000Z" % (index // 60, index % 60),
"source": "10.180.254.17",
"client_ip": ip,
"status": status,
"request_method": method,
"request_uri": uri,
"http_user_agent": agent,
"message": '%s - "%s %s HTTP/1.1" %d' % (ip, method, uri, status),
})
add("203.0.113.66", 40, 401, "/api/login", "python-requests/2.31", "POST")
add("192.0.2.99", 25, 404, "/wp-login.php", "sqlmap/1.7")
add("198.51.100.200", 50, 200, "/index", "Mozilla/5.0")
add("10.180.5.5", 30, 401, "/api/login", "Mozilla/5.0", "POST")
return rows
JUDGE_MESSAGES = _build_judge_messages()
CLAUSE = re.compile(r'^([A-Za-z_][A-Za-z0-9_.]*):(\([^)]*\)|"[^"]*"|[^\s]+)$')
def _clause_values(raw):
raw = raw.strip()
if raw.startswith("(") and raw.endswith(")"):
return [part.strip().strip('"') for part in raw[1:-1].split(" OR ") if part.strip()]
return [raw.strip('"')]
def matches(message, query):
"""Very small subset of Lucene, enough to exercise the judge signal queries."""
query = (query or "*").strip()
if query in ("", "*"):
return True
clause = CLAUSE.match(query)
if clause is None:
return True # unsupported syntax: do not filter (test double only)
field, raw = clause.groups()
value = message.get(field)
if value is None:
return False
text = str(value).lower()
for candidate in _clause_values(raw):
# Mirror Lucene: `foo*` is a prefix match, `\/` is an escaped slash.
needle = candidate.lower().replace("\\/", "/")
if needle.endswith("*"):
if needle[:-1] and text.startswith(needle[:-1]):
return True
continue
if needle and needle in text:
return True
return False
def aggregate(messages, body):
"""Minimal views-pivot implementation (one query, one search type)."""
query = (body.get("queries") or [{}])[0]
query_string = (query.get("query") or {}).get("query_string") or "*"
filtered = [message for message in messages if matches(message, query_string)]
stream_filter = query.get("filter")
stream_ids = set()
if isinstance(stream_filter, dict):
for item in stream_filter.get("filters") or []:
if isinstance(item, dict) and item.get("id"):
stream_ids.add(item["id"])
if stream_ids:
filtered = [message for message in filtered
if set(message.get("streams") or []) & stream_ids]
search_type = (query.get("search_types") or [{}])[0]
series_id = ((search_type.get("series") or [{}])[0]).get("id", "c1")
row_groups = search_type.get("row_groups") or []
column_groups = search_type.get("column_groups") or []
rows = []
total = 0
if row_groups:
field = row_groups[0].get("field")
limit = int(row_groups[0].get("limit") or 10)
counts = {}
for message in filtered:
value = message.get(field)
if value is None:
continue
counts[str(value)] = counts.get(str(value), 0) + 1
total = sum(counts.values())
ordered = sorted(counts.items(), key=lambda item: (-item[1], item[0]))[:limit]
for key, count in ordered:
rows.append({"key": [key], "source": "leaf",
"values": [{"key": [series_id], "value": count,
"rollup": True, "source": "row-leaf"}]})
rows.append({"key": [], "source": "rollup",
"values": [{"key": [series_id], "value": total,
"rollup": True, "source": "row-leaf"}]})
elif column_groups:
buckets = {}
for message in filtered:
stamp = str(message.get("timestamp", ""))
key = stamp[:16] + ":00.000Z"
buckets[key] = buckets.get(key, 0) + 1
total = sum(buckets.values())
for key in sorted(buckets):
rows.append({"key": [], "source": "leaf",
"values": [{"key": [key, series_id], "value": buckets[key],
"rollup": False, "source": "col-leaf"}]})
return {"id": search_type.get("id", "st1"), "type": "pivot",
"total": total, "rows": rows}
class FakeGraylog:
def __init__(self, require_auth=True, messages=None, views_enabled=True):
self.require_auth = require_auth
self.messages = list(MESSAGES if messages is None else messages)
self.views_enabled = views_enabled
self.terms_size_param = "size"
self.requests = []
owner = self
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args): # silence test output
pass
def _send(self, code, payload):
body = json.dumps(payload).encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_POST(self):
parsed = urlparse(self.path)
path = parsed.path
query = {key: values[0] for key, values in parse_qs(parsed.query).items()}
length = int(self.headers.get("Content-Length") or 0)
raw_body = self.rfile.read(length) if length else b""
owner.requests.append((path, query))
if owner.require_auth and self.headers.get("Authorization") != EXPECTED_AUTH:
return self._send(401, {"message": "Unauthorized"})
if not owner.views_enabled:
return self._send(404, {"message": "Sorry, Page Not Found"})
if path == "/api/views/search/sync":
try:
body = json.loads(raw_body or b"{}")
except ValueError:
return self._send(400, {"message": "invalid json"})
result = aggregate(owner.messages, body)
return self._send(200, {
"execution": {"done": True, "cancelled": False,
"completed_exceptionally": False},
"results": {"q1": {"errors": [], "search_types": {"st1": result}}},
"search_id": "mock-search",
})
return self._send(404, {"message": "not found: %s" % path})
def do_GET(self):
parsed = urlparse(self.path)
path = parsed.path
query = {key: values[0] for key, values in parse_qs(parsed.query).items()}
owner.requests.append((path, query))
if owner.require_auth and self.headers.get("Authorization") != EXPECTED_AUTH:
return self._send(401, {"message": "Unauthorized"})
if path == "/api/system":
return self._send(200, {
"version": VERSION,
"cluster_id": "6b8c55d0-dead-beef-0000-000000000001",
"hostname": "edu-master",
"is_processing": True,
"timezone": "Asia/Shanghai",
"locale": "zh_CN",
})
if path == "/api/system/cluster/nodes":
return self._send(200, {"total": 1, "nodes": [{
"node_id": "node-1",
"hostname": "edu-master",
"version": VERSION,
"lifecycle": "RUNNING",
"is_leader": True,
"lb_status": "ALIVE",
}]})
if path == "/api/system/indexer/cluster/health":
return self._send(200, {
"status": "green",
"cluster_name": "graylog",
"number_of_nodes": 1,
})
if path == "/api/system/deflector":
return self._send(200, {"index": "graylog_0", "is_up": True})
if path == "/api/system/fields":
return self._send(200, {"fields": [
{"name": "client_ip", "type": "ip"},
{"name": "status", "type": "long"},
{"name": "request_uri", "type": "string"},
]})
if path == "/api/system/license":
return self._send(200, {"license": {"type": "opensource"}})
if path == "/api/system/inputs":
return self._send(200, {"total": 1, "inputs": [{
"id": "5f0000000000000000000001",
"title": "GELF UDP",
"type": "org.graylog2.inputs.gelf.udp.GELFUDPInput",
"global": True,
"configuration": {"bind_address": "0.0.0.0", "port": 12201},
}]})
if path == "/api/streams":
return self._send(200, {"total": 2, "streams": [
{
"id": "5f00000000000000000000aa",
"title": "业务访问日志",
"description": "nginx access log",
"disabled": False,
"rules": [],
},
{
"id": "000000000000000000000001",
"title": "Default Stream",
"description": "default",
"disabled": False,
"rules": [],
},
]})
if path == "/api/system/indices/index_sets":
return self._send(200, {"total": 1, "index_sets": [{
"id": "5f00000000000000000000bb",
"title": "Default index set",
"index_prefix": "graylog",
"shards": 4,
"replicas": 0,
"rotation_strategy": "time",
"rotation_strategy_configuration": {"type": "time", "period": "P1D"},
"retention_strategy": "delete",
"retention_strategy_configuration": {"max_number_of_indices": 10},
}]})
if path == "/api/system/indexer/indices":
return self._send(200, {"indices": [{
"index_name": "graylog_0",
"index_set_id": "5f00000000000000000000bb",
"size": 1048576,
"begin": "2026-10-17T00:00:00.000Z",
"end": "2026-10-18T00:00:00.000Z",
}]})
if path == "/api/events/definitions":
return self._send(200, {
"total": 1, "count": 1, "page": 1, "per_page": 50,
"event_definitions": [{
"id": "5f00000000000000000000cc",
"title": "401 风暴",
"priority": 3,
"disabled": False,
"config": {"type": "aggregation-v1"},
"notifications": [],
}],
})
if path == "/api/events/notifications":
return self._send(200, {"total": 1, "notifications": [{
"id": "5f00000000000000000000dd",
"title": "ban webhook",
"config": {"type": "http-notification-v1",
"url": "http://127.0.0.1:8080/ban"},
}]})
if path == "/api/events":
return self._send(200, {"total_events": 0, "events": []})
if path == "/api/sidecars":
return self._send(200, {"total": 0, "sidecars": []})
if path in ("/api/system/pipelines/pipeline", "/api/system/pipelines/rule"):
return self._send(200, [])
if path == "/api/views":
return self._send(200, {"total": 0, "views": [], "page": 1, "per_page": 50})
if path == "/api/search/universal/relative":
try:
limit = int(query.get("limit", 100))
except ValueError:
limit = 100
try:
offset = int(query.get("offset", 0))
except ValueError:
offset = 0
matched = [item for item in owner.messages
if matches(item, query.get("query", "*"))]
page = matched[offset:offset + limit]
return self._send(200, {
"messages": [{"message": dict(item), "index": "graylog_0"} for item in page],
"total_results": len(matched),
"time": 3,
"built_query": "mock",
})
if path == "/api/search/universal/terms":
if owner.terms_size_param == "limit" and "size" in query:
return self._send(400, {"message": "Parameter 'size' is not allowed"})
field = query.get("field", "source")
param = "size" if "size" in query else ("limit" if "limit" in query else None)
try:
size = int(query.get(param, 50)) if param else 50
except ValueError:
size = 50
counts = {}
query_string = query.get("query", "*")
for message in owner.messages:
if not matches(message, query_string):
continue
value = message.get(field)
if value is None:
continue
counts[str(value)] = counts.get(str(value), 0) + 1
terms = sorted(
({"term": key, "count": value} for key, value in counts.items()),
key=lambda row: (-row["count"], row["term"]),
)[:size]
return self._send(200, {
"field": field,
"terms": terms,
"missing": 0,
"other": 0,
"total": len(terms),
})
if path == "/api/search/universal/histogram":
return self._send(200, {
"results": {
"2026-10-17T15:00:00.000Z": {"count": 3},
"2026-10-17T15:01:00.000Z": {"count": 5},
},
"interval": "1m",
"time": 2,
})
return self._send(404, {"message": "not found: %s" % path})
self.server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
self.port = self.server.server_address[1]
self.thread = None
@property
def url(self):
return "http://127.0.0.1:%d" % self.port
def start(self):
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
self.thread.start()
return self
def stop(self):
self.server.shutdown()
self.server.server_close()
if self.thread is not None:
self.thread.join(timeout=5)
tests/test_asn.py
+49
View File
@@ -0,0 +1,49 @@
import os
import sys
import tempfile
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.dirname(HERE))
from graylog_ops.asn import AsnDB, build_asn_db, classify_org # noqa: E402
class AsnTests(unittest.TestCase):
def test_build_and_lookup(self):
with tempfile.TemporaryDirectory() as directory:
csv_path = os.path.join(directory, "asn.csv")
with open(csv_path, "w", encoding="utf-8") as handle:
handle.write('1.0.1.0,1.0.3.255,4134,"Chinanet"\n')
handle.write('8.8.8.0,8.8.8.255,15169,"Google LLC"\n')
handle.write('47.0.0.0,47.0.0.255,37963,'
'"Hangzhou Alibaba Advertising Co.,Ltd."\n')
handle.write('2001:db8::,2001:db8::ffff,15169,"Google LLC"\n')
db_path = os.path.join(directory, "asn.db")
count = build_asn_db(csv_path, db_path)
self.assertEqual(count, 3) # IPv6 skipped
asn = AsnDB(db_path)
try:
china = asn.lookup("1.0.2.1")
self.assertEqual(china["brand"], "中国电信")
self.assertEqual(china["type"], "broadband")
aliyun = asn.lookup("47.0.0.1")
self.assertEqual(aliyun["brand"], "阿里云")
self.assertEqual(aliyun["type"], "cloud")
self.assertEqual(aliyun["icon"], "☁️")
self.assertIsNone(asn.lookup("9.9.9.9"))
finally:
asn.close()
def test_classify_org(self):
self.assertEqual(classify_org("China Mobile Communications Corporation")[:2],
("中国移动", "mobile"))
self.assertEqual(classify_org("China Education and Research Network Center")[:2],
("教育网", "education"))
self.assertEqual(classify_org("Wowrack.com")[:2], ("IDC 机房", "idc"))
self.assertEqual(classify_org("Some Unknown ISP")[:2], ("未知", "unknown"))
if __name__ == "__main__":
unittest.main()
tests/test_behavior.py
+103
View File
@@ -0,0 +1,103 @@
import os
import sys
import tempfile
import time
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.dirname(HERE))
from graylog_ops import watch # noqa: E402
NOW = "2026-10-17T16:00:00Z"
def _behavior(ip, verdict="建议封禁", score=75):
return {
"ip": ip, "score": score, "verdict": verdict,
"signals": ["高失败率"], "reason_text": "24 小时内扫描了大量路径。",
"share_note": "", "behavior_type": "目录扫描",
"asn": "4538", "asn_org": "教育网", "shared_type": "education",
"total": 50, "pct4": 60.0, "uri_card": 20, "uri_templates": 15,
"ua_card": 1, "host_card": 2, "coverage": 100.0, "peak_share": 50.0,
"top_uri": "/.env", "top_ua": "curl/8.0",
"country_code": "CN", "country_name": "中国", "region": "北京",
"city": "Beijing",
}
class FusedTests(unittest.TestCase):
def test_feature_and_behavior_merge(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
try:
store.upsert({"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS,
"score": 91.0, "total": 120, "window": 3600,
"rules": {"probe_java": 10},
"reasons": ["探测 10 次"]}, NOW)
store.sync_fused(NOW)
fused = store.fused_items()
self.assertEqual(len(fused), 1)
self.assertEqual(fused[0]["sources"], "feature")
self.assertEqual(fused[0]["first_banned_at"], NOW)
self.assertIn("特征命中", fused[0]["fused_reason"])
# behaviour-only ban is merged
store.replace_behavior_profiles([_behavior("198.51.100.9")])
store.sync_fused("2026-10-17T16:10:00Z")
merged = {item["ip"]: item for item in store.fused_items()}
self.assertIn("198.51.100.9", merged)
self.assertEqual(merged["198.51.100.9"]["sources"], "behavior")
self.assertEqual(merged["203.0.113.9"]["sources"], "feature")
# same IP on both sides becomes "both"
store.upsert({"ip": "198.51.100.9", "tier": watch.TIER_MALICIOUS,
"score": 80.0, "total": 50, "window": 3600,
"rules": {"probe_java": 6},
"reasons": ["探测 6 次"]}, "2026-10-17T16:11:00Z")
store.sync_fused("2026-10-17T16:12:00Z")
merged = {item["ip"]: item for item in store.fused_items()}
self.assertEqual(merged["198.51.100.9"]["sources"], "both")
# watch-only behaviour must not enter the block list
store.replace_behavior_profiles([
_behavior("192.0.2.44", verdict="仅告警（共享出口）")])
store.sync_fused("2026-10-17T16:20:00Z")
ips = {item["ip"] for item in store.fused_items()}
self.assertNotIn("192.0.2.44", ips)
self.assertIn("203.0.113.9", ips) # feature rows stay forever
finally:
store.close()
def test_push_throttle_24h(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
try:
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
store.replace_behavior_profiles([_behavior("198.51.100.7")])
store.sync_fused(now)
# first sync suppresses historical rows; a fresh detection is due
with store.lock, store.conn:
store.conn.execute(
"UPDATE ip_fused SET last_pushed_at=NULL WHERE ip=?",
("198.51.100.7",))
due = store.due_pushes(active_hours=24, cooldown_hours=24)
self.assertEqual([row["ip"] for row in due], ["198.51.100.7"])
store.mark_pushed(["198.51.100.7"], now)
self.assertEqual(store.due_pushes(24, 24), []) # throttled
old = time.strftime("%Y-%m-%dT%H:%M:%SZ",
time.gmtime(time.time() - 25 * 3600))
with store.lock, store.conn:
store.conn.execute(
"UPDATE ip_fused SET last_pushed_at=? WHERE ip=?",
(old, "198.51.100.7"))
due = store.due_pushes(active_hours=24, cooldown_hours=24)
self.assertEqual([row["ip"] for row in due], ["198.51.100.7"])
finally:
store.close()
if __name__ == "__main__":
unittest.main()
tests/test_detect.py
+127
View File
@@ -0,0 +1,127 @@
"""Unit tests for the client-IP field heuristics."""
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from graylog_ops import detect # noqa: E402
def message(client_ip, source="10.180.254.17", **extra):
payload = {
"timestamp": "2026-10-17T15:00:00.000Z",
"source": source,
"client_ip": client_ip,
"status": 401,
"request_uri": "/api/login",
"message": '%s - - "POST /api/login HTTP/1.1" 401 0.01' % client_ip,
}
payload.update(extra)
return payload
class AsIpTests(unittest.TestCase):
def test_ipv4(self):
self.assertEqual(str(detect.as_ip("203.0.113.5")), "203.0.113.5")
def test_ipv6(self):
self.assertEqual(str(detect.as_ip("2001:db8::1")), "2001:db8::1")
def test_bracketed_ipv6_with_port(self):
self.assertEqual(str(detect.as_ip("[2001:db8::1]:443")), "2001:db8::1")
def test_ipv4_with_port(self):
self.assertEqual(str(detect.as_ip("192.0.2.44:8080")), "192.0.2.44")
def test_rejects_cidr(self):
self.assertIsNone(detect.as_ip("10.0.0.0/8"))
def test_rejects_hostname(self):
self.assertIsNone(detect.as_ip("edu-master"))
def test_rejects_bad_octet(self):
self.assertIsNone(detect.as_ip("999.1.1.1"))
def test_rejects_empty_and_number(self):
self.assertIsNone(detect.as_ip(""))
self.assertIsNone(detect.as_ip(12345))
class AnalyseTests(unittest.TestCase):
def test_empty_sample(self):
report = detect.analyse_messages([])
self.assertEqual(report["sampled"], 0)
self.assertIsNone(report["recommendation"]["field"])
self.assertEqual(report["recommendation"]["confidence"], "none")
def test_recommends_client_ip_over_source(self):
messages = [message(ip) for ip in (
"203.0.113.5", "203.0.113.5", "203.0.113.5",
"198.51.100.7", "198.51.100.7", "192.0.2.44", "192.0.2.44",
"10.180.2.9",
)]
report = detect.analyse_messages(messages)
self.assertEqual(report["recommendation"]["field"], "client_ip")
self.assertEqual(report["recommendation"]["confidence"], "high")
candidates = {row["field"]: row for row in report["candidates"]}
self.assertIn("source", candidates)
self.assertNotIn("message", candidates)
self.assertLess(candidates["source"]["score"], candidates["client_ip"]["score"])
# The constant `source` must not outrank the varying client address.
self.assertEqual(candidates["source"]["distinct_ips"], 1)
def test_warns_when_source_is_a_proxy(self):
messages = [message("203.0.113.%d" % index) for index in range(1, 7)]
report = detect.analyse_messages(messages)
joined = " ".join(report["recommendation"]["warnings"])
self.assertIn("source", joined)
def test_private_ratio_note(self):
messages = [message("10.180.2.%d" % index) for index in range(1, 7)]
report = detect.analyse_messages(messages)
best = report["recommendation"]["field"]
self.assertEqual(best, "client_ip")
joined = " ".join(report["recommendation"]["warnings"])
self.assertIn("内网", joined)
def test_no_ip_field_at_all(self):
messages = [{"source": "edu-master", "message": "no address here"}] * 3
report = detect.analyse_messages(messages)
self.assertIsNone(report["recommendation"]["field"])
self.assertEqual(report["candidates"], [])
def test_http_fields_detected(self):
report = detect.analyse_messages([message("203.0.113.5")])
self.assertIn("status", report["http_fields"])
self.assertIn("uri", report["http_fields"])
class InternalAddressTests(unittest.TestCase):
"""`is_internal` must not use ipaddress.is_private: that flag also covers
documentation ranges such as 203.0.113.0/24."""
def test_rfc1918(self):
self.assertTrue(detect.is_internal(detect.as_ip("10.1.2.3")))
self.assertTrue(detect.is_internal(detect.as_ip("172.16.5.5")))
self.assertTrue(detect.is_internal(detect.as_ip("192.168.1.1")))
def test_documentation_range_is_not_internal(self):
self.assertFalse(detect.is_internal(detect.as_ip("203.0.113.5")))
self.assertFalse(detect.is_internal(detect.as_ip("2001:db8::1")))
def test_loopback_link_local_and_cgnat(self):
self.assertTrue(detect.is_internal(detect.as_ip("127.0.0.1")))
self.assertTrue(detect.is_internal(detect.as_ip("169.254.1.1")))
self.assertTrue(detect.is_internal(detect.as_ip("100.64.1.1")))
self.assertTrue(detect.is_internal(detect.as_ip("::1")))
def test_ipv6_unique_local(self):
self.assertTrue(detect.is_internal(detect.as_ip("fd00::1")))
self.assertFalse(detect.is_internal(detect.as_ip("2400:cb00::1")))
if __name__ == "__main__":
unittest.main()
tests/test_e2e.py
+212
View File
@@ -0,0 +1,212 @@
"""End-to-end tests: run the real CLI against the fake Graylog server."""
import io
import json
import os
import sys
import tempfile
import unittest
from contextlib import redirect_stderr, redirect_stdout
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
sys.path.insert(0, ROOT)
sys.path.insert(0, HERE)
from fake_graylog import MESSAGES, FakeGraylog # noqa: E402
from graylog_ops import cli # noqa: E402
class CliEndToEndTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.server = FakeGraylog().start()
@classmethod
def tearDownClass(cls):
cls.server.stop()
def setUp(self):
self.server.terms_size_param = "size"
self.server.requests.clear()
def run_cli(self, *argv, json_mode=True):
args = ["--url", self.server.url, "--token", "testtoken"]
if json_mode:
args.append("--json")
args.extend(argv)
stdout, stderr = io.StringIO(), io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
code = cli.main(args)
return code, stdout.getvalue(), stderr.getvalue()
# ------------------------------------------------------------- probe
def test_probe_json(self):
code, out, err = self.run_cli("probe")
self.assertEqual(code, 0, err)
data = json.loads(out)
self.assertIn("4.0.8", data["system"]["version"])
self.assertEqual(data["indexer"]["status"], "green")
failed = [check["name"] for check in data["checks"] if not check["ok"]]
self.assertEqual(failed, [], "unexpected failing endpoints: %s" % failed)
def test_probe_human_output(self):
code, out, err = self.run_cli("probe", json_mode=False)
self.assertEqual(code, 0, err)
self.assertIn("检查项", out)
self.assertIn("4.0.8", out)
self.assertIn("全部", out)
# ---------------------------------------------------------- discover
def test_discover_finds_client_ip(self):
code, out, err = self.run_cli("discover", "--range", "3600")
self.assertEqual(code, 0, err)
data = json.loads(out)
self.assertEqual(data["sampled"], len(MESSAGES))
self.assertEqual(data["recommendation"]["field"], "client_ip")
self.assertEqual(data["recommendation"]["confidence"], "high")
self.assertNotIn("message", [row["field"] for row in data["candidates"]])
self.assertIn("status", data["http_fields"])
self.assertIn("uri", data["http_fields"])
def test_discover_stream_title_resolution(self):
code, out, err = self.run_cli("discover", "--stream", "业务访问")
self.assertEqual(code, 0, err)
data = json.loads(out)
self.assertEqual(data["stream_id"], "5f00000000000000000000aa")
def test_discover_unknown_stream_fails_cleanly(self):
stdout, stderr = io.StringIO(), io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
with self.assertRaises(SystemExit) as ctx:
cli.main(["--url", self.server.url, "--token", "testtoken",
"discover", "--stream", "does-not-exist"])
self.assertNotEqual(ctx.exception.code, 0)
# ------------------------------------------------------------- terms
def test_top_ips(self):
code, out, err = self.run_cli("top-ips", "--field", "client_ip", "--size", "10")
self.assertEqual(code, 0, err)
data = json.loads(out)
self.assertEqual(data["field"], "client_ip")
self.assertEqual(len(data["terms"]), 4)
self.assertEqual(data["terms"][0]["term"], "203.0.113.5")
self.assertEqual(data["terms"][0]["count"], 3)
def test_terms_use_views_backend(self):
"""4.x removed the legacy terms endpoint, so aggregation must POST."""
code, out, err = self.run_cli("top-ips", "--field", "client_ip")
self.assertEqual(code, 0, err)
self.assertTrue(json.loads(out)["terms"])
posts = [path for path, _ in self.server.requests
if path == "/api/views/search/sync"]
self.assertTrue(posts, "expected a views search POST")
def test_legacy_fallback_when_views_missing(self):
"""Graylog 3.x has the legacy terms endpoint but no views API."""
server = FakeGraylog(views_enabled=False).start()
server.terms_size_param = "limit"
try:
stdout, stderr = io.StringIO(), io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
code = cli.main(["--url", server.url, "--token", "testtoken", "--json",
"terms", "--field", "client_ip"])
self.assertEqual(code, 0, stderr.getvalue())
self.assertTrue(json.loads(stdout.getvalue())["terms"])
term_calls = [query for path, query in server.requests
if path.endswith("/universal/terms")]
self.assertEqual(len(term_calls), 2)
self.assertIn("size", term_calls[0])
self.assertIn("limit", term_calls[1])
self.assertNotIn("size", term_calls[1])
finally:
server.stop()
# ----------------------------------------------------------- snapshot
def test_snapshot(self):
with tempfile.TemporaryDirectory() as directory:
code, out, err = self.run_cli("snapshot", "--dir", directory, "--sample", "5")
self.assertEqual(code, 0, err)
data = json.loads(out)
failed = [row for row in data["resources"] if row["status"] != "ok"]
self.assertEqual(failed, [])
self.assertTrue(os.path.exists(os.path.join(directory, "manifest.json")))
self.assertTrue(os.path.exists(os.path.join(directory, "system.json")))
self.assertEqual(data["sampled_messages"], 5)
messages_path = os.path.join(directory, "messages.jsonl")
with open(messages_path, "r", encoding="utf-8") as handle:
lines = [line for line in handle if line.strip()]
self.assertEqual(len(lines), 5)
json.loads(lines[0])
# -------------------------------------------------------------- raw
def test_raw_endpoint(self):
code, out, err = self.run_cli("raw", "/api/system/inputs")
self.assertEqual(code, 0, err)
data = json.loads(out)
self.assertIn("inputs", data)
# ---------------------------------------------------------- messages
def test_messages_jsonl(self):
code, out, err = self.run_cli("messages", "--limit", "3", "--jsonl")
self.assertEqual(code, 0, err)
lines = [line for line in out.splitlines() if line.strip()]
self.assertEqual(len(lines), 3)
for line in lines:
json.loads(line)
# --------------------------------------------------------------- auth
def test_bad_token_returns_1(self):
stdout, stderr = io.StringIO(), io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
code = cli.main(["--url", self.server.url, "--token", "wrong", "probe"])
self.assertEqual(code, 1)
self.assertIn("401", stderr.getvalue())
def test_missing_config_returns_2(self):
saved = {key: os.environ.pop(key, None) for key in (
"GRAYLOG_URL", "GRAYLOG_TOKEN", "GRAYLOG_USER", "GRAYLOG_PASSWORD")}
original_cwd = os.getcwd()
try:
# Run from an empty directory: a real ./graylog-ops.json in the
# project root must not leak into this test.
with tempfile.TemporaryDirectory() as directory:
os.chdir(directory)
stderr = io.StringIO()
with redirect_stderr(stderr):
code = cli.main(["system"])
self.assertEqual(code, 2)
self.assertIn("配置错误", stderr.getvalue())
finally:
os.chdir(original_cwd)
for key, value in saved.items():
if value is not None:
os.environ[key] = value
# --------------------------------------------------------------- collect
def test_collect_bundle(self):
with tempfile.TemporaryDirectory() as directory:
code, out, err = self.run_cli("collect", "--dir", directory, "--range", "3600")
self.assertEqual(code, 0, err)
for name in ("probe.json", "discover.json", "judge.json",
"profile.json", "SUMMARY.txt"):
self.assertTrue(os.path.exists(os.path.join(directory, name)), name)
self.assertTrue(os.path.isdir(os.path.join(directory, "snapshot")))
data = json.loads(out)
self.assertTrue(all(step["ok"] for step in data["steps"]), data["steps"])
self.assertEqual(data["recommendation"]["field"], "client_ip")
# ---------------------------------------------------------------- out
def test_out_file(self):
with tempfile.TemporaryDirectory() as directory:
path = os.path.join(directory, "ips.json")
code, out, err = self.run_cli("top-ips", "--field", "client_ip", "--out", path)
self.assertEqual(code, 0, err)
self.assertTrue(os.path.exists(path))
with open(path, "r", encoding="utf-8") as handle:
payload = json.load(handle)
self.assertEqual(payload["field"], "client_ip")
if __name__ == "__main__":
unittest.main()
tests/test_geo.py
+48
View File
@@ -0,0 +1,48 @@
import os
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from graylog_ops.geo import GeoIP, build_geo_db, country_name, flag, region_name
class GeoIPTests(unittest.TestCase):
def test_build_and_lookup(self):
with tempfile.TemporaryDirectory() as directory:
csv_path = os.path.join(directory, "city.csv")
with open(csv_path, "w", encoding="utf-8") as fh:
fh.write("1.0.0.0,1.0.0.255,OC,AU,Queensland,\"South Brisbane\",0,0\n")
fh.write("1.0.1.0,1.0.3.255,AS,CN,Fujian,Wenquan,0,0\n")
fh.write("8.8.8.0,8.8.8.255,NA,US,California,\"Mountain View\",0,0\n")
fh.write("2001:db8::,2001:db8::ffff,NA,US,California,X,0,0\n")
db_path = os.path.join(directory, "geo.db")
count = build_geo_db(csv_path, db_path)
self.assertEqual(count, 3) # IPv6 range skipped
geo = GeoIP(db_path)
try:
cn = geo.lookup("1.0.2.1")
self.assertEqual(cn["country_code"], "CN")
self.assertEqual(cn["country_name"], "中国")
self.assertEqual(cn["region"], "福建")
self.assertEqual(geo.lookup("8.8.8.8")["country_name"], "美国")
self.assertIsNone(geo.lookup("9.9.9.9"))
self.assertIsNone(geo.lookup("not-an-ip"))
finally:
geo.close()
def test_helpers(self):
self.assertEqual(flag("CN"), "\U0001F1E8\U0001F1F3")
self.assertEqual(flag("ZZ"), "")
self.assertEqual(flag(""), "")
self.assertEqual(country_name("CN"), "中国")
self.assertEqual(country_name("US"), "美国")
self.assertEqual(country_name("XX"), "XX")
self.assertEqual(region_name("CN", "Guangdong"), "广东")
self.assertEqual(region_name("US", "California"), "California")
if __name__ == "__main__":
unittest.main()
tests/test_judge.py
+168
View File
@@ -0,0 +1,168 @@
"""Tests for the judgement layer (ban candidate scoring)."""
import io
import json
import os
import sys
import tempfile
import unittest
from contextlib import redirect_stderr, redirect_stdout
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
sys.path.insert(0, ROOT)
sys.path.insert(0, HERE)
from fake_graylog import JUDGE_MESSAGES, FakeGraylog # noqa: E402
from graylog_ops import cli # noqa: E402
class JudgeTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.server = FakeGraylog(messages=JUDGE_MESSAGES).start()
@classmethod
def tearDownClass(cls):
cls.server.stop()
def run_cli(self, *argv, json_mode=True):
args = ["--url", self.server.url, "--token", "testtoken"]
if json_mode:
args.append("--json")
args.extend(argv)
stdout, stderr = io.StringIO(), io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
code = cli.main(args)
return code, stdout.getvalue(), stderr.getvalue()
def judged(self, *argv):
code, out, err = self.run_cli("judge", "--range", "300", *argv)
self.assertEqual(code, 0, err)
return json.loads(out)
# ------------------------------------------------------------ behaviour
def test_bruteforce_client_is_recommended_for_ban(self):
data = self.judged()
rows = {row["ip"]: row for row in data["candidates"]}
self.assertIn("203.0.113.66", rows)
row = rows["203.0.113.66"]
self.assertEqual(row["action"], "ban_1h")
self.assertGreaterEqual(row["score"], 75)
self.assertIn("auth_fail", row["signals"])
self.assertTrue(any("失败请求" in reason for reason in row["reasons"]))
def test_scanner_client_is_recommended_for_ban(self):
data = self.judged()
row = {item["ip"]: item for item in data["candidates"]}["192.0.2.99"]
self.assertEqual(row["action"], "ban_1h")
self.assertIn("scanner_ua", row["signals"])
self.assertIn("sensitive_path", row["signals"])
def test_busy_but_legitimate_client_is_not_banned(self):
data = self.judged("--all")
row = {item["ip"]: item for item in data["candidates"]}["198.51.100.200"]
self.assertIn(row["action"], ("ignore", "observe"))
self.assertFalse(row["evidence"])
def test_internal_addresses_are_excluded_by_default(self):
data = self.judged()
reasons = {row["ip"]: row["reason"] for row in data["excluded"]}
self.assertIn("10.180.5.5", reasons)
self.assertIn("内部", reasons["10.180.5.5"])
self.assertNotIn("10.180.5.5", {row["ip"] for row in data["candidates"]})
def test_include_internal_flag(self):
data = self.judged("--include-internal", "--all")
ips = {row["ip"] for row in data["candidates"]}
self.assertIn("10.180.5.5", ips)
def test_allowlist_file(self):
with tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False) as handle:
handle.write("# 深信服 AD / VIP\n10.180.254.17\n203.0.113.66\n")
path = handle.name
try:
data = self.judged("--allow", path)
excluded = {row["ip"]: row["reason"] for row in data["excluded"]}
self.assertIn("203.0.113.66", excluded)
self.assertIn("白名单", excluded["203.0.113.66"])
self.assertNotIn("203.0.113.66", {row["ip"] for row in data["candidates"]})
finally:
os.unlink(path)
def test_explain_attaches_samples(self):
data = self.judged("--explain", "2")
sampled = [row for row in data["candidates"] if row.get("samples")]
self.assertTrue(sampled)
self.assertLessEqual(len(sampled[0]["samples"]), 3)
def test_verify_downgrades_automation_client(self):
"""A steady single-endpoint poller with 403s must not be auto-banned."""
automation = [{
"timestamp": "2026-10-17T15:%02d:%02d.000Z" % (index // 60, index % 60),
"source": "10.180.254.17",
"client_ip": "203.0.113.77",
"status": 403,
"request_method": "POST",
"request_uri": "/prod-api/biz/liveCourseChapter/msgSubscription",
"http_user_agent": "okhttp/4.9",
"message": '203.0.113.77 - "POST /prod-api/biz/x" 403',
} for index in range(40)]
server = FakeGraylog(messages=list(JUDGE_MESSAGES) + automation).start()
try:
stdout, stderr = io.StringIO(), io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
code = cli.main(["--url", server.url, "--token", "testtoken",
"--json", "judge", "--range", "300",
"--verify", "10", "--all"])
self.assertEqual(code, 0, stderr.getvalue())
data = json.loads(stdout.getvalue())
row = {item["ip"]: item for item in data["candidates"]}["203.0.113.77"]
self.assertEqual(row["action"], "observe")
self.assertTrue(row["verify"]["automation_suspected"])
self.assertFalse(row["verify"]["login_like"])
finally:
server.stop()
def test_verify_keeps_login_bruteforce(self):
"""Failures against a login path stay bannable even if concentrated."""
data = self.judged("--verify", "5")
row = {item["ip"]: item for item in data["candidates"]}["203.0.113.66"]
self.assertEqual(row["action"], "ban_1h")
self.assertTrue(row["verify"]["login_like"])
def test_profile_roundtrip(self):
with tempfile.TemporaryDirectory() as directory:
profile = os.path.join(directory, "profile.json")
code, _, err = self.run_cli("discover", "--range", "3600",
"--save-profile", profile)
self.assertEqual(code, 0, err)
with open(profile, "r", encoding="utf-8") as handle:
saved = json.load(handle)
self.assertEqual(saved["fields"]["client_ip"], "client_ip")
self.assertEqual(saved["fields"]["status"], "status")
self.assertEqual(saved["fields"]["uri"], "request_uri")
data = self.judged("--profile", profile)
self.assertTrue(data["candidates"])
self.assertEqual(data["field"], "client_ip")
# -------------------------------------------------------------- output
def test_human_output(self):
code, out, err = self.run_cli("judge", json_mode=False)
self.assertEqual(code, 0, err)
self.assertIn("信号命中", out)
self.assertIn("本命令只读", out)
self.assertIn("ban_1h", out)
def test_signal_queries_are_reported(self):
data = self.judged()
names = {stat["name"]: stat for stat in data["signal_stats"]}
self.assertIn("auth_fail", names)
self.assertGreater(names["auth_fail"]["ips_matched"], 0)
self.assertGreater(names["scanner_ua"]["ips_matched"], 0)
self.assertIn("query", names["auth_fail"])
if __name__ == "__main__":
unittest.main()
tests/test_notify.py
+209
View File
@@ -0,0 +1,209 @@
import json
import os
import sys
import tempfile
import threading
import unittest
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
for path in (ROOT, HERE):
if path not in sys.path:
sys.path.insert(0, path)
from graylog_ops import watch # noqa: E402
from graylog_ops.behavior import notify_fused # noqa: E402
from graylog_ops.notify import Notifier, load_notifier # noqa: E402
from test_watch import _StubClient # noqa: E402
class _CaptureNotifier:
def __init__(self):
self.batches = []
self.posts = []
def send(self, entries):
self.batches.append(entries)
def link_for(self, ip):
return ""
def post(self, text):
self.posts.append(text)
class _Receiver(BaseHTTPRequestHandler):
received = []
def log_message(self, *args):
pass
def do_POST(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length).decode("utf-8")
_Receiver.received.append(body)
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(b'{"errcode":0}')
class NotifierTests(unittest.TestCase):
def _config(self, kind="generic", url="http://127.0.0.1:1/hook"):
return {"type": kind, "url": url, "link_prefix": "http://example/panel/"}
def test_payload_per_kind(self):
text = "**新增恶意 IP**\n\n1.2.3.4"
self.assertEqual(
Notifier(self._config("wecom")).payload(text),
{"msgtype": "markdown", "markdown": {"content": text}})
self.assertEqual(
Notifier(self._config("dingtalk")).payload(text)["msgtype"], "markdown")
self.assertEqual(
Notifier(self._config("feishu")).payload(text)["msg_type"], "text")
self.assertEqual(Notifier(self._config("slack")).payload(text), {"text": text})
generic = Notifier(self._config("generic")).payload(text)
self.assertEqual(generic["text"], text)
self.assertEqual(generic["title"], "新增恶意 IP")
def test_render_includes_geo_rules_and_link(self):
notifier = Notifier(self._config())
text = notifier.render([{
"ip": "203.0.113.5", "country_name": "美国", "region": "California",
"score": 91.0, "total": 120, "first_seen": "2026-10-17T16:00:00Z",
"rules_text": "Spring/中间件探测 4 次",
}])
self.assertIn("203.0.113.5", text)
self.assertIn("美国 · California", text)
self.assertIn("Spring/中间件探测", text)
self.assertIn("http://example/panel/#/ip/203.0.113.5", text)
def test_send_to_http_endpoint(self):
_Receiver.received = []
server = ThreadingHTTPServer(("127.0.0.1", 0), _Receiver)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
url = "http://127.0.0.1:%d/hook" % server.server_address[1]
notifier = Notifier(self._config("generic", url))
notifier.send_test()
self.assertEqual(len(_Receiver.received), 1)
payload = json.loads(_Receiver.received[0])
self.assertIn("测试推送", payload["text"])
finally:
server.shutdown()
server.server_close()
def test_dingtalk_signing(self):
import base64
import hashlib
import hmac
import urllib.parse
secret = "SECtest"
url = "https://oapi.dingtalk.com/robot/send?access_token=abc"
notifier = Notifier({"type": "dingtalk", "url": url, "secret": secret})
stamp = 1700000000000
expected = base64.b64encode(hmac.new(
secret.encode(), ("%d\n%s" % (stamp, secret)).encode(),
hashlib.sha256).digest()).decode()
signed = notifier.signed_url(stamp)
self.assertIn("timestamp=1700000000000", signed)
self.assertIn("sign=" + urllib.parse.quote_plus(expected), signed)
self.assertTrue(signed.startswith(url + "&"))
def test_dingtalk_without_secret_keeps_url(self):
url = "https://oapi.dingtalk.com/robot/send?access_token=abc"
self.assertEqual(Notifier({"type": "dingtalk", "url": url}).signed_url(1), url)
def test_dingtalk_signature(self):
import base64
import hashlib
import hmac
import urllib.parse
notifier = Notifier({
"type": "dingtalk",
"url": "https://oapi.dingtalk.com/robot/send?access_token=abc",
"secret": "SECtest-secret",
})
url = notifier.signed_url()
self.assertIn("timestamp=", url)
self.assertIn("sign=", url)
query = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
timestamp = query["timestamp"][0]
expected = base64.b64encode(hmac.new(
b"SECtest-secret",
("%s\nSECtest-secret" % timestamp).encode("utf-8"),
hashlib.sha256).digest()).decode("ascii")
self.assertEqual(query["sign"][0], expected)
unsigned = Notifier({"type": "dingtalk",
"url": "https://example/hook?access_token=abc"})
self.assertEqual(unsigned.signed_url(),
"https://example/hook?access_token=abc")
def test_load_notifier(self):
with tempfile.TemporaryDirectory() as directory:
path = os.path.join(directory, "notify.json")
json.dump(self._config("wecom"), open(path, "w", encoding="utf-8"))
notifier = load_notifier(path)
self.assertEqual(notifier.kind, "wecom")
class UpsertStatusTests(unittest.TestCase):
def test_upsert_returns_new_upgraded_unchanged(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
try:
base = {"ip": "203.0.113.9", "total": 10, "rules": {},
"reasons": []}
self.assertEqual(store.upsert(dict(base, tier=watch.TIER_OBSERVE,
score=5.0),
"2026-10-17T16:00:00Z"), "new")
self.assertEqual(store.upsert(dict(base, tier=watch.TIER_OBSERVE,
score=5.0),
"2026-10-17T16:01:00Z"), "unchanged")
self.assertEqual(store.upsert(dict(base, tier=watch.TIER_MALICIOUS,
score=80.0),
"2026-10-17T16:02:00Z"), "upgraded")
self.assertEqual(store.upsert(dict(base, tier=watch.TIER_MALICIOUS,
score=80.0),
"2026-10-17T16:03:00Z"), "unchanged")
finally:
store.close()
class DetectorNotificationTests(unittest.TestCase):
def test_new_malicious_notifies_once(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
try:
probe = watch.Detector(None, store, dict(watch.DEFAULT_FIELDS))
queries = {rule["name"]: rule["query"] for rule in probe.rules}
by_query = {"*": [("203.0.113.5", 40)],
queries["probe_java"]: [("203.0.113.5", 40)]}
for name, query in queries.items():
by_query.setdefault(query, [])
capture = _CaptureNotifier()
detector = watch.Detector(_StubClient(by_query), store,
dict(watch.DEFAULT_FIELDS), window=300)
detector.run_once()
store.sync_fused()
# first sync suppresses historical rows; make it a fresh detection
with store.lock, store.conn:
store.conn.execute(
"UPDATE ip_fused SET last_pushed_at=NULL")
notify_fused(store, capture)
self.assertEqual(len(capture.posts), 1)
self.assertIn("203.0.113.5", capture.posts[0])
detector.run_once()
store.sync_fused()
notify_fused(store, capture)
self.assertEqual(len(capture.posts), 1) # 24h cooldown
finally:
store.close()
if __name__ == "__main__":
unittest.main()
tests/test_watch.py
+436
View File
@@ -0,0 +1,436 @@
"""Tests for the continuous detector, the SQLite store and the dashboard."""
import io
import json
import os
import sys
import tempfile
import threading
import time
import unittest
import urllib.error
import urllib.request
from http.server import ThreadingHTTPServer
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
sys.path.insert(0, ROOT)
from graylog_ops import watch # noqa: E402
class ClassifyTests(unittest.TestCase):
def test_sensitive_probe_is_malicious(self):
tier, score, reasons = watch.classify(44, {"probe_java": 44})
self.assertEqual(tier, watch.TIER_MALICIOUS)
self.assertGreaterEqual(score, 60)
self.assertTrue(any("探测" in reason for reason in reasons))
def test_single_sensitive_hit_is_enough(self):
tier, score, _ = watch.classify(18, {"probe_java": 18})
self.assertEqual(tier, watch.TIER_MALICIOUS)
def test_dotbot_crawler_is_malicious(self):
tier, score, reasons = watch.classify(352, {"bot_ua": 352, "not_found": 176})
self.assertEqual(tier, watch.TIER_MALICIOUS)
self.assertTrue(any("404" in reason for reason in reasons))
def test_login_burst_alone_is_only_observed(self):
"""Students logging in repeatedly must not look suspicious."""
tier, score, _ = watch.classify(2253, {"login_burst": 660})
self.assertEqual(tier, watch.TIER_OBSERVE)
def test_login_attack_plus_burst_is_malicious(self):
tier, score, _ = watch.classify(666, {"login_attack": 333, "login_burst": 666})
self.assertEqual(tier, watch.TIER_MALICIOUS)
def test_busy_clean_ip_scores_zero(self):
tier, score, reasons = watch.classify(5000, {})
self.assertEqual(score, 0.0)
self.assertEqual(reasons, [])
def test_404_ratio_needs_volume(self):
_, score, _ = watch.classify(10, {"not_found": 5})
self.assertEqual(score, 0.0)
class _StubClient:
"""Returns canned counts keyed by the exact query string."""
def __init__(self, by_query, totals=None):
self.by_query = by_query
self.totals = totals or {}
def terms(self, field, query, size, range_seconds, filter_str=None):
if query.startswith("client_ip:("):
ips = [part.strip().strip('"')
for part in query[len("client_ip:("):-1].split(" OR ")]
items = [(ip, self.totals[ip]) for ip in ips if ip in self.totals]
else:
items = self.by_query.get(query, [])
return {"terms": [{"term": ip, "count": count} for ip, count in items],
"total": sum(count for _, count in items)}
def search_views_messages(self, query="*", range_seconds=300, limit=5, offset=0,
filter_str=None, from_ts=None, to_ts=None,
timeout_ms=60000, sort_order="DESC"):
return []
class StoreTests(unittest.TestCase):
def test_upsert_list_forget(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
try:
record = {"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS,
"score": 88.0, "total": 120,
"rules": {"sensitive_probe": 4}, "reasons": ["敏感路径探测 4 次"]}
store.upsert(record, "2026-10-17T16:00:00Z")
store.upsert(dict(record, score=90.0), "2026-10-17T16:01:00Z")
rows = store.list()
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["score"], 90.0)
self.assertEqual(rows[0]["hits"], 2)
self.assertEqual(rows[0]["rules"]["sensitive_probe"], 4)
self.assertEqual(store.stats()["by_tier"][watch.TIER_MALICIOUS], 1)
self.assertEqual(store.stats()["events"], 1)
store.upsert({"ip": "198.51.100.7", "tier": watch.TIER_SUSPICIOUS,
"score": 40.0, "total": 10, "rules": {}, "reasons": []},
"2026-10-17T16:02:00Z")
self.assertEqual(len(store.list(tier=watch.TIER_SUSPICIOUS)), 1)
finally:
store.close()
class PermanentRetentionTests(unittest.TestCase):
def _store(self, directory):
return watch.Store(os.path.join(directory, "ips.db"))
def test_ever_malicious_survives_downgrade(self):
with tempfile.TemporaryDirectory() as directory:
store = self._store(directory)
try:
base = {"ip": "203.0.113.9", "total": 10, "rules": {}, "reasons": []}
store.upsert(dict(base, tier=watch.TIER_MALICIOUS, score=80.0),
"2026-10-17T16:00:00Z")
store.upsert(dict(base, tier=watch.TIER_OBSERVE, score=5.0),
"2026-10-17T16:01:00Z")
row = store.get("203.0.113.9")
self.assertEqual(row["tier"], watch.TIER_OBSERVE)
self.assertEqual(row["ever_malicious"], 1)
self.assertEqual(row["last_malicious_at"], "2026-10-17T16:00:00Z")
core = store.list(scope="core")
self.assertEqual([item["ip"] for item in core], ["203.0.113.9"])
finally:
store.close()
def test_forget_keeps_convicted_ips(self):
with tempfile.TemporaryDirectory() as directory:
store = self._store(directory)
try:
store.upsert({"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS,
"score": 80.0, "total": 10, "rules": {},
"reasons": []}, "2026-10-01T00:00:00Z")
store.upsert({"ip": "198.51.100.7", "tier": watch.TIER_OBSERVE,
"score": 5.0, "total": 10, "rules": {},
"reasons": []}, "2026-10-01T00:00:00Z")
store.forget(3600, time.time())
self.assertIsNotNone(store.get("203.0.113.9"))
self.assertIsNone(store.get("198.51.100.7"))
finally:
store.close()
def test_sorting_modes(self):
with tempfile.TemporaryDirectory() as directory:
store = self._store(directory)
try:
store.upsert({"ip": "203.0.113.2", "tier": watch.TIER_MALICIOUS,
"score": 70.0, "total": 10, "rules": {},
"reasons": []}, "2026-10-17T16:00:00Z")
store.upsert({"ip": "203.0.113.1", "tier": watch.TIER_MALICIOUS,
"score": 90.0, "total": 5, "rules": {},
"reasons": []}, "2026-10-17T17:00:00Z")
first = [row["ip"] for row in store.list(sort="first_seen", order="asc")]
self.assertEqual(first, ["203.0.113.2", "203.0.113.1"])
recent = [row["ip"] for row in store.list(sort="last_malicious",
order="desc")]
self.assertEqual(recent, ["203.0.113.1", "203.0.113.2"])
score = [row["ip"] for row in store.list(sort="score", order="desc")]
self.assertEqual(score, ["203.0.113.1", "203.0.113.2"])
finally:
store.close()
class TextSanitisingTests(unittest.TestCase):
def test_strips_control_and_replacement_chars(self):
self.assertEqual(watch._clean_text("a\x00b\ufffdc"), "a·b·c")
def test_scrubs_binary_header_field(self):
self.assertEqual(watch._scrub_binary("\ufffd\ufffdk\ufffdU\ufffd\ufffd"),
"[非 UTF-8 数据]")
self.assertEqual(watch._scrub_binary("GET"), "GET")
class DetectorTests(unittest.TestCase):
def _detector(self, by_query, store):
fields = dict(watch.DEFAULT_FIELDS)
probe = watch.Detector(None, store, fields)
queries = {rule["name"]: rule["query"] for rule in probe.rules}
mapping = dict(by_query)
mapping.setdefault(queries["not_found"], [])
return watch.Detector(_StubClient(mapping), store, fields, window=300)
def test_automation_is_downgraded(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
try:
probe = watch.Detector(None, store, dict(watch.DEFAULT_FIELDS))
q = {rule["name"]: rule["query"] for rule in probe.rules}
by_query = {
"*": [("124.70.2.111", 666)],
q["login_attack"]: [("124.70.2.111", 333)],
q["login_burst"]: [("124.70.2.111", 666)],
q["not_found"]: [],
q["probe_java"]: [],
q["bad_ua"]: [],
q["bot_ua"]: [],
q["probe_config"]: [],
q["probe_cms"]: [],
q["probe_iot"]: [],
q["probe_devops"]: [],
q["probe_webshell"]: [],
q["exploit_payload"]: [],
'client_ip:"124.70.2.111"': [
("/prod-api/biz/liveCourseChapter/msgSubscription", 666)],
}
detector = watch.Detector(_StubClient(by_query), store,
dict(watch.DEFAULT_FIELDS), window=300)
result = detector.run_once()
self.assertEqual(result["automation"], 1)
row = store.get("124.70.2.111")
self.assertEqual(row["tier"], watch.TIER_AUTOMATION)
self.assertTrue(any("单一接口" in r for r in row["reasons"]))
finally:
store.close()
def test_low_volume_scanner_is_still_caught(self):
"""A scanner with 18 requests must not need a top-volume slot."""
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
try:
probe = watch.Detector(None, store, dict(watch.DEFAULT_FIELDS))
q = {rule["name"]: rule["query"] for rule in probe.rules}
by_query = {
"*": [("10.185.9.41", 5000), ("202.201.16.87", 4000)],
q["probe_java"]: [("5.188.86.234", 18)],
}
detector = watch.Detector(
_StubClient(by_query, totals={"5.188.86.234": 18}), store,
dict(watch.DEFAULT_FIELDS), window=300)
result = detector.run_once()
self.assertEqual(result["malicious"], 1)
self.assertEqual(store.get("5.188.86.234")["tier"], watch.TIER_MALICIOUS)
finally:
store.close()
def test_internal_and_allowlisted_are_skipped(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
try:
probe = watch.Detector(None, store, dict(watch.DEFAULT_FIELDS))
q = {rule["name"]: rule["query"] for rule in probe.rules}
by_query = {
"*": [("10.185.9.41", 5000), ("203.0.113.5", 30), ("198.51.100.9", 30)],
q["probe_java"]: [("203.0.113.5", 30), ("198.51.100.9", 30)],
}
detector = watch.Detector(
_StubClient(by_query), store, dict(watch.DEFAULT_FIELDS),
window=300, allowlist=lambda ip: ip == "198.51.100.9")
detector.run_once()
self.assertIsNone(store.get("10.185.9.41"))
self.assertIsNone(store.get("198.51.100.9"))
self.assertEqual(store.get("203.0.113.5")["tier"], watch.TIER_MALICIOUS)
finally:
store.close()
class SampleTests(unittest.TestCase):
def test_sample_keeps_all_source_fields(self):
message = {
"timestamp": "2026-10-17T05:08:58.046Z",
"client_ip": "51.158.145.102", "client_port": 46990,
"server_ip": "10.185.9.90", "server_port": 80,
"host": "61.178.20.46:443", "bigip": "www.gsedu.edu.cn_2_4600",
"vs": "/Common/SmartEDU_https", "method": "GET",
"uri": "/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php",
"status": "403", "ua": "libredtail-http",
"client_ip_country_code": "FR", "rt": "0.031",
"message": 'event_type="req",client_ip="51.158.145.102"',
}
sample = watch._sample(message)
for key in ("client_port", "server_port", "bigip", "vs", "rt",
"client_ip_country_code", "message"):
self.assertIn(key, sample)
self.assertEqual(sample["client_port"], "46990")
def test_sample_drops_empty_and_na_values(self):
sample = watch._sample({"client_ip": "1.2.3.4",
"client_ip_city_name": "N/A",
"referer": ""})
self.assertEqual(sample, {"client_ip": "1.2.3.4"})
class SnapshotTests(unittest.TestCase):
def test_compact_evidence_is_bounded(self):
evidence = {
"window": 89600,
"uris": [{"term": "/probe/%d" % i, "count": i} for i in range(30)],
"statuses": [{"term": str(code), "count": 1} for code in range(10)],
"methods": [{"term": "GET", "count": 1}],
"hosts": [{"term": "h", "count": 1}],
"server_ips": [{"term": "10.0.0.1", "count": 1}],
"vips": [{"term": "vip", "count": 1}],
"uas": [{"term": "ua", "count": 1}],
"samples": [{} for _ in range(5)],
"timeline": [{"timestamp": "t%d" % i, "count": 1} for i in range(200)],
"peak": {"timestamp": "t", "count": 3},
}
compact = watch._compact_evidence(evidence)
self.assertTrue(compact["compact"])
self.assertLessEqual(len(compact["uris"]), 10)
self.assertLessEqual(len(compact["timeline"]), 72)
self.assertEqual(compact["sample_count"], 5)
self.assertEqual(compact["window"], 89600)
class PathPrefixTests(unittest.TestCase):
def test_root_is_hidden_and_prefix_works(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
store.upsert({"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS,
"score": 91.0, "total": 120, "window": 3600,
"rules": {"probe_java": 4},
"reasons": ["探测 4 次"]}, "2026-10-17T16:00:00Z")
store.sync_fused()
server = ThreadingHTTPServer(
("127.0.0.1", 0),
watch.make_handler(store, 3600, 60, None, "/s/secret123"))
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
base = "http://127.0.0.1:%d" % server.server_address[1]
try:
with self.assertRaises(urllib.error.HTTPError) as ctx:
urllib.request.urlopen(base + "/", timeout=5)
self.assertEqual(ctx.exception.code, 404)
with urllib.request.urlopen(base + "/s/secret123/", timeout=5) as response:
page = response.read().decode("utf-8")
self.assertIn('const BASE = "/s/secret123"', page)
with urllib.request.urlopen(base + "/s/secret123/api/ips", timeout=5) as response:
payload = json.loads(response.read().decode("utf-8"))
self.assertEqual(payload["count"], 1)
with self.assertRaises(urllib.error.HTTPError):
urllib.request.urlopen(base + "/api/ips", timeout=5)
finally:
server.shutdown()
server.server_close()
store.close()
class StatusIndicatorTests(unittest.TestCase):
def test_status_endpoint(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
class DetectorStub:
last_success = time.time() - 5
last_error = None
last_run = "2026-10-17T16:00:00Z"
server = ThreadingHTTPServer(
("127.0.0.1", 0),
watch.make_handler(store, 3600, 60, DetectorStub()))
threading.Thread(target=server.serve_forever, daemon=True).start()
base = "http://127.0.0.1:%d" % server.server_address[1]
try:
with urllib.request.urlopen(base + "/api/status", timeout=5) as response:
payload = json.loads(response.read().decode("utf-8"))
self.assertEqual(payload["graylog"], "ok")
DetectorStub.last_error = "connection refused"
DetectorStub.last_success = None
with urllib.request.urlopen(base + "/api/status", timeout=5) as response:
payload = json.loads(response.read().decode("utf-8"))
self.assertEqual(payload["graylog"], "error")
finally:
server.shutdown()
server.server_close()
store.close()
class DashboardTests(unittest.TestCase):
def test_api_and_html(self):
with tempfile.TemporaryDirectory() as directory:
store = watch.Store(os.path.join(directory, "ips.db"))
store.upsert({"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS,
"score": 91.0, "total": 120, "window": 3600,
"rules": {"sensitive_probe": 4},
"reasons": ["敏感路径探测 4 次"]}, "2026-10-17T16:00:00Z")
store.upsert({"ip": "198.51.100.77", "tier": watch.TIER_MALICIOUS,
"score": 70.0, "total": 653344, "window": 3600,
"risk": "shared_egress",
"rules": {"sensitive_probe": 4},
"reasons": ["疑似共享出口"]}, "2026-10-17T16:00:00Z")
store.sync_fused()
server = ThreadingHTTPServer(("127.0.0.1", 0),
watch.make_handler(store, 3600, 60))
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
base = "http://127.0.0.1:%d" % server.server_address[1]
try:
with urllib.request.urlopen(base + "/", timeout=5) as response:
page = response.read().decode("utf-8")
self.assertIn("恶意 IP 清单", page)
self.assertIn("/api/ips", page)
with urllib.request.urlopen(base + "/api/ips", timeout=5) as response:
payload = json.loads(response.read().decode("utf-8"))
self.assertEqual(payload["count"], 2)
self.assertEqual({item["ip"] for item in payload["items"]},
{"203.0.113.9", "198.51.100.77"})
self.assertTrue(all(item["sources"] == "feature"
for item in payload["items"]))
with urllib.request.urlopen(base + "/api/stats", timeout=5) as response:
stats = json.loads(response.read().decode("utf-8"))
self.assertEqual(stats["core"], 2)
self.assertNotIn("review", stats)
with urllib.request.urlopen(base + "/api/ip/203.0.113.9", timeout=5) as response:
detail = json.loads(response.read().decode("utf-8"))
self.assertEqual(detail["item"]["tier"], watch.TIER_MALICIOUS)
self.assertFalse(detail["evidence"]["available"])
with urllib.request.urlopen(base + "/api/export.csv", timeout=5) as response:
body = response.read().decode("utf-8")
self.assertIn("text/csv", response.headers.get("Content-Type", ""))
self.assertIn("ip,tier,score", body)
self.assertIn("203.0.113.9", body)
self.assertIn("198.51.100.77", body)
with urllib.request.urlopen(base + "/api/export.txt", timeout=5) as response:
text = response.read().decode("utf-8")
self.assertEqual(set(text.split()),
{"203.0.113.9", "198.51.100.77"})
finally:
server.shutdown()
server.server_close()
store.close()
if __name__ == "__main__":
unittest.main()