commit 2d76ed667238adc67dc9fbe1c34d94f15bedf6f3 Author: DeepFal Date: Sun Oct 18 05:03:05 2026 +0800 初始提交：恶意 IP 特征+行为融合检测系统 - 特征检测（规则/签名）与行为检测（24h 统计）单进程运行 - 融合名单按首次发现时间倒序，展示位置/归属（GeoIP + ASN） - 钉钉融合推送（加签、24h 节流、去重） - 页面状态灯、CSV 导出、Docker 镜像构建文件 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..c5e3024 --- /dev/null +++ b/.gitignore @@ -0,0 +1,24 @@ +# 凭据 +graylog-ops.json +notify.json +config.local.json +*.token + +# 运行时数据/采集产物 +data/ +collect-*/ +graylog-snapshot-*/ +graylog-collect-*/ +snapshots/ +*.tar.gz +*.log +.DS_Store + +# Python +__pycache__/ +*.pyc +.venv/ + +# Docker 构建临时目录（由 Dockerfile 生成 staging） +docker-build/app/ +docker-build/seed/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..4332e2a --- /dev/null +++ b/README.md @@ -0,0 +1,672 @@ +# graylog-ops + +Graylog **4.0.x** 只读信息采集 + 封禁判断工具，服务于"用业务访问日志自动封 IP"这条链路。 + +## 边界（先说清楚） + +Graylog 是**旁路日志系统**：它能看到、能聚合、能告警，但**不在业务流量链路上，不能自己封 IP**。 +本工具同样只读：不建流、不改索引、不建事件、不下发任何封禁。 + +``` +外部客户端 + │ + ▼ +深信服 AD 7.0.8R5（应用交付 / 负载均衡 / 反代） + │ ▲ + │ └── 将来封禁的执行点大概率在这里（黑名单 / 访问控制） + ▼ +业务系统(nginx 等) ──日志──► Graylog ──► graylog-ops（本工具：采集 + 判断） + │ + └──► {ip, action, score, reasons} +``` + +## 深信服 AD 在链路里的影响（关键） + +AD 挡在业务系统前面，会直接改变"日志里的客户端 IP"的含义，排查顺序： + +1. **AD 是否透传真实客户端 IP**：HTTP 类型的业务看 AD 的"真实客户端 IP / X-Forwarded-For"配置； + 四层转发通常保留源地址，七层反代不配置就会变成 AD 自己的地址。 +2. **后端 nginx 的 `$remote_addr` 是不是 AD 地址**。若是，日志里必须加 + `$http_x_forwarded_for`（或 AD 提供的专用头），并在采集端落成独立字段。 +3. 用 `discover` 验证：如果候选字段的"私有 IP 占比"很高、且 `source` 只有一个取值， + 基本可以判定拿到的是 AD/反代地址，而不是真实客户端。 +4. 把 AD 的管理地址、业务 VIP、回源地址写进白名单，避免把自己的设备封掉。 + +## 为什么先做"信息获取 + 判断" + +封禁规则依赖四个事实，猜错任何一个都会封错人： + +1. **真实客户端 IP 落在哪个字段**（`source` 往往是采集器/反代地址，不是访问者） +2. 日志里有哪些可做判据的字段（`status` / `request_uri` / `user_agent` / `response_time` …） +3. 量级（每秒多少条，决定窗口和阈值） +4. 现有 stream / 索引 / 事件定义是怎么组织的 + +`discover` 回答 1、2；`histogram` / `top-ips` 回答 3；`snapshot` / `streams` / `alerts` 回答 4； +`judge` 把这些聚合成"封谁、为什么、怎么处理"。 + +## 快速开始 + +零第三方依赖，只用 Python 3.7+ 标准库。 + +```bash +cd graylog-ops + +# 方式 A：环境变量 +export GRAYLOG_URL=http://edu-master:9002 +export GRAYLOG_TOKEN='从 Graylog UI 创建的 API token' +bin/graylog-ops probe + +# 方式 B：配置文件（推荐，token 不落 shell history） +cp config.example.json graylog-ops.json +# 编辑 graylog-ops.json 填 url / token +bin/graylog-ops --config graylog-ops.json probe + +# 方式 C：临时参数 +bin/graylog-ops --url http://edu-master:9002 --token "$TOKEN" probe +``` + +> 认证方式：**API token 当用户名，字面量密码 `token`**（4.0 文档的写法）。 +> 工具内部已按此拼 `Authorization: Basic base64(:token)`，你只需要给 `--token`。 +> 非 GET 请求会自动带 `X-Requested-By`（Graylog 的 CSRF 保护要求）。 +> +> **token 放哪**：写进 `graylog-ops.json`（已在 `.gitignore` 里）并 `chmod 600`。 +> 只需要读权限的 token，**不需要管理员账号密码**；用完在 UI 里一键吊销。 +> 不要把 token 贴到聊天、工单或截图里——`--config` 读文件时它不会出现在命令行或输出中。 + +## 命令 + +| 命令 | 作用 | +| --- | --- | +| `probe` | 连通性 / 认证 / 16 个接口的可用性矩阵，**第一步跑这个** | +| `system` | 版本、节点、索引器健康、deflector、license | +| `streams [--rules]` | stream 列表（分流与权限边界），`--rules` 额外拉规则 | +| `inputs` | input 类型、监听地址、端口 | +| `fields` | Graylog 已注册字段及类型 | +| `indices` | index set 轮转/保留策略、索引大小 | +| `alerts` | 现有事件定义与通知方式（改造前先看，别重复建） | +| `discover` | **采样日志，推断真实客户端 IP 字段**（核心） | +| `judge` | **把聚合信号判成封禁候选**（只读，dry-run 语义） | +| `top-ips` | `terms` 聚合，默认 `--field client_ip`，看 top IP 与计数 | +| `terms` | 任意字段的 terms 聚合 | +| `messages` | 拉原始消息（表格 / `--jsonl`） | +| `histogram` | 消息量时间分布，估算阈值与容量 | +| `snapshot` | 全量环境信息落盘（每个资源一个 JSON + manifest） | +| `raw` | 直接调用任意 API 路径（支持 `--method`/`--body`），应对版本差异 | +| `collect` | 一条命令产出 probe/discover/judge/snapshot 整套信息 | +| `watch` | 持续检测恶意公网 IP + SQLite 清单 + 本地 Web 面板 | + +通用参数：`--json`（机器可读）、`--out FILE`（写 JSON）、`-v`（打印每个 HTTP 请求）。 + +## 五步摸底 + +```bash +# 1) 能不能通、这个版本到底有哪些接口 +bin/graylog-ops probe + +# 2) 客户端 IP 在哪个字段（最关键的结论），同时生成 profile +bin/graylog-ops discover --range 3600 --stream 业务访问日志 \ + --save-profile profiles/edu.json + +# 3) 用真实数据看阈值是否合理 +bin/graylog-ops top-ips --field client_ip --query 'status:401' --range 300 --size 30 +bin/graylog-ops histogram --range 3600 --interval 5m + +# 4) 判断：谁该封、为什么 +bin/graylog-ops judge --profile profiles/edu.json --range 300 \ + --allow allowlist.example.txt --explain 5 + +# 5) 整体快照存档，方便离线分析和复盘 +bin/graylog-ops snapshot --dir snapshots/$(date +%F) --sample 500 +``` + +> 机器上要能同时访问 `10.180.x`（Graylog）和 `10.230.x`（AD），否则先在跳板机跑。 + +`discover` 的输出（示例数据）： + +``` +采样 8 条消息 query=* range=3600s stream=(all) + +分数 字段 出现率 IP占比 不同IP 私有IP IP版本 样例 +---- --------- ------ ------ ------ ------ ------ ---------------------------- +95.0 client_ip 100% 100% 4 12% v4 203.0.113.5, 198.51.100.7 +45.0 source 100% 100% 1 100% v4 10.180.254.17 + +推荐字段: client_ip (confidence=high) + - 字段 `client_ip` 的 IP 占比 100%，样例 203.0.113.5, 198.51.100.7, 192.0.2.44 + ! `source` 只有 1 个不同取值，而 `client_ip` 有 4 个：`source` 是采集器/反代，封禁必须用 `client_ip` +``` + +## 判断规则（judge） + +固定的 5 个 `terms` 查询 + 3 个统计量，不随 IP 数量增加查询次数： + +| 信号 | 查询（字段名来自 profile） | 权重 | 饱和点 | 说明 | +| --- | --- | --- | --- | --- | +| `auth_fail` | `status:(401 OR 403)` | 45 | 10 次 | 认证失败 | +| `scanner_ua` | `http_user_agent:(sqlmap OR nikto OR nuclei OR nmap OR …)` | 50 | 2 次 | 扫描器特征 | +| `sensitive_path` | `request_uri:(wp-login.php OR phpmyadmin OR actuator OR …)` | 25 | 5 次 | 敏感路径探测 | +| `not_found` | `status:404` | 20 | 20 次 | 目录/文件探测 | +| `bad_method` | `request_method:(PUT OR DELETE OR TRACE OR CONNECT)` | 10 | 5 次 | 异常方法 | +| `rate` | 请求数 / 200 | 25 | 200 次 | 绝对量 | +| `spike` | 请求量 ≥ 候选中位数 4 倍 | 15 | 64 倍 | 相对异常 | +| `auth_ratio` | 失败占比 ≥ 50% 且 ≥ 10 次 | 25 | — | 几乎全是失败 | + +动作映射： + +| 条件 | 动作 | +| --- | --- | +| 有攻击证据 且 score ≥ 75 | `ban_1h` | +| 有攻击证据 且 score ≥ 55 | `ban_30m` | +| score ≥ 40 | `observe`（只记录，不封） | +| 其余 | `ignore` | + +三条安全约束（比阈值本身更重要）： + +1. **内网地址默认硬排除**（RFC1918 / 回环 / 链路本地 / CGNAT），避免把 AD、LB、NAT 出口封掉； + `--include-internal` 可关闭，但一般不该关。 +2. **白名单硬排除**：`--allow` 支持多个文件，一行一个 IP/CIDR，`#` 注释。 + 把 AD 管理地址、业务 VIP、回源地址、办公出口、监控探针都写进去。 +3. **纯流量不封**（`require_evidence`）：没有任何攻击特征、只是请求量大的客户端， + 最多判 `observe`，不会因为"量大"被自动拉黑。 + +`judge` 输出示例： + +``` +窗口 300s query=* field=client_ip stream=(all) 候选池 4（中位请求数 35.0） + +得分 动作 客户端 IP 请求数 依据 +---- ------ ------------ ------ ----------------------------------------------- +98.1 ban_1h 192.0.2.99 25 扫描器 UA 25 次; 敏感路径 25 次; 404 探测 25 次 +75.0 ban_1h 203.0.113.66 40 认证失败 40 次; 失败请求占 100% + +已排除（不参与判分）: + 10.180.5.5 30 内部地址（RFC1918/回环/CGNAT） +``` + +**信号命中 0 个 IP 时要注意**：可能是该窗口确实没有这类流量，也可能是查询模板跟字段分词不匹配 +（比如 `request_uri` 在 ES 里的分词方式和你以为的不一样）。用 profile 覆盖 `query` 后再跑， +`judge` 会把每个信号的实际查询和命中数都打出来。 + +## 持续检测服务（watch） + +```bash +# 持续运行（面板 + 检测循环） +bin/graylog-ops --config graylog-ops.json watch \ + --stream F5_Accesslog --profile profiles/f5.json \ + --interval 60 --window 900 --db data/ips.db \ + --port 8088 --allow allowlist.example.txt + +# 只跑一轮（离线分析，比如 8 小时） +bin/graylog-ops --config graylog-ops.json watch --once \ + --stream F5_Accesslog --profile profiles/f5.json \ + --window 28800 --db data/ips-8h.db + +# 只提供面板，浏览已有数据库 +bin/graylog-ops --config graylog-ops.json watch --serve-only \ + --db data/ips-8h.db --port 8089 +``` + +- 面板：`http://127.0.0.1:8088`（30 秒自动刷新） +- JSON API：`/api/ips?tier=malicious&limit=100`、`/api/stats`、`/api/ip/` +- 存储：`data/ips.db`（SQLite）。IP 连续 `--forget` 秒（默认 7200）不再出现就自动移除 +- 名单：`--export PATH` 每轮把高置信恶意 IP 写成纯文本（默认剔除 `shared_egress`） + +### 面板（给同事看的） + +**只有一个地址** `http://<本机IP>:8088`（只读，无鉴权，仅限内网使用）。 +历史（17–18 日 25 小时）与实时检测共用同一个 SQLite（`data/ips.db`）， +所以打开就是一个完整的大列表： + +- **核心名单**（默认视图）：只列高置信恶意 IP，默认按首次攻击倒序（最新在前），点任意一行进详情 +- 所有判定为恶意的 IP（含此前标为「疑似共享出口」的）统一归入核心名单，不再有「待确认」档 +- **详情页**：评分环、请求总量、平均频率、峰值时段、活跃时段数、命中规则与次数、 + 请求频率分布图、状态码分布、请求方法分布、访问域名分布，以及**按判定类型抽样的证据** + （优先抽攻击类请求；高流量 IP 不会把日志刷屏），附处置建议 +- 搜索框可按 IP 或判定原因过滤；**排序字段与方向是两个独立下拉**（首次攻击 / 最近攻击 / 评分 / 请求数 × 升序 / 降序） +- **单击任意 IP 地址即可复制**（列表和详情页标题都支持，带 toast 提示） +- 列表 30 秒自动刷新（可关闭） + +界面截图：[列表页](docs/screenshots/list.png) · [详情页](docs/screenshots/detail.png) +- **一键导出**（右上角两个下拉，都作用于当前筛选/排序结果）： + - 「复制 IP ▾」→ 逗号分隔 / 换行分隔（换行版可直接粘贴给 ipset、防火墙批量导入） + - 「导出 ▾」→ CSV（完整字段） / 仅 IP 列表（每行一个） +- 只读 API：`/api/ips?scope=core|all`、`/api/ip/`、`/api/stats` +- 导出接口（脚本可直接拉）： + - `GET /api/export.csv?scope=core|all&sort=&order=` → CSV（含等级/评分/首次攻击/原因） + - `GET /api/export.txt?scope=...` → 纯 IP 列表，每行一个 + +证据**完全按需生成**，本地库只存判定结果（56 个 IP 约 36 KB），点开详情时才连 Graylog： + +- **9 个查询并发**（URI/UA/状态码/方法/域名/服务端 IP/VIP + 时间分布 + 样本），首次打开约 0.3–0.5 秒， + 结果在内存缓存 10 分钟，再打开约 1 毫秒；进程重启后重新拉取 +- 样本走 views 的 `messages` 搜索：旧版 `/search/universal/relative` 在 25 小时窗口 + 复合查询下 + 会被 Graylog 断连，表现就是"只有分布没有样本"（WebShell 这类 IP 尤其明显） +- 样本按**判定类型**分组：每种类型各保留最近 3 条（两种类型 6 条、三种类型 9 条，以此类推）， + 同一秒内的重复请求会合并；没有命中规则时退化为最近 5 条通用请求 +- 每条样本展示源日志的**完整字段**：客户端/服务端 `IP:端口`、Host、VIP(bigip)、VS、 + 地区、响应时间 `rt`、请求长度、Referer、采集来源，以及原始日志行 +- **长字段可点击弹窗**查看完整内容（URI / UA / Referer / Host / 原始日志行），Esc 或点遮罩关闭 +- **轻量快照兜底**：每次查看详情时顺手存一份 1–2 KB 的摘要（URI/状态/方法/VIP/时间分布）， + Graylog 索引轮转（约 10 天）后自动回退到快照，并明确标注"日志已过期，以下为判定时的快照" +- 负载：每轮固定 6 个 terms 查询 + 最多 `--verify` 次核验，**与 IP 数量无关** + （本环境 15 分钟窗口约 3.5–5 秒/轮） + +规则（速率类信号按窗口归一化到 15 分钟基准，行为类信号不缩放）： + +| 规则 | 权重 | 饱和 | 说明 | +| --- | --- | --- | --- | +| 敏感路径探测 | 65 | 2 次 | `/actuator`、`/.env`、`/.git`、`/wp-login`、`/druid`、`/nacos`、`/swagger`、`/console`、`/ISAPI`… | +| 扫描器 UA | 55 | 1 次 | sqlmap / nikto / nuclei / nmap / zgrab … | +| 登录失败（401/403） | 55 | 8 次 | 登录类 URI 上的认证失败 | +| 爬虫 UA | 50 | 20 次 | DotBot 等 | +| 登录接口高频 | 15 | 2000 次 | 只作上下文，单独不足以定性（学生也会反复登录） | +| 404 占比 | 25 | 比例 ≥50% 且 ≥20 次 | 目录/文件探测 | + +分级：**≥60 恶意 / ≥30 可疑 / 其余观察**（观察级分数 <`--min-observe-score`(10) 不入库）。 +安全约束：内网与白名单硬排除；单一接口高频的自动化客户端自动降级为 `automation`； +候选池取「所有规则命中的 IP ∪ 访问量 top」，因此**低流量扫描器不会被漏掉**。 + +### 地理信息（离线 GeoIP） + +- 列表新增「国家/地区」列（国旗 + 中文国名 + 省/州），详情页也显示 +- 数据来自 **DB-IP Lite**（免费、CC BY 4.0），已构建为本地 SQLite：`data/geo/geo.db`（144MB，360 万条 IPv4 段） +- 检测时自动补全；已有 IP 在启动时自动回填 +- 更新地理库（每月）： + +```bash +cd data/geo +curl -O https://download.db-ip.com/free/dbip-city-lite-2026-10.csv.gz +gunzip -kf dbip-city-lite-2026-10.csv.gz +cd ../.. && python3 -m graylog_ops.geo \ + --city-csv data/geo/dbip-city-lite-2026-10.csv --out data/geo/geo.db +``` + +- 关闭地理功能：`--geo-db ""` + +### 新增 IP 推送通知 + +- 只有 **新增恶意 IP** 或 **观察/可疑 → 恶意升级** 才会推送；同一 IP 不会重复推 +- 支持企业微信 / 钉钉 / 飞书 / Slack / 通用 JSON（`type` 字段选择） +- 推送失败只打印日志，不影响检测循环 +- 配置：复制 `notify.example.json` 为 `notify.json`（已在 .gitignore 中），填 webhook： + +```json +{ + "type": "dingtalk", + "url": "https://oapi.dingtalk.com/robot/send?access_token=你的TOKEN", + "secret": "SEC你的加签密钥", + "link_prefix": "http://179.253.234.124:12139/s/301b.../", + "title": "新增恶意 IP" +} +``` + +| type | 说明 | +| --- | --- | +| `wecom` | 企业微信群机器人（markdown） | +| `dingtalk` | 钉钉自定义机器人（markdown，支持 `secret` 加签） | +| `feishu` | 飞书自定义机器人（text） | +| `slack` | Slack Incoming Webhook | +| `generic` | POST `{"title","text"}`，可接自己的网关 | + +测试与自检： + +```bash +# 发一条测试消息（验证 webhook 是否通） +./bin/graylog-ops --config graylog-ops.json watch --notify-config notify.json --notify-test \ + --stream F5_Accesslog --profile profiles/f5.json --db data/ips.db +``` + +推送内容示例： + +``` +**新增恶意 IP** +**112.32.1.211**（中国 · 广东） +评分 100.0 · 请求 108 次 +首次攻击 2026-10-17T18:24:09Z +命中：Spring/中间件探测 40 次；配置/凭证探测 8 次 +[查看详情](http://179.253.234.124:12139/s/301b.../#/ip/112.32.1.211) +``` + +### 状态指示灯 + +页面右上角有一个圆点，表示后端与 Graylog 的连接状态： + +- 绿色：最近一轮检测成功 +- 红色：最近一轮检测失败 +- 黄色：状态过期 +- 灰色：尚未开始检测 + +不展示文字，悬停可看提示。 + +### 行为检测（已并入主服务，单进程） + +特征检测和行为检测现在跑在**同一个进程**里： + +- 特征检测线程：每 60 秒一轮（原有逻辑） +- 行为检测线程：每 15 分钟一轮（24h 窗口，纯行为评分，不依赖规则和名单） +- 同一个 Web 前端：融合首页 + 融合详情 + +**融合名单规则**： + +- 只有「建议封禁」才展示和推送：特征命中攻击证据，或行为判定为建议封禁（非共享出口） +- 共享出口/NAT、人工确认、数据不足等行为结果**不展示、不推送** +- 一个 IP 只出现一行，来源标「特征 / 行为 / 双来源」 +- 首页按 `first_banned_at`（首次进入建议封禁的时间）倒序，**每个 IP 只置顶一次**，后续再命中不改变位置 +- 历史封禁 IP 永久保留在列表里（不参与 24h 再推） + +**推送**： + +- 融合成一条消息，同一 IP 永远不重复推 +- 同一 IP 24 小时内最多推一次；24 小时后仍在活动（最近 24h 有流量）的 IP 会再推一条 +- 首次部署时历史 IP 静默（不补推） + +**详情页**：同一 IP 融合展示「行为检测画像」和「特征检测证据」。 + +相关参数：`--behavior-interval 900`（行为刷新间隔）、`--no-behavior`（关闭行为检测）。 + +### ASN / IP 归属 + +- 列表「位置 / 归属」合并为一列、上下两行：第一行国家/地区，第二行图标 + 厂商/机构（如 ☁️ 阿里云、📱 中国移动） +- 悬停显示 ASN 号、完整组织名和网络类型 +- 详情页显示 ASN 号 + 厂商 + 组织全名 + 类型 +- CSV 导出包含 `asn, asn_org, asn_brand, asn_type` +- 品牌映射在 `asn_brand_map.json`（可继续加厂商），数据库 `data/geo/asn.db` +- 更新方式（DB-IP ASN Lite，每月）： + +```bash +curl -O https://download.db-ip.com/free/dbip-asn-lite-2026-10.csv.gz +gunzip -kf dbip-asn-lite-2026-10.csv.gz +python3 -m graylog_ops.asn --csv data/geo/dbip-asn-lite-2026-10.csv --out data/geo/asn.db +``` + +关闭 ASN：`--asn-db ""` + +### 名单语义（重要） + +- **永久保留**：一旦某个 IP 被判定为恶意（`ever_malicious=1` + `last_malicious_at`）， + 就永远不会被 `--forget` 清理，即使它后来降级为可疑/观察，也仍留在核心名单和导出名单里。 + 只有"从未判过恶意"的观察级 IP 才会按 `--forget` 过期删除。 +- **时间字段全部来自日志，不是检测时间**： + - 首次攻击 = 日志中最早命中攻击规则的时间（最多回溯 36 小时） + - 最近攻击 = 日志中最近一次命中攻击规则的时间 + - 最近出现 = 该 IP 最后一条日志的时间 + - `hits` 只是"被检测到的轮次"，不是攻击次数 +- **只有窗口内计数发生变化时才回查并更新时间**，避免滚动窗口把"最近攻击"不断刷成当前时间 +- 历史数据用 `watch --fix-times`（别名 `--fix-first-seen`）修正 +- **默认排序 = 首次攻击倒序（最新攻击的在最上面）**，同一发现时间下最近还在攻击的排前面； + 面板右上角可切换「最近攻击（最新优先）」「评分」「请求数」。 +- `blocklist.txt` 导出的是**历史累计**的全部恶意 IP，不是当前窗口快照。 +- 乱码处理：源日志里非 UTF-8 的字段会被清洗，识别为二进制时显示 `[非 UTF-8 数据]`， + URI/方法/状态码等字段不会再把乱码带进页面。 + +### 启动方式（按需，不挂后台服务） + +按当前约定**不安装 launchd/systemd**，需要看的时候手动起（DSH/终端重启后进程会消失，重新执行即可）： + +```bash +cd "/Users/deepfal/Code/Work/ip sec/graylog-ops" +./bin/graylog-ops --config graylog-ops.json watch \ + --stream F5_Accesslog --profile profiles/f5.json \ + --interval 60 --window 3600 --forget 604800 \ + --db data/ips.db --host 0.0.0.0 --port 8088 \ + --allow allowlist.example.txt --export data/blocklist.txt --warm-limit 5 +``` + +停止：Ctrl+C（或用 `pkill -f "graylog_ops watch"`）。 + +维护命令： + +```bash +# 用日志重新校准历史 IP 的首次攻击 / 最近攻击 / 最近出现时间（幂等） +./bin/graylog-ops --config graylog-ops.json watch --fix-times \ + --stream F5_Accesslog --profile profiles/f5.json --db data/ips.db + +# 为名单里的恶意 IP 预生成证据快照 +./bin/graylog-ops --config graylog-ops.json watch --warm-only --warm-limit 200 \ + --stream F5_Accesslog --profile profiles/f5.json --db data/ips.db +``` + +## 两日实测结果（本地 10-17 00:00 – 10-18 01:00） + +窗口 89,600 秒（约 24.9 小时），12 条规则，`--size 600`： + +**55 个候选 → 48 个恶意 = 44 个高置信 + 4 个疑似共享出口；4 个可疑。** + +| 探测类型 | 典型路径 | 规则 | +| --- | --- | --- | +| Spring/中间件 | `/actuator*`、`/druid*`、`/nacos*`、`/swagger*`、`/jmx*`、`/heapdump*`、`/console*`、`/manager/html*`、`/solr*` | probe_java | +| 配置/凭证 | `/.env*`、`/.git*`、`/.svn*`、`/.aws*`、`/.ssh*`、`/appsettings*`、`/WEB-INF*`、`/backup*` | probe_config | +| CMS | `/wp-*`、`/xmlrpc.php`、`/phpmyadmin*`、`/administrator*`、`/joomla*`、`/thinkphp*` | probe_cms | +| IoT/路由器 | `/ISAPI*`、`/LAPI*`、`/HNAP1*`、`/boaform*`、`/GponForm*`、`/cgi-bin/luci*` | probe_iot | +| DevOps/监控 | `/jenkins*`、`/grafana*`、`/kibana*`、`/zabbix*`、`/prometheus*`、`/consul*`、`/harbor*` | probe_devops | +| WebShell/上传 | `/shell.php`、`/cmd.php`、`/eval-stdin.php`、`/phpinfo*`、`/vendor/phpunit*` | probe_webshell | +| 注入/穿越 | `etc/passwd`、`jndi:`、`union select`、`%2e%2e`、`%252f`、`base64_decode` | exploit_payload | +| 登录撞库 | `/login*`、`/passport*`、`/captcha*` + 401/403 | login_attack / login_burst | +| 扫描器/爬虫 | sqlmap/nikto/nuclei/ffuf…、DotBot | bad_ua / bot_ua | + +高置信名单（截至本轮回填，已写入 `data/blocklist.txt`）： + +``` + 117.136.27.147 113.31.155.142 113.31.155.219 116.176.186.63 + 39.144.210.4 120.232.115.57 39.144.210.0 186.241.107.91 + 91.92.242.37 36.159.247.84 113.201.250.120 42.91.116.131 + 124.152.12.48 120.83.247.23 223.198.163.145 217.60.242.58 + 112.32.8.112 119.42.146.230 120.214.218.174 213.165.242.55 + 216.219.95.217 190.122.240.80 91.194.91.202 182.16.82.226 + 109.105.209.20 42.94.151.105 5.188.86.234 209.99.189.217 + 165.154.218.226 115.190.159.225 51.158.145.102 111.229.129.233 + 223.123.92.48 222.83.114.16 109.105.209.18 43.136.79.172 + 91.92.243.72 35.209.176.178 47.121.125.205 109.205.176.10 + 130.12.180.171 130.12.180.168 +``` + +疑似共享出口 / NAT（4 个，**排除在名单外**）：42.92.144.29(653344)、27.185.218.213(125241)、118.181.187.248(30978)、42.94.148.133(22757) + +判定公式：`(请求量 >= 50000 且 探测占比 < 2%) 或 (请求量 >= 20000 且 探测占比 < 0.5%)` +→ 标记 `risk=shared_egress`，导出名单时自动剔除。 + +> 规则收紧记录：`/uploadfiles/*`（正常文件目录）和 `/database/*`（正常业务路径）曾被 +> 误判为 WebShell/配置探测，已从前缀里移除；`/config.json`、`/settings.json` 同理。 + +## profile 格式 + +`discover --save-profile` 生成的骨架，`judge --profile` 读取： + +```json +{ + "fields": { + "client_ip": "client_ip", + "status": "status", + "uri": "request_uri", + "ua": "http_user_agent", + "method": "request_method" + }, + "signals": { + "sensitive_path": {"query": "request_uri:wp-login", "weight": 30, "saturate": 3} + }, + "thresholds": { + "min_requests": 30, + "ban_score": 70 + } +} +``` + +`fields` 可被命令行覆盖（`--field` / `--status-field` / `--uri-field` / `--ua-field` / `--method-field`）， +`signals` 里 `enabled: false` 可以关掉某个信号。 + +## Graylog 4.0.8 接口笔记 + +| 用途 | 路径 | +| --- | --- | +| 版本/集群 | `GET /api/system` | +| 节点 | `GET /api/system/cluster/nodes` | +| 索引器健康 | `GET /api/system/indexer/cluster/health` | +| 字段类型 | `GET /api/system/fields` | +| stream | `GET /api/streams` | +| input | `GET /api/system/inputs` | +| index set | `GET /api/system/indices/index_sets?skip=0&limit=100` | +| 索引 | `GET /api/system/indexer/indices` | +| 事件定义 | `GET /api/events/definitions?page=1&per_page=50` | +| 通知 | `GET /api/events/notifications` | +| 相对时间搜索 | `GET /api/search/universal/relative?query=*&range=300&limit=100&sort=timestamp:desc` | +| 绝对时间搜索 | `GET /api/search/universal/absolute?query=*&from=...&to=...` | +| terms 聚合 | `GET /api/search/universal/terms?field=client_ip&range=300&size=50` | +| 直方图 | `GET /api/search/universal/histogram?query=*&range=3600&interval=1m` | +| 统计 | `GET /api/search/universal/stats?field=...&stats=cardinality,count` | +| stream 过滤 | 追加 `filter=streams:` | + +4.0 的关键点： + +- **legacy 搜索 API 是主路径**。`/api/views/search`（pivot）在 4.0 已经存在，但 DTO 与 5.x/6.x 差异大， + 本工具不用它，避免被版本升级打断。需要时可用 `raw` 直接调。 +- `size` / `limit` 在 terms 端点上的参数名历史上不一致，客户端遇到 400 会自动换另一个再试，`probe` 会显示实际生效的那个。 +- 时间一律 **UTC ISO8601**，返回形如 `2026-10-17T15:00:00.000Z`。 +- 搜索返回是 `{"messages": [{"message": {...}}, ...]}`，工具已摊平。 +- 4.0.8 跑在 Java 8 / CentOS 7 上，`/api/system` 不返回 ES 版本；后端是 ES 6.8 或 7.10。 + +## 4.0.8 实测踩坑（本环境已验证） + +这几条是拿着真实 token 打 edu-master 得出的，不是文档抄的： + +1. **`terms` / `histogram` / `stats` 在 4.0 已移除**（返回 404）。 + 聚合必须走 views API：`POST /api/views/search/sync`（pivot）。 + 工具会自动探测：views 可用就用 views，3.x 上自动回退到 legacy 端点。 + pivot 的行组是 `{"type":"values","field":"uri","limit":50}`（注意是 `field` 不是 `fields`）。 +2. **字段是 keyword 语义**，查询有三条铁律： + - `uri:ketang` → 0 条；`uri:/ketang-studyapi/apis*` → 46 万条（只有**尾部通配符**有效） + - **未转义的 `/` 会被 Lucene 当成正则分隔符**：`uri:/manager/html*` 会误命中所有含 + `html` token 的文档（实测 9079 条，全是 UEditor 的 JS 静态文件）。必须写成 + `uri:\/manager\/html*`，或引号精确匹配 `uri:"/wp-login.php"` + - 前导通配符 `uri:*admin*` 被禁用（直接 500/400） + judge 的默认信号查询已按此修正（`ua:sqlmap*`、`uri:\/actuator*`）。 +3. **部分 F5 VS 的 HTTP 字段为空串**：1 小时样本里 `uri` 约 98% 有值、`status` 约 48%、 + `ua` 约 48%。四层 VS 只有 `client_ip/server_ip/method` 等基础字段。 +4. `status` 是混格式字符串（`200 OK` / `404` / `502 Bad Gateway`），`status:404` 仍能按 + token 命中；`status_code` 字段全空，不要用。 +5. `event_type` 只有 `req`，不存在 req/resp 双份，不需要去重。 + +## 本环境实测结论（2026-10-18） + +| 项 | 结论 | +| --- | --- | +| 集群 | 4 节点：`edu-master`(master) + `edu-slave1/3/4`，2354 个 active shard，deflector `graylog_457` | +| 数据链路 | **F5**(`10.180.241.2/3`) --syslog--> Graylog。深信服 AD 不在 F5 访问日志链路上（它对应的是 DNS 类业务） | +| 活跃 stream | `F5_Accesslog`（约 11 万条/5 分钟）、`F5 Attack Logs`、`有源业务数据-Nat-Stream`（NAT 会话）、`syslog_vpn_stream`（VPN）；`站群`、`全量站点数据` **24h 内无数据** | +| 量级 | `F5_Accesslog` 约 130 万条/小时（≈360 条/秒） | +| 字段 | `client_ip` / `server_ip` / `uri` / `host` / `ua` / `status` / `method` / `bigip` / `vs`；**没有 XFF 字段** | +| 关键事实 | `10.185.9.41`(46%) 和 `10.185.9.15`(45%) 两个**内网 IP 占 91% 流量**，是内部 API 轮询（`/ketang-studyapi/apis/*`，一半是 404），必须白名单/内网排除；外网用户 F5 直见真实 IP，所以 `client_ip` 可用于封禁 | + +真实判定样例（1 小时窗口，`--verify 5`）： + +``` +100.0 ban_1h 186.241.107.91 1164 认证失败 24; 敏感路径 36; 404 探测 156 + 68.5 ban_30m 113.201.250.120 612 认证失败 3; 敏感路径 6; 404 探测 5 + 54.0 observe 124.70.2.111 666 认证失败 333（集中在单接口，已降级） +``` + +- `186.241.107.91` 挖到 `/actuator/env`、`/actuator/xyzabc`（fuzz）、`/actuator;/health`（分号绕过尝试） +- `113.201.250.120` 同样在打 `/actuator/env` +- `124.70.2.111` 每 **10.6 秒** POST 同一个业务 API、稳定 403 —— 自动化客户端 token 过期， + 被 `--verify` 识别并降级为 `observe`，不误封 + +`--verify N` 会对前 N 个 ban 候选多做一次 URI 聚合：若流量 90% 以上集中在单一非登录接口， +判定为"疑似业务客户端/自动化任务"并降级。这是本环境实测后加的必要规则。 + +## 权限 + +只读账号给这些权限就够（缺哪个会报 403，`probe` 会逐条标出来）： + +``` +messages:read searches:relative searches:absolute +searches:keyword streams:read inputs:read +indexsets:read eventdefinitions:read eventnotifications:read +``` + +首次摸底建议直接用管理员 token，确认链路通了再收窄到只读 token。 + +## 已知限制 + +- 纯只读，不提供写入/告警创建（下一阶段再做，会单独走 `--apply` 路径）。 +- `judge` 是单字段聚合：只按 `client_ip` 分组，不做多字段联合（如 IP×URI）。 + 这是刻意的取舍——固定查询次数，代价是拿不到"同一 IP 命中几种不同敏感路径"这类信号。 + 需要时用 `--explain` 抽样人工看。 +- 每类聚合最多看 top `--size`（默认 200）个 IP；超出部分不参与判分。 +- `discover` 用 offset 分页采样，适合几千条量级；十万级请改用 `terms` / `histogram`。 +- 大时间窗 + 高基数 `terms` 会打 ES，`--range` 别一上来就 7d；先 `histogram` 看量。 + +## 与执行侧衔接（下一步） + +判断层已经产出 `{ip, action, score, reasons, signals}`。执行侧有两个候选： + +1. **Graylog 事件 + HTTP 通知**：事件定义按 `client_ip` 分组 count 超阈值 → 通知到 ban-service。 + 优点是近实时；缺点是阈值表达力有限，复杂判定还得在服务里做。 +2. **定时跑 `judge`**（如每 1 分钟）+ 直接调 AD 接口： + +``` +cron: graylog-ops judge --range 120 --json --out /var/lib/ban/candidates.json + → ban-service 读 candidates + → 白名单/冷却/幂等校验 + → 调 AD 黑名单 API 下发 + → TTL 到期自动删除 / 复核 +``` + +**AD 接口这块先不猜**。截图里管理入口是 +`https://10.230.13.25:4433/index?x_id=<...>#mod=/mod_runtime/homepage/index`， +`x_id` 看着像会话/鉴权参数，前端是 SPA 路由，真正的 XHR 端点得抓包确认。需要你做的只有一件事： + +> 在能访问 AD 的机器上打开 Chrome DevTools → Network → 勾选 Fetch/XHR → +> 手工做一遍"添加/删除一条 IP 黑名单" → 右键 `Copy as cURL`（或导出 HAR）发我。 + +需要覆盖的请求：登录、会话刷新、业务/节点列表、黑名单增删查、下发后的生效确认。 +拿到之后我补一个 `sangfor` 模块（登录态维持、黑名单 CRUD、幂等、限流），复用现有的 HTTP 重试/错误归一。 + +## 环境备忘 + +| 项 | 值 | +| --- | --- | +| Graylog | 4.0.8+6b8c55d @ `edu-master:9002`（4 节点，Java 8 / CentOS 7） | +| 深信服 AD | 7.0.8R5，UI `https://10.230.13.25:4433`，HA（主）`10.1.1.1`，业务 5 / 节点 3 | +| 网络 | 开发机可访问 `10.180.254.17:9002` 与 `10.230.13.25:4433` | +| 凭据 | `graylog-ops.json`（chmod 600，已 gitignore），只读 token | + +## 开发 + +```bash +python3 tests/demo.py # 用假服务预览各命令输出，不需要真 Graylog +python3 -m unittest discover -s tests # 41 项单测 + 端到端测试 +``` + +测试不需要真实 Graylog：`tests/fake_graylog.py` 起一个本地 HTTP 服务，按 4.0 的响应结构回包， +并实现了一个很小的 Lucene 子集（`field:value`、`field:(a OR b)`）来验证 `judge` 的查询构造。 +要加新接口，先在这个假服务里加路由，再补一条 e2e。 + +``` +graylog_ops/ + http.py urllib 封装：Basic auth、X-Requested-By、重试、错误归一 + config.py CLI / 环境变量 / JSON 文件三级配置 + client.py 4.0 REST 接口 -> 普通 dict/list，容忍响应形状差异 + detect.py 客户端 IP 字段推断（IP 解析、内部地址判定、评分） + judge.py 判断层：信号采集、基线、评分、动作映射、白名单 + commands.py 各子命令（构建 JSON + 人读渲染） + render.py CJK 宽度对齐的表格 +tests/ + fake_graylog.py 假 Graylog 4.0 服务（含查询子集） + test_detect.py 字段推断单测 + test_judge.py 判断层测试（暴力破解/扫描/正常大流量/内网/白名单） + test_e2e.py CLI 端到端 +``` + +## 排错 + +| 现象 | 原因 / 处理 | +| --- | --- | +| 连接超时 | 服务在 **9002** 端口（9000 是空等）。9002 也不通时走 VPN / 内网跳板机 | +| 401 | token 错或已吊销；确认是"API token"不是 session token | +| 403 | 账号缺权限，看 `probe` 里哪项 FAIL | +| 404 | 该版本没这个接口，用 `raw /api/...` 验证，或 `probe` 看矩阵 | +| 400 / 422 | 参数不被接受，加 `-v` 看实际请求 URL | +| `discover` 没结果 | 时间窗内无数据 / stream 过滤错 / 查询语法错 / 无权限（静默返回空） | +| `discover` 推荐的还是代理 IP | 采集端没解析 XFF；需要 AD 透传 + nginx `log_format` 落真实 IP | +| `judge` 没候选 | 窗口内 IP 数低于 `--min-requests`，或分数低于 `--min-score`；先 `--all` 看全量 | +| `judge` 信号命中 0 | 查询模板与字段分词不匹配（见"判断规则"末尾），用 profile 覆盖 query | +| 封禁候选里出现自己的设备 | 白名单没写全；把 AD 地址/VIP/回源/监控补进 `--allow` 文件 | diff --git a/allowlist.example.txt b/allowlist.example.txt new file mode 100644 index 0000000..917e11b --- /dev/null +++ b/allowlist.example.txt @@ -0,0 +1,22 @@ +# 封禁白名单（judge --allow 使用） +# 一行一个 IP 或 CIDR，# 之后是注释。 +# 这里列的东西永远不会进入封禁候选，请按实际环境补全。 + +# --- 基础设施 --- +10.180.254.17 # Graylog 主机 edu-master +10.230.13.25 # 深信服 AD 管理地址（UI :4433） + +# --- 深信服 AD --- +10.1.1.1 # AD HA/设备地址（截图显示，按实际确认） +# 10.230.13.0/24 # AD 所在管理网段，酌情放开 + +# --- 业务 VIP / 反代回源 --- +# 填 AD 上配置的业务 VIP 与后端真实服务器地址 +# 10.1.1.0/24 + +# --- 办公出口 / 集中 NAT --- +# 注意：整个出口网段拉黑会中断整栋楼的上网，务必确认 +# 10.180.0.0/16 + +# --- 监控 / 探测 --- +# 监控探针、拨测、健康检查来源 diff --git a/analysis/behavior_prototype.py b/analysis/behavior_prototype.py new file mode 100644 index 0000000..5ffa8d0 --- /dev/null +++ b/analysis/behavior_prototype.py @@ -0,0 +1,951 @@ +#!/usr/bin/env python3 +"""Behaviour analysis prototype v2 (read-only). + +Computes 24h traffic features per client IP from Graylog pivot aggregations, +derives a behaviour profile, combines it with the existing rule score into a +single risk score with an action recommendation, and serves the result on a +separate port. It never writes to the production sqlite database. +""" +import argparse +import csv +import ipaddress +import json +import os +import re +import statistics +import sys +import sqlite3 +import time +from concurrent.futures import ThreadPoolExecutor +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, ROOT) +from graylog_ops.http import HttpClient # noqa: E402 +from graylog_ops.client import GraylogClient, normalize_histogram # noqa: E402 +from graylog_ops import watch # noqa: E402 + +STREAM = "streams:6ac83059a6878e26256fcacc" +WINDOW = 86400 +DATA_FILE = os.path.join(ROOT, "data", "behavior_prototype.json") +DB_FILE = os.path.join(ROOT, "data", "ips.db") + +INTERNAL = ("10.", "192.168.", "172.16.", "172.17.", "172.18.", "172.19.", + "172.2", "172.3") + +UA_SDK = ("curl", "wget", "python-requests", "python-urllib", "go-http-client", + "okhttp", "java/", "apache-httpclient", "httpcomponents", + "libredtail", "axios", "node-fetch", "aiohttp", "scrapy", + "postmanruntime", "insomnia", "unityplayer", "dart/", "httpie", + "guzzle", "faraday", "restsharp", "libwww", "eeo/", "academy_widget") +UA_BROWSER = ("chrome", "firefox", "safari", "edg/", "opr/", "gecko/20100101") + + +ASN_CSV = os.path.join(ROOT, "data", "geo", "dbip-asn-lite-2026-10.csv") +_ASN_CACHE = {} + + +def _load_asn(): + if _ASN_CACHE: + return _ASN_CACHE + import bisect + starts, ranges = [], [] + try: + with open(ASN_CSV, encoding="utf-8", errors="replace") as fh: + for row in csv.reader(fh): + if len(row) < 4 or ":" in row[0]: + continue + try: + first = int(ipaddress.IPv4Address(row[0])) + last = int(ipaddress.IPv4Address(row[1])) + except ValueError: + continue + starts.append(first) + ranges.append((last, row[2], row[3])) + except FileNotFoundError: + pass + order = sorted(range(len(starts)), key=lambda i: starts[i]) + starts = [starts[i] for i in order] + ranges = [ranges[i] for i in order] + _ASN_CACHE.update({"starts": starts, "ranges": ranges, "bisect": bisect}) + return _ASN_CACHE + + +def asn_info(ip): + cache = _load_asn() + starts = cache.get("starts") or [] + if not starts: + return None + try: + value = int(ipaddress.IPv4Address(ip)) + except ValueError: + return None + index = cache["bisect"].bisect_right(starts, value) - 1 + if index >= 0: + last, asn, org = cache["ranges"][index] + if last >= value: + return {"asn": asn, "org": org, "shared": shared_type(org)} + return None + + +def shared_type(org): + text = (org or "").lower() + cloud = ("alibaba", "aliyun", "tencent", "huawei cloud", "aws", "amazon", + "google", "microsoft", "digitalocean", "vultr", "linode", "hetzner", + "ovh", "wowrack", "idc", "data center", "datacenter", "cloud", + "hosting", "colo") + if any(token in text for token in cloud): + return "hosting" + if "china mobile" in text or "中国移动" in text or "cmnet" in text: + return "mobile" + if ("china unicom" in text or "china telecom" in text + or "chinanet" in text or "联通" in text or "电信" in text + or "broadband" in text): + return "broadband" + if "education" in text or "cernet" in text or "教育" in text: + return "education" + return "unknown" + + +ORG_CN = (("china mobile", "中国移动"), ("china unicom", "中国联通"), + ("china telecom", "中国电信"), ("chinanet", "中国电信"), + ("cernet", "中国教育网"), ("education", "教育网"), + ("alibaba", "阿里云"), ("aliyun", "阿里云"), + ("huawei cloud", "华为云"), ("tencent", "腾讯云"), + ("wowrack", "Wowrack 机房"), ("vmiss", "VMISS 机房"), + ("idc", "IDC 机房")) + + +def org_cn(org): + text = (org or "").lower() + for token, name in ORG_CN: + if token in text: + return name + return org or "未知" + + +SHARE_LABEL = {"mobile": "移动网络", "broadband": "宽带 ISP", + "education": "教育网", "hosting": "云主机/机房", + "unknown": "未知网络"} + + +def humanize(item): + total = item["total"] + parts = [] + if item["uri_card"] >= 20: + parts.append("访问了 %s 个不同路径" % format(item["uri_card"], ",")) + if item["four_xx"] >= 20: + parts.append("产生 %s 次失败请求（失败率 %.0f%%）" + % (format(int(item["four_xx"]), ","), item["pct4"])) + elif item["pct4"] >= 20: + parts.append("失败率 %.0f%%" % item["pct4"]) + if item["coverage"] <= 30 and item["peak_share"] >= 25: + parts.append("流量集中在很短的时间内") + if item["login_ratio"] >= 20: + parts.append("大量请求集中在登录接口") + if item["host_card"] >= 2: + parts.append("同时访问了 %d 个站点" % item["host_card"]) + if item["ua_family"] in ("sdk", "fake-browser", "other"): + parts.append("使用脚本/非浏览器客户端") + behavior = item.get("behavior_type") or "异常自动化" + tail = { + "目录扫描": "符合目录扫描特征", + "短时扫描": "符合短时扫描特征", + "登录爆破": "符合登录爆破特征", + "伪造爬虫 UA": "伪装成搜索引擎爬虫，且请求大量失败", + "采集爬虫": "是采集爬虫", + "异常自动化": "行为高度自动化，不像正常用户", + }.get(behavior, "行为异常") + if parts: + return "24 小时内" + "，".join(parts) + "——" + tail + "。" + return "24 小时内请求 %d 次，%s。" % (total, tail) + + +def load_client(): + cfg = json.load(open(os.path.join(ROOT, "graylog-ops.json"))) + return GraylogClient(HttpClient(cfg["url"], cfg["token"], "token")) + + +def pivot(client, series, row_groups, column_groups=None, query="*", + limit=300, timeout=180000): + body = {"queries": [{ + "id": "q1", + "timerange": {"type": "relative", "range": WINDOW}, + "query": {"type": "elasticsearch", "query_string": query}, + "filter": {"type": "or", "filters": [{"type": "stream", + "id": STREAM.split(":")[-1]}]}, + "search_types": [{ + "type": "pivot", "id": "st1", "series": series, + "row_groups": [{"type": "values", "field": f, "limit": limit} + for f in row_groups], + "column_groups": column_groups or [], "sort": [], "rollup": True, + }], + }], "parameters": []} + data = client.http.request("POST", "/api/views/search/sync", + params={"timeout": timeout}, body=body) + return data["results"]["q1"]["search_types"]["st1"] + + +def rows_to_dict(search_type, series_id): + out = {} + for row in search_type.get("rows") or []: + key = row.get("key") or [] + if not key: + continue + ip = str(key[0]) + for value in row.get("values") or []: + if (value.get("key") or [None])[0] == series_id: + try: + out[ip] = float(value.get("value") or 0) + except (TypeError, ValueError): + out[ip] = 0.0 + return out + + +def hourly_matrix(search_type): + out = {} + for row in search_type.get("rows") or []: + key = row.get("key") or [] + if not key: + continue + counts = [] + for value in row.get("values") or []: + stamp = (value.get("key") or [None])[0] + if not stamp or "cnt" not in (value.get("key") or []): + continue + try: + counts.append((stamp, float(value.get("value") or 0))) + except (TypeError, ValueError): + pass + out[str(key[0])] = counts + return out + + +def bucket_stats(counts): + active = [c for c in counts if c > 0] + if not counts: + return {"coverage": 0.0, "cv": None, "peak_share": 0.0} + cv = (statistics.pstdev(active) / statistics.fmean(active) + if len(active) > 2 and statistics.fmean(active) else None) + peak = max(counts) if counts else 0 + return {"coverage": len(active) / len(counts) * 100, + "cv": cv, + "peak_share": (peak / sum(counts) * 100) if sum(counts) else 0.0} + + +def uri_template(uri): + path = str(uri or "").split("?")[0] + path = re.sub(r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-" + r"[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "{uuid}", path) + path = re.sub(r"[0-9a-fA-F]{16,}", "{hex}", path) + path = re.sub(r"\d+", "{n}", path) + return path + + +def ua_family(ua): + text = str(ua or "").strip() + if not text: + return "empty" + lower = text.lower() + if any(token in lower for token in UA_SDK): + return "sdk" + if lower.startswith("mozilla/") and any(t in lower for t in UA_BROWSER): + return "browser" + if lower.startswith("mozilla/5.0"): + return "fake-browser" + return "other" + + +def ua_garbage(ua_card, top_ua): + """The F5 UA field carries binary garbage on some VS: detect it.""" + if ua_card > 50: + return True + text = str(top_ua or "") + if not text: + return False + printable = sum(1 for char in text if 32 <= ord(char) < 127 or ord(char) > 127) + return printable / max(1, len(text)) < 0.85 + + +def rule_data(client, fields): + rules = {r["name"]: {"title": r["title"], "query": r["query"](fields), + "weight": r["weight"]} + for r in watch.RULE_DEFS} + hits = {} + + def run(item): + name, meta = item + try: + st = pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query=meta["query"], limit=300) + return name, rows_to_dict(st, "cnt") + except Exception: + return name, {} + + with ThreadPoolExecutor(max_workers=6) as pool: + for name, counts in pool.map(run, rules.items()): + for ip, count in counts.items(): + if count > 0: + hits.setdefault(ip, {})[name] = int(count) + return rules, hits + + +def detail_for(client, ip): + q = 'client_ip:"%s"' % ip + detail = {"uri": [], "ua": [], "host": []} + try: + st = pivot(client, [{"type": "count", "id": "cnt"}], ["uri"], + query=q, limit=40) + for row in st.get("rows") or []: + key = row.get("key") or [] + value = (row.get("values") or [{}])[0].get("value") + if key: + detail["uri"].append((str(key[0]), int(float(value or 0)))) + except Exception: + pass + for label, field, size in (("ua", "ua", 3), ("host", "host", 2)): + try: + st = pivot(client, [{"type": "count", "id": "cnt"}], [field], + query=q, limit=size) + for row in st.get("rows") or []: + key = row.get("key") or [] + value = (row.get("values") or [{}])[0].get("value") + if key: + detail[label].append((str(key[0]), int(float(value or 0)))) + except Exception: + pass + try: + hist = client.histogram(query=q, range_seconds=WINDOW, interval="5m", + filter_str=STREAM) + detail["buckets5m"] = [b["count"] for b in normalize_histogram(hist)] + except Exception: + detail["buckets5m"] = [] + return ip, detail + + +def gather_details(client, ips): + out = {} + with ThreadPoolExecutor(max_workers=10) as pool: + for ip, detail in pool.map(lambda value: detail_for(client, value), ips): + out[ip] = detail + return out + + +def rule_score_for(total, hits): + tier, score, reasons = watch.classify(total, hits) + return tier, score, reasons + + +def behavior_profile(feat): + """Return (category, adjustment, confidence, notes) from behaviour only.""" + notes = [] + total = feat["total"] + coverage = feat["coverage"] + cv = feat["cv"] + pct2, pct4 = feat["pct2"], feat["pct4"] + status_ok = feat["status_known"] >= 10 + ua_ok = not feat["ua_garbage"] + ua_card = feat["ua_card"] + + if not ua_ok: + notes.append("UA 字段异常（%d 种）" % ua_card) + if not status_ok: + notes.append("status 字段缺失（仅 %.0f%%）" % feat["status_known"]) + if total < 100: + notes.append("24h 仅 %d 次请求，行为样本不足" % total) + return "insufficient", -10, "低", notes + + if coverage >= 80 and cv is not None and cv <= 1.2 and ua_ok and ua_card <= 2: + if status_ok and pct4 >= 50: + notes.append("长期稳定轮询、%.0f%% 失败" % pct4) + return "integration_error", -35, "中", notes + notes.append("覆盖率 %.0f%%、小时CV %.2f、UA %d 个%s" + % (coverage, cv, ua_card, + ("、2xx %.0f%%" % pct2) if status_ok else "")) + return "server", -35, "高" if status_ok else "中", notes + if coverage >= 80 and cv is not None and cv > 1.2 and ua_ok and ua_card <= 2: + if (status_ok and pct2 >= 85) or not status_ok: + notes.append("覆盖率 %.0f%%、小时CV %.2f，周期性批量任务" % (coverage, cv)) + return "batch_job", -30, "中", notes + if ua_ok and ua_card >= 5 and feat["total"] >= 300: + notes.append("%d 种 UA、请求 %d 次，疑似共享出口/NAT" + % (ua_card, total)) + return "shared_nat", -40, "中", notes + if ((feat["uri_templates"] >= 200 and total >= 500 and ua_ok + and ua_card <= 3 and (not status_ok or pct2 >= 80)) + or (feat["get_ratio"] >= 85 and feat["uri_templates"] >= 100 + and total >= 1000 and ua_ok)): + notes.append("访问 %d 个 URI 模板，UA 稳定%s" + % (feat["uri_templates"], + ("、2xx %.0f%%" % pct2) if status_ok else "")) + return "crawler", -15, "中", notes + if status_ok and pct4 >= 50 and total >= 1000 and ua_ok and ua_card <= 3: + notes.append("请求 %d 次、%.0f%% 失败，疑似鉴权/配置异常" % (total, pct4)) + return "integration_error", -35, "中", notes + if coverage <= 30 and ua_card >= 3 and (not status_ok or pct4 >= 20): + notes.append("短时活跃、%d 种 UA、%.0f%% 4xx" % (ua_card, pct4)) + return "script_candidate", 10, "低", notes + if coverage <= 30 and (not status_ok or pct2 >= 90): + notes.append("短时活跃、成功率 %.0f%%" % pct2) + return "human", -25, "低", notes + notes.append("覆盖率 %.0f%%、CV %s、4xx %.0f%%" + % (coverage, ("%.2f" % cv) if cv is not None else "-", pct4)) + return "uncertain", 0, "低", notes + + +def combine(rule_tier, rule_score, behavior_cat, behavior_adj): + """Hybrid risk score + action recommendation.""" + score = rule_score + behavior_adj + server_like = behavior_cat in ("server", "batch_job", "shared_nat", + "integration_error") + if behavior_cat == "insufficient": + server_like = False + risk = max(0.0, min(100.0, score)) + if rule_score >= 60: + action = "人工确认" if server_like else "封禁" + elif risk >= 45: + action = "人工确认" + elif risk >= 25: + action = "观察" + else: + action = "正常" + if server_like and action == "封禁": + action = "人工确认" + return round(risk, 1), action, server_like + + +def behavior_score(feat): + """Behaviour-only risk score. Uses no rule hits and no known list.""" + total = feat["total"] + if total < 10: + return 0, ["请求过少"] + score = 0 + signals = [] + pct4 = feat["pct4"] + if pct4 >= 50: + score += 30 + signals.append("高失败率 %.0f%%" % pct4) + elif pct4 >= 30: + score += 15 + signals.append("失败率 %.0f%%" % pct4) + if feat["uri_card"] >= 20: + score += 20 + signals.append("路径分散（%d 个 URI）" % feat["uri_card"]) + elif feat["uri_card"] >= 8: + score += 8 + if feat["coverage"] <= 30 and feat["peak_share"] >= 25: + score += 20 + signals.append("短时爆发（覆盖 %.0f%%、峰值 %.0f%%）" + % (feat["coverage"], feat["peak_share"])) + if feat["login_ratio"] >= 20: + score += 15 + signals.append("登录接口集中 %.0f%%" % feat["login_ratio"]) + if feat["host_card"] >= 2: + score += 5 + signals.append("多站点（%d 个 Host）" % feat["host_card"]) + if feat["ua_card"] == 1: + score += 5 + if feat["ua_family"] in ("sdk", "fake-browser", "other"): + score += 10 + signals.append("自动化 UA（%s）" % feat["ua_family"]) + elif feat["ua_family"] == "browser": + score -= 5 + if feat.get("four_xx", 0) >= 100 and pct4 >= 20: + score += 10 + signals.append("失败请求 %d 次" % feat["four_xx"]) + # business/integration client: mostly POST or a few steady endpoints + if feat["get_ratio"] < 30: + score -= 25 + signals.append("非 GET 为主（业务接口）") + if pct4 < 15 and feat["pct2"] >= 80: + score -= 20 + signals.append("低失败率（采集/浏览）") + if (feat["coverage"] >= 80 and feat["cv"] is not None + and feat["cv"] <= 1.2 and feat["uri_templates"] <= 8): + score -= 20 + signals.append("长期稳定单接口") + if feat["uri_templates"] >= 15: + score += 10 + signals.append("路径模板分散（%d 个）" % feat["uri_templates"]) + if feat["status_known"] < 10: + score -= 10 + signals.append("状态字段缺失") + return max(0, min(100, score)), signals + + +def empty_feat(ip): + return { + "ip": ip, "total": 0, "uri_card": 0, "ua_card": 0, "host_card": 0, + "method_card": 0, "get_ratio": 0.0, "login_ratio": 0.0, + "pct2": 0.0, "pct3": 0.0, "pct4": 0.0, "pct5": 0.0, + "status_known": 0.0, "coverage": 0.0, "cv": None, "peak_share": 0.0, + "coverage5m": 0.0, "cv5m": None, "peak_share5m": 0.0, + "uri_templates": 0, "ua_family": "", "ua_garbage": False, + "top_uri": "", "top_ua": "", + } + + +def finalize_feat(feat, status_map, get_map, login_map, hourly_map): + ip = feat["ip"] + buckets = [c for _, c in hourly_map.get(ip, [])] + hour = bucket_stats(buckets) + sum_status = sum(status_map[cls].get(ip, 0) for cls in status_map) + total = feat["total"] + feat.update({ + "get_ratio": get_map.get(ip, 0) / total * 100 if total else 0.0, + "login_ratio": login_map.get(ip, 0) / total * 100 if total else 0.0, + "pct2": status_map["2*"].get(ip, 0) / max(1, sum_status) * 100, + "pct3": status_map["3*"].get(ip, 0) / max(1, sum_status) * 100, + "pct4": status_map["4*"].get(ip, 0) / max(1, sum_status) * 100, + "pct5": status_map["5*"].get(ip, 0) / max(1, sum_status) * 100, + "status_known": (sum_status / total * 100) if total else 0.0, + "coverage": hour["coverage"], + "cv": hour["cv"], + "peak_share": hour["peak_share"], + "four_xx": status_map["4*"].get(ip, 0), + }) + return feat + + +def analyze(): + client = load_client() + profile = json.load(open(os.path.join(ROOT, "profiles", "f5.json"))) + fields = profile.get("fields", profile) + + known = {} + try: + db = sqlite3.connect(DB_FILE) + db.row_factory = sqlite3.Row + for row in db.execute("select ip, ever_malicious from ip_stats"): + known[row["ip"]] = bool(row["ever_malicious"]) + except sqlite3.Error: + pass + + print("拉取 24h 全量聚合 ...", flush=True) + st = pivot(client, + [{"type": "count", "id": "cnt"}, + {"type": "card", "field": "uri", "id": "uri_card"}, + {"type": "card", "field": "ua", "id": "ua_card"}, + {"type": "card", "field": "host", "id": "host_card"}, + {"type": "card", "field": "method", "id": "method_card"}], + ["client_ip"], limit=600) + total = rows_to_dict(st, "cnt") + uri_card = rows_to_dict(st, "uri_card") + ua_card = rows_to_dict(st, "ua_card") + host_card = rows_to_dict(st, "host_card") + method_card = rows_to_dict(st, "method_card") + + status = {cls: {} for cls in ("2*", "3*", "4*", "5*")} + for cls in status: + st_cls = pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query="status:%s" % cls, limit=600) + status[cls] = rows_to_dict(st_cls, "cnt") + get_count = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query="method:GET", limit=600), "cnt") + LOGIN_QUERY = ("uri:(\\/login* OR \\/passport* OR \\/captcha* " + "OR \\/jsxy\\/api\\/login*)") + login_count = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query=LOGIN_QUERY, limit=600), "cnt") + st_hour = pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + column_groups=[{"type": "time", "field": "timestamp", + "interval": "1h"}], limit=600) + hourly = hourly_matrix(st_hour) + + # IPs with the most 4xx are the ones a behaviour model should look at; + # they are usually outside the volume top-600. + st_4xx = pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query="status:4*", limit=800) + four_xx = rows_to_dict(st_4xx, "cnt") + + base = {} + for ip, count in total.items(): + if count <= 0: + continue + feat = empty_feat(ip) + feat["total"] = int(count) + feat["uri_card"] = int(uri_card.get(ip, 0)) + feat["ua_card"] = int(ua_card.get(ip, 0)) + feat["host_card"] = int(host_card.get(ip, 0)) + feat["method_card"] = int(method_card.get(ip, 0)) + base[ip] = finalize_feat(feat, status, get_count, login_count, hourly) + + extra = sorted(set(four_xx) - set(total), key=lambda ip: -four_xx[ip]) + extra = [ip for ip in extra if not ip.startswith(INTERNAL)][:600] + print("补充 %d 个高 4xx IP 的计数 ..." % len(extra), flush=True) + for offset in range(0, len(extra), 80): + chunk = extra[offset:offset + 80] + chunk_query = " OR ".join('client_ip:"%s"' % ip for ip in chunk) + st = pivot(client, + [{"type": "count", "id": "cnt"}, + {"type": "card", "field": "uri", "id": "uri_card"}, + {"type": "card", "field": "ua", "id": "ua_card"}, + {"type": "card", "field": "host", "id": "host_card"}, + {"type": "card", "field": "method", "id": "method_card"}], + ["client_ip"], query=chunk_query, limit=200) + totals = rows_to_dict(st, "cnt") + uris = rows_to_dict(st, "uri_card") + uas = rows_to_dict(st, "ua_card") + hosts = rows_to_dict(st, "host_card") + methods = rows_to_dict(st, "method_card") + chunk_status = {} + for cls in ("2*", "3*", "4*", "5*"): + chunk_status[cls] = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query='status:%s AND (%s)' % (cls, chunk_query), limit=200), + "cnt") + chunk_get = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query='method:GET AND (%s)' % chunk_query, limit=200), "cnt") + chunk_login = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query='%s AND (%s)' % (LOGIN_QUERY, chunk_query), limit=200), + "cnt") + chunk_hour = hourly_matrix(pivot( + client, [{"type": "count", "id": "cnt"}], ["client_ip"], + column_groups=[{"type": "time", "field": "timestamp", + "interval": "1h"}], query=chunk_query, limit=200)) + for ip in chunk: + if ip not in totals or totals[ip] <= 0: + continue + for cls in chunk_status: + status[cls].update(chunk_status[cls]) + ctotal = totals[ip] + feat = empty_feat(ip) + feat["total"] = int(ctotal) + feat["uri_card"] = int(uris.get(ip, 0)) + feat["ua_card"] = int(uas.get(ip, 0)) + feat["host_card"] = int(hosts.get(ip, 0)) + feat["method_card"] = int(methods.get(ip, 0)) + base[ip] = finalize_feat(feat, status, chunk_get, chunk_login, chunk_hour) + get_count.update(chunk_get) + login_count.update(chunk_login) + + print("候选 %d 个，预筛细节 ..." % len(base), flush=True) + shortlist = [] + for ip, feat in base.items(): + if ip.startswith(INTERNAL): + continue + if feat["total"] >= 10 and (feat["pct4"] >= 20 or feat["four_xx"] >= 20 + or feat["uri_card"] >= 12): + shortlist.append(ip) + shortlist.sort(key=lambda ip: -base[ip]["four_xx"]) + shortlist = shortlist[:260] + details = gather_details(client, shortlist) + + geo = None + try: + from graylog_ops.geo import GeoIP + geo = GeoIP(os.path.join(ROOT, "data", "geo", "geo.db")) + except (FileNotFoundError, sqlite3.Error): + geo = None + + flagged = [] + for ip, feat in base.items(): + if ip.startswith(INTERNAL): + continue + detail = details.get(ip) or {} + five = bucket_stats(detail.get("buckets5m") or []) + uris = detail.get("uri") or [] + uas = detail.get("ua") or [] + feat["coverage5m"] = five["coverage"] + feat["cv5m"] = five["cv"] + feat["peak_share5m"] = five["peak_share"] + feat["uri_templates"] = len({uri_template(uri) for uri, _ in uris}) + feat["top_uri"] = uris[0][0] if uris else "" + feat["top_ua"] = uas[0][0] if uas else "" + feat["ua_family"] = ua_family(feat["top_ua"]) + feat["ua_garbage"] = ua_garbage(feat["ua_card"], feat["top_ua"]) + + score, signals = behavior_score(feat) + if score < 60 or len(signals) < 2: + continue + crawler_tokens = ("sogou", "baiduspider", "googlebot", "bingbot", + "yandex", "bytespider", "ahrefs", "semrush", "mj12", + "dotbot", "petalbot") + is_crawler_ua = any(token in feat["top_ua"].lower() + for token in crawler_tokens) + if is_crawler_ua: + behavior_type = ("伪造爬虫 UA" if feat["pct4"] >= 50 + else "采集爬虫") + elif feat["login_ratio"] >= 20: + behavior_type = "登录爆破" + elif feat["pct4"] >= 50 and feat["uri_card"] >= 20: + behavior_type = "目录扫描" + elif feat["coverage"] <= 30 and feat["peak_share"] >= 25: + behavior_type = "短时扫描" + elif feat["uri_templates"] >= 100 and feat["pct2"] >= 70: + behavior_type = "内容抓取" + else: + behavior_type = "异常自动化" + geo_info = geo.lookup(ip) if geo else None + feat.update({ + "score": score, + "signals": signals, + "behavior_type": behavior_type, + "in_list": known.get(ip, False), + "country_code": (geo_info or {}).get("country_code", ""), + "country_name": (geo_info or {}).get("country_name", ""), + "region": (geo_info or {}).get("region", ""), + "city": (geo_info or {}).get("city", ""), + }) + info = asn_info(ip) or {"asn": "", "org": "未知", "shared": "unknown"} + shared = info["shared"] + org = info["org"] or "未知" + multi_device = feat["ua_card"] >= 5 or feat["host_card"] >= 10 + if shared in ("mobile", "education"): + verdict, vclass = "仅告警（共享出口）", "v-watch" + share_note = ("出口属于%s（%s），大概率有大量用户共享；" + "整体封禁会误伤，建议只告警或按 UA/路径单独拦截。" + % (org_cn(org), SHARE_LABEL[shared])) + elif shared == "broadband" and multi_device: + verdict, vclass = "仅告警（疑似多用户）", "v-watch" + share_note = ("出口属于%s（宽带 ISP），且该 IP 出现 %d 种 UA / %d 个站点，" + "疑似多用户共享，整体封禁可能误伤。" + % (org_cn(org), feat["ua_card"], feat["host_card"])) + elif shared == "broadband": + if feat["score"] >= 75: + verdict, vclass = "建议封禁", "v-ban" + else: + verdict, vclass = "建议人工确认", "v-review" + share_note = ("出口属于%s（宽带 ISP），但该 IP 只有 %d 种 UA，" + "更像单台设备/家庭网络，封禁影响面小。" + % (org_cn(org), feat["ua_card"])) + elif shared == "hosting": + verdict, vclass = "建议封禁", "v-ban" + share_note = "出口属于%s（云主机/机房），不是家庭或移动网络，封禁影响面小。" % org_cn(org) + else: + verdict, vclass = "建议人工确认", "v-review" + share_note = "出口属于%s，归属不明确，建议人工核实后再决定。" % org_cn(org) + feat.update({ + "verdict": verdict, "vclass": vclass, + "reason_text": humanize(feat), + "share_note": share_note, + "asn": info["asn"], "asn_org": org, "shared_type": shared, + }) + flagged.append(feat) + + flagged.sort(key=lambda item: (-item["score"], -item["total"])) + metrics = { + "flagged": len(flagged), + "new": sum(1 for item in flagged if not item["in_list"]), + "known": sum(1 for item in flagged if item["in_list"]), + "high": sum(1 for item in flagged if item["score"] >= 80), + "ban": sum(1 for item in flagged if item["verdict"] == "建议封禁"), + "watch": sum(1 for item in flagged if item["verdict"].startswith("仅告警")), + "review": sum(1 for item in flagged if item["verdict"] == "建议人工确认"), + "candidates": len(base), + } + payload = { + "generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "window_seconds": WINDOW, + "items": flagged, + "metrics": metrics, + } + os.makedirs(os.path.dirname(DATA_FILE), exist_ok=True) + with open(DATA_FILE, "w", encoding="utf-8") as fh: + json.dump(payload, fh, ensure_ascii=False) + if geo is not None: + geo.close() + return payload + + +def serve(port): + class Handler(BaseHTTPRequestHandler): + def log_message(self, *args): + pass + + def _send(self, code, body, ctype="application/json; charset=utf-8"): + raw = body.encode("utf-8") + self.send_response(code) + self.send_header("Content-Type", ctype) + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) + + def do_GET(self): + if self.path.startswith("/data.json"): + try: + with open(DATA_FILE, encoding="utf-8") as fh: + return self._send(200, fh.read()) + except FileNotFoundError: + return self._send(404, '{"error":"no data"}') + return self._send(200, PAGE, "text/html; charset=utf-8") + + server = ThreadingHTTPServer(("0.0.0.0", port), Handler) + print("行为分析原型: http://0.0.0.0:%d/ (Ctrl+C 退出)" % port, flush=True) + server.serve_forever() + + +PAGE = r""" + + +行为检测 · 独立判定 + +

行为检测 · 独立判定

+
不用规则、不看名单，纯行为特征算出的恶意 IP · 只读
+
加载中…
+
+
+
+
+ +
+
+
+
+ 算法说明（纯行为评分，不依赖规则和名单） +
    +
  • 范围：24h 内全部公网 IP（约 1000+ 个候选），先按失败率/路径分散预筛，再抽样细节。
    • +
  • 评分信号：高失败率、路径分散、短时爆发、登录接口集中、多站点、自动化 UA、 + 路径模板分散、失败请求量大；业务接口（非 GET 为主）和长期稳定单接口会降权。
    • +
  • 判定：行为分 ≥60 且至少 2 个信号 → 行为判定恶意。
    • +
  • 建议动作：出口是移动/宽带/教育网（共享网络）→ 仅告警，避免误伤； + 出口是云主机/机房 → 建议封禁；其余 → 人工确认。
    • +
  • 名单列只用于事后对比：红色「新发现」= 不在现有规则名单。
    • +
+
+
+ +""" + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--analyze", action="store_true") + parser.add_argument("--serve", action="store_true") + parser.add_argument("--port", type=int, default=8090) + args = parser.parse_args() + if args.analyze or not os.path.exists(DATA_FILE): + payload = analyze() + print("分析完成: %d 个 IP" % len(payload["items"]), flush=True) + if args.serve: + serve(args.port) diff --git a/asn_brand_map.json b/asn_brand_map.json new file mode 100644 index 0000000..6ae7ba2 --- /dev/null +++ b/asn_brand_map.json @@ -0,0 +1,43 @@ +{ + "items": [ + {"keywords": ["alibaba", "aliyun"], "brand": "阿里云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["tencent"], "brand": "腾讯云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["huawei cloud", "huaweicloud"], "brand": "华为云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["baidu"], "brand": "百度云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["kingsoft", "ksyun"], "brand": "金山云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["jd cloud", "jingdong"], "brand": "京东云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["ucloud"], "brand": "UCloud", "type": "cloud", "icon": "☁️"}, + {"keywords": ["ctyun", "china telecom cloud"], "brand": "天翼云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["china mobile cloud", "cmcloud"], "brand": "移动云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["cucloud", "china unicom cloud"], "brand": "联通云", "type": "cloud", "icon": "☁️"}, + {"keywords": ["amazon", "aws"], "brand": "AWS", "type": "cloud", "icon": "☁️"}, + {"keywords": ["microsoft"], "brand": "Azure", "type": "cloud", "icon": "☁️"}, + {"keywords": ["google"], "brand": "Google Cloud", "type": "cloud", "icon": "☁️"}, + {"keywords": ["oracle"], "brand": "Oracle Cloud", "type": "cloud", "icon": "☁️"}, + {"keywords": ["ibm"], "brand": "IBM Cloud", "type": "cloud", "icon": "☁️"}, + {"keywords": ["digitalocean", "digital ocean"], "brand": "DigitalOcean", "type": "cloud", "icon": "☁️"}, + {"keywords": ["vultr", "constant company"], "brand": "Vultr", "type": "cloud", "icon": "☁️"}, + {"keywords": ["linode", "akamai"], "brand": "Akamai/Linode", "type": "cloud", "icon": "☁️"}, + {"keywords": ["hetzner"], "brand": "Hetzner", "type": "cloud", "icon": "☁️"}, + {"keywords": ["ovh"], "brand": "OVH", "type": "cloud", "icon": "☁️"}, + {"keywords": ["contabo"], "brand": "Contabo", "type": "cloud", "icon": "☁️"}, + {"keywords": ["scaleway"], "brand": "Scaleway", "type": "cloud", "icon": "☁️"}, + {"keywords": ["rackspace"], "brand": "Rackspace", "type": "cloud", "icon": "☁️"}, + {"keywords": ["upcloud"], "brand": "UpCloud", "type": "cloud", "icon": "☁️"}, + {"keywords": ["equinix"], "brand": "Equinix", "type": "idc", "icon": "🏢"}, + {"keywords": ["digital realty"], "brand": "Digital Realty", "type": "idc", "icon": "🏢"}, + {"keywords": ["21vianet", "vianet", "世纪互联"], "brand": "世纪互联", "type": "idc", "icon": "🏢"}, + {"keywords": ["chinanet idc", "idc network"], "brand": "IDC 机房", "type": "idc", "icon": "🏢"}, + {"keywords": ["wowrack", "colocation", "colo"], "brand": "IDC 机房", "type": "idc", "icon": "🏢"}, + {"keywords": ["data center", "datacenter", "hosting"], "brand": "IDC 机房", "type": "idc", "icon": "🏢"}, + {"keywords": ["china mobile", "cmnet"], "brand": "中国移动", "type": "mobile", "icon": "📱"}, + {"keywords": ["china unicom", "china169"], "brand": "中国联通", "type": "broadband", "icon": "🏠"}, + {"keywords": ["chinanet", "china telecom", "chinatelecom"], "brand": "中国电信", "type": "broadband", "icon": "🏠"}, + {"keywords": ["china broadcasting", "cbnet", "广电"], "brand": "中国广电", "type": "broadband", "icon": "🏠"}, + {"keywords": ["great wall broadband", "gwbn"], "brand": "长城宽带", "type": "broadband", "icon": "🏠"}, + {"keywords": ["dr.peng", "pengbo", "鹏博士"], "brand": "鹏博士", "type": "broadband", "icon": "🏠"}, + {"keywords": ["cernet", "education network", "education and research", "university", "college", "教育"], "brand": "教育网", "type": "education", "icon": "🎓"}, + {"keywords": ["government", "gov.cn"], "brand": "政府机构", "type": "enterprise", "icon": "🏛️"} + ], + "defaults": {"brand": "未知", "type": "unknown", "icon": "❓"} +} diff --git a/bin/graylog-ops b/bin/graylog-ops new file mode 100755 index 0000000..3db7f7d --- /dev/null +++ b/bin/graylog-ops @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Wrapper so the tool can be called as `bin/graylog-ops ...` from anywhere. +set -euo pipefail +here="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +export PYTHONPATH="${here}${PYTHONPATH:+:${PYTHONPATH}}" +exec python3 -m graylog_ops "$@" diff --git a/config.example.json b/config.example.json new file mode 100644 index 0000000..624edc1 --- /dev/null +++ b/config.example.json @@ -0,0 +1,7 @@ +{ + "url": "http://edu-master:9002", + "token": "REPLACE_WITH_API_TOKEN", + "insecure": false, + "timeout": 30, + "retries": 2 +} diff --git a/docker-build/Dockerfile b/docker-build/Dockerfile new file mode 100644 index 0000000..0126dad --- /dev/null +++ b/docker-build/Dockerfile @@ -0,0 +1,30 @@ +FROM python:3.12-slim + +ENV PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 \ + DATA_DIR=/data + +WORKDIR /app + +# application code (no credentials, no config files) +COPY app/graylog_ops /app/graylog_ops +COPY app/bin /app/bin +COPY app/profiles /app/profiles +COPY app/allowlist.example.txt /app/allowlist.example.txt +COPY app/asn_brand_map.json /app/asn_brand_map.json + +# seed data: current production snapshot (sqlite + geo + asn + blocklist) +COPY seed /app/seed + +COPY entrypoint.sh /app/entrypoint.sh +RUN chmod +x /app/entrypoint.sh + +VOLUME ["/data"] +EXPOSE 8088 + +HEALTHCHECK --interval=60s --timeout=5s --start-period=30s \ + CMD python3 -c "import os,urllib.request,sys; \ +url='http://127.0.0.1:%s/healthz' % os.environ.get('APP_PORT','8088'); \ +sys.exit(0 if urllib.request.urlopen(url, timeout=4).status == 200 else 1)" + +ENTRYPOINT ["/app/entrypoint.sh"] diff --git a/docker-build/entrypoint.sh b/docker-build/entrypoint.sh new file mode 100755 index 0000000..2dabcc2 --- /dev/null +++ b/docker-build/entrypoint.sh @@ -0,0 +1,30 @@ +#!/bin/sh +set -eu +DATA_DIR="${DATA_DIR:-/data}" +if [ ! -f "$DATA_DIR/ips.db" ]; then + echo "[init] 初始化数据目录: $DATA_DIR" + mkdir -p "$DATA_DIR" + cp -a /app/seed/. "$DATA_DIR"/ +fi + +set -- watch \ + --stream "${GRAYLOG_STREAM:-F5_Accesslog}" \ + --profile "${PROFILE:-/app/profiles/f5.json}" \ + --interval "${CHECK_INTERVAL:-60}" \ + --window "${WINDOW:-3600}" \ + --forget "${FORGET:-604800}" \ + --db "$DATA_DIR/ips.db" \ + --host 0.0.0.0 \ + --port "${APP_PORT:-8088}" \ + --path-prefix "${APP_PATH_PREFIX:-}" \ + --geo-db "$DATA_DIR/geo/geo.db" \ + --asn-db "$DATA_DIR/geo/asn.db" \ + --allow /app/allowlist.example.txt \ + --export "$DATA_DIR/blocklist.txt" \ + --behavior-interval "${BEHAVIOR_INTERVAL:-900}" \ + --warm-limit "${WARM_LIMIT:-5}" + +if [ "${NO_BEHAVIOR:-0}" = "1" ]; then + set -- "$@" --no-behavior +fi +exec python3 -m graylog_ops "$@" diff --git a/docs/screenshots/behavior-prototype.png b/docs/screenshots/behavior-prototype.png new file mode 100644 index 0000000..062557c Binary files /dev/null and b/docs/screenshots/behavior-prototype.png differ diff --git a/docs/screenshots/detail-asn.png b/docs/screenshots/detail-asn.png new file mode 100644 index 0000000..35a48c2 Binary files /dev/null and b/docs/screenshots/detail-asn.png differ diff --git a/docs/screenshots/detail.png b/docs/screenshots/detail.png new file mode 100644 index 0000000..c8c9708 Binary files /dev/null and b/docs/screenshots/detail.png differ diff --git a/docs/screenshots/list-fused.png b/docs/screenshots/list-fused.png new file mode 100644 index 0000000..f30f518 Binary files /dev/null and b/docs/screenshots/list-fused.png differ diff --git a/docs/screenshots/list-geo.png b/docs/screenshots/list-geo.png new file mode 100644 index 0000000..0e6fe04 Binary files /dev/null and b/docs/screenshots/list-geo.png differ diff --git a/docs/screenshots/list.png b/docs/screenshots/list.png new file mode 100644 index 0000000..7e2a358 Binary files /dev/null and b/docs/screenshots/list.png differ diff --git a/graylog_ops/__init__.py b/graylog_ops/__init__.py new file mode 100644 index 0000000..ba876f6 --- /dev/null +++ b/graylog_ops/__init__.py @@ -0,0 +1,8 @@ +"""Read-only Graylog inspection toolkit. + +Target API: Graylog 4.0.x (legacy ``/api/search/universal`` surface). +No third-party dependencies: stdlib only, so it also runs on old CentOS 7 +python3 installations. +""" + +__version__ = "0.2.0" diff --git a/graylog_ops/__main__.py b/graylog_ops/__main__.py new file mode 100644 index 0000000..dbdd066 --- /dev/null +++ b/graylog_ops/__main__.py @@ -0,0 +1,6 @@ +import sys + +from .cli import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/graylog_ops/asn.py b/graylog_ops/asn.py new file mode 100644 index 0000000..967ac22 --- /dev/null +++ b/graylog_ops/asn.py @@ -0,0 +1,107 @@ +"""OFFLINE ASN lookup + provider branding (DB-IP ASN Lite).""" +import bisect +import csv +import ipaddress +import json +import os +import sqlite3 + +DEFAULT_MAP = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), + "asn_brand_map.json") + + +def load_brand_map(path=None): + try: + with open(path or DEFAULT_MAP, encoding="utf-8") as handle: + data = json.load(handle) + except (OSError, ValueError): + data = {"items": [], "defaults": {"brand": "未知", "type": "unknown", + "icon": "❓"}} + return data + + +def classify_org(org, brand_map=None): + brand_map = brand_map or load_brand_map() + text = (org or "").lower() + for item in brand_map.get("items", []): + if any(keyword in text for keyword in item.get("keywords", [])): + return item["brand"], item["type"], item["icon"] + fallback = brand_map.get("defaults") or {} + return (fallback.get("brand", "未知"), fallback.get("type", "unknown"), + fallback.get("icon", "❓")) + + +def build_asn_db(csv_path, out_path): + os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) + conn = sqlite3.connect(out_path) + conn.execute("DROP TABLE IF EXISTS asn") + conn.execute("CREATE TABLE asn (start INTEGER PRIMARY KEY, end INTEGER, " + "asn TEXT, org TEXT)") + batch, total = [], 0 + + def flush(): + nonlocal batch + if batch: + conn.executemany("INSERT OR REPLACE INTO asn VALUES (?,?,?,?)", batch) + batch = [] + + with open(csv_path, encoding="utf-8", errors="replace") as handle: + for row in csv.reader(handle): + if len(row) < 4 or ":" in row[0]: + continue + try: + first = int(ipaddress.IPv4Address(row[0])) + last = int(ipaddress.IPv4Address(row[1])) + except ValueError: + continue + batch.append((first, last, row[2], row[3])) + total += 1 + if len(batch) >= 100000: + flush() + conn.commit() + flush() + conn.commit() + conn.execute("PRAGMA journal_mode=WAL") + conn.close() + return total + + +class AsnDB: + def __init__(self, path, brand_map_path=None): + self.conn = sqlite3.connect(path, check_same_thread=False) + self.brand_map = load_brand_map(brand_map_path) + self._prefix = None + with self.conn: + try: + self.conn.execute("PRAGMA query_only=ON") + except sqlite3.Error: + pass + + def lookup(self, ip): + try: + value = int(ipaddress.IPv4Address(ip)) + except ValueError: + return None + row = self.conn.execute( + "SELECT end, asn, org FROM asn WHERE start <= ? " + "ORDER BY start DESC LIMIT 1", (value,)).fetchone() + if not row or row[0] < value: + return None + brand, kind, icon = classify_org(row[2], self.brand_map) + return {"asn": row[1], "org": row[2], "brand": brand, + "type": kind, "icon": icon} + + def close(self): + try: + self.conn.close() + except sqlite3.Error: + pass + + +if __name__ == "__main__": + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--csv", default="data/geo/dbip-asn-lite-2026-10.csv") + parser.add_argument("--out", default="data/geo/asn.db") + args = parser.parse_args() + print("imported %d ranges -> %s" % (build_asn_db(args.csv, args.out), args.out)) diff --git a/graylog_ops/behavior.py b/graylog_ops/behavior.py new file mode 100644 index 0000000..e80c5f7 --- /dev/null +++ b/graylog_ops/behavior.py @@ -0,0 +1,695 @@ +"""Behaviour detection running inside the main service. + +It scans 24h of F5 traffic with Graylog pivot aggregations, scores each IP by +behaviour only (no rule hits, no known list), and stores the result in SQLite. +A background thread keeps it refreshed; the web UI reads the stored result. +""" +import csv +import ipaddress +import json +import os +import statistics +import threading +import time + +from .http import GraylogError +from .asn import AsnDB + +WINDOW = 86400 +INTERNAL = ("10.", "192.168.", "172.16.", "172.17.", "172.18.", "172.19.", + "172.2", "172.3") + +UA_SDK = ("curl", "wget", "python-requests", "python-urllib", "go-http-client", + "okhttp", "java/", "apache-httpclient", "httpcomponents", + "libredtail", "axios", "node-fetch", "aiohttp", "scrapy", + "postmanruntime", "insomnia", "unityplayer", "dart/", "httpie", + "guzzle", "faraday", "restsharp", "libwww", "eeo/", "academy_widget") +UA_BROWSER = ("chrome", "firefox", "safari", "edg/", "opr/", "gecko/20100101") + +CRAWLER_TOKENS = ("sogou", "baiduspider", "googlebot", "bingbot", "yandex", + "bytespider", "ahrefs", "semrush", "mj12", "dotbot", + "petalbot") + +RULE_TITLES = { + "probe_java": "Spring/中间件探测", "probe_config": "配置/凭证探测", + "probe_cms": "CMS 探测", "probe_iot": "IoT/路由器探测", + "probe_devops": "DevOps/监控探测", "probe_webshell": "WebShell/上传探测", + "exploit_payload": "漏洞利用 payload", "login_attack": "登录失败尝试", + "login_burst": "登录接口高频", "bad_ua": "扫描器 UA", + "bot_ua": "僵尸网络 UA", "not_found": "404 探测", +} + +ASN_CACHE = {} + + +def _asn_csv(root): + geo_dir = os.path.join(root, "data", "geo") + try: + names = sorted(name for name in os.listdir(geo_dir) + if name.startswith("dbip-asn-lite-") and name.endswith(".csv")) + except FileNotFoundError: + return None + return os.path.join(geo_dir, names[-1]) if names else None + + +def _load_asn(path): + if ASN_CACHE: + return ASN_CACHE + starts, ranges = [], [] + try: + with open(path, encoding="utf-8", errors="replace") as fh: + for row in csv.reader(fh): + if len(row) < 4 or ":" in row[0]: + continue + try: + first = int(ipaddress.IPv4Address(row[0])) + last = int(ipaddress.IPv4Address(row[1])) + except ValueError: + continue + starts.append(first) + ranges.append((last, row[2], row[3])) + except (FileNotFoundError, TypeError): + pass + order = sorted(range(len(starts)), key=lambda i: starts[i]) + ASN_CACHE.update({"starts": [starts[i] for i in order], + "ranges": [ranges[i] for i in order]}) + return ASN_CACHE + + +def shared_type(org): + text = (org or "").lower() + cloud = ("alibaba", "aliyun", "tencent", "huawei cloud", "aws", "amazon", + "google", "microsoft", "digitalocean", "vultr", "linode", "hetzner", + "ovh", "wowrack", "idc", "data center", "datacenter", "cloud", + "hosting", "colo") + if any(token in text for token in cloud): + return "hosting" + if "china mobile" in text or "中国移动" in text or "cmnet" in text: + return "mobile" + if ("china unicom" in text or "china telecom" in text or "chinanet" in text + or "联通" in text or "电信" in text or "broadband" in text): + return "broadband" + if "education" in text or "cernet" in text or "教育" in text: + return "education" + return "unknown" + + +ORG_CN = (("china mobile", "中国移动"), ("china unicom", "中国联通"), + ("china telecom", "中国电信"), ("chinanet", "中国电信"), + ("cernet", "中国教育网"), ("education", "教育网"), + ("alibaba", "阿里云"), ("aliyun", "阿里云"), ("huawei cloud", "华为云"), + ("tencent", "腾讯云"), ("wowrack", "Wowrack 机房"), + ("vmiss", "VMISS 机房"), ("idc", "IDC 机房")) + +TYPE_TO_SHARED = {"mobile": "mobile", "broadband": "broadband", + "education": "education", "cloud": "hosting", "idc": "hosting"} + +SHARE_LABEL = {"mobile": "移动网络", "broadband": "宽带 ISP", + "education": "教育网", "hosting": "云主机/机房", "unknown": "未知网络"} + + +def org_cn(org): + text = (org or "").lower() + for token, name in ORG_CN: + if token in text: + return name + return org or "未知" + + +def asn_info(ip, root): + cache = _load_asn(_asn_csv(root)) + starts = cache.get("starts") or [] + if not starts: + return {"asn": "", "org": "未知", "shared": "unknown"} + try: + value = int(ipaddress.IPv4Address(ip)) + except ValueError: + return {"asn": "", "org": "未知", "shared": "unknown"} + import bisect + index = bisect.bisect_right(starts, value) - 1 + if index >= 0: + last, asn, org = cache["ranges"][index] + if last >= value: + return {"asn": asn, "org": org, "shared": shared_type(org)} + return {"asn": "", "org": "未知", "shared": "unknown"} + + +def pivot(client, series, row_groups, column_groups=None, query="*", + limit=300, timeout=180000): + body = {"queries": [{ + "id": "q1", + "timerange": {"type": "relative", "range": WINDOW}, + "query": {"type": "elasticsearch", "query_string": query}, + "filter": {"type": "or", "filters": [{"type": "stream", + "id": "6ac83059a6878e26256fcacc"}]}, + "search_types": [{ + "type": "pivot", "id": "st1", "series": series, + "row_groups": [{"type": "values", "field": f, "limit": limit} + for f in row_groups], + "column_groups": column_groups or [], "sort": [], "rollup": True, + }], + }], "parameters": []} + data = client.http.request("POST", "/api/views/search/sync", + params={"timeout": timeout}, body=body) + return data["results"]["q1"]["search_types"]["st1"] + + +def rows_to_dict(search_type, series_id): + out = {} + for row in search_type.get("rows") or []: + key = row.get("key") or [] + if not key: + continue + ip = str(key[0]) + for value in row.get("values") or []: + if (value.get("key") or [None])[0] == series_id: + try: + out[ip] = float(value.get("value") or 0) + except (TypeError, ValueError): + out[ip] = 0.0 + return out + + +def hourly_matrix(search_type): + out = {} + for row in search_type.get("rows") or []: + key = row.get("key") or [] + if not key: + continue + counts = [] + for value in row.get("values") or []: + stamp = (value.get("key") or [None])[0] + if not stamp or "cnt" not in (value.get("key") or []): + continue + try: + counts.append(float(value.get("value") or 0)) + except (TypeError, ValueError): + pass + out[str(key[0])] = counts + return out + + +def bucket_stats(counts): + active = [c for c in counts if c > 0] + if not counts: + return {"coverage": 0.0, "cv": None, "peak_share": 0.0} + cv = (statistics.pstdev(active) / statistics.fmean(active) + if len(active) > 2 and statistics.fmean(active) else None) + total = sum(counts) + return {"coverage": len(active) / len(counts) * 100, + "cv": cv, + "peak_share": (max(counts) / total * 100) if total else 0.0} + + +def uri_template(uri): + import re + path = str(uri or "").split("?")[0] + path = re.sub( + r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-" + r"[0-9a-fA-F]{12}", "{uuid}", path) + path = re.sub(r"[0-9a-fA-F]{16,}", "{hex}", path) + path = re.sub(r"\d+", "{n}", path) + return path + + +def ua_family(ua): + text = str(ua or "").strip() + if not text: + return "empty" + lower = text.lower() + if any(token in lower for token in UA_SDK): + return "sdk" + if lower.startswith("mozilla/") and any(t in lower for t in UA_BROWSER): + return "browser" + if lower.startswith("mozilla/5.0"): + return "fake-browser" + return "other" + + +def ua_garbage(ua_card, top_ua): + if ua_card > 50: + return True + text = str(top_ua or "") + if not text: + return False + printable = sum(1 for char in text if 32 <= ord(char) < 127 or ord(char) > 127) + return printable / max(1, len(text)) < 0.85 + + +def behavior_score(feat): + total = feat["total"] + if total < 10: + return 0, ["请求过少"] + score = 0 + signals = [] + pct4 = feat["pct4"] + if pct4 >= 50: + score += 30 + signals.append("高失败率 %.0f%%" % pct4) + elif pct4 >= 30: + score += 15 + signals.append("失败率 %.0f%%" % pct4) + if feat["uri_card"] >= 20: + score += 20 + signals.append("路径分散（%d 个 URI）" % feat["uri_card"]) + elif feat["uri_card"] >= 8: + score += 8 + if feat["coverage"] <= 30 and feat["peak_share"] >= 25: + score += 20 + signals.append("短时爆发（覆盖 %.0f%%、峰值 %.0f%%）" + % (feat["coverage"], feat["peak_share"])) + if feat["login_ratio"] >= 20: + score += 15 + signals.append("登录接口集中 %.0f%%" % feat["login_ratio"]) + if feat["host_card"] >= 2: + score += 5 + signals.append("多站点（%d 个 Host）" % feat["host_card"]) + if feat["ua_card"] == 1: + score += 5 + if feat["ua_family"] in ("sdk", "fake-browser", "other"): + score += 10 + signals.append("自动化 UA（%s）" % feat["ua_family"]) + elif feat["ua_family"] == "browser": + score -= 5 + if feat.get("four_xx", 0) >= 100 and pct4 >= 20: + score += 10 + signals.append("失败请求 %d 次" % int(feat["four_xx"])) + if feat["get_ratio"] < 30: + score -= 25 + signals.append("非 GET 为主（业务接口）") + if pct4 < 15 and feat["pct2"] >= 80: + score -= 20 + signals.append("低失败率（采集/浏览）") + if (feat["coverage"] >= 80 and feat["cv"] is not None + and feat["cv"] <= 1.2 and feat["uri_templates"] <= 8): + score -= 20 + signals.append("长期稳定单接口") + if feat["uri_templates"] >= 15: + score += 10 + signals.append("路径模板分散（%d 个）" % feat["uri_templates"]) + if feat["status_known"] < 10: + score -= 10 + signals.append("状态字段缺失") + return max(0, min(100, score)), signals + + +def humanize(item): + parts = [] + if item["uri_card"] >= 20: + parts.append("访问了 %s 个不同路径" % format(item["uri_card"], ",")) + if item["four_xx"] >= 20: + parts.append("产生 %s 次失败请求（失败率 %.0f%%）" + % (format(int(item["four_xx"]), ","), item["pct4"])) + elif item["pct4"] >= 20: + parts.append("失败率 %.0f%%" % item["pct4"]) + if item["coverage"] <= 30 and item["peak_share"] >= 25: + parts.append("流量集中在很短的时间内") + if item["login_ratio"] >= 20: + parts.append("大量请求集中在登录接口") + if item["host_card"] >= 2: + parts.append("同时访问了 %d 个站点" % item["host_card"]) + if item["ua_family"] in ("sdk", "fake-browser", "other"): + parts.append("使用脚本/非浏览器客户端") + behavior = item.get("behavior_type") or "异常自动化" + tail = {"目录扫描": "符合目录扫描特征", "短时扫描": "符合短时扫描特征", + "登录爆破": "符合登录爆破特征", + "伪造爬虫 UA": "伪装成搜索引擎爬虫，且请求大量失败", + "采集爬虫": "是采集爬虫", "异常自动化": "行为高度自动化，不像正常用户"}.get( + behavior, "行为异常") + if parts: + return "24 小时内" + "，".join(parts) + "——" + tail + "。" + return "24 小时内请求 %d 次，%s。" % (item["total"], tail) + + +def behavior_type(feat, top_ua): + tokens = CRAWLER_TOKENS + if any(token in (top_ua or "").lower() for token in tokens): + return "伪造爬虫 UA" if feat["pct4"] >= 50 else "采集爬虫" + if feat["login_ratio"] >= 20: + return "登录爆破" + if feat["pct4"] >= 50 and feat["uri_card"] >= 20: + return "目录扫描" + if feat["coverage"] <= 30 and feat["peak_share"] >= 25: + return "短时扫描" + if feat["uri_templates"] >= 100 and feat["pct2"] >= 70: + return "内容抓取" + return "异常自动化" + + +def empty_feat(ip): + return {"ip": ip, "total": 0, "uri_card": 0, "ua_card": 0, "host_card": 0, + "method_card": 0, "get_ratio": 0.0, "login_ratio": 0.0, + "pct2": 0.0, "pct3": 0.0, "pct4": 0.0, "pct5": 0.0, + "status_known": 0.0, "coverage": 0.0, "cv": None, "peak_share": 0.0, + "uri_templates": 0, "ua_family": "", "ua_garbage": False, + "top_uri": "", "top_ua": "", "four_xx": 0} + + +class BehaviorAnalyzer: + def __init__(self, client, store, fields, filter_str=None, root=None, + interval=900, window=WINDOW, notifier=None): + self.client = client + self.store = store + self.fields = fields + self.filter_str = filter_str + self.root = root or os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + self.interval = max(120, int(interval or 900)) + self.window = int(window or WINDOW) + self.notifier = notifier + self.last_run = None + self.last_error = None + self.runs = 0 + self.asn = None + try: + asn_path = os.path.join(self.root, "data", "geo", "asn.db") + if os.path.exists(asn_path): + self.asn = AsnDB(asn_path) + except Exception: + self.asn = None + self.geo = None + try: + from .geo import GeoIP + geo_path = os.path.join(self.root, "data", "geo", "geo.db") + if os.path.exists(geo_path): + self.geo = GeoIP(geo_path) + except Exception: + self.geo = None + + # ---------- data collection ---------- + def _details(self, ips): + def run(ip): + q = 'client_ip:"%s"' % ip + detail = {"uri": [], "ua": [], "buckets5m": []} + try: + st = pivot(self.client, [{"type": "count", "id": "cnt"}], + ["uri"], query=q, limit=40) + for row in st.get("rows") or []: + key = row.get("key") or [] + value = (row.get("values") or [{}])[0].get("value") + if key: + detail["uri"].append((str(key[0]), int(float(value or 0)))) + except GraylogError: + pass + try: + st = pivot(self.client, [{"type": "count", "id": "cnt"}], + ["ua"], query=q, limit=3) + for row in st.get("rows") or []: + key = row.get("key") or [] + value = (row.get("values") or [{}])[0].get("value") + if key: + detail["ua"].append((str(key[0]), int(float(value or 0)))) + except GraylogError: + pass + try: + hist = self.client.histogram(query=q, range_seconds=self.window, + interval="5m", + filter_str=self.filter_str) + from .client import normalize_histogram + detail["buckets5m"] = [b["count"] + for b in normalize_histogram(hist)] + except GraylogError: + pass + return ip, detail + + from concurrent.futures import ThreadPoolExecutor + out = {} + with ThreadPoolExecutor(max_workers=8) as pool: + for ip, detail in pool.map(run, ips): + out[ip] = detail + return out + + def analyze(self): + from concurrent.futures import ThreadPoolExecutor + fields = self.fields + client = self.client + + st = pivot(client, + [{"type": "count", "id": "cnt"}, + {"type": "card", "field": "uri", "id": "uri_card"}, + {"type": "card", "field": "ua", "id": "ua_card"}, + {"type": "card", "field": "host", "id": "host_card"}, + {"type": "card", "field": "method", "id": "method_card"}], + ["client_ip"], limit=600) + total = rows_to_dict(st, "cnt") + uri_card = rows_to_dict(st, "uri_card") + ua_card = rows_to_dict(st, "ua_card") + host_card = rows_to_dict(st, "host_card") + method_card = rows_to_dict(st, "method_card") + + status = {} + for cls in ("2*", "3*", "4*", "5*"): + status[cls] = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query="status:%s" % cls, limit=600), "cnt") + get_count = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query="method:GET", limit=600), "cnt") + LOGIN_QUERY = ("uri:(\\\\/login* OR \\\\/passport* OR \\\\/captcha* " + "OR \\\\/jsxy\\\\/api\\\\/login*)") + login_count = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query=LOGIN_QUERY, limit=600), "cnt") + hourly = hourly_matrix(pivot( + client, [{"type": "count", "id": "cnt"}], ["client_ip"], + column_groups=[{"type": "time", "field": "timestamp", "interval": "1h"}], + limit=600)) + four_xx = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query="status:4*", limit=800), "cnt") + + def build(ip, count, uris, uas, hosts, methods, status_map, get_map, + login_map, hourly_map): + hour = bucket_stats(hourly_map.get(ip, [])) + sum_status = sum(status_map[cls].get(ip, 0) for cls in status_map) + feat = empty_feat(ip) + feat.update({ + "total": int(count), + "uri_card": int(uris.get(ip, 0)), + "ua_card": int(uas.get(ip, 0)), + "host_card": int(hosts.get(ip, 0)), + "method_card": int(methods.get(ip, 0)), + "get_ratio": get_map.get(ip, 0) / count * 100 if count else 0.0, + "login_ratio": login_map.get(ip, 0) / count * 100 if count else 0.0, + "pct2": status_map["2*"].get(ip, 0) / max(1, sum_status) * 100, + "pct3": status_map["3*"].get(ip, 0) / max(1, sum_status) * 100, + "pct4": status_map["4*"].get(ip, 0) / max(1, sum_status) * 100, + "pct5": status_map["5*"].get(ip, 0) / max(1, sum_status) * 100, + "status_known": (sum_status / count * 100) if count else 0.0, + "coverage": hour["coverage"], "cv": hour["cv"], + "peak_share": hour["peak_share"], + "four_xx": status_map["4*"].get(ip, 0), + }) + return feat + + base = {} + for ip, count in total.items(): + if count <= 0: + continue + base[ip] = build(ip, count, uri_card, ua_card, host_card, + method_card, status, get_count, login_count, hourly) + + extra = sorted(set(four_xx) - set(total), key=lambda ip: -four_xx[ip]) + extra = [ip for ip in extra if not ip.startswith(INTERNAL)][:600] + for offset in range(0, len(extra), 80): + chunk = extra[offset:offset + 80] + chunk_query = " OR ".join('client_ip:"%s"' % ip for ip in chunk) + st = pivot(client, + [{"type": "count", "id": "cnt"}, + {"type": "card", "field": "uri", "id": "uri_card"}, + {"type": "card", "field": "ua", "id": "ua_card"}, + {"type": "card", "field": "host", "id": "host_card"}, + {"type": "card", "field": "method", "id": "method_card"}], + ["client_ip"], query=chunk_query, limit=200) + totals = rows_to_dict(st, "cnt") + uris = rows_to_dict(st, "uri_card") + uas = rows_to_dict(st, "ua_card") + hosts = rows_to_dict(st, "host_card") + methods = rows_to_dict(st, "method_card") + chunk_status = {} + for cls in ("2*", "3*", "4*", "5*"): + chunk_status[cls] = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query='status:%s AND (%s)' % (cls, chunk_query), + limit=200), "cnt") + chunk_get = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query='method:GET AND (%s)' % chunk_query, limit=200), "cnt") + chunk_login = rows_to_dict( + pivot(client, [{"type": "count", "id": "cnt"}], ["client_ip"], + query='%s AND (%s)' % (LOGIN_QUERY, chunk_query), + limit=200), "cnt") + chunk_hour = hourly_matrix(pivot( + client, [{"type": "count", "id": "cnt"}], ["client_ip"], + column_groups=[{"type": "time", "field": "timestamp", + "interval": "1h"}], query=chunk_query, limit=200)) + for cls in chunk_status: + status[cls].update(chunk_status[cls]) + for ip in chunk: + if ip not in totals or totals[ip] <= 0: + continue + get_count[ip] = chunk_get.get(ip, 0) + login_count[ip] = chunk_login.get(ip, 0) + base[ip] = build(ip, totals[ip], uris, uas, hosts, methods, + status, get_count, login_count, chunk_hour) + + shortlist = [ip for ip, feat in base.items() + if not ip.startswith(INTERNAL) and feat["total"] >= 10 + and (feat["pct4"] >= 20 or feat["four_xx"] >= 20 + or feat["uri_card"] >= 12)] + shortlist.sort(key=lambda ip: -base[ip]["four_xx"]) + shortlist = shortlist[:260] + details = self._details(shortlist) + + items = [] + for ip, feat in base.items(): + if ip.startswith(INTERNAL): + continue + detail = details.get(ip) or {} + five = bucket_stats(detail.get("buckets5m") or []) + uris = detail.get("uri") or [] + uas = detail.get("ua") or [] + feat["uri_templates"] = len({uri_template(uri) for uri, _ in uris}) + feat["top_uri"] = uris[0][0] if uris else "" + feat["top_ua"] = uas[0][0] if uas else "" + feat["ua_family"] = ua_family(feat["top_ua"]) + feat["ua_garbage"] = ua_garbage(feat["ua_card"], feat["top_ua"]) + feat["coverage5m"] = five["coverage"] + score, signals = behavior_score(feat) + if score < 60 or len(signals) < 2: + continue + feat["score"] = score + feat["signals"] = signals + feat["behavior_type"] = behavior_type(feat, feat["top_ua"]) + info = (self.asn.lookup(ip) if self.asn is not None else None) or { + "asn": "", "org": "未知", "brand": "未知", + "type": "unknown", "icon": "❓"} + shared = TYPE_TO_SHARED.get(info["type"], "unknown") + multi = feat["ua_card"] >= 5 or feat["host_card"] >= 10 + if shared in ("mobile", "education"): + verdict = "仅告警（共享出口）" + note = ("出口属于%s（%s），大概率有大量用户共享。" + % (info["brand"], SHARE_LABEL[shared])) + elif shared == "broadband" and multi: + verdict = "仅告警（疑似多用户）" + note = ("出口属于%s（宽带 ISP），但该 IP 出现 %d 种 UA / %d 个站点，" + "疑似多用户共享。" % (info["brand"], feat["ua_card"], + feat["host_card"])) + elif shared == "broadband": + verdict = "建议封禁" if score >= 75 else "建议人工确认" + note = ("出口属于%s（宽带 ISP），但只有 %d 种 UA，更像单台设备。" + % (info["brand"], feat["ua_card"])) + elif shared == "hosting": + verdict = "建议封禁" + note = "出口属于%s（云主机/机房）。" % info["brand"] + else: + verdict = "建议人工确认" + note = "出口属于%s，归属不明确。" % info["brand"] + geo = None + if self.geo is not None: + geo = self.geo.lookup(ip) + feat.update({ + "country_code": (geo or {}).get("country_code", ""), + "country_name": (geo or {}).get("country_name", ""), + "region": (geo or {}).get("region", ""), + "city": (geo or {}).get("city", ""), + "verdict": verdict, "share_note": note, + "reason_text": humanize(feat), + "asn": info["asn"], "asn_org": info["org"], + "asn_brand": info["brand"], "asn_type": info["type"], + "asn_icon": info["icon"], "shared_type": shared, + }) + items.append(feat) + items.sort(key=lambda item: (-item["score"], -item["total"])) + return items + + def run_once(self): + started = time.time() + try: + items = self.analyze() + self.store.replace_behavior_profiles(items) + self.store.sync_fused() + self.last_run = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + self.last_error = None + self.runs += 1 + print("[%s] 行为分析: %d 个违规 IP，%d 个建议封禁，耗时 %.1fs" + % (self.last_run, len(items), + sum(1 for item in items if item["verdict"] == "建议封禁"), + time.time() - started), flush=True) + if self.notifier is not None: + notify_fused(self.store, self.notifier, self.root) + return items + except Exception as exc: # keep the service alive + self.last_error = str(exc) + print("[行为分析失败] %s" % exc, flush=True) + return [] + + def run_forever(self): + time.sleep(8) + while True: + self.run_once() + time.sleep(self.interval) + + +TYPE_LABEL = {"cloud": "云服务商", "idc": "IDC/机房", "mobile": "移动网络", + "broadband": "宽带", "education": "教育网", + "enterprise": "企业/政府", "unknown": "未知"} + + +def _local_time(value): + import datetime as _dt + try: + stamp = _dt.datetime.fromisoformat(str(value).replace("Z", "+00:00")) + return stamp.astimezone().strftime("%m-%d %H:%M") + except (TypeError, ValueError): + return str(value or "-") + + +_PUSH_ERROR_AT = 0 + + +def notify_fused(store, notifier, root=None): + """Push a single merged message for IPs due for a reminder (24h).""" + due = store.due_pushes(active_hours=24, cooldown_hours=24) + if not due: + return 0 + now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + lines = ["**建议封禁 · 融合检测**", ""] + for row in due[:20]: + geo = " · ".join(part for part in (row.get("country_name"), + row.get("region")) if part) + lines.append("**%s**%s" % (row["ip"], ("（%s）" % geo) if geo else "")) + source = ("双来源" if row.get("sources") == "both" + else ("行为检测" if row.get("sources") == "behavior" + else "特征检测")) + brand = row.get("asn_brand") or "" + if brand or row.get("asn"): + lines.append("归属：%s %s%s" + % (row.get("asn_icon") or "", brand or "未知", + (" · AS" + str(row["asn"])) if row.get("asn") else "")) + score = max(float(row.get("feature_score") or 0), + float(row.get("behavior_score") or 0)) + lines.append("来源：%s · 发现：%s" + % (source, _local_time(row.get("first_banned_at")))) + lines.append("评分 %.0f · 请求 %s 次" + % (score, format(int(row.get("total") or 0), ","))) + if int(row.get("push_count") or 0) > 0: + lines.append("⏰ 该 IP 仍在活动（第 %d 次提醒）" + % (int(row["push_count"]) + 1)) + if row.get("fused_reason"): + lines.append("依据：" + row["fused_reason"]) + link = notifier.link_for(row["ip"]) + if link: + lines.append("[查看详情](%s)" % link) + lines.append("") + try: + notifier.post("\n".join(lines).strip()) + except Exception as exc: + # 失败时不刷屏：每 5 分钟最多提示一次；下一轮会自动重试 + global _PUSH_ERROR_AT + now = time.time() + if now - _PUSH_ERROR_AT > 300: + print("推送失败（每 5 分钟最多提示一次）: %s" % exc, flush=True) + _PUSH_ERROR_AT = now + return 0 + store.mark_pushed([row["ip"] for row in due[:20]], now) + return len(due[:20]) diff --git a/graylog_ops/cli.py b/graylog_ops/cli.py new file mode 100644 index 0000000..fccee6d --- /dev/null +++ b/graylog_ops/cli.py @@ -0,0 +1,289 @@ +"""CLI entry point.""" + +import argparse +import sys + +from . import __version__, commands +from .client import GraylogClient +from .config import ConfigError, load_config +from .http import GraylogError, HttpClient + +EPILOG = """\ +示例: + # 1) 先确认连通性、权限和这个版本支持哪些接口 + graylog-ops --url http://edu-master:9000 --token $GRAYLOG_TOKEN probe + + # 2) 摸清"客户端 IP 到底在哪个字段"（最关键的一步） + graylog-ops discover --range 3600 --stream 业务访问日志 + + # 3) 按条件看 top IP，判断阈值是否合理 + graylog-ops top-ips --field client_ip --query 'status:401' --range 300 --size 30 + + # 3.5) 生成 profile，然后做封禁判断（只读，dry-run 语义） + graylog-ops discover --range 3600 --save-profile profiles/edu.json + graylog-ops judge --profile profiles/edu.json --range 300 --allow allowlist.txt --explain 5 + + # 4) 把环境信息整体落盘，便于离线分析/复盘 + graylog-ops snapshot --dir snapshots/$(date +%F) --sample 500 + + # 5) 探测未知接口（版本差异时用） + graylog-ops raw /api/system/inputs + graylog-ops raw /api/search/universal/terms --param field=source --param range=60 + + # 6) 一次性产出整套信息（probe/discover/judge/snapshot），便于离线分析 + graylog-ops collect --dir collect-2026-10-18 --range 3600 --allow allowlist.example.txt + +环境变量: GRAYLOG_URL GRAYLOG_TOKEN GRAYLOG_USER GRAYLOG_PASSWORD + GRAYLOG_INSECURE GRAYLOG_TIMEOUT GRAYLOG_RETRIES +配置文件: ~/.config/graylog-ops/config.json 或 ./graylog-ops.json +""" + + +def _add_common(parser, suppress): + """Add connection/output flags. + + With ``suppress=True`` the defaults are SUPPRESS so a sub-parser never + clobbers a value that was already parsed by the root parser. + """ + + def default(value): + return argparse.SUPPRESS if suppress else value + + parser.add_argument("--url", metavar="URL", default=default(None), + help="Graylog 地址，如 http://edu-master:9000") + parser.add_argument("--token", metavar="TOKEN", default=default(None), + help="API token（推荐；设置后忽略 --user/--password）") + parser.add_argument("--user", metavar="USER", default=default(None), + help="用户名（配合 --password）") + parser.add_argument("--password", metavar="PASSWORD", default=default(None), + help="密码") + parser.add_argument("--insecure", action="store_true", + default=argparse.SUPPRESS if suppress else False, + help="跳过 HTTPS 证书校验") + parser.add_argument("--timeout", type=float, metavar="SECONDS", default=default(None), + help="单请求超时，默认 30") + parser.add_argument("--retries", type=int, metavar="N", default=default(None), + help="失败重试次数，默认 2") + parser.add_argument("--config", metavar="PATH", default=default(None), + help="JSON 配置文件路径") + parser.add_argument("--json", action="store_true", + default=argparse.SUPPRESS if suppress else False, + help="输出 JSON（便于管道/脚本消费）") + parser.add_argument("--out", metavar="PATH", default=default(None), + help="把结果 JSON 写入文件") + parser.add_argument("-v", "--verbose", action="store_true", + default=argparse.SUPPRESS if suppress else False, + help="打印每个 HTTP 请求") + + +def build_parser(): + common = argparse.ArgumentParser(add_help=False) + _add_common(common, suppress=True) + + parser = argparse.ArgumentParser( + prog="graylog-ops", + description="Graylog 4.0 只读信息采集工具（为 IP 封禁规则摸底）", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=EPILOG, + ) + _add_common(parser, suppress=False) + parser.add_argument("--version", action="version", version="graylog-ops %s" % __version__) + + sub = parser.add_subparsers(dest="command", metavar="") + + def add(name, handler, help_text): + cmd = sub.add_parser(name, parents=[common], help=help_text, description=help_text) + cmd.set_defaults(func=handler) + return cmd + + add("probe", commands.cmd_probe, "检查连通性、认证、接口可用性矩阵") + add("system", commands.cmd_system, "节点/索引器/版本/license 概要") + add("streams", commands.cmd_streams, "列出 stream（分流与权限边界）").add_argument( + "--rules", action="store_true", help="同时拉取每个 stream 的规则（N+1 请求）") + add("inputs", commands.cmd_inputs, "列出 input 及监听地址") + add("fields", commands.cmd_fields, "列出 Graylog 已注册字段与类型") + add("indices", commands.cmd_indices, "index set 与索引大小/保留策略") + add("alerts", commands.cmd_alerts, "现有事件定义与通知方式（改造前先看）").add_argument( + "--limit", type=int, default=None, help="最多返回条数") + + discover = add("discover", commands.cmd_discover, "采样日志，推断真实客户端 IP 字段") + discover.add_argument("--stream", help="stream id 或标题子串") + discover.add_argument("--query", default="*", help="查询条件，默认 *") + discover.add_argument("--range", type=int, default=3600, help="时间窗（秒），默认 3600") + discover.add_argument("--sample", type=int, default=500, help="采样消息数，默认 500") + discover.add_argument("--save-profile", metavar="PATH", + help="把推断出的字段映射写成 profile，供 judge 使用") + + judge = add("judge", commands.cmd_judge, + "把聚合信号判成封禁候选（只读，不执行封禁）") + judge.add_argument("--profile", metavar="PATH", help="discover --save-profile 生成的文件") + judge.add_argument("--range", type=int, default=300, help="时间窗（秒），默认 300") + judge.add_argument("--query", default="*", help="限定查询范围，默认 *") + judge.add_argument("--stream", help="stream id 或标题子串") + judge.add_argument("--field", help="客户端 IP 字段（覆盖 profile）") + judge.add_argument("--status-field", help="状态码字段") + judge.add_argument("--uri-field", help="URI 字段") + judge.add_argument("--ua-field", help="User-Agent 字段") + judge.add_argument("--method-field", help="HTTP 方法字段") + judge.add_argument("--allow", action="append", metavar="FILE", + help="白名单文件（IP/CIDR，可重复）") + judge.add_argument("--include-internal", action="store_true", + help="允许内网地址进入候选（默认排除）") + judge.add_argument("--min-requests", type=int, help="低于该请求数的 IP 不参与判分") + judge.add_argument("--min-score", type=float, help="低于该分数的候选不输出") + judge.add_argument("--top", type=int, default=30, help="最多输出条数，默认 30") + judge.add_argument("--size", type=int, default=200, help="每类聚合返回的 IP 数上限，默认 200") + judge.add_argument("--verify", type=int, default=0, metavar="N", + help="对前 N 个候选反查 URI 集中度（识别自动化客户端，每候选多 1 次查询）") + judge.add_argument("--explain", type=int, default=0, metavar="N", + help="为前 N 个候选附带样例消息") + judge.add_argument("--all", action="store_true", help="输出全部候选（含 ignore）") + + watch = add("watch", commands.cmd_watch, + "持续检测恶意公网 IP + 本地 Web 面板（SQLite 持久化）") + watch.add_argument("--profile", metavar="PATH", help="字段映射 profile") + watch.add_argument("--stream", help="只检测某个 stream") + watch.add_argument("--interval", type=int, default=60, help="检测间隔（秒），默认 60") + watch.add_argument("--window", type=int, default=900, help="检测时间窗（秒），默认 900") + watch.add_argument("--host", default="127.0.0.1", help="面板监听地址") + watch.add_argument("--port", type=int, default=8088, help="面板端口，默认 8088") + watch.add_argument("--db", default="data/ips.db", help="SQLite 路径") + watch.add_argument("--behavior-interval", type=int, default=900, + help="行为检测刷新间隔（秒，默认 900）") + watch.add_argument("--no-behavior", action="store_true", + help="关闭行为检测线程，只跑特征检测") + watch.add_argument("--notify-config", default="notify.json", + help="新增恶意 IP 的 webhook 配置（不存在则不推送）") + watch.add_argument("--notify-test", action="store_true", + help="发送一条测试推送后退出") + watch.add_argument("--asn-db", default="data/geo/asn.db", + help="ASN 库路径（空字符串禁用）") + watch.add_argument("--geo-db", default="data/geo/geo.db", + help="DB-IP Lite 生成的离线地理库路径（空字符串禁用）") + watch.add_argument("--path-prefix", metavar="PATH", default="", + help="只在该路径前缀下提供服务，根路径返回 404（用于对外分享）") + watch.add_argument("--forget", type=int, default=7200, help="多久没再出现就从列表移除（秒）") + watch.add_argument("--allow", action="append", metavar="FILE", help="白名单文件，可重复") + watch.add_argument("--field", help="客户端 IP 字段") + watch.add_argument("--status-field", help="状态码字段") + watch.add_argument("--uri-field", help="URI 字段") + watch.add_argument("--ua-field", help="User-Agent 字段") + watch.add_argument("--min-requests", type=int, default=3, help="低于该请求数不参与判定") + watch.add_argument("--size", type=int, default=200, help="每类聚合返回的 IP 数上限") + watch.add_argument("--verify", type=int, default=8, help="对前 N 个候选做 URI 核验") + watch.add_argument("--min-observe-score", type=float, default=10.0, + help="观察级候选低于该分数不入库，默认 10") + watch.add_argument("--top", type=int, default=50, help="--once 模式下打印的条数") + watch.add_argument("--export", metavar="PATH", + help="把高置信恶意 IP 导出为纯文本名单（排除疑似 NAT）") + watch.add_argument("--once", action="store_true", help="只跑一轮后退出（用于离线分析）") + watch.add_argument("--warm-limit", type=int, default=5, + help="每轮为多少个恶意 IP 预生成证据快照，默认 5") + watch.add_argument("--fix-times", "--fix-first-seen", dest="fix_first_seen", + action="store_true", + help="回查日志，修正首次攻击 / 最近攻击 / 最近出现时间") + watch.add_argument("--warm-only", action="store_true", + help="只预热证据快照后退出（配合 --warm-limit）") + watch.add_argument("--serve-only", action="store_true", + help="只提供面板，不做检测（用于浏览已有数据库）") + + collect = add("collect", commands.cmd_collect, + "一条命令产出 probe/discover/judge/snapshot 整套信息") + collect.add_argument("--dir", help="输出目录，默认 graylog-collect-<时间戳>") + collect.add_argument("--range", type=int, default=3600, help="discover 的时间窗（秒）") + collect.add_argument("--judge-range", type=int, default=300, help="judge 的时间窗（秒）") + collect.add_argument("--stream", help="只针对某个 stream（传给 discover/judge）") + collect.add_argument("--allow", action="append", metavar="FILE", help="白名单文件，可重复") + + for name, default_field in (("terms", None), ("top-ips", "client_ip")): + cmd = add(name, commands.cmd_terms, + "按字段做 terms 聚合（top-ips 是 field=client_ip 的快捷方式）") + cmd.add_argument("--field", required=(default_field is None), default=default_field, + help="聚合字段") + cmd.add_argument("--query", default="*", help="查询条件，默认 *") + cmd.add_argument("--range", type=int, default=300, help="时间窗（秒），默认 300") + cmd.add_argument("--size", type=int, default=50, help="返回条数，默认 50") + cmd.add_argument("--min-count", type=int, default=None, help="过滤低于该计数的项") + cmd.add_argument("--stream", help="stream id 或标题子串") + cmd.add_argument("--from", dest="from_ts", metavar="ISO8601", help="绝对时间起（UTC）") + cmd.add_argument("--to", dest="to_ts", metavar="ISO8601", help="绝对时间止（UTC）") + + messages = add("messages", commands.cmd_messages, "拉取原始消息（JSON/表格/JSONL）") + messages.add_argument("--query", default="*", help="查询条件，默认 *") + messages.add_argument("--range", type=int, default=300, help="时间窗（秒），默认 300") + messages.add_argument("--limit", type=int, default=50, help="条数，默认 50") + messages.add_argument("--offset", type=int, default=0, help="偏移") + messages.add_argument("--fields", help="只取这些字段，逗号分隔") + messages.add_argument("--stream", help="stream id 或标题子串") + messages.add_argument("--jsonl", action="store_true", help="每行一个 JSON 消息") + + histogram = add("histogram", commands.cmd_histogram, "消息量时间分布（估算阈值/容量）") + histogram.add_argument("--query", default="*", help="查询条件，默认 *") + histogram.add_argument("--range", type=int, default=3600, help="时间窗（秒），默认 3600") + histogram.add_argument("--interval", default="1m", help="时间桶，如 1m/5m/1h") + histogram.add_argument("--stream", help="stream id 或标题子串") + + snapshot = add("snapshot", commands.cmd_snapshot, "整体环境快照落到目录（每个资源一个 json）") + snapshot.add_argument("--dir", help="输出目录，默认 graylog-snapshot-<时间戳>") + snapshot.add_argument("--sample", type=int, default=0, help="附带采样消息条数，默认 0") + + raw = add("raw", commands.cmd_raw, "直接调用任意 API 路径（探索未封装的接口）") + raw.add_argument("path", help="如 /api/system/inputs") + raw.add_argument("--method", default="GET", help="HTTP 方法，默认 GET") + raw.add_argument("--body", metavar="JSON", help="JSON 请求体，或 @文件路径") + raw.add_argument("--param", action="append", metavar="K=V", help="查询参数，可重复") + raw.add_argument("--text", action="store_true", help="按文本返回（非 JSON）") + + return parser + + +def main(argv=None): + parser = build_parser() + args = parser.parse_args(argv) + if not getattr(args, "command", None): + parser.print_help() + return 2 + + try: + config = load_config(args) + except ConfigError as exc: + print("配置错误: %s" % exc, file=sys.stderr) + return 2 + + http_client = HttpClient( + config.url, + username=config.credentials[0], + password=config.credentials[1], + timeout=config.timeout, + retries=config.retries, + insecure=config.insecure, + verbose=bool(getattr(args, "verbose", False)), + ) + ctx = commands.Context(args, config, GraylogClient(http_client)) + + try: + return args.func(ctx, args) or 0 + except GraylogError as exc: + print("Graylog API 错误: %s" % exc, file=sys.stderr) + if exc.status == 401: + print( + "提示: token 认证是把 token 当用户名、字面量 token 当密码；" + "确认 token 未吊销且账号有对应权限", + file=sys.stderr, + ) + elif exc.status == 403: + print("提示: 账号缺少该资源权限（如 messages:read / searches:*）", file=sys.stderr) + elif exc.status == 404: + print( + "提示: 该版本可能没有这个接口。先跑 `probe` 看可用性，" + "或用 `raw /api/...` 直接验证", + file=sys.stderr, + ) + elif exc.status in (400, 422): + print("提示: 参数不被该版本接受，加 -v 查看实际请求的 URL", file=sys.stderr) + return 1 + except BrokenPipeError: + return 0 + except KeyboardInterrupt: + print("", file=sys.stderr) + return 130 diff --git a/graylog_ops/client.py b/graylog_ops/client.py new file mode 100644 index 0000000..e36a937 --- /dev/null +++ b/graylog_ops/client.py @@ -0,0 +1,463 @@ +"""Thin, defensive wrappers around the Graylog 4.0 REST API. + +Every method returns plain Python data (dict/list) that has been normalised +just enough for callers not to care about cosmetic response differences. +Anything that is version specific and ambiguous is passed through untouched +and documented as such. + +Endpoint reference for 4.0 lives in the server itself: +``http://:9002/api/api-browser/`` +""" + +import json + +from .http import GraylogError + + +def as_list(data, *keys): + """Best effort extraction of a list from a Graylog response.""" + if isinstance(data, list): + return data + if isinstance(data, dict): + for key in keys: + value = data.get(key) + if isinstance(value, list): + return value + for value in data.values(): + if isinstance(value, list): + return value + return [] + + +def dig(data, *keys, **kwargs): + default = kwargs.get("default") + current = data + for key in keys: + if not isinstance(current, dict): + return default + current = current.get(key) + if current is None: + return default + return current + + +def normalize_terms(data): + """Turn a terms aggregation response into ``[{"term", "count"}, ...]``. + + Graylog has shipped a couple of shapes for this over the years, so accept + the common ones instead of assuming a single layout. + """ + items = as_list(data, "terms", "values", "results") + rows = [] + for item in items: + if isinstance(item, dict): + term = item.get("term", item.get("value", item.get("name"))) + count = item.get("count", item.get("doc_count", item.get("hits", 0))) + rows.append({"term": term, "count": count}) + elif isinstance(item, (list, tuple)) and len(item) >= 2: + rows.append({"term": item[0], "count": item[1]}) + return rows + + +def normalize_histogram(data): + """Turn a histogram response into ``[{"timestamp", "count"}, ...]``.""" + if not isinstance(data, dict): + return [] + results = data.get("results") + rows = [] + if isinstance(results, dict): + for timestamp, value in results.items(): + if isinstance(value, dict): + count = value.get("count", value.get("doc_count", 0)) + else: + count = value + rows.append({"timestamp": timestamp, "count": count or 0}) + elif isinstance(results, list): + for item in results: + if isinstance(item, dict): + rows.append({ + "timestamp": item.get("timestamp", item.get("time")), + "count": item.get("count", 0), + }) + rows.sort(key=lambda row: str(row["timestamp"])) + return rows + + +class GraylogClient: + def __init__(self, http): + self.http = http + # Which aggregation backend to use. Graylog 4.0 removed the legacy + # /api/search/universal/{terms,histogram,stats} endpoints, so 4.x must + # use the views API (POST /api/views/search/sync). 3.x has no views API. + self.agg_backend = None + # Legacy fallback only: some builds only accept "size" on terms, + # others tolerate/require "limit". + self.terms_size_param = "size" + + # ---------------------------------------------------------------- system + def system_info(self): + return self.http.get("/api/system") or {} + + def cluster_nodes(self): + return as_list(self.http.get("/api/system/cluster/nodes"), "nodes") + + def indexer_health(self): + return self.http.get("/api/system/indexer/cluster/health") or {} + + def indexer_cluster_name(self): + return self.http.get("/api/system/indexer/cluster/name") or "" + + def deflector(self): + return self.http.get("/api/system/deflector") or {} + + def field_types(self): + data = self.http.get("/api/system/fields") + if isinstance(data, dict): + fields = data.get("fields") + if isinstance(fields, list): + return fields + if isinstance(fields, dict): + return [{"name": name, "type": kind} for name, kind in fields.items()] + return as_list(data, "fields") + + def license_info(self): + return self.http.get("/api/system/license") or {} + + def index_sets(self): + data = self.http.get("/api/system/indices/index_sets", {"skip": 0, "limit": 100}) + return as_list(data, "index_sets") + + def indices(self): + data = self.http.get("/api/system/indexer/indices") + if isinstance(data, dict): + # Strict per-key access: as_list()'s "first list in the dict" + # fallback would merge the same list three times here. + merged = [] + for key in ("indices", "closed_indices", "reopened_indices"): + value = data.get(key) + if isinstance(value, list): + merged.extend(value) + return merged + return as_list(data) + + # --------------------------------------------------------------- streams + def streams(self): + return as_list(self.http.get("/api/streams"), "streams") + + def stream(self, stream_id): + return self.http.get("/api/streams/%s" % stream_id) or {} + + # ---------------------------------------------------------------- inputs + def inputs(self): + return as_list(self.http.get("/api/system/inputs"), "inputs") + + def extractors(self, input_id): + data = self.http.get("/api/system/inputs/%s/extractors" % input_id) + return as_list(data, "extractors") + + # ------------------------------------------------------------- pipelines + def pipelines(self): + return as_list(self.http.get("/api/system/pipelines/pipeline"), "pipelines") + + def pipeline_rules(self): + return as_list(self.http.get("/api/system/pipelines/rule"), "rules") + + # ---------------------------------------------------------------- events + def _paged(self, path, key, limit=None, per_page=50, params=None): + items = [] + page = 1 + while True: + query = dict(params or {}) + query.update({"page": page, "per_page": per_page}) + data = self.http.get(path, query) + batch = as_list(data, key) + if not batch: + return items + items.extend(batch) + if limit is not None and len(items) >= limit: + return items[:limit] + # Trust the server's total when it tells us there is nothing left; + # this also stops runaway pagination on servers that ignore `page`. + total = dig(data, "total") + if isinstance(total, int) and len(items) >= total: + return items + if len(batch) < per_page: + return items + page += 1 + if page > 500: # defensive stop + return items + + def event_definitions(self, limit=None, per_page=50): + return self._paged( + "/api/events/definitions", + "event_definitions", + limit=limit, + per_page=per_page, + params={"sort": "title", "order": "asc"}, + ) + + def notifications(self, limit=None, per_page=50): + return self._paged( + "/api/events/notifications", "notifications", limit=limit, per_page=per_page + ) + + def events(self, limit=None, per_page=50): + data = self.http.get("/api/events", {"page": 1, "per_page": per_page}) + items = as_list(data, "events") + return items[:limit] if limit else items + + # --------------------------------------------------------------- sidecar + def sidecars(self): + return as_list(self.http.get("/api/sidecars"), "sidecars") + + # ---------------------------------------------------------------- search + @staticmethod + def _time_params(range_seconds=None, from_ts=None, to_ts=None): + if from_ts and to_ts: + return {"from": from_ts, "to": to_ts} + return {"range": int(range_seconds if range_seconds is not None else 300)} + + def search_relative(self, query="*", range_seconds=300, limit=100, offset=0, + sort="timestamp:desc", fields=None, filter_str=None): + params = {"query": query or "*", "limit": int(limit), "offset": int(offset), "sort": sort} + params.update(self._time_params(range_seconds=range_seconds)) + if fields: + params["fields"] = ",".join(fields) + if filter_str: + params["filter"] = filter_str + data = self.http.get("/api/search/universal/relative", params) + return self._messages(data) + + def search_absolute(self, query="*", from_ts=None, to_ts=None, limit=100, offset=0, + sort="timestamp:desc", fields=None, filter_str=None): + params = {"query": query or "*", "limit": int(limit), "offset": int(offset), "sort": sort} + params.update(self._time_params(from_ts=from_ts, to_ts=to_ts)) + if fields: + params["fields"] = ",".join(fields) + if filter_str: + params["filter"] = filter_str + data = self.http.get("/api/search/universal/absolute", params) + return self._messages(data) + + @staticmethod + def _messages(data): + if not isinstance(data, dict): + return [] + messages = data.get("messages") + if not isinstance(messages, list): + return [] + result = [] + for item in messages: + if isinstance(item, dict) and isinstance(item.get("message"), dict): + message = dict(item["message"]) + if item.get("index") and "_index" not in message: + message["_index"] = item["index"] + result.append(message) + elif isinstance(item, dict): + result.append(item) + return result + + # ------------------------------------------------------- views aggregation + BASE_PATH = "/api/views/search" + + @staticmethod + def _timerange(range_seconds=None, from_ts=None, to_ts=None): + if from_ts and to_ts: + return {"type": "absolute", "from": from_ts, "to": to_ts} + return {"type": "relative", "range": int(range_seconds or 300)} + + @staticmethod + def _stream_filter(filter_str): + """Translate the legacy ``streams:[,]`` filter to a views filter.""" + if not filter_str or not filter_str.startswith("streams:"): + return None + ids = [part.strip() for part in filter_str[len("streams:"):].split(",") if part.strip()] + if not ids: + return None + return {"type": "or", "filters": [{"type": "stream", "id": sid} for sid in ids]} + + def _views_pivot(self, query, timerange, row_groups, column_groups, series, + sort, filter_str=None, timeout_ms=60000): + search_type = { + "type": "pivot", + "id": "st1", + "series": series, + "row_groups": row_groups, + "column_groups": column_groups, + "sort": sort, + "rollup": True, + } + query_object = { + "id": "q1", + "timerange": timerange, + "query": {"type": "elasticsearch", "query_string": query or "*"}, + "search_types": [search_type], + } + stream_filter = self._stream_filter(filter_str) + if stream_filter: + query_object["filter"] = stream_filter + body = {"queries": [query_object], "parameters": []} + data = self.http.request( + "POST", self.BASE_PATH + "/sync", + params={"timeout": int(timeout_ms)}, body=body, + ) + results = ((data or {}).get("results") or {}).get("q1") or {} + errors = results.get("errors") or [] + if errors: + raise GraylogError( + "views search error: %s" % json.dumps(errors, ensure_ascii=False)[:400], + path=self.BASE_PATH + "/sync", method="POST", + ) + search_types = results.get("search_types") or {} + if isinstance(search_types, list): + search_types = {item.get("id"): item for item in search_types + if isinstance(item, dict)} + return search_types.get("st1") or {} + + def _terms_views(self, field, query, size, range_seconds, from_ts, to_ts, filter_str): + search_type = self._views_pivot( + query=query, + timerange=self._timerange(range_seconds, from_ts, to_ts), + row_groups=[{"type": "values", "field": field, "limit": int(size)}], + column_groups=[], + series=[{"type": "count", "id": "c1"}], + sort=[{"type": "pivot", "field": "c1", "direction": "Descending"}], + filter_str=filter_str, + ) + terms = [] + for row in search_type.get("rows") or []: + key = row.get("key") or [] + if not key: + continue # rollup row (total) + values = row.get("values") or [] + count = values[0].get("value") if values else 0 + terms.append({"term": key[0], "count": count}) + return { + "field": field, + "terms": terms, + "total": search_type.get("total", sum(row["count"] or 0 for row in terms)), + } + + def _histogram_views(self, query, range_seconds, interval, filter_str, from_ts, to_ts): + search_type = self._views_pivot( + query=query, + timerange=self._timerange(range_seconds, from_ts, to_ts), + row_groups=[], + column_groups=[{"type": "time", "field": "timestamp", "interval": interval}], + series=[{"type": "count", "id": "c1"}], + sort=[], + filter_str=filter_str, + ) + results = {} + for row in search_type.get("rows") or []: + for item in row.get("values") or []: + key = item.get("key") or [] + # Column keys are [timestamp, series_id]; the pivot rollup row + # only carries [series_id] and must not become a time bucket. + if len(key) < 2: + continue + results[key[0]] = {"count": item.get("value", 0)} + return {"results": results, "interval": interval} + + # ------------------------------------------------------------- aggregation + def search_views_messages(self, query="*", range_seconds=300, limit=5, + offset=0, filter_str=None, from_ts=None, to_ts=None, + timeout_ms=60000, sort_order="DESC"): + """Message search through the views API (the legacy endpoint is flaky + on long windows with complex queries and drops the connection).""" + search_type = { + "type": "messages", + "id": "st1", + "limit": int(limit), + "offset": int(offset), + "sort": [{"field": "timestamp", + "order": "ASC" if str(sort_order).upper() == "ASC" else "DESC"}], + } + query_object = { + "id": "q1", + "timerange": self._timerange(range_seconds, from_ts, to_ts), + "query": {"type": "elasticsearch", "query_string": query or "*"}, + "search_types": [search_type], + } + stream_filter = self._stream_filter(filter_str) + if stream_filter: + query_object["filter"] = stream_filter + body = {"queries": [query_object], "parameters": []} + data = self.http.request( + "POST", self.BASE_PATH + "/sync", + params={"timeout": int(timeout_ms)}, body=body, + ) + results = ((data or {}).get("results") or {}).get("q1") or {} + errors = results.get("errors") or [] + if errors: + raise GraylogError( + "views message search error: %s" % json.dumps(errors, ensure_ascii=False)[:300], + path=self.BASE_PATH + "/sync", method="POST", + ) + search_types = results.get("search_types") or {} + if isinstance(search_types, list): + search_types = {item.get("id"): item for item in search_types + if isinstance(item, dict)} + search_type_result = search_types.get("st1") or {} + return self._messages({"messages": search_type_result.get("messages") or []}) + + def terms(self, field, query="*", size=50, range_seconds=300, + from_ts=None, to_ts=None, filter_str=None): + if self.agg_backend in (None, "views"): + try: + data = self._terms_views(field, query, size, range_seconds, + from_ts, to_ts, filter_str) + self.agg_backend = "views" + return data + except GraylogError as exc: + if exc.status != 404 or self.agg_backend == "views": + raise + self.agg_backend = "legacy" + return self._terms_legacy(field, query, size, range_seconds, + from_ts, to_ts, filter_str) + + def _terms_legacy(self, field, query, size, range_seconds, from_ts, to_ts, filter_str): + params = {"query": query or "*", "field": field} + params.update(self._time_params(range_seconds, from_ts, to_ts)) + if filter_str: + params["filter"] = filter_str + params[self.terms_size_param] = int(size) + try: + return self.http.get("/api/search/universal/terms", params) + except GraylogError as exc: + # Older/newer builds disagree on the parameter name; retry once. + if exc.status in (400, 422) and self.terms_size_param == "size": + params.pop("size", None) + params["limit"] = int(size) + data = self.http.get("/api/search/universal/terms", params) + self.terms_size_param = "limit" + return data + raise + + def histogram(self, query="*", range_seconds=3600, interval="1m", + filter_str=None, from_ts=None, to_ts=None): + if self.agg_backend in (None, "views"): + try: + data = self._histogram_views(query, range_seconds, interval, + filter_str, from_ts, to_ts) + self.agg_backend = "views" + return data + except GraylogError as exc: + if exc.status != 404 or self.agg_backend == "views": + raise + self.agg_backend = "legacy" + params = {"query": query or "*", "interval": interval} + params.update(self._time_params(range_seconds, from_ts, to_ts)) + if filter_str: + params["filter"] = filter_str + return self.http.get("/api/search/universal/histogram", params) + + def stats(self, field, query="*", range_seconds=300, stats=None, + filter_str=None, from_ts=None, to_ts=None): + params = {"query": query or "*", "field": field} + params.update(self._time_params(range_seconds, from_ts, to_ts)) + if stats: + params["stats"] = ",".join(stats) + if filter_str: + params["filter"] = filter_str + return self.http.get("/api/search/universal/stats", params) diff --git a/graylog_ops/commands.py b/graylog_ops/commands.py new file mode 100644 index 0000000..a02ee7f --- /dev/null +++ b/graylog_ops/commands.py @@ -0,0 +1,1174 @@ +"""Sub-command implementations. + +Every command builds one JSON-serialisable result dict and then either prints +it (``--json`` / ``--out``) or renders it for a terminal. Keep stdout parseable +when ``--json`` is set: diagnostics go to stderr. +""" + +import datetime +import json +import os +import sys +import time + +from . import __version__, detect, render +from .client import normalize_histogram, normalize_terms +from .http import GraylogError +from .judge import ( + DEFAULT_THRESHOLDS, + Allowlist, + build_fields, + build_signals, + collect_signals, + load_allowlist, + load_profile, + score_candidates, + verify_candidates, +) + + +# --------------------------------------------------------------------- utils +def now_iso(): + return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def write_json(path, payload): + directory = os.path.dirname(os.path.abspath(path)) + if directory and not os.path.isdir(directory): + os.makedirs(directory, exist_ok=True) + with open(path, "w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2, ensure_ascii=False) + handle.write("\n") + + +def _safe(func, default): + """Call *func*, returning *default* on API errors (for optional resources).""" + try: + return func() + except GraylogError: + return default + + +def resolve_stream(client, value): + """Accept a stream id or a (unique) substring of its title.""" + if not value: + return None + streams = _safe(client.streams, []) + for stream in streams: + if stream.get("id") == value: + return value + needle = value.lower() + matches = [s for s in streams if needle in (s.get("title") or "").lower()] + if len(matches) == 1: + return matches[0].get("id") + if not matches: + raise SystemExit( + "找不到匹配的 stream: %r（先用 `graylog-ops streams` 查看可用 stream）" % value + ) + titles = ", ".join(repr(s.get("title", "?")) for s in matches[:5]) + raise SystemExit("stream 匹配到多个（%s），请改用完整 stream id" % titles) + + +def _filter_for(stream_id): + return "streams:%s" % stream_id if stream_id else None + + +class Context: + def __init__(self, args, config, client): + self.args = args + self.config = config + self.client = client + self.json_mode = bool(getattr(args, "json", False)) + self.out_path = getattr(args, "out", None) + + def emit(self, data, human=None): + if self.out_path: + write_json(self.out_path, data) + if not self.json_mode: + print("JSON 已写入 %s" % self.out_path) + if self.json_mode: + print(json.dumps(data, indent=2, ensure_ascii=False)) + elif human is not None: + print(human(data)) + else: + print(json.dumps(data, indent=2, ensure_ascii=False)) + + def note(self, text): + if not self.json_mode: + print(text) + sys.stdout.flush() + + +# ----------------------------------------------------------------- fingerprint +def _short_version(value): + return (value or "unknown").split("+", 1)[0] + + +def cmd_probe(ctx, args): + client = ctx.client + started = time.time() + try: + system = client.system_info() + except GraylogError as exc: + print("无法连接 Graylog: %s" % exc, file=sys.stderr) + print( + "排查建议: 1) 内网/VPN 是否可达 2) --url 是否带 http(s):// 3) token 是否有效", + file=sys.stderr, + ) + return 1 + + checks = [] + + def check(name, func, detail=None): + begin = time.time() + try: + value = func() + item = { + "name": name, + "ok": True, + "ms": int((time.time() - begin) * 1000), + "detail": detail(value) if detail else _describe(value), + } + checks.append(item) + return value + except GraylogError as exc: + checks.append({ + "name": name, + "ok": False, + "ms": int((time.time() - begin) * 1000), + "status": exc.status, + "error": exc.message, + }) + return None + + def count(value): + if isinstance(value, (list, tuple)): + return "%d 项" % len(value) + if isinstance(value, dict): + return "%d 个键" % len(value) + return "ok" + + health = check("indexer health", client.indexer_health) + check("cluster nodes", client.cluster_nodes, count) + fields = check("fields", client.field_types, count) + check("streams", client.streams, count) + check("inputs", client.inputs, count) + check("index sets", client.index_sets, count) + check("event definitions", lambda: client.event_definitions(limit=1), count) + check("notifications", lambda: client.notifications(limit=1), count) + check("pipelines", client.pipelines, count) + check("pipeline rules", client.pipeline_rules, count) + check("sidecars", client.sidecars, count) + check("license", client.license_info) + check("views api", lambda: client.http.get("/api/views", {"page": 1, "per_page": 1}), count) + + relative = check( + "search relative", + lambda: client.search_relative(limit=1, range_seconds=300), + lambda value: "%d 条样例消息" % len(value), + ) + terms_value = check( + "search terms", + lambda: client.terms("source", size=1, range_seconds=300), + lambda value: "%d 个 term" % len(normalize_terms(value)), + ) + check( + "search histogram", + lambda: client.histogram(range_seconds=300, interval="1m"), + lambda value: "%d 个时间桶" % len(normalize_histogram(value)), + ) + + data = { + "system": system, + "indexer": health or {}, + "aggregation": client.agg_backend or "views", + "checks": checks, + "elapsed_ms": int((time.time() - started) * 1000), + } + + def human(payload): + info = payload["system"] + indexer = payload["indexer"] or {} + missing = [item for item in payload["checks"] + if not item["ok"] and item.get("status") == 404] + failed = [item for item in payload["checks"] + if not item["ok"] and item.get("status") != 404] + lines = [] + lines.append(render.kv_block([ + ("版本", _short_version(info.get("version"))), + ("主机", info.get("hostname", "-")), + ("集群 ID", info.get("cluster_id", "-")), + ("时区", info.get("timezone", "-")), + ("索引器", "%s / %s" % (indexer.get("status", "?"), indexer.get("cluster_name", "?"))), + ("聚合接口", payload["aggregation"]), + ])) + lines.append("") + rows = [ + (item["name"], "OK" if item["ok"] else "FAIL", item.get("ms", 0), + item.get("detail") or item.get("error") or "") + for item in payload["checks"] + ] + lines.append(render.table(["检查项", "结果", "耗时(ms)", "说明"], rows, ["l", "l", "r", "l"])) + lines.append("") + if failed: + lines.append("失败 %d 项（权限不足或服务端异常）: %s" + % (len(failed), ", ".join(item["name"] for item in failed))) + if missing: + lines.append("接口不存在 %d 项（版本差异，正常）: %s" + % (len(missing), ", ".join(item["name"] for item in missing))) + if not failed and not missing: + lines.append("全部 %d 项检查通过" % len(payload["checks"])) + return "\n".join(lines) + + ctx.emit(data, human) + required_ok = system and relative is not None + return 0 if required_ok else 1 + + +def _describe(value): + if isinstance(value, list): + return "%d 项" % len(value) + if isinstance(value, dict): + status = value.get("status") + if status: + return str(status) + return "ok" + if value is None: + return "-" + return render.clip(value, 40) + + +# ---------------------------------------------------------------------- system +def cmd_system(ctx, args): + client = ctx.client + data = { + "system": client.system_info(), + "indexer": _safe(client.indexer_health, {}), + "deflector": _safe(client.deflector, {}), + "license": _safe(client.license_info, {}), + "field_types": _safe(client.field_types, []), + "nodes": _safe(client.cluster_nodes, []), + "collected_at": now_iso(), + } + + def human(payload): + info = payload["system"] + indexer = payload["indexer"] or {} + deflector = payload["deflector"] or {} + shards = indexer.get("shards") or {} + shard_text = "" + if shards: + shard_text = "(active=%s, unassigned=%s)" % ( + shards.get("active", "?"), shards.get("unassigned", "?")) + lines = [ + render.kv_block([ + ("版本", info.get("version", "-")), + ("主机名", info.get("hostname", "-")), + ("集群 ID", info.get("cluster_id", "-")), + ("操作系统", info.get("operating_system", "-")), + ("时区", info.get("timezone", "-")), + ("启动时间", info.get("started_at", "-")), + ("生命周期 / LB", "%s / %s" % (info.get("lifecycle", "-"), + info.get("lb_status", "-"))), + ("处理中", info.get("is_processing")), + ("索引器", "%s %s" % (indexer.get("status", "-"), shard_text)), + ("Deflector", "%s (up=%s)" % ( + deflector.get("current_target", deflector.get("index", "-")), + deflector.get("is_up", "-"))), + ("已注册字段", len(payload["field_types"])), + ]) + ] + nodes = payload["nodes"] or [] + if nodes: + rows = [] + for node in nodes: + rows.append(( + node.get("hostname", "-"), + node.get("short_node_id", (node.get("node_id") or "-")[:8]), + node.get("transport_address", "-"), + "yes" if node.get("is_master") else "", + )) + lines.append("") + lines.append(render.table(["节点", "short id", "transport", "master"], rows)) + return "\n".join(lines) + + ctx.emit(data, human) + return 0 + + +# --------------------------------------------------------------------- streams +def cmd_streams(ctx, args): + client = ctx.client + streams = client.streams() + if getattr(args, "rules", False): + for stream in streams: + stream_id = stream.get("id") + if not stream_id: + continue + try: + detail = client.stream(stream_id) + stream["rules"] = detail.get("rules", []) + except GraylogError as exc: + stream["rules"] = [] + stream["rules_error"] = exc.message + + data = {"count": len(streams), "streams": streams} + + def human(payload): + rows = [] + for stream in payload["streams"]: + rules = stream.get("rules") + rule_count = "-" if rules is None else len(rules) + if stream.get("rules_error"): + rule_count = "err" + rows.append(( + stream.get("id", "-"), + render.clip(stream.get("title", "-"), 28), + "disabled" if stream.get("disabled") else "active", + rule_count, + render.clip(stream.get("description", ""), 40), + )) + return render.table(["stream id", "标题", "状态", "规则", "描述"], rows, + ["l", "l", "l", "r", "l"]) + + ctx.emit(data, human) + return 0 + + +# ---------------------------------------------------------------------- inputs +def cmd_inputs(ctx, args): + client = ctx.client + inputs = client.inputs() + data = {"count": len(inputs), "inputs": inputs} + + def human(payload): + rows = [] + for item in payload["inputs"]: + config = item.get("configuration") or {} + bind = config.get("bind_address") or config.get("bindAddress") or "-" + port = config.get("port") or config.get("Port") or "-" + kind = (item.get("type") or "-").rsplit(".", 1)[-1] + scope = "global" if item.get("global") else "node" + rows.append(( + render.clip(kind, 26), + render.clip(item.get("title", "-"), 26), + str(bind), + str(port), + scope, + render.clip(item.get("id", "-"), 26), + )) + return render.table(["类型", "标题", "绑定", "端口", "作用域", "input id"], rows, + ["l", "l", "l", "r", "l", "l"]) + + ctx.emit(data, human) + return 0 + + +# ---------------------------------------------------------------------- fields +def cmd_fields(ctx, args): + client = ctx.client + fields = client.field_types() + rows = [] + for item in fields: + if isinstance(item, dict): + rows.append({"name": item.get("name", "-"), "type": item.get("type", "-")}) + else: + rows.append({"name": str(item), "type": "-"}) + data = {"count": len(rows), "fields": rows} + + def human(payload): + body = render.table( + ["字段", "类型"], + [(row["name"], row["type"]) for row in payload["fields"]], + ) + return body + "\n\n共 %d 个已注册字段" % payload["count"] + + ctx.emit(data, human) + return 0 + + +# --------------------------------------------------------------------- indices +def cmd_indices(ctx, args): + client = ctx.client + index_sets = client.index_sets() + indices = client.indices() + data = {"index_sets": index_sets, "indices": indices} + + def human(payload): + lines = [] + set_rows = [] + for item in payload["index_sets"]: + rotation = item.get("rotation_strategy") or "-" + rotation_cfg = item.get("rotation_strategy_configuration") or {} + if rotation == "time": + rotation = "%s/%s" % (rotation, rotation_cfg.get("period", "?")) + else: + rotation = "%s/%s" % (rotation, rotation_cfg.get("max_docs_per_index", + rotation_cfg.get("max_size_per_index", "?"))) + retention = item.get("retention_strategy") or "-" + retention_cfg = item.get("retention_strategy_configuration") or {} + keep = retention_cfg.get("max_number_of_indices", retention_cfg.get("max_time_per_index", "?")) + set_rows.append(( + item.get("id", "-"), + render.clip(item.get("title", "-"), 22), + item.get("index_prefix", "-"), + "%s/%s" % (item.get("shards", "-"), item.get("replicas", "-")), + render.clip(rotation, 24), + "%s/%s" % (retention, keep), + )) + lines.append(render.table( + ["index set id", "标题", "前缀", "分片/副本", "轮转", "保留"], set_rows, + ["l", "l", "l", "l", "l", "l"], + )) + + index_rows = [] + total_size = 0 + for item in payload["indices"]: + size = item.get("size") or 0 + if isinstance(size, (int, float)): + total_size += size + index_rows.append(( + item.get("index_name", "-"), + render.clip(item.get("index_set_id", "-"), 26), + render.human_bytes(size), + item.get("docs_count", item.get("documents_count", "-")), + render.clip(item.get("status", item.get("state", "-")), 12), + )) + lines.append("") + lines.append(render.table(["索引", "index set", "大小", "文档数", "状态"], index_rows, + ["l", "l", "r", "r", "l"])) + lines.append("") + lines.append("共 %d 个索引，合计 %s" % (len(index_rows), render.human_bytes(total_size))) + return "\n".join(lines) + + ctx.emit(data, human) + return 0 + + +# -------------------------------------------------------------------- discover +def cmd_discover(ctx, args): + client = ctx.client + stream_id = resolve_stream(client, getattr(args, "stream", None)) + filter_str = _filter_for(stream_id) + query = getattr(args, "query", "*") or "*" + range_seconds = int(getattr(args, "range", 3600) or 3600) + wanted = int(getattr(args, "sample", 500) or 500) + + messages = [] + offset = 0 + while len(messages) < wanted: + batch = client.search_relative( + query=query, + range_seconds=range_seconds, + limit=min(500, wanted - len(messages)), + offset=offset, + filter_str=filter_str, + ) + if not batch: + break + messages.extend(batch) + offset += len(batch) + if offset > wanted * 4: + break + + report = detect.analyse_messages(messages) + report.update({ + "query": query, + "range_seconds": range_seconds, + "stream_id": stream_id, + "collected_at": now_iso(), + }) + + profile_path = getattr(args, "save_profile", None) + if profile_path: + http_fields = report.get("http_fields") or {} + + def first(kind): + names = http_fields.get(kind) or [] + return names[0] if names else None + + write_json(profile_path, { + "generated_at": now_iso(), + "sampled": report["sampled"], + "fields": { + "client_ip": report["recommendation"].get("field"), + "status": first("status"), + "uri": first("uri"), + "ua": first("user_agent"), + "method": first("method"), + }, + }) + ctx.note("profile 已写入 %s" % profile_path) + + data = report + + def human(payload): + rec = payload["recommendation"] + lines = [ + "采样 %d 条消息 query=%s range=%ds stream=%s" + % (payload["sampled"], payload["query"], payload["range_seconds"], + payload["stream_id"] or "(all)"), + "", + ] + if payload["candidates"]: + rows = [] + for item in payload["candidates"]: + rows.append(( + "%.1f" % item["score"], + item["field"], + "%.0f%%" % (item["presence"] * 100), + "%.0f%%" % (item["ip_rate"] * 100), + item["distinct_ips"], + "%.0f%%" % (item["private_ratio"] * 100), + "v%s" % "/".join(str(v) for v in item["versions"]), + ", ".join(item["examples"][:2]), + )) + lines.append(render.table( + ["分数", "字段", "出现率", "IP占比", "不同IP", "私有IP", "IP版本", "样例"], + rows, ["r", "l", "r", "r", "r", "r", "l", "l"], + )) + else: + lines.append("没有字段存放纯 IP 值。") + lines.append("") + + http_fields = payload.get("http_fields") or {} + if http_fields: + lines.append("HTTP 相关字段:") + for kind, names in sorted(http_fields.items()): + lines.append(" %-10s -> %s" % (kind, ", ".join(names))) + lines.append("") + + confidence = rec["confidence"] + lines.append("推荐字段: %s (confidence=%s)" % (rec["field"] or "-", confidence)) + for note in rec["notes"]: + lines.append(" - %s" % note) + for warning in rec["warnings"]: + lines.append(" ! %s" % warning) + + if rec["field"]: + lines.append("") + lines.append("下一步: graylog-ops top-ips --field %s --range %d%s" + % (rec["field"], payload["range_seconds"], + (" --stream %s" % payload["stream_id"]) if payload["stream_id"] else "")) + if payload["sample_message"]: + lines.append("") + lines.append("样例行:") + sample = json.dumps(payload["sample_message"], indent=2, ensure_ascii=False) + lines.extend(" " + line for line in sample.splitlines()) + return "\n".join(lines) + + ctx.emit(data, human) + return 0 + + +# ----------------------------------------------------------------------- terms +def cmd_terms(ctx, args): + client = ctx.client + stream_id = resolve_stream(client, getattr(args, "stream", None)) + field = args.field + raw = client.terms( + field=field, + query=getattr(args, "query", "*") or "*", + size=int(getattr(args, "size", 50) or 50), + range_seconds=getattr(args, "range", 300), + from_ts=getattr(args, "from_ts", None), + to_ts=getattr(args, "to_ts", None), + filter_str=_filter_for(stream_id), + ) + rows = [row for row in normalize_terms(raw) if row["term"] not in (None, "")] + min_count = getattr(args, "min_count", None) + if min_count is not None: + rows = [row for row in rows if (row["count"] or 0) >= int(min_count)] + rows.sort(key=lambda row: (-(row["count"] or 0), str(row["term"]))) + total = sum((row["count"] or 0) for row in rows) + data = { + "field": field, + "query": getattr(args, "query", "*") or "*", + "range_seconds": getattr(args, "range", 300), + "stream_id": stream_id, + "total": total, + "terms": rows, + "collected_at": now_iso(), + } + + def human(payload): + lines = ["字段 %s top %d total=%d range=%ss" + % (payload["field"], len(payload["terms"]), payload["total"], + payload["range_seconds"]), ""] + table_rows = [] + for index, row in enumerate(payload["terms"], 1): + table_rows.append(( + index, + row["term"], + row["count"], + render.percent(row["count"] or 0, payload["total"]), + )) + lines.append(render.table(["#", "值", "计数", "占比"], table_rows, + ["r", "l", "r", "r"])) + return "\n".join(lines) + + ctx.emit(data, human) + return 0 + + +# -------------------------------------------------------------------- collect +def cmd_collect(ctx, args): + """Run probe/discover/judge/snapshot into one directory for offline review. + + Calling the CLI recursively keeps this command thin: every step reuses the + exact same code path (and therefore the same connection/auth handling) as + running it by hand. + """ + import io + from contextlib import redirect_stderr, redirect_stdout + + from .cli import main as cli_main # lazy import: avoids a module cycle + + if getattr(args, "dir", None): + out_dir = args.dir + elif ctx.out_path: + # --out points at a directory for this command, not at a JSON file. + out_dir = ctx.out_path + ctx.out_path = None + else: + out_dir = "graylog-collect-" + time.strftime("%Y%m%d-%H%M%S") + os.makedirs(out_dir, exist_ok=True) + + connection = ["--url", ctx.config.url] + if ctx.config.token: + connection += ["--token", ctx.config.token] + else: + connection += ["--user", ctx.config.user or "", + "--password", ctx.config.password or ""] + if ctx.config.insecure: + connection.append("--insecure") + connection += ["--timeout", str(ctx.config.timeout), + "--retries", str(ctx.config.retries)] + + window = int(getattr(args, "range", 3600) or 3600) + judge_window = int(getattr(args, "judge_range", 300) or 300) + profile_path = os.path.join(out_dir, "profile.json") + stream = getattr(args, "stream", None) + stream_args = ["--stream", stream] if stream else [] + + judge_args = ["judge", "--all", "--json", + "--range", str(judge_window), + "--profile", profile_path, + "--out", os.path.join(out_dir, "judge.json")] + stream_args + for allow in (getattr(args, "allow", None) or []): + judge_args += ["--allow", allow] + + steps = ( + ("probe", ["probe", "--json", "--out", os.path.join(out_dir, "probe.json")]), + ("discover", ["discover", "--json", "--range", str(window), + "--save-profile", profile_path, + "--out", os.path.join(out_dir, "discover.json")] + stream_args), + ("judge", judge_args), + ("snapshot", ["snapshot", "--dir", os.path.join(out_dir, "snapshot")]), + ) + + results = [] + for name, argv in steps: + buffer, error_buffer = io.StringIO(), io.StringIO() + begin = time.time() + with redirect_stdout(buffer), redirect_stderr(error_buffer): + code = cli_main(connection + argv) + results.append({ + "step": name, + "ok": code == 0, + "ms": int((time.time() - begin) * 1000), + "stderr": error_buffer.getvalue().strip()[-600:], + }) + + def load(filename): + try: + with open(os.path.join(out_dir, filename), "r", encoding="utf-8") as handle: + return json.load(handle) + except (OSError, ValueError): + return {} + + probe = load("probe.json") + discover = load("discover.json") + judge = load("judge.json") + + recommendation = discover.get("recommendation") or {} + candidates = judge.get("candidates") or [] + bans = [row for row in candidates + if str(row.get("action", "")).startswith("ban")] + summary = [ + "graylog-ops collect", + "generated_at: %s" % now_iso(), + "graylog: %s (%s)" % ( + _short_version((probe.get("system") or {}).get("version")), + ctx.config.url, + ), + "", + "字段推断: %s (confidence=%s, sampled=%s)" % ( + recommendation.get("field") or "-", + recommendation.get("confidence") or "none", + discover.get("sampled", "-"), + ), + "封禁候选: %d 个（ban_1h/ban_30m），其余候选 %d 个，排除 %d 个" % ( + len(bans), len(candidates) - len(bans), len(judge.get("excluded") or []), + ), + "", + "步骤:", + ] + for item in results: + summary.append(" %-10s %-4s %6dms %s" % ( + item["step"], "OK" if item["ok"] else "FAIL", item["ms"], + item["stderr"].splitlines()[-1] if item["stderr"] else "", + )) + summary.append("") + summary.append("该目录不含凭据，可直接打包发送。") + with open(os.path.join(out_dir, "SUMMARY.txt"), "w", encoding="utf-8") as handle: + handle.write("\n".join(summary) + "\n") + + report = { + "directory": os.path.abspath(out_dir), + "steps": results, + "recommendation": recommendation, + "ban_candidates": len(bans), + "candidate_total": len(candidates), + "excluded": len(judge.get("excluded") or []), + "collected_at": now_iso(), + } + + def human(payload): + lines = [ + "采集目录: %s" % payload["directory"], + "", + render.table( + ["步骤", "结果", "耗时(ms)", "备注"], + [(item["step"], "OK" if item["ok"] else "FAIL", item["ms"], + render.clip(item["stderr"].splitlines()[-1] if item["stderr"] else "", 60)) + for item in payload["steps"]], + ["l", "l", "r", "l"], + ), + "", + "字段推断: %s (confidence=%s)" + % (payload["recommendation"].get("field") or "-", + payload["recommendation"].get("confidence") or "none"), + "封禁候选: %d 个，其余候选 %d 个，排除 %d 个" + % (payload["ban_candidates"], payload["candidate_total"] - payload["ban_candidates"], + payload["excluded"]), + "", + "该目录不含凭据，压缩后发出来即可。", + ] + return "\n".join(lines) + + ctx.emit(report, human) + return 0 + + +# ----------------------------------------------------------------------- watch +def cmd_watch(ctx, args): + from . import watch + + ctx.stream_id = resolve_stream(ctx.client, getattr(args, "stream", None)) + return watch.run(ctx, args) + + +# ----------------------------------------------------------------------- judge +def cmd_judge(ctx, args): + client = ctx.client + profile = load_profile(getattr(args, "profile", None)) + fields = build_fields(profile, args) + signals = build_signals(fields, profile.get("signals")) + + thresholds = dict(DEFAULT_THRESHOLDS) + thresholds.update(profile.get("thresholds") or {}) + if getattr(args, "min_requests", None) is not None: + thresholds["min_requests"] = int(args.min_requests) + if getattr(args, "min_score", None) is not None: + thresholds["min_score"] = float(args.min_score) + + allowlist = Allowlist( + load_allowlist(getattr(args, "allow", None)), + exclude_internal=not getattr(args, "include_internal", False), + ) + + stream_id = resolve_stream(client, getattr(args, "stream", None)) + filter_str = _filter_for(stream_id) + window = int(getattr(args, "range", 300) or 300) + query = getattr(args, "query", "*") or "*" + size = int(getattr(args, "size", 200) or 200) + + totals, per_signal, errors = collect_signals( + client, fields["client_ip"], signals, window, + query=query, filter_str=filter_str, size=size, + ) + report = score_candidates(totals, per_signal, signals, thresholds, allowlist) + + verify_count = int(getattr(args, "verify", 0) or 0) + if verify_count > 0: + verify_candidates(client, report["candidates"], fields, window, + filter_str=filter_str, limit=verify_count) + report["candidates"].sort( + key=lambda row: (-row["score"], -row["requests"], row["ip"])) + + report.update({ + "window_seconds": window, + "query": query, + "field": fields["client_ip"], + "stream_id": stream_id, + "fields": fields, + "thresholds": dict(thresholds), + "signals": [ + {key: signal[key] for key in ("name", "title", "query", "weight", "saturate")} + for signal in signals + ], + "errors": errors, + "collected_at": now_iso(), + }) + + explain = int(getattr(args, "explain", 0) or 0) + if explain > 0: + for candidate in report["candidates"][:explain]: + try: + samples = client.search_relative( + query='%s:"%s"' % (fields["client_ip"], candidate["ip"]), + range_seconds=window, + limit=3, + filter_str=filter_str, + ) + except GraylogError: + samples = [] + keep = {"timestamp", "message"} | {value for value in fields.values() if value} + candidate["samples"] = [ + {key: value for key, value in message.items() if key in keep} + for message in samples + ] + + total_candidates = len(report["candidates"]) + shown = report["candidates"] + if not getattr(args, "all", False): + shown = [row for row in shown if row["score"] >= thresholds["min_score"]] + report["total_candidates"] = total_candidates + report["candidates"] = shown[: int(getattr(args, "top", 30) or 30)] + report["shown"] = len(report["candidates"]) + + def human(payload): + lines = [ + "窗口 %ds query=%s field=%s stream=%s 候选池 %d（中位请求数 %s）" + % (payload["window_seconds"], payload["query"], payload["field"], + payload["stream_id"] or "(all)", payload["candidate_pool"], + payload["median_requests"]), + "", + ] + if payload["candidates"]: + rows = [] + for row in payload["candidates"]: + rows.append(( + "%.1f" % row["score"], + row["action"], + row["ip"], + row["requests"], + render.clip("; ".join(row["reasons"]), 74), + )) + lines.append(render.table(["得分", "动作", "客户端 IP", "请求数", "依据"], rows, + ["r", "l", "l", "r", "l"])) + else: + lines.append("没有达到 --min-score 的候选。先跑 discover 确认字段，或降低阈值观察。") + + if payload["excluded"]: + lines.append("") + lines.append("已排除（不参与判分）:") + for row in payload["excluded"]: + lines.append(" %-20s %-6s %s" % (row["ip"], row["requests"], row["reason"])) + + lines.append("") + lines.append("信号命中:") + for stat in payload["signal_stats"]: + lines.append( + " %-14s %-10s %3d 个 IP / %5d 次" + % (stat["name"], stat["title"], stat["ips_matched"], + stat["requests_matched"]) + ) + lines.append(" %-25s query: %s" % ("", render.clip(stat["query"], 66))) + + zero = [stat["name"] for stat in payload["signal_stats"] if stat["ips_matched"] == 0] + if zero: + lines.append("") + lines.append("! 信号命中 0：%s —— 若窗口内确实有这类流量，说明查询模板与字段分词不匹配，" + "需要用 profile 覆盖 query" % ", ".join(zero)) + for error in payload["errors"]: + lines.append("! 信号 %s 查询失败: %s" % (error["signal"], error["error"])) + + samples = [(row["ip"], item) for row in payload["candidates"][:3] + for item in (row.get("samples") or [])] + seen = set() + unique = [] + for ip, item in samples: + text = item.get("message", "") + if text in seen: + continue + seen.add(text) + unique.append((ip, item)) + if unique: + lines.append("") + lines.append("样例消息（前 3 个候选）:") + for ip, item in unique[:6]: + lines.append(" %-16s %s" % (ip, render.clip(item.get("message", ""), 88))) + + lines.append("") + lines.append("本命令只读，不会下发封禁；下一步是把 action 交给执行侧（AD 黑名单 / WAF）。") + return "\n".join(lines) + + ctx.emit(report, human) + return 0 + + +# -------------------------------------------------------------------- messages +def cmd_messages(ctx, args): + client = ctx.client + stream_id = resolve_stream(client, getattr(args, "stream", None)) + fields = None + if getattr(args, "fields", None): + fields = [name.strip() for name in args.fields.split(",") if name.strip()] + messages = client.search_relative( + query=getattr(args, "query", "*") or "*", + range_seconds=getattr(args, "range", 300), + limit=int(getattr(args, "limit", 50) or 50), + offset=int(getattr(args, "offset", 0) or 0), + fields=fields, + filter_str=_filter_for(stream_id), + ) + data = {"count": len(messages), "query": getattr(args, "query", "*"), + "stream_id": stream_id, "messages": messages} + + if getattr(args, "jsonl", False): + for message in messages: + print(json.dumps(message, ensure_ascii=False)) + return 0 + + def human(payload): + if not payload["messages"]: + return "没有匹配的消息（检查时间范围 / 查询语法 / stream 权限）" + columns = ["timestamp", "source"] + [name for name in (fields or []) + if name not in ("timestamp", "source")] + rows = [] + for message in payload["messages"]: + row = [render.clip(message.get(name, "-"), 22) for name in columns] + row.append(render.clip(message.get("message", ""), 70)) + rows.append(row) + header = [render.clip(name, 22) for name in columns] + ["message"] + return render.table(header, rows) + "\n\n共 %d 条" % payload["count"] + + ctx.emit(data, human) + return 0 + + +# ------------------------------------------------------------------- histogram +def cmd_histogram(ctx, args): + client = ctx.client + stream_id = resolve_stream(client, getattr(args, "stream", None)) + raw = client.histogram( + query=getattr(args, "query", "*") or "*", + range_seconds=getattr(args, "range", 3600), + interval=getattr(args, "interval", "1m") or "1m", + filter_str=_filter_for(stream_id), + ) + rows = normalize_histogram(raw) + total = sum((row["count"] or 0) for row in rows) + seconds = int(getattr(args, "range", 3600) or 3600) + data = { + "interval": getattr(args, "interval", "1m"), + "range_seconds": seconds, + "total": total, + "average_per_second": round(total / seconds, 3) if seconds else 0, + "buckets": rows, + } + + def human(payload): + table_rows = [(row["timestamp"], row["count"]) for row in payload["buckets"]] + lines = [ + render.table(["时间桶", "消息数"], table_rows, ["l", "r"]), + "", + "总计 %d 条，平均 %.2f 条/秒" % (payload["total"], payload["average_per_second"]), + ] + return "\n".join(lines) + + ctx.emit(data, human) + return 0 + + +# ---------------------------------------------------------------------- alerts +def cmd_alerts(ctx, args): + client = ctx.client + definitions = client.event_definitions(limit=getattr(args, "limit", None)) + notifications = client.notifications(limit=getattr(args, "limit", None)) + data = {"event_definitions": definitions, "notifications": notifications} + + def human(payload): + lines = [] + rows = [] + for item in payload["event_definitions"]: + config = item.get("config") or {} + rows.append(( + render.clip(item.get("title", "-"), 30), + item.get("priority", "-"), + "disabled" if item.get("disabled") else "active", + render.clip(str(config.get("type", "-")), 24), + len((item.get("notifications") or [])), + )) + lines.append("事件定义") + lines.append(render.table(["标题", "优先级", "状态", "类型", "通知数"], rows, + ["l", "r", "l", "l", "r"])) + lines.append("") + notif_rows = [] + for item in payload["notifications"]: + config = item.get("config") or {} + notif_rows.append(( + render.clip(item.get("title", "-"), 30), + render.clip(str(config.get("type", "-")), 26), + render.clip(str(config.get("url", config.get("uri", ""))), 50), + )) + lines.append("通知方式") + lines.append(render.table(["标题", "类型", "目标"], notif_rows, ["l", "l", "l"])) + return "\n".join(lines) + + ctx.emit(data, human) + return 0 + + +# -------------------------------------------------------------------- snapshot +SNAPSHOT_RESOURCES = ( + ("system", lambda client: client.system_info()), + ("cluster_nodes", lambda client: client.cluster_nodes()), + ("indexer_health", lambda client: client.indexer_health()), + ("deflector", lambda client: client.deflector()), + ("field_types", lambda client: client.field_types()), + ("license", lambda client: client.license_info()), + ("streams", lambda client: client.streams()), + ("inputs", lambda client: client.inputs()), + ("index_sets", lambda client: client.index_sets()), + ("indices", lambda client: client.indices()), + ("pipelines", lambda client: client.pipelines()), + ("pipeline_rules", lambda client: client.pipeline_rules()), + ("event_definitions", lambda client: client.event_definitions()), + ("notifications", lambda client: client.notifications()), + ("sidecars", lambda client: client.sidecars()), + ("views", lambda client: client.http.get("/api/views", {"page": 1, "per_page": 50})), +) + + +def cmd_snapshot(ctx, args): + client = ctx.client + out_dir = (getattr(args, "dir", None) or ctx.out_path + or ("graylog-snapshot-" + time.strftime("%Y%m%d-%H%M%S"))) + os.makedirs(out_dir, exist_ok=True) + + resources = [] + for name, fetch in SNAPSHOT_RESOURCES: + started = time.time() + try: + payload = fetch(client) + status, error = "ok", None + except GraylogError as exc: + payload, status, error = None, "error", str(exc) + write_json(os.path.join(out_dir, name + ".json"), { + "resource": name, + "status": status, + "error": error, + "data": payload, + }) + resources.append({ + "resource": name, + "status": status, + "error": error, + "ms": int((time.time() - started) * 1000), + }) + + sample = int(getattr(args, "sample", 0) or 0) + sampled = 0 + if sample > 0: + try: + messages = client.search_relative(limit=sample, range_seconds=3600) + path = os.path.join(out_dir, "messages.jsonl") + with open(path, "w", encoding="utf-8") as handle: + for message in messages: + handle.write(json.dumps(message, ensure_ascii=False) + "\n") + sampled = len(messages) + except GraylogError as exc: + resources.append({"resource": "messages.jsonl", "status": "error", + "error": str(exc), "ms": 0}) + + system = {} + for item in resources: + if item["resource"] == "system" and item["status"] == "ok": + try: + with open(os.path.join(out_dir, "system.json"), "r", encoding="utf-8") as handle: + system = json.load(handle).get("data") or {} + except (OSError, ValueError): + system = {} + + manifest = { + "tool": "graylog-ops", + "tool_version": __version__, + "generated_at": now_iso(), + "graylog": { + "url": ctx.config.url, + "version": system.get("version"), + "hostname": system.get("hostname"), + "cluster_id": system.get("cluster_id"), + }, + "sampled_messages": sampled, + "resources": resources, + } + write_json(os.path.join(out_dir, "manifest.json"), manifest) + + def human(payload): + rows = [(item["resource"], item["status"], item.get("ms", 0), + render.clip(item.get("error") or "", 60)) for item in payload["resources"]] + lines = [ + "Graylog %s @ %s" % (_short_version(payload["graylog"].get("version")), + payload["graylog"].get("hostname", "-")), + "", + render.table(["资源", "状态", "耗时(ms)", "备注"], rows, ["l", "l", "r", "l"]), + "", + "快照目录: %s" % out_dir, + ] + if payload["sampled_messages"]: + lines.append("附带采样消息: %d 条 -> messages.jsonl" % payload["sampled_messages"]) + failed = [item for item in payload["resources"] if item["status"] != "ok"] + lines.append("失败 %d 项（快照仍可用，缺失项见各自 json 的 error 字段）" % len(failed)) + return "\n".join(lines) + + ctx.emit(manifest, human) + return 0 + + +# ------------------------------------------------------------------------- raw +def cmd_raw(ctx, args): + params = {} + for item in (getattr(args, "param", None) or []): + if "=" not in item: + raise SystemExit("--param 需要 key=value 形式, 收到: %r" % item) + key, value = item.split("=", 1) + params[key] = value + + method = (getattr(args, "method", None) or "GET").upper() + body = None + raw_body = getattr(args, "body", None) + if raw_body: + if raw_body.startswith("@"): + try: + with open(raw_body[1:], "r", encoding="utf-8") as handle: + raw_body = handle.read() + except OSError as exc: + raise SystemExit("无法读取 --body 文件 %s: %s" % (raw_body[1:], exc)) + try: + body = json.loads(raw_body) + except ValueError as exc: + raise SystemExit("--body 不是合法 JSON: %s" % exc) + + if getattr(args, "text", False): + data = ctx.client.http.request(method, args.path, params=params, + body=body, expect_json=False) + print(data) + return 0 + + payload = ctx.client.http.request(method, args.path, params=params, body=body) + print(json.dumps(payload, indent=2, ensure_ascii=False)) + return 0 diff --git a/graylog_ops/config.py b/graylog_ops/config.py new file mode 100644 index 0000000..48d2205 --- /dev/null +++ b/graylog_ops/config.py @@ -0,0 +1,161 @@ +"""Configuration resolution. + +Precedence (highest wins): + 1. command line flags + 2. environment variables (GRAYLOG_URL, GRAYLOG_TOKEN, ...) + 3. JSON config file (--config, ./graylog-ops.json, + ~/.config/graylog-ops/config.json) + +File format:: + + { + "url": "http://edu-master:9000", + "token": "xxxxxxxx", + "insecure": false, + "timeout": 30 + } +""" + +import json +import os + +ENV_KEYS = { + "url": "GRAYLOG_URL", + "token": "GRAYLOG_TOKEN", + "user": "GRAYLOG_USER", + "password": "GRAYLOG_PASSWORD", + "insecure": "GRAYLOG_INSECURE", + "timeout": "GRAYLOG_TIMEOUT", + "retries": "GRAYLOG_RETRIES", +} + +DEFAULT_CONFIG_PATHS = ( + os.path.expanduser("~/.config/graylog-ops/config.json"), + "graylog-ops.json", +) + + +class ConfigError(Exception): + """User facing configuration problem.""" + + +def _truthy(value): + return str(value).strip().lower() in ("1", "true", "yes", "on", "y") + + +def _read_file(path): + try: + with open(path, "r", encoding="utf-8") as handle: + data = json.load(handle) + except OSError as exc: + raise ConfigError("cannot read config %s: %s" % (path, exc)) + except ValueError as exc: + raise ConfigError("config %s is not valid JSON: %s" % (path, exc)) + if not isinstance(data, dict): + raise ConfigError("config %s must contain a JSON object" % path) + return data + + +class Config: + def __init__(self, url, token=None, user=None, password=None, + insecure=False, timeout=30.0, retries=2, source=None): + self.url = url + self.token = token + self.user = user + self.password = password + self.insecure = bool(insecure) + self.timeout = float(timeout) + self.retries = int(retries) + self.source = source + + @property + def credentials(self): + """(username, password) pair for HTTP Basic auth.""" + if self.token: + # Graylog token auth: username is the token, password is "token". + return self.token, "token" + return self.user, self.password + + @property + def auth_kind(self): + return "token" if self.token else "user/password" + + +def _pick(values, key, default=None): + value = values.get(key, default) + return value + + +def load_config(args): + """Build a Config from CLI args, env vars and the config file.""" + values = {} + chosen = None + + explicit = getattr(args, "config", None) + if explicit: + if not os.path.exists(explicit): + raise ConfigError("config file not found: %s" % explicit) + values.update(_read_file(explicit)) + chosen = explicit + else: + for candidate in DEFAULT_CONFIG_PATHS: + if os.path.exists(candidate): + values.update(_read_file(candidate)) + chosen = candidate + break + + for key, env_name in ENV_KEYS.items(): + env_value = os.environ.get(env_name) + if env_value is not None and env_value.strip() != "": + values[key] = env_value + + for key in ("url", "token", "user", "password", "timeout", "retries"): + flag_value = getattr(args, key, None) + if flag_value is not None and flag_value != "": + values[key] = flag_value + if getattr(args, "insecure", False): + values["insecure"] = True + + url = values.get("url") + if not url: + raise ConfigError( + "no Graylog URL: pass --url, set GRAYLOG_URL, or add \"url\" to the config file" + ) + url = str(url).strip() + if "://" not in url: + url = "http://" + url + + token = (values.get("token") or "").strip() or None + user = (values.get("user") or "").strip() or None + password = values.get("password") + if not token and not (user and password): + raise ConfigError( + "no credentials: pass --token or --user/--password " + "(env GRAYLOG_TOKEN / GRAYLOG_USER + GRAYLOG_PASSWORD)" + ) + + timeout = values.get("timeout", 30) + retries = values.get("retries", 2) + try: + timeout = float(timeout) + except (TypeError, ValueError): + raise ConfigError("timeout must be a number, got %r" % (timeout,)) + try: + retries = int(retries) + except (TypeError, ValueError): + raise ConfigError("retries must be an integer, got %r" % (retries,)) + + insecure = values.get("insecure", False) + if isinstance(insecure, str): + insecure = _truthy(insecure) + + return Config( + url=url, + token=token, + user=user, + password=password, + insecure=bool(insecure), + timeout=timeout, + retries=retries, + source=chosen, + ) diff --git a/graylog_ops/detect.py b/graylog_ops/detect.py new file mode 100644 index 0000000..50e7aaf --- /dev/null +++ b/graylog_ops/detect.py @@ -0,0 +1,300 @@ +"""Field heuristics: find the field that actually holds the client IP. + +The ban pipeline is only as good as the field it keys on. In a typical +deployment behind a reverse proxy / CDN, the field you *think* is the client +IP is the proxy's address: + +* ``source`` -- the host that shipped the log line to Graylog +* ``gl2_remote_ip`` -- the host that opened the GELF/Beats connection +* ``remote_addr`` -- could be the proxy if XFF is not parsed + +So instead of assuming, sample real messages and score every field by +"how often does this field hold an IP address" plus "does the field name look +like a client address". +""" + +import ipaddress +import re + +# Field names that strongly suggest a client address. +STRONG_HINT = re.compile( + r"(?i)client[_. -]?ip|remote[_. -]?addr|real[_. -]?ip|" + r"(^|[_. -])ip([_. -]|$)|x[_. -]?forwarded[_. -]?for|" + r"(^|[_. -])xff([_. -]|$)|src[_. -]?ip|c[_. -]?ip" +) + +WEAK_HINT = re.compile( + r"(?i)addr|host|client|proxy|peer|origin|caller|visitor|remote|forward|endpoint" +) + +# Graylog internals that are never the client IP (gl2_remote_ip is the shipper). +INTERNAL_PREFIXES = ("gl2_", "_") +INTERNAL_NAMES = ("timestamp", "streams", "message") + +HTTP_HINTS = { + "status": re.compile(r"(?i)^(http[_. ]?)?status([_. ]?code)?$|^response([_. ]?(code|status))?$|^code$"), + "uri": re.compile(r"(?i)(request[_. ]?)?uri$|^url$|^path$|^request$|^endpoint$|^target$"), + "method": re.compile(r"(?i)^(http[_. ]?)?method$|^verb$"), + "user_agent": re.compile(r"(?i)user[_. -]?agent|^ua$|^agent$"), + "host": re.compile(r"(?i)^(http[_. ]?)?host$|^vhost$|^domain$|^server_name$"), + "bytes": re.compile(r"(?i)bytes|^size$|^body_bytes_sent$"), + "referer": re.compile(r"(?i)refer(r)?er$"), + "duration": re.compile(r"(?i)request[_. ]?time|response[_. ]?time|duration|latency|elapsed"), +} + +# Deliberately not using ``ipaddress.is_private``: that property also covers +# documentation ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24) and +# other special-use blocks, which would make every public test address look +# like an internal proxy. For ban decisions "internal" means RFC1918 plus +# loopback, link-local and CGNAT. +_INTERNAL_V4 = tuple(ipaddress.ip_network(cidr) for cidr in ( + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16", + "100.64.0.0/10", # CGNAT + "127.0.0.0/8", + "169.254.0.0/16", +)) +_INTERNAL_V6 = tuple(ipaddress.ip_network(cidr) for cidr in ( + "::1/128", + "fc00::/7", # unique local + "fe80::/10", # link local +)) + + +def is_internal(ip): + networks = _INTERNAL_V4 if ip.version == 4 else _INTERNAL_V6 + return any(ip in network for network in networks) + + +def as_ip(value): + """Return an ``ipaddress`` object if *value* looks like a bare IP. + + Accepts ``1.2.3.4``, ``2001:db8::1``, ``[2001:db8::1]:443`` and + ``1.2.3.4:5678``. Rejects CIDRs, hostnames and arbitrary integers. + """ + if not isinstance(value, str): + return None + text = value.strip().strip('"') + if not text or len(text) > 64: + return None + try: + return ipaddress.ip_address(text) + except ValueError: + pass + if text.startswith("[") and "]" in text: + try: + return ipaddress.ip_address(text[1:text.index("]")]) + except ValueError: + return None + if text.count(":") == 1 and "." in text: + host = text.split(":", 1)[0] + try: + return ipaddress.ip_address(host) + except ValueError: + return None + return None + + +def _empty(name): + return { + "name": name, + "present": 0, + "ip_messages": 0, + "ip_values": 0, + "distinct": set(), + "distinct_ip": set(), + "private": 0, + "versions": set(), + "examples": [], + } + + +def _analyse(messages): + total = len(messages) + stats = {} + embedded = 0 + for message in messages: + embedded_here = False + for name, raw in message.items(): + values = raw if isinstance(raw, list) else [raw] + entry = stats.get(name) + if entry is None: + entry = stats[name] = _empty(name) + emitted = False + has_ip = False + for value in values: + if value is None or value == "": + continue + text = str(value)[:160] + entry["distinct"].add(text) + emitted = True + ip = as_ip(value) + if ip is not None: + has_ip = True + entry["ip_values"] += 1 + entry["distinct_ip"].add(str(ip)) + entry["versions"].add(ip.version) + if is_internal(ip): + entry["private"] += 1 + text_ip = str(ip) + if len(entry["examples"]) < 3 and text_ip not in entry["examples"]: + entry["examples"].append(text_ip) + if name == "message": + embedded_here = True + if emitted: + entry["present"] += 1 + if has_ip: + entry["ip_messages"] += 1 + if embedded_here: + embedded += 1 + return stats, total, embedded + + +def _score(name, entry, total): + present = max(entry["present"], 1) + ip_rate = entry["ip_messages"] / present + presence = entry["present"] / max(total, 1) + score = 60.0 * ip_rate + 5.0 * presence + notes = [] + + if STRONG_HINT.search(name): + score += 30 + elif WEAK_HINT.search(name): + score += 10 + + if name == "source": + score -= 20 + notes.append("`source` 是日志发送方（shipper/反代），通常不是真实客户端") + elif name == "gl2_remote_ip": + score -= 25 + notes.append("`gl2_remote_ip` 是向 Graylog 发送消息的对端地址，不是访问者") + return score, notes + + +def _confidence(best): + if not best: + return "none" + if best["score"] >= 90 and best["ip_rate"] >= 0.95: + return "high" + if best["score"] >= 60: + return "medium" + return "low" + + +def analyse_messages(messages): + """Build the field report used by ``graylog-ops discover``.""" + stats, total, embedded = _analyse(messages) + if total == 0: + return { + "sampled": 0, + "fields": [], + "candidates": [], + "http_fields": {}, + "embedded_ip_in_message_rate": 0.0, + "recommendation": { + "field": None, + "confidence": "none", + "notes": ["没有采到消息：检查时间范围、查询条件、stream 过滤或权限"], + "warnings": [], + }, + "sample_message": None, + } + + fields = [] + candidates = [] + for name, entry in stats.items(): + if entry["present"] == 0: + continue + ip_rate = entry["ip_messages"] / entry["present"] + row = { + "name": name, + "present": entry["present"], + "presence": round(entry["present"] / total, 4), + "distinct": len(entry["distinct"]), + "distinct_ips": len(entry["distinct_ip"]), + "ip_rate": round(ip_rate, 4), + "examples": entry["examples"], + } + fields.append(row) + + if name in INTERNAL_NAMES or name.startswith(INTERNAL_PREFIXES): + continue + if ip_rate <= 0: + continue + score, notes = _score(name, entry, total) + private_ratio = (entry["private"] / entry["ip_values"]) if entry["ip_values"] else 0.0 + candidates.append({ + "field": name, + "score": round(score, 1), + "ip_rate": round(ip_rate, 4), + "presence": round(entry["present"] / total, 4), + "distinct_ips": len(entry["distinct_ip"]), + "private_ratio": round(private_ratio, 4), + "versions": sorted(entry["versions"]), + "examples": entry["examples"], + "notes": notes, + }) + + fields.sort(key=lambda row: (-row["present"], row["name"])) + candidates.sort(key=lambda row: (-row["score"], -row["distinct_ips"])) + + http_fields = {kind: [] for kind in HTTP_HINTS} + for name in stats: + for kind, pattern in HTTP_HINTS.items(): + if pattern.search(name): + http_fields[kind].append(name) + http_fields = {kind: names for kind, names in http_fields.items() if names} + + best = candidates[0] if candidates else None + source_entry = stats.get("source") + warnings = [] + notes = [] + + if best: + notes.append( + "字段 `%s` 的 IP 占比 %.0f%%，样例 %s" + % (best["field"], best["ip_rate"] * 100, ", ".join(best["examples"][:3]) or "-") + ) + if source_entry and len(source_entry["distinct"]) <= 2 and best["distinct_ips"] > 2: + warnings.append( + "`source` 只有 %d 个不同取值，而 `%s` 有 %d 个：`source` 是采集器/反代，" + "封禁必须用 `%s`" % (len(source_entry["distinct"]), best["field"], + best["distinct_ips"], best["field"]) + ) + if best["private_ratio"] >= 0.5: + warnings.append( + "`%s` 中 %.0f%% 是内网地址（私有/回环），可能仍是 LB 或代理地址，" + "需要确认 X-Forwarded-For 是否已在采集侧解析" + % (best["field"], best["private_ratio"] * 100) + ) + else: + notes.append("没有发现任何字段存放纯 IP 值") + + # Only worth flagging when there is no usable field: then the IP lives in + # the free-text `message` and has to be extracted server side. + if embedded / max(total, 1) >= 0.5 and not best: + warnings.append( + "%.0f%% 的消息在 `message` 文本里内嵌了 IP。要么在 input 上加 extractor / " + "pipeline 解析成独立字段，要么在采集端（nginx log_format）直接输出 client_ip 字段" + % (100.0 * embedded / total) + ) + + # HTTP-ish fields help with the next phase (per-URI thresholds), surface them. + if not best and http_fields: + warnings.append("只找到 HTTP 相关字段，没有 IP 字段，见 http_fields") + + return { + "sampled": total, + "fields": fields, + "candidates": candidates, + "http_fields": http_fields, + "embedded_ip_in_message_rate": round(embedded / total, 4), + "recommendation": { + "field": best["field"] if best else None, + "confidence": _confidence(best), + "notes": notes, + "warnings": warnings, + }, + "sample_message": messages[0] if messages else None, + } diff --git a/graylog_ops/geo.py b/graylog_ops/geo.py new file mode 100644 index 0000000..110768b --- /dev/null +++ b/graylog_ops/geo.py @@ -0,0 +1,138 @@ +"""Offline GeoIP lookup backed by the free DB-IP Lite CSV dumps. + +The city dump has no country name, so a small code -> name table is shipped +here; anything unknown falls back to the ISO code. +""" +import csv +import ipaddress +import os +import sqlite3 + +COUNTRY_CN = { + "CN": "中国", "HK": "中国香港", "TW": "中国台湾", "MO": "中国澳门", + "US": "美国", "CA": "加拿大", "MX": "墨西哥", "BR": "巴西", "AR": "阿根廷", + "CL": "智利", "CO": "哥伦比亚", "PE": "秘鲁", "VE": "委内瑞拉", + "GB": "英国", "IE": "爱尔兰", "FR": "法国", "DE": "德国", "NL": "荷兰", + "BE": "比利时", "LU": "卢森堡", "CH": "瑞士", "AT": "奥地利", "IT": "意大利", + "ES": "西班牙", "PT": "葡萄牙", "GR": "希腊", "PL": "波兰", "CZ": "捷克", + "SK": "斯洛伐克", "HU": "匈牙利", "RO": "罗马尼亚", "BG": "保加利亚", + "HR": "克罗地亚", "RS": "塞尔维亚", "SI": "斯洛文尼亚", "LT": "立陶宛", + "LV": "拉脱维亚", "EE": "爱沙尼亚", "FI": "芬兰", "SE": "瑞典", + "NO": "挪威", "DK": "丹麦", "IS": "冰岛", "UA": "乌克兰", "BY": "白俄罗斯", + "RU": "俄罗斯", "TR": "土耳其", "IL": "以色列", "SA": "沙特阿拉伯", + "AE": "阿联酋", "QA": "卡塔尔", "KW": "科威特", "IR": "伊朗", "IQ": "伊拉克", + "EG": "埃及", "ZA": "南非", "NG": "尼日利亚", "KE": "肯尼亚", "MA": "摩洛哥", + "ET": "埃塞俄比亚", "IN": "印度", "PK": "巴基斯坦", "BD": "孟加拉国", + "LK": "斯里兰卡", "NP": "尼泊尔", "MM": "缅甸", "KH": "柬埔寨", "LA": "老挝", + "TH": "泰国", "VN": "越南", "MY": "马来西亚", "SG": "新加坡", "ID": "印度尼西亚", + "PH": "菲律宾", "JP": "日本", "KR": "韩国", "MN": "蒙古", "KZ": "哈萨克斯坦", + "UZ": "乌兹别克斯坦", "GE": "格鲁吉亚", "AM": "亚美尼亚", "AZ": "阿塞拜疆", + "AU": "澳大利亚", "NZ": "新西兰", "ZZ": "未知", +} + +CN_REGION = { + "Beijing": "北京", "Tianjin": "天津", "Hebei": "河北", "Shanxi": "山西", + "Inner Mongolia": "内蒙古", "Liaoning": "辽宁", "Jilin": "吉林", + "Heilongjiang": "黑龙江", "Shanghai": "上海", "Jiangsu": "江苏", + "Zhejiang": "浙江", "Anhui": "安徽", "Fujian": "福建", "Jiangxi": "江西", + "Shandong": "山东", "Henan": "河南", "Hubei": "湖北", "Hunan": "湖南", + "Guangdong": "广东", "Guangxi": "广西", "Hainan": "海南", "Chongqing": "重庆", + "Sichuan": "四川", "Guizhou": "贵州", "Yunnan": "云南", "Tibet": "西藏", + "Shaanxi": "陕西", "Gansu": "甘肃", "Qinghai": "青海", "Ningxia": "宁夏", + "Xinjiang": "新疆", "Taiwan": "台湾", "Hong Kong": "香港", "Macau": "澳门", +} + + +def flag(cc): + if not cc or len(cc) != 2 or not cc.isalpha() or cc == "ZZ": + return "" + return "".join(chr(0x1F1E6 + ord(char) - 65) for char in cc.upper()) + + +def country_name(cc): + return COUNTRY_CN.get(cc, cc or "") + + +def region_name(cc, region): + if cc == "CN" and region: + return CN_REGION.get(region, region) + return region or "" + + +def build_geo_db(city_csv, out_db, country_csv=None): + """Import the DB-IP city CSV (IPv4 only) into a small SQLite database.""" + os.makedirs(os.path.dirname(out_db) or ".", exist_ok=True) + conn = sqlite3.connect(out_db) + conn.execute("DROP TABLE IF EXISTS geo") + conn.execute("CREATE TABLE geo (start INTEGER PRIMARY KEY, end INTEGER, " + "cc TEXT, region TEXT, city TEXT)") + batch = [] + total = 0 + + def flush(): + nonlocal batch + if batch: + conn.executemany("INSERT OR REPLACE INTO geo VALUES (?,?,?,?,?)", batch) + batch = [] + + with open(city_csv, newline="", encoding="utf-8", errors="replace") as fh: + for row in csv.reader(fh): + if len(row) < 6: + continue + start, end, cc = row[0], row[1], row[3] + if ":" in start or ":" in end or not cc: + continue + try: + first = int(ipaddress.IPv4Address(start)) + last = int(ipaddress.IPv4Address(end)) + except ValueError: + continue + batch.append((first, last, cc, row[4], row[5])) + total += 1 + if len(batch) >= 100000: + flush() + conn.commit() + flush() + conn.commit() + conn.execute("PRAGMA journal_mode=WAL") + conn.close() + return total + + +class GeoIP: + """Read-only lookup over the generated geo.db.""" + + def __init__(self, path): + if not path or not os.path.exists(path): + raise FileNotFoundError(path) + self.conn = sqlite3.connect("file:%s?mode=ro" % path, uri=True, + check_same_thread=False) + + def lookup(self, ip): + try: + value = int(ipaddress.ip_address(ip)) + except ValueError: + return None + row = self.conn.execute( + "SELECT end, cc, region, city FROM geo WHERE start <= ? " + "ORDER BY start DESC LIMIT 1", (value,)).fetchone() + if not row or row[0] < value: + return None + return {"country_code": row[1], "country_name": country_name(row[1]), + "region": region_name(row[1], row[2]), "city": row[3] or ""} + + def close(self): + try: + self.conn.close() + except sqlite3.Error: + pass + + +if __name__ == "__main__": + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--city-csv", default="data/geo/dbip-city-lite-2026-10.csv") + parser.add_argument("--out", default="data/geo/geo.db") + args = parser.parse_args() + count = build_geo_db(args.city_csv, args.out) + print("imported %d IPv4 ranges -> %s" % (count, args.out)) diff --git a/graylog_ops/http.py b/graylog_ops/http.py new file mode 100644 index 0000000..28a7322 --- /dev/null +++ b/graylog_ops/http.py @@ -0,0 +1,214 @@ +"""Minimal HTTP layer for the Graylog REST API. + +Deliberately built on urllib so the tool has no third-party dependencies and +can be copied onto old jump hosts. Handles the three things Graylog's API +demands that plain urllib does not do for you: + +* HTTP Basic auth (API token used as username with the literal password + ``token``, or a real username/password pair), +* the ``X-Requested-By`` header required by Graylog's CSRF protection on all + non-GET requests, +* bounded retries with backoff for 429/5xx and transient connection errors. +""" + +import base64 +import json +import random +import socket +import ssl +import sys +import time +import urllib.error +import urllib.parse +import urllib.request + +USER_AGENT = "graylog-ops/0.1 (read-only recon)" +RETRY_STATUS = frozenset((429, 500, 502, 503, 504)) + + +class GraylogError(Exception): + """Any failure talking to the Graylog API.""" + + def __init__(self, message, status=None, body=None, path=None, method=None): + super().__init__(message) + self.message = message + self.status = status + self.body = body + self.path = path + self.method = method + + def __str__(self): + parts = [] + if self.status is not None: + parts.append("HTTP %s" % self.status) + if self.method and self.path: + parts.append("%s %s" % (self.method, self.path)) + parts.append(self.message) + return " | ".join(parts) + + +def _basic_auth(username, password): + raw = ("%s:%s" % (username, password)).encode("utf-8") + return "Basic " + base64.b64encode(raw).decode("ascii") + + +def _error_detail(body): + """Pull a human readable message out of a Graylog/ES error body.""" + if not body: + return "" + try: + data = json.loads(body) + except (TypeError, ValueError): + return body.strip()[:300] + if isinstance(data, dict): + for key in ("message", "error", "description"): + value = data.get(key) + if isinstance(value, str) and value.strip(): + return value.strip() + if isinstance(data.get("error"), dict): + reason = data["error"].get("reason") + if isinstance(reason, str): + return reason + return body.strip()[:300] + + +class HttpClient: + def __init__( + self, + base_url, + username=None, + password=None, + timeout=30.0, + retries=2, + insecure=False, + verbose=False, + ): + self.base_url = base_url.rstrip("/") + self.timeout = float(timeout) + self.retries = max(0, int(retries)) + self.verbose = verbose + self.auth = None + if username is not None: + self.auth = _basic_auth(username, "" if password is None else password) + + handlers = [] + if insecure: + context = ssl.create_default_context() + context.check_hostname = False + context.verify_mode = ssl.CERT_NONE + handlers.append(urllib.request.HTTPSHandler(context=context)) + # build_opener keeps the default ProxyHandler, so HTTP(S)_PROXY env vars + # keep working for environments that need a proxy to reach Graylog. + self.opener = urllib.request.build_opener(*handlers) + + # -- url helpers ----------------------------------------------------- + def url(self, path, params=None): + if not path.startswith("/"): + path = "/" + path + target = self.base_url + path + if params: + clean = {} + for key, value in params.items(): + if value is None: + continue + if isinstance(value, bool): + value = "true" if value else "false" + clean[key] = value + if clean: + sep = "&" if "?" in target else "?" + target += sep + urllib.parse.urlencode(clean, doseq=True) + return target + + # -- request core ---------------------------------------------------- + def request(self, method, path, params=None, body=None, expect_json=True): + method = method.upper() + url = self.url(path, params) + payload = None + headers = { + "Accept": "application/json", + "Accept-Encoding": "identity", + "User-Agent": USER_AGENT, + # Graylog rejects state changing requests without this header. + "X-Requested-By": "graylog-ops", + } + if body is not None: + payload = json.dumps(body).encode("utf-8") + headers["Content-Type"] = "application/json" + if self.auth: + headers["Authorization"] = self.auth + + started = time.time() + last_error = None + for attempt in range(self.retries + 1): + request = urllib.request.Request( + url, data=payload, headers=headers, method=method + ) + try: + with self.opener.open(request, timeout=self.timeout) as response: + status = response.getcode() + raw = response.read() + self._debug(method, url, status, started) + if not raw.strip(): + return None + text = raw.decode("utf-8", "replace") + if not expect_json: + return text + try: + return json.loads(text) + except ValueError: + raise GraylogError( + "response was not JSON", + status=status, + body=text[:500], + path=path, + method=method, + ) + except urllib.error.HTTPError as exc: + text = exc.read().decode("utf-8", "replace") + detail = _error_detail(text) + self._debug(method, url, exc.code, started) + last_error = GraylogError( + detail or ("request rejected" if exc.code != 401 else "authentication failed"), + status=exc.code, + body=text[:2000], + path=path, + method=method, + ) + if exc.code in RETRY_STATUS and attempt < self.retries: + self._sleep(attempt, exc.headers.get("Retry-After")) + continue + raise last_error + except (urllib.error.URLError, socket.timeout, ConnectionError, OSError) as exc: + reason = getattr(exc, "reason", exc) + self._debug(method, url, "ERR", started) + last_error = GraylogError( + "connection failed: %s" % reason, + path=path, + method=method, + ) + if attempt < self.retries: + self._sleep(attempt, None) + continue + raise last_error + raise last_error # pragma: no cover - loop always raises or returns + + # -- internals ------------------------------------------------------- + def _sleep(self, attempt, retry_after): + delay = (1.5 ** attempt) + random.uniform(0, 0.3) + try: + if retry_after is not None: + delay = max(delay, float(retry_after)) + except (TypeError, ValueError): + pass + time.sleep(min(delay, 30.0)) + + def _debug(self, method, url, status, started): + if not self.verbose: + return + elapsed = int((time.time() - started) * 1000) + sys.stderr.write("[http] %s %s -> %s (%dms)\n" % (method, url, status, elapsed)) + sys.stderr.flush() + + # -- convenience ----------------------------------------------------- + def get(self, path, params=None): + return self.request("GET", path, params=params) diff --git a/graylog_ops/judge.py b/graylog_ops/judge.py new file mode 100644 index 0000000..dcecdf5 --- /dev/null +++ b/graylog_ops/judge.py @@ -0,0 +1,397 @@ +"""Judgement layer: aggregated Graylog signals -> ranked ban candidates. + +This module is **read only**. It answers one question: + + "在这个时间窗里，哪些客户端 IP 看起来像在攻击，为什么，建议怎么处理？" + +It does not touch the enforcement side (Sangfor AD / WAF / iptables). The +output is a list of ``{ip, score, action, reasons, signals}`` which the next +phase can feed into whatever blacklist API we end up using. + +Design notes +------------ +* A **fixed number of ``terms`` queries** (one per signal) keeps the load on + Elasticsearch predictable no matter how many IPs show up in the window. +* Signal queries are templates over field names. The report exposes each + signal's query and hit count, so they can be tuned against real data. A + signal that matched 0 IPs usually means the query does not fit the field + mapping (e.g. ``request_uri`` is a keyword-ish analyzed field). +* **Volume alone never bans** (``require_evidence``). A busy but legitimate + client must not be blackholed by a rate threshold alone; at least one + behavioural signal (auth failures / scanner UA / sensitive path / 404 + probing) has to be present for a ban recommendation. +""" + +import ipaddress +import json +import math +import statistics + +from . import detect +from .client import normalize_terms +from .http import GraylogError + +DEFAULT_FIELDS = { + "client_ip": "client_ip", + "status": "status", + "uri": "request_uri", + "ua": "http_user_agent", + "method": "request_method", +} + +# name, title, required field keys, query template, weight, saturation count +SIGNAL_DEFS = ( + { + "name": "auth_fail", + "title": "认证失败", + "requires": ("status",), + "query": lambda fields: "%s:(401 OR 403)" % fields["status"], + "weight": 45.0, + "saturate": 10.0, + }, + { + "name": "scanner_ua", + "title": "扫描器 UA", + "requires": ("ua",), + "query": lambda fields: ( + "%s:(sqlmap* OR nikto* OR nuclei* OR nmap* OR masscan* OR zgrab* OR " + "wpscan* OR gobuster* OR dirbuster* OR dirsearch* OR acunetix*)" + % fields["ua"] + ), + "weight": 50.0, + "saturate": 2.0, + }, + { + "name": "sensitive_path", + "title": "敏感路径", + "requires": ("uri",), + # Slashes must be escaped, otherwise Lucene treats /.../ as a regex. + "query": lambda fields: ( + "%s:(\\/wp-login* OR \\/phpmyadmin* OR \\/.env* OR \\/actuator* " + "OR \\/cgi-bin* OR \\/manager\\/html*)" + % fields["uri"] + ), + "weight": 25.0, + "saturate": 5.0, + }, + { + "name": "not_found", + "title": "404 探测", + "requires": ("status",), + "query": lambda fields: "%s:404" % fields["status"], + "weight": 20.0, + "saturate": 20.0, + }, + { + "name": "bad_method", + "title": "异常方法", + "requires": ("method",), + "query": lambda fields: "%s:(PUT OR DELETE OR TRACE OR CONNECT)" % fields["method"], + "weight": 10.0, + "saturate": 5.0, + }, +) + +# URIs containing these hints mean auth failures are likely credential +# attacks rather than an integration with a broken/expired token. +AUTH_PATH_HINTS = ("login", "signin", "auth", "token", "captcha", "oauth", "password") + +DEFAULT_THRESHOLDS = { + "min_requests": 20, # ignore IPs with fewer requests in the window + "min_score": 40.0, # below this: ignore + "short_ban_score": 55.0, # >= this: short ban recommendation + "ban_score": 75.0, # >= this: ban recommendation + "rate_weight": 25.0, + "rate_saturate": 200.0, # requests in window for full rate points + "spike_weight": 15.0, + "auth_ratio_weight": 25.0, + "require_evidence": True, + "evidence_signals": ("auth_fail", "scanner_ua", "sensitive_path", "not_found", "bad_method"), +} + + +# --------------------------------------------------------------------- profile +def load_profile(path): + if not path: + return {} + try: + with open(path, "r", encoding="utf-8") as handle: + data = json.load(handle) + except OSError as exc: + raise SystemExit("无法读取 profile %s: %s" % (path, exc)) + except ValueError as exc: + raise SystemExit("profile %s 不是合法 JSON: %s" % (path, exc)) + if not isinstance(data, dict): + raise SystemExit("profile %s 必须是 JSON 对象" % path) + return data + + +def build_fields(profile, args=None): + fields = dict(DEFAULT_FIELDS) + for key, value in (profile.get("fields") or {}).items(): + if value: + fields[key] = value + + flag_map = { + "client_ip": "field", + "status": "status_field", + "uri": "uri_field", + "ua": "ua_field", + "method": "method_field", + } + for key, attribute in flag_map.items(): + value = getattr(args, attribute, None) if args is not None else None + if value: + fields[key] = value + return fields + + +def build_signals(fields, overrides=None): + overrides = overrides or {} + signals = [] + for definition in SIGNAL_DEFS: + if any(not fields.get(key) for key in definition["requires"]): + continue + signal = { + "name": definition["name"], + "title": definition["title"], + "query": definition["query"](fields), + "weight": float(definition["weight"]), + "saturate": float(definition["saturate"]), + } + override = overrides.get(signal["name"]) or {} + if override.get("enabled") is False: + continue + if override.get("query"): + signal["query"] = override["query"] + if override.get("weight") is not None: + signal["weight"] = float(override["weight"]) + if override.get("saturate") is not None: + signal["saturate"] = float(override["saturate"]) + signals.append(signal) + return signals + + +# ------------------------------------------------------------------- allowlist +class Allowlist: + """Never ban infrastructure, internal addresses or explicitly listed nets.""" + + def __init__(self, entries=None, exclude_internal=True): + self.entries = list(entries or []) + self.exclude_internal = exclude_internal + + def reason(self, ip_text): + try: + ip = ipaddress.ip_address(ip_text) + except ValueError: + return None + for network, source in self.entries: + if ip.version == network.version and ip in network: + return "白名单 %s" % source + if self.exclude_internal and detect.is_internal(ip): + return "内部地址（RFC1918/回环/CGNAT）" + return None + + +def load_allowlist(paths): + entries = [] + for path in paths or []: + try: + with open(path, "r", encoding="utf-8") as handle: + lines = handle.readlines() + except OSError as exc: + raise SystemExit("无法读取白名单 %s: %s" % (path, exc)) + for number, line in enumerate(lines, 1): + text = line.split("#", 1)[0].strip() + if not text: + continue + try: + network = ipaddress.ip_network(text, strict=False) + except ValueError: + raise SystemExit("白名单 %s:%d 不是合法 IP/CIDR: %r" % (path, number, text)) + entries.append((network, "%s:%d" % (path, number))) + return entries + + +# ----------------------------------------------------------------- collection +def collect_signals(client, field, signals, window_seconds, query="*", + filter_str=None, size=200): + """Run one terms query per signal plus one for the total request counts.""" + term_frame = lambda data: { + row["term"]: int(row["count"] or 0) + for row in normalize_terms(data) + if row["term"] not in (None, "") + } + + totals = term_frame(client.terms( + field=field, query=query, size=size, + range_seconds=window_seconds, filter_str=filter_str, + )) + + per_signal = {} + errors = [] + for signal in signals: + try: + data = client.terms( + field=field, query=signal["query"], size=size, + range_seconds=window_seconds, filter_str=filter_str, + ) + per_signal[signal["name"]] = term_frame(data) + except GraylogError as exc: + per_signal[signal["name"]] = {} + errors.append({ + "signal": signal["name"], + "status": exc.status, + "error": exc.message, + }) + return totals, per_signal, errors + + +# ------------------------------------------------------------------- verify +def verify_candidates(client, candidates, fields, window_seconds, + filter_str=None, limit=5, size=10): + """Drill into top candidates: is the traffic one automated endpoint poller? + + A client that hammers a *single* business API at a steady rate is usually an + integration with a broken/expired token, not an attacker. Banning it breaks + a business system, so such candidates are downgraded to ``observe`` and the + decision is left to a human. + """ + uri_field = fields.get("uri") + client_field = fields.get("client_ip") + if not uri_field or not client_field: + return + + for candidate in candidates[:limit]: + if not candidate["action"].startswith("ban"): + continue + query = '%s:"%s"' % (client_field, candidate["ip"]) + try: + data = client.terms( + field=uri_field, query=query, size=size, + range_seconds=window_seconds, filter_str=filter_str, + ) + except GraylogError as exc: + candidate["verify_error"] = exc.message + continue + + rows = [(row["term"], row["count"]) for row in normalize_terms(data) + if row["term"] not in (None, "")] + total = sum(count for _, count in rows) + candidate["verify"] = { + "distinct_uris": len(rows), + "total_requests": total, + "top_uri": rows[0][0] if rows else None, + "top_uri_ratio": round(rows[0][1] / total, 4) if rows and total else 0.0, + } + if not rows or not total: + continue + + top_uri, top_count = rows[0] + ratio = top_count / float(total) + login_like = any(hint in str(top_uri).lower() for hint in AUTH_PATH_HINTS) + candidate["verify"]["login_like"] = login_like + + if ratio >= 0.9 and len(rows) <= 2 and not login_like: + candidate["verify"]["automation_suspected"] = True + candidate["action"] = "observe" + candidate["score"] = round(min(candidate["score"], 54.0), 1) + candidate["reasons"].append( + "流量集中在单一接口 %s（%.0f%%），疑似业务客户端/自动化任务，已降级为 observe" + % (top_uri, ratio * 100) + ) + + +# -------------------------------------------------------------------- scoring +def score_candidates(totals, per_signal, signals, thresholds, allowlist): + pool = {ip: count for ip, count in totals.items() + if count >= thresholds["min_requests"]} + median = statistics.median(pool.values()) if pool else 0 + + candidates = [] + excluded = [] + + for ip, requests in pool.items(): + reason = allowlist.reason(ip) + if reason: + excluded.append({"ip": ip, "requests": requests, "reason": reason}) + continue + + points = {} + counts = {} + reasons = [] + + for signal in signals: + count = per_signal.get(signal["name"], {}).get(ip, 0) + if not count: + continue + counts[signal["name"]] = count + contributed = signal["weight"] * min(1.0, count / signal["saturate"]) + points[signal["name"]] = round(contributed, 1) + reasons.append("%s %d 次" % (signal["title"], count)) + + rate_points = thresholds["rate_weight"] * min( + 1.0, requests / float(thresholds["rate_saturate"])) + points["rate"] = round(rate_points, 1) + + if median and requests >= max(50, median * 4): + ratio = requests / float(median) + spike = thresholds["spike_weight"] * min(1.0, math.log(ratio, 2) / 6.0) + points["spike"] = round(spike, 1) + reasons.append("请求量是候选中位数的 %.0f 倍" % ratio) + + auth = counts.get("auth_fail", 0) + auth_ratio = (auth / float(requests)) if requests else 0.0 + if auth >= 10 and auth_ratio >= 0.5: + points["auth_ratio"] = thresholds["auth_ratio_weight"] + reasons.append("失败请求占 %.0f%%" % (auth_ratio * 100)) + + score = min(100.0, sum(points.values())) + evidence = any(counts.get(name) for name in thresholds["evidence_signals"]) + + if evidence and score >= thresholds["ban_score"]: + action = "ban_1h" + elif evidence and score >= thresholds["short_ban_score"]: + action = "ban_30m" + elif score >= thresholds["min_score"]: + action = "observe" + else: + action = "ignore" + + if not evidence and thresholds["require_evidence"] and action in ("ban_1h", "ban_30m"): + action = "observe" + reasons.append("仅请求量异常、无攻击特征，不自动封禁") + + candidates.append({ + "ip": ip, + "score": round(score, 1), + "requests": requests, + "action": action, + "evidence": evidence, + "auth_ratio": round(auth_ratio, 4), + "signals": counts, + "points": points, + "reasons": reasons, + }) + + candidates.sort(key=lambda row: (-row["score"], -row["requests"], row["ip"])) + excluded.sort(key=lambda row: (-row["requests"], row["ip"])) + + signal_stats = [] + for signal in signals: + matched = per_signal.get(signal["name"], {}) + signal_stats.append({ + "name": signal["name"], + "title": signal["title"], + "query": signal["query"], + "ips_matched": len(matched), + "requests_matched": sum(matched.values()), + }) + + return { + "median_requests": median, + "candidate_pool": len(pool), + "candidates": candidates, + "excluded": excluded, + "signal_stats": signal_stats, + } diff --git a/graylog_ops/notify.py b/graylog_ops/notify.py new file mode 100644 index 0000000..4d3b626 --- /dev/null +++ b/graylog_ops/notify.py @@ -0,0 +1,167 @@ +"""Webhook notifications for newly convicted malicious IPs. + +Supported payload flavours: WeCom (企业微信), DingTalk, Feishu, Slack and a +generic JSON body. Delivery failures are reported but never break detection. +""" +import base64 +import hashlib +import re +import hmac +import json +import time +import urllib.error +import urllib.parse +import urllib.request +from datetime import datetime + +from .geo import flag + +DEFAULT_TITLE = "新增恶意 IP" + + +class Notifier: + def __init__(self, config): + self.kind = (config.get("type") or "generic").lower() + self.url = config.get("url") or "" + if not self.url: + raise ValueError("notify config is missing 'url'") + self.secret = config.get("secret") or "" + self.link_prefix = config.get("link_prefix") or "" + self.timeout = int(config.get("timeout") or 10) + self.title = config.get("title") or DEFAULT_TITLE + + def link_for(self, ip): + if not self.link_prefix: + return "" + return self.link_prefix.rstrip("/") + "/#/ip/" + ip + + @staticmethod + def local_time(value): + try: + text = str(value).replace("Z", "+00:00") + return datetime.fromisoformat(text).astimezone().strftime("%m-%d %H:%M") + except (ValueError, TypeError): + return str(value) + + @staticmethod + def risk_label(score): + if score >= 90: + return "🔴 高危" + if score >= 70: + return "🟠 中危" + return "🟡 关注" + + def render(self, entries): + count = len(entries) + heading = self.title if count == 1 else "新增 %d 个恶意 IP" % count + lines = ["### 🚨 %s" % heading, ""] + for index, entry in enumerate(entries, 1): + ip = entry.get("ip", "?") + geo = " · ".join(part for part in + (entry.get("country_name"), entry.get("region")) if part) + flag_text = flag(entry.get("country_code")) + place = ("%s %s" % (flag_text, geo)).strip() if geo else "" + score = float(entry.get("score") or 0) + lead = ("**%d. %s**" % (index, ip)) if count > 1 else ("**%s**" % ip) + lines.append("%s%s" % (lead, (" " + place) if place else "")) + lines.append("评分 **%.1f**（%s）｜请求 **%s** 次" + % (score, self.risk_label(score), entry.get("total") or 0)) + if entry.get("first_seen"): + lines.append("首次攻击 **%s**" % self.local_time(entry["first_seen"])) + rules = entry.get("rules_text") + if rules: + lines.append("") + lines.append("**命中类型**") + for part in rules.split("；"): + part = part.strip() + if not part: + continue + part = re.sub(r"\s+(\d+)\s*次$", r" × \1", part) + lines.append("- " + part) + link = self.link_for(ip) + if link: + lines.append("") + lines.append("[🔗 查看详情与证据](%s)" % link) + if index != count: + lines.append("") + lines.append("---") + lines.append("") + return "\n".join(lines) + + def signed_url(self, timestamp=None): + """DingTalk custom robots with 加签 need timestamp + HMAC-SHA256.""" + if self.kind != "dingtalk" or not self.secret: + return self.url + stamp = str(timestamp if timestamp is not None + else int(time.time() * 1000)) + string_to_sign = stamp + "\n" + self.secret + digest = hmac.new(self.secret.encode("utf-8"), + string_to_sign.encode("utf-8"), + hashlib.sha256).digest() + sign = urllib.parse.quote_plus(base64.b64encode(digest).decode("ascii")) + separator = "&" if "?" in self.url else "?" + return "%s%stimestamp=%s&sign=%s" % (self.url, separator, stamp, sign) + + def payload(self, text, title=None): + title = title or self.title + if self.kind == "wecom": + return {"msgtype": "markdown", "markdown": {"content": text}} + if self.kind == "dingtalk": + return {"msgtype": "markdown", "markdown": {"title": title, "text": text}} + if self.kind == "feishu": + return {"msg_type": "text", "content": {"text": text}} + if self.kind == "slack": + return {"text": text} + return {"title": title, "text": text} + + def post(self, text, title=None): + body = json.dumps(self.payload(text, title), ensure_ascii=False).encode("utf-8") + request = urllib.request.Request( + self.signed_url(), data=body, method="POST", + headers={"Content-Type": "application/json; charset=utf-8"}) + try: + with urllib.request.urlopen(request, timeout=self.timeout) as response: + reply = response.read().decode("utf-8", "replace") + except urllib.error.HTTPError as exc: + raise RuntimeError("webhook HTTP %s: %s" + % (exc.code, exc.read()[:200].decode("utf-8", "replace"))) + except urllib.error.URLError as exc: + raise RuntimeError("webhook unreachable: %s" % exc.reason) + if reply and self.kind in ("wecom", "dingtalk", "feishu"): + try: + parsed = json.loads(reply) + except ValueError: + return reply + code = parsed.get("errcode", parsed.get("code", 0)) + if code: + raise RuntimeError("webhook rejected: %s" % reply[:200]) + return reply + + def send(self, entries): + if not entries: + return "" + return self.post(self.render(entries)) + + def send_test(self): + return self.post("**%s**\n\n这是一条测试推送，说明通知通道工作正常。" + % self.title, title=self.title + " · 测试") + + +def load_notifier(path): + with open(path, "r", encoding="utf-8") as handle: + return Notifier(json.load(handle)) + + +def notifier_from_env(): + """Build a notifier from NOTIFY_* environment variables (container mode).""" + import os + url = (os.environ.get("NOTIFY_URL") or "").strip() + if not url: + return None + return Notifier({ + "type": os.environ.get("NOTIFY_TYPE", "generic"), + "url": url, + "secret": os.environ.get("NOTIFY_SECRET", ""), + "link_prefix": os.environ.get("NOTIFY_LINK_PREFIX", ""), + "title": os.environ.get("NOTIFY_TITLE", "新增恶意 IP"), + }) diff --git a/graylog_ops/render.py b/graylog_ops/render.py new file mode 100644 index 0000000..094b783 --- /dev/null +++ b/graylog_ops/render.py @@ -0,0 +1,77 @@ +"""Terminal rendering helpers (CJK-aware column widths).""" + +import unicodedata + + +def display_width(text): + width = 0 + for char in str(text): + if unicodedata.east_asian_width(char) in ("W", "F"): + width += 2 + else: + width += 1 + return width + + +def clip(text, limit=60): + text = "" if text is None else str(text) + text = text.replace("\n", " ").replace("\r", " ").replace("\t", " ") + if len(text) <= limit: + return text + return text[: max(0, limit - 1)] + "…" + + +def table(headers, rows, align=None): + """Render a plain-text table. *align* is a list of ``l``/``r`` per column.""" + rows = [["" if cell is None else str(cell) for cell in row] for row in rows] + if align is None: + align = ["l"] * len(headers) + widths = [display_width(header) for header in headers] + for row in rows: + for index, cell in enumerate(row[: len(widths)]): + widths[index] = max(widths[index], display_width(cell)) + + def render(cells): + parts = [] + for index, cell in enumerate(cells): + padding = widths[index] - display_width(cell) + if align[index] == "r": + parts.append(" " * padding + cell) + else: + parts.append(cell + " " * padding) + return " ".join(parts).rstrip() + + lines = [render(headers), " ".join("-" * width for width in widths)] + lines.extend(render(row) for row in rows) + return "\n".join(lines) + + +def human_bytes(size): + if not isinstance(size, (int, float)): + return str(size) + value = float(size) + for unit in ("B", "KiB", "MiB", "GiB", "TiB"): + if value < 1024 or unit == "TiB": + if unit == "B": + return "%d B" % value + return "%.1f %s" % (value, unit) + value /= 1024.0 + return str(size) + + +def percent(part, total): + if not total: + return "-" + return "%.1f%%" % (100.0 * part / total) + + +def kv_block(pairs): + """Render aligned ``key: value`` lines.""" + if not pairs: + return "" + width = max(display_width(key) for key, _ in pairs) + lines = [] + for key, value in pairs: + padding = " " * (width - display_width(key)) + lines.append("%s%s : %s" % (key, padding, "" if value is None else value)) + return "\n".join(lines) diff --git a/graylog_ops/watch.py b/graylog_ops/watch.py new file mode 100644 index 0000000..3aecfb9 --- /dev/null +++ b/graylog_ops/watch.py @@ -0,0 +1,2768 @@ +"""Continuous malicious-IP detection, persistent list, local web dashboard. + +Design +------ +* Every ``interval`` seconds the detector runs a **fixed set of terms queries** + over the last ``window`` seconds, so the load on Elasticsearch does not grow + with the number of IPs. +* Rules were derived from an 8 hour exploration of the F5 access logs + (2026-10-17). The four observed patterns: + + 1. 漏洞探测 -- low volume, hits /actuator, /.env, /wp-login, /druid, /nacos, + /swagger, /ISAPI, /console ... (very high signal, few requests) + 2. 登录爆破 -- repeated 401/403 on /login, /passport, /captcha endpoints + 3. 爬虫/机器人 -- known bot UA (DotBot) + high 404 ratio + many content URIs + 4. 自动化客户端 -- one single business API at a steady rate (a broken token, + not an attacker) -> downgraded, never bans + +* Every candidate is persisted in SQLite (``data/ips.db``) so the list survives + restarts, and the same process serves a small read-only dashboard on + ``http://127.0.0.1:8088``. +""" + +import csv +import html +import io +import ipaddress +import re +import json +import os +import sqlite3 +import threading +import time +from datetime import datetime, timezone +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from urllib.parse import parse_qs, urlparse + +from . import detect +from .client import normalize_histogram, normalize_terms +from .http import GraylogError +from .notify import load_notifier, notifier_from_env +from .behavior import BehaviorAnalyzer, notify_fused, RULE_TITLES +from .asn import AsnDB + +# --------------------------------------------------------------------- rules +# Queries run against Graylog's query_string. Field names are substituted from +# the profile (client_ip / status / uri / ua). Slashes must be escaped: an +# unescaped /wp-login* would be parsed as a Lucene regex. +PROBE_RULES = ("probe_java", "probe_config", "probe_cms", "probe_iot", + "probe_devops", "probe_webshell", "exploit_payload") + +RULE_DEFS = ( + { + "name": "probe_java", + "title": "Spring/中间件探测", + "weight": 60.0, + "saturate": 2.0, + "query": lambda f: ( + "%s:(\\/actuator* OR \\/druid* OR \\/nacos* OR \\/swagger* " + "OR \\/v2\\/api-docs* OR \\/v3\\/api-docs* OR \\/openapi* OR \\/jmx* " + "OR \\/heapdump* OR \\/threaddump* OR \\/console* OR \\/manager\\/html* " + "OR \\/solr* OR \\/struts* OR \\/weblogic* OR \\/log4j*)" + ) % f["uri"], + }, + { + "name": "probe_config", + "title": "配置/凭证探测", + "weight": 60.0, + "saturate": 2.0, + "query": lambda f: ( + "%s:(\\/.env* OR \\/.git* OR \\/.svn* OR \\/.hg* OR \\/.aws* OR \\/.ssh* " + "OR \\/.kube* OR \\/.docker* OR \\/appsettings* OR \\/web.config " + "OR \\/WEB-INF* OR \\/META-INF* OR \\/backup* OR \\/dump* " + "OR \\/db.sql* OR \\/.DS_Store)" + ) % f["uri"], + }, + { + "name": "probe_cms", + "title": "CMS 探测", + "weight": 60.0, + "saturate": 2.0, + "query": lambda f: ( + "%s:(\\/wp-* OR \\/xmlrpc.php OR \\/phpmyadmin* OR \\/pma* " + "OR \\/administrator* OR \\/joomla* OR \\/drupal* OR \\/thinkphp* " + "OR \\/uc_server* OR \\/dede* OR \\/index.php?s=*)" + ) % f["uri"], + }, + { + "name": "probe_iot", + "title": "IoT/路由器探测", + "weight": 60.0, + "saturate": 2.0, + "query": lambda f: ( + "%s:(\\/ISAPI* OR \\/LAPI* OR \\/HNAP1* OR \\/boaform* OR \\/GponForm* " + "OR \\/cgi-bin\\/luci* OR \\/goform* OR \\/setup.cgi* OR \\/login.cgi*)" + ) % f["uri"], + }, + { + "name": "probe_devops", + "title": "DevOps/监控探测", + "weight": 60.0, + "saturate": 2.0, + "query": lambda f: ( + "%s:(\\/jenkins* OR \\/grafana* OR \\/kibana* OR \\/zabbix* " + "OR \\/prometheus* OR \\/consul* OR \\/eureka* OR \\/harbor* " + "OR \\/nexus* OR \\/sonarqube* OR \\/gitlab* OR \\/portainer*)" + ) % f["uri"], + }, + { + "name": "probe_webshell", + "title": "WebShell/上传探测", + "weight": 60.0, + "saturate": 2.0, + "query": lambda f: ( + "%s:(\\/shell.php OR \\/shell.asp OR \\/shell.jsp OR \\/cmd.php " + "OR \\/cmd.asp OR \\/webshell* OR \\/phpinfo* OR \\/dbadmin* " + "OR \\/eval-stdin* OR \\/vendor\\/phpunit*)" + ) % f["uri"], + }, + { + # Regex: only over 1 hour max, a 25h regex aggregation kills the ES node. + "name": "exploit_payload", + "title": "注入/穿越载荷", + "weight": 65.0, + "saturate": 2.0, + "window_cap": 3600, + "query": lambda f: ( + "%s:/.*(etc\\/passwd|proc\\/self|jndi:|union.*select|information_schema" + "|sleep\\(|benchmark\\(|\\.\\.%%2f|%%2e%%2e|%%252f|base64_decode|xp_cmdshell).*/" + ) % f["uri"], + }, + { + "name": "login_attack", + "title": "登录失败尝试", + "weight": 55.0, + "saturate": 8.0, + "query": lambda f: ( + "(%s:(\\/login* OR \\/passport* OR \\/captcha* OR \\/jsxy\\/api\\/login*)) " + "AND %s:(401 OR 403)" + ) % (f["uri"], f["status"]), + }, + { + "name": "login_burst", + "title": "登录接口高频", + "weight": 15.0, + "saturate": 2000.0, + "query": lambda f: ( + "%s:(\\/login* OR \\/passport* OR \\/captcha* OR \\/jsxy\\/api\\/login*)" + ) % f["uri"], + }, + { + "name": "bad_ua", + "title": "扫描器 UA", + "weight": 55.0, + "saturate": 1.0, + "query": lambda f: ( + "%s:(sqlmap* OR nikto* OR nuclei* OR nmap* OR masscan* OR zgrab* " + "OR wpscan* OR gobuster* OR dirsearch* OR dirbuster* OR ffuf* " + "OR feroxbuster* OR wfuzz* OR joomscan* OR droopescan* OR whatweb* " + "OR xray* OR vulmap* OR afrog* OR hydra*)" + ) % f["ua"], + }, + { + "name": "bot_ua", + "title": "爬虫 UA", + "weight": 50.0, + "saturate": 20.0, + "query": lambda f: ( + '%s:"Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)"' + ) % f["ua"], + }, + { + "name": "not_found", + "title": "404", + "weight": 0.0, # scored via the 404 ratio, not the raw count + "saturate": 1.0, + "query": lambda f: "%s:404" % f["status"], + }, +) + +DEFAULT_FIELDS = { + "client_ip": "client_ip", + "status": "status", + "uri": "uri", + "ua": "ua", + "method": "method", + "host": "host", + "server_ip": "server_ip", + "bigip": "bigip", +} + +AUTH_PATH_HINTS = ("login", "signin", "auth", "token", "captcha", "oauth", "password") + +# How far back to look when dating the first attack of an IP. +BACKFILL_WINDOW = 129600 # ~36h, as far back as the index retention allows + +TIER_MALICIOUS = "malicious" +TIER_SUSPICIOUS = "suspicious" +TIER_AUTOMATION = "automation" +TIER_OBSERVE = "observe" + +TIER_LABELS = { + TIER_MALICIOUS: "恶意", + TIER_SUSPICIOUS: "可疑", + TIER_AUTOMATION: "自动化", + TIER_OBSERVE: "观察", +} + + +# ------------------------------------------------------------------ classify +RATE_RULES = ("login_attack", "login_burst") + + +def classify(total, hits, window_scale=1.0): + """Pure scoring function: request counts per rule -> (tier, score, reasons). + + ``window_scale`` normalises rate-like rules to a 15 minute baseline, so an + 8 hour analysis does not turn a student who mistyped a password six times + into a "suspicious IP". Behavioural rules (probing / bad UA / crawler) are + never scaled: any probing is bad regardless of window length. + """ + scale = max(1.0, float(window_scale)) + score = 0.0 + reasons = [] + + def add(rule, title, count): + nonlocal score + if count <= 0: + return + saturate = rule["saturate"] + if rule["name"] in RATE_RULES: + saturate *= scale + points = rule["weight"] * min(1.0, count / saturate) + score += points + reasons.append("%s %d 次" % (title, count)) + + for rule in RULE_DEFS: + if rule["weight"] <= 0: + continue + add(rule, rule["title"], hits.get(rule["name"], 0)) + + not_found = hits.get("not_found", 0) + if not_found and total: + ratio = not_found / float(total) + if ratio >= 0.5 and not_found >= 20: + score += 25.0 * min(1.0, ratio / 0.8) + reasons.append("404 占比 %.0f%%（%d/%d）" % (ratio * 100, not_found, total)) + + score = min(100.0, round(score, 1)) + if score >= 60: + tier = TIER_MALICIOUS + elif score >= 30: + tier = TIER_SUSPICIOUS + elif score > 0: + tier = TIER_OBSERVE + else: + tier = TIER_OBSERVE + return tier, score, reasons + + +def is_public(text): + try: + return not detect.is_internal(detect.as_ip(text)) + except (TypeError, ValueError): + return False + + +# --------------------------------------------------------------------- store +class Store: + def __init__(self, path): + directory = os.path.dirname(os.path.abspath(path)) + if directory: + os.makedirs(directory, exist_ok=True) + self.path = path + self.lock = threading.Lock() + self.conn = sqlite3.connect(path, check_same_thread=False) + try: + self.conn.execute("PRAGMA journal_mode=WAL") + self.conn.execute("PRAGMA busy_timeout=5000") + except sqlite3.Error: + pass + self.conn.row_factory = sqlite3.Row + self._init() + + def _init(self): + with self.conn: + self.conn.execute( + """CREATE TABLE IF NOT EXISTS ip_stats ( + ip TEXT PRIMARY KEY, + tier TEXT NOT NULL, + score REAL NOT NULL, + total INTEGER NOT NULL DEFAULT 0, + rules TEXT NOT NULL DEFAULT '{}', + reasons TEXT NOT NULL DEFAULT '[]', + first_seen TEXT, + last_seen TEXT, + risk TEXT, + window INTEGER, + evidence TEXT, + evidence_at TEXT, + ever_malicious INTEGER NOT NULL DEFAULT 0, + last_malicious_at TEXT, + hits INTEGER NOT NULL DEFAULT 1, + updated_at TEXT + )""" + ) + for column in ("risk TEXT", "window INTEGER", "evidence TEXT", + "evidence_at TEXT", "ever_malicious INTEGER DEFAULT 0", + "last_malicious_at TEXT", "country_code TEXT", + "country_name TEXT", "region TEXT", "city TEXT", + "asn TEXT", "asn_org TEXT", "asn_brand TEXT", + "asn_type TEXT", "asn_icon TEXT"): + try: + self.conn.execute("ALTER TABLE ip_stats ADD COLUMN %s" % column) + except sqlite3.OperationalError: + pass # column already exists + try: + self.conn.execute( + + """UPDATE ip_stats SET ever_malicious = 1, + last_malicious_at = COALESCE( + last_malicious_at, + (SELECT MAX(e.ts) FROM events e + WHERE e.ip = ip_stats.ip AND e.tier = 'malicious'), + last_seen) + WHERE COALESCE(ever_malicious, 0) = 0 AND ( + tier = 'malicious' OR EXISTS ( + SELECT 1 FROM events e + WHERE e.ip = ip_stats.ip + AND (e.tier = 'malicious' + OR e.detail LIKE '%malicious%')))""" + ) + except sqlite3.OperationalError: + pass # events table may not exist yet on a fresh db + self.conn.execute( + """CREATE TABLE IF NOT EXISTS behavior_profiles ( + ip TEXT PRIMARY KEY, + score REAL, verdict TEXT, signals TEXT, reason_text TEXT, + share_note TEXT, behavior_type TEXT, + asn TEXT, asn_org TEXT, shared_type TEXT, + asn_brand TEXT, asn_type TEXT, asn_icon TEXT, + total INTEGER, pct4 REAL, uri_card INTEGER, + uri_templates INTEGER, ua_card INTEGER, host_card INTEGER, + coverage REAL, peak_share REAL, top_uri TEXT, top_ua TEXT, + last_ts TEXT, updated_at TEXT, + country_code TEXT, country_name TEXT, region TEXT, city TEXT + )""" + ) + for column in ("asn_brand TEXT", "asn_type TEXT", "asn_icon TEXT"): + try: + self.conn.execute( + "ALTER TABLE behavior_profiles ADD COLUMN %s" % column) + except sqlite3.OperationalError: + pass + self.conn.execute( + """CREATE TABLE IF NOT EXISTS ip_fused ( + ip TEXT PRIMARY KEY, + sources TEXT NOT NULL, + feature_score REAL, behavior_score REAL, + first_banned_at TEXT, last_activity_at TEXT, + last_pushed_at TEXT, push_count INTEGER NOT NULL DEFAULT 0, + fused_reason TEXT, updated_at TEXT + )""" + ) + self.conn.execute( + """CREATE TABLE IF NOT EXISTS events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + ts TEXT, ip TEXT, tier TEXT, score REAL, detail TEXT + )""" + ) + + def upsert(self, record, now): + with self.lock, self.conn: + row = self.conn.execute( + "SELECT tier, score, ever_malicious, last_malicious_at, " + "last_seen, rules, total FROM ip_stats WHERE ip = ?", + (record["ip"],), + ).fetchone() + malicious = record["tier"] == TIER_MALICIOUS + geo = record.get("geo") or {} + info = record.get("asn") or {} + if row is None: + last_seen = record.get("last_seen") or now + last_malicious_at = record.get("last_malicious_at") or ( + last_seen if malicious else None) + self.conn.execute( + """INSERT INTO ip_stats + (ip, tier, score, total, rules, reasons, + first_seen, last_seen, risk, window, hits, updated_at, + ever_malicious, last_malicious_at, + country_code, country_name, region, city, + asn, asn_org, asn_brand, asn_type, asn_icon) + VALUES (?,?,?,?,?,?,?,?,?,?,1,?,?,?,?,?,?,?,?,?,?,?,?)""", + (record["ip"], record["tier"], record["score"], record["total"], + json.dumps(record["rules"], ensure_ascii=False), + json.dumps(record["reasons"], ensure_ascii=False), + record.get("first_seen") or now, last_seen, + record.get("risk"), record.get("window"), now, + 1 if malicious else 0, last_malicious_at, + geo.get("country_code"), geo.get("country_name"), + geo.get("region"), geo.get("city"), + info.get("asn"), info.get("org"), info.get("brand"), + info.get("type"), info.get("icon")), + ) + self._event(now, record, "new") + return "new" + else: + ever = 1 if (malicious or row["ever_malicious"]) else 0 + # Timestamps only move when the detector verified new evidence, + # otherwise a stale window would keep refreshing them to "now". + last_seen = record.get("last_seen") or row["last_seen"] or now + if record.get("last_malicious_at"): + last_malicious_at = record["last_malicious_at"] + elif malicious and row["tier"] != TIER_MALICIOUS: + last_malicious_at = last_seen + else: + last_malicious_at = row["last_malicious_at"] + self.conn.execute( + """UPDATE ip_stats + SET tier=?, score=?, total=?, rules=?, reasons=?, + last_seen=?, risk=?, window=?, hits=hits+1, updated_at=?, + ever_malicious=?, last_malicious_at=?, + country_code=COALESCE(?, country_code), + country_name=COALESCE(?, country_name), + region=COALESCE(?, region), + city=COALESCE(?, city), + asn=COALESCE(?, asn), asn_org=COALESCE(?, asn_org), + asn_brand=COALESCE(?, asn_brand), + asn_type=COALESCE(?, asn_type), + asn_icon=COALESCE(?, asn_icon) + WHERE ip=?""", + (record["tier"], record["score"], record["total"], + json.dumps(record["rules"], ensure_ascii=False), + json.dumps(record["reasons"], ensure_ascii=False), + last_seen, record.get("risk"), record.get("window"), now, + ever, last_malicious_at, + geo.get("country_code"), geo.get("country_name"), + geo.get("region"), geo.get("city"), + info.get("asn"), info.get("org"), info.get("brand"), + info.get("type"), info.get("icon"), record["ip"]), + ) + if row["tier"] != record["tier"]: + self._event(now, record, "%s->%s" % (row["tier"], record["tier"])) + if malicious and row["tier"] != TIER_MALICIOUS: + return "upgraded" + return "unchanged" if row["tier"] == record["tier"] else "changed" + + def _event(self, now, record, detail): + self.conn.execute( + "INSERT INTO events (ts, ip, tier, score, detail) VALUES (?,?,?,?,?)", + (now, record["ip"], record["tier"], record["score"], detail), + ) + + SORT_COLUMNS = {"first_seen": "first_seen", + "last_malicious": "last_malicious_at", + "score": "score", + "total": "total"} + + def list(self, limit=500, tier=None, scope=None, sort="first_seen", order="desc"): + query = ("SELECT ip, tier, score, total, rules, reasons, first_seen, " + "last_seen, risk, window, hits, updated_at, evidence_at, " + "ever_malicious, last_malicious_at, country_code, country_name, " + "region, city, asn, asn_org, asn_brand, asn_type, asn_icon " + "FROM ip_stats") + args = [] + where = [] + if tier: + where.append("tier = ?") + args.append(tier) + if scope == "core": + where.append("ever_malicious = 1") + if where: + query += " WHERE " + " AND ".join(where) + column = self.SORT_COLUMNS.get(sort, "first_seen") + direction = "DESC" if str(order).lower() == "desc" else "ASC" + query += (" ORDER BY (%s IS NULL) ASC, %s %s, " + "COALESCE(last_malicious_at, last_seen) DESC, ip ASC LIMIT ?" + % (column, column, direction)) + args.append(limit) + with self.lock: + rows = self.conn.execute(query, args).fetchall() + result = [] + for row in rows: + item = dict(row) + item["rules"] = json.loads(item["rules"] or "{}") + item["reasons"] = json.loads(item["reasons"] or "[]") + result.append(item) + return result + + def get(self, ip): + with self.lock: + row = self.conn.execute("SELECT * FROM ip_stats WHERE ip = ?", (ip,)).fetchone() + if row is None: + return None + item = dict(row) + item["rules"] = json.loads(item["rules"] or "{}") + item["reasons"] = json.loads(item["reasons"] or "[]") + return item + + # ---------------------------------------------------------- behaviour + def replace_behavior_profiles(self, items): + stamp = now_iso() + with self.lock, self.conn: + self.conn.execute("DELETE FROM behavior_profiles") + for item in items: + self.conn.execute( + """INSERT OR REPLACE INTO behavior_profiles + (ip, score, verdict, signals, reason_text, share_note, + behavior_type, asn, asn_org, shared_type, asn_brand, + asn_type, asn_icon, total, pct4, uri_card, uri_templates, + ua_card, host_card, coverage, peak_share, top_uri, top_ua, + last_ts, updated_at, country_code, country_name, region, + city) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""", + (item["ip"], item.get("score"), item.get("verdict"), + json.dumps(item.get("signals") or [], ensure_ascii=False), + item.get("reason_text"), item.get("share_note"), + item.get("behavior_type"), item.get("asn"), + item.get("asn_org"), item.get("shared_type"), + item.get("asn_brand"), item.get("asn_type"), + item.get("asn_icon"), + int(item.get("total") or 0), item.get("pct4"), + int(item.get("uri_card") or 0), + int(item.get("uri_templates") or 0), + int(item.get("ua_card") or 0), + int(item.get("host_card") or 0), item.get("coverage"), + item.get("peak_share"), item.get("top_uri"), + item.get("top_ua"), stamp, stamp, + item.get("country_code"), item.get("country_name"), + item.get("region"), item.get("city")), + ) + + def behavior_profile(self, ip): + with self.lock: + row = self.conn.execute( + "SELECT * FROM behavior_profiles WHERE ip = ?", (ip,)).fetchone() + if row is None: + return None + item = dict(row) + try: + item["signals"] = json.loads(item.get("signals") or "[]") + except ValueError: + item["signals"] = [] + return item + + def fused_row(self, ip): + with self.lock: + row = self.conn.execute( + "SELECT * FROM ip_fused WHERE ip = ?", (ip,)).fetchone() + return dict(row) if row else None + + def fused_items(self, limit=500): + with self.lock: + rows = self.conn.execute( + """SELECT f.*, s.tier AS tier, s.total AS total, s.rules AS s_rules, + b.total AS b_total, + s.asn AS s_asn, s.asn_org AS s_asn_org, + s.asn_brand AS s_asn_brand, s.asn_type AS s_asn_type, + s.asn_icon AS s_asn_icon, + b.asn AS b_asn, b.asn_org AS b_asn_org, + b.asn_brand AS b_asn_brand, b.asn_type AS b_asn_type, + b.asn_icon AS b_asn_icon, + s.country_code AS s_cc, s.country_name AS s_cn, + s.region AS s_region, s.city AS s_city, + b.country_code AS b_cc, b.country_name AS b_cn, + b.region AS b_region, b.city AS b_city + FROM ip_fused f + LEFT JOIN ip_stats s ON s.ip = f.ip + LEFT JOIN behavior_profiles b ON b.ip = f.ip + ORDER BY f.first_banned_at DESC, f.ip ASC + LIMIT ?""", (int(limit),)).fetchall() + items = [] + for row in rows: + item = dict(row) + item["country_code"] = item.get("s_cc") or item.get("b_cc") or "" + item["country_name"] = item.get("s_cn") or item.get("b_cn") or "" + item["region"] = item.get("s_region") or item.get("b_region") or "" + item["city"] = item.get("s_city") or item.get("b_city") or "" + item["asn"] = item.get("s_asn") or item.get("b_asn") or "" + item["asn_org"] = item.get("s_asn_org") or item.get("b_asn_org") or "" + item["asn_brand"] = item.get("s_asn_brand") or item.get("b_asn_brand") or "" + item["asn_type"] = item.get("s_asn_type") or item.get("b_asn_type") or "" + item["asn_icon"] = item.get("s_asn_icon") or item.get("b_asn_icon") or "" + item["total"] = item.get("total") or item.get("b_total") or 0 + items.append(item) + return items + + def _fused_reason(self, feature_rules, behavior_reason): + parts = [] + if feature_rules: + try: + rules = json.loads(feature_rules or "{}") + except ValueError: + rules = {} + if rules: + ranked = sorted(rules.items(), key=lambda kv: -kv[1])[:3] + parts.append("特征命中：" + "、".join( + "%s %d 次" % (RULE_TITLES.get(name, name), count) + for name, count in ranked)) + if behavior_reason: + parts.append("行为表现：" + behavior_reason) + return "；".join(parts) + + def sync_fused(self, now=None): + """Merge feature + behaviour verdicts into the fused block list. + + Rule: feature-confirmed IPs always stay (option A, historical list); + behaviour adds IPs only when its verdict is "建议封禁" (shared/NAT + networks never enter the block list). first_banned_at is pinned so an + IP only reaches the top of the list once. + """ + now = now or now_iso() + with self.lock, self.conn: + feature_rows = self.conn.execute( + "SELECT ip, score, rules, first_seen, last_seen, " + "last_malicious_at, country_code, country_name, region, city " + "FROM ip_stats WHERE ever_malicious = 1").fetchall() + behavior_rows = self.conn.execute( + "SELECT * FROM behavior_profiles").fetchall() + existing = {row["ip"]: row for row in self.conn.execute( + "SELECT * FROM ip_fused").fetchall()} + initial = not existing + behavior = {row["ip"]: row for row in behavior_rows} + wanted = {} + + def iso(value): + return value or now + + for row in feature_rows: + ip = row["ip"] + beh = behavior.get(ip) + beh_ban = bool(beh and beh["verdict"] == "建议封禁") + prev = existing.get(ip) + first = (prev["first_banned_at"] if prev + else iso(row["first_seen"] or row["last_malicious_at"])) + activity = max(filter(None, [row["last_seen"], + beh["last_ts"] if beh else None]), + default=now) + wanted[ip] = { + "sources": "both" if beh_ban else "feature", + "feature_score": row["score"], + "behavior_score": beh["score"] if beh else None, + "first_banned_at": first, + "last_activity_at": iso(activity), + "fused_reason": self._fused_reason( + row["rules"], beh["reason_text"] if beh else None), + } + + for ip, beh in behavior.items(): + if beh["verdict"] != "建议封禁" or ip in wanted: + continue + prev = existing.get(ip) + wanted[ip] = { + "sources": "behavior", + "feature_score": None, + "behavior_score": beh["score"], + "first_banned_at": prev["first_banned_at"] if prev else now, + "last_activity_at": iso(beh["last_ts"]), + "fused_reason": "行为表现：" + (beh["reason_text"] or ""), + } + + for ip, data in wanted.items(): + prev = existing.get(ip) + if prev: + self.conn.execute( + """UPDATE ip_fused + SET sources=?, feature_score=?, behavior_score=?, + last_activity_at=?, fused_reason=?, updated_at=? + WHERE ip=?""", + (data["sources"], data["feature_score"], + data["behavior_score"], data["last_activity_at"], + data["fused_reason"], now, ip)) + else: + self.conn.execute( + """INSERT INTO ip_fused + (ip, sources, feature_score, behavior_score, + first_banned_at, last_activity_at, last_pushed_at, + push_count, fused_reason, updated_at) + VALUES (?,?,?,?,?,?,?,0,?,?)""", + (ip, data["sources"], data["feature_score"], + data["behavior_score"], data["first_banned_at"], + data["last_activity_at"], now if initial else None, + data["fused_reason"], now)) + + stale = [ip for ip in existing if ip not in wanted] + for ip in stale: + self.conn.execute("DELETE FROM ip_fused WHERE ip = ?", (ip,)) + return len(wanted) + + def due_pushes(self, active_hours=24, cooldown_hours=24): + now = time.time() + active_cut = time.strftime("%Y-%m-%dT%H:%M:%SZ", + time.gmtime(now - active_hours * 3600)) + push_cut = time.strftime("%Y-%m-%dT%H:%M:%SZ", + time.gmtime(now - cooldown_hours * 3600)) + with self.lock: + rows = self.conn.execute( + """SELECT f.*, s.country_name AS s_cn, s.region AS s_region, + s.total AS s_total, s.asn AS s_asn, + s.asn_brand AS s_asn_brand, s.asn_type AS s_asn_type, + s.asn_icon AS s_asn_icon, + b.country_name AS b_cn, b.region AS b_region, + b.total AS b_total, b.asn AS b_asn, + b.asn_brand AS b_asn_brand, b.asn_type AS b_asn_type, + b.asn_icon AS b_asn_icon + FROM ip_fused f + LEFT JOIN ip_stats s ON s.ip = f.ip + LEFT JOIN behavior_profiles b ON b.ip = f.ip + WHERE f.last_activity_at >= ? + AND (f.last_pushed_at IS NULL OR f.last_pushed_at <= ?) + ORDER BY f.first_banned_at DESC LIMIT 50""", + (active_cut, push_cut)).fetchall() + items = [] + for row in rows: + item = dict(row) + item["country_name"] = item.get("s_cn") or item.get("b_cn") or "" + item["region"] = item.get("s_region") or item.get("b_region") or "" + item["total"] = item.get("s_total") or item.get("b_total") or 0 + item["asn"] = item.get("s_asn") or item.get("b_asn") or "" + item["asn_brand"] = (item.get("s_asn_brand") + or item.get("b_asn_brand") or "") + item["asn_type"] = (item.get("s_asn_type") + or item.get("b_asn_type") or "") + item["asn_icon"] = (item.get("s_asn_icon") + or item.get("b_asn_icon") or "") + items.append(item) + return items + + def mark_pushed(self, ips, stamp): + with self.lock, self.conn: + for ip in ips: + self.conn.execute( + "UPDATE ip_fused SET last_pushed_at=?, push_count=push_count+1 " + "WHERE ip=?", (stamp, ip)) + + def stats(self): + with self.lock: + rows = self.conn.execute( + "SELECT tier, COUNT(*) AS n FROM ip_stats GROUP BY tier" + ).fetchall() + core = self.conn.execute( + "SELECT COUNT(*) AS n FROM ip_stats WHERE ever_malicious = 1" + ).fetchone()["n"] + permanent = self.conn.execute( + "SELECT COUNT(*) AS n FROM ip_stats WHERE ever_malicious = 1" + ).fetchone()["n"] + events = self.conn.execute("SELECT COUNT(*) AS n FROM events").fetchone()["n"] + updated = self.conn.execute( + "SELECT MAX(updated_at) AS ts FROM ip_stats" + ).fetchone()["ts"] + counts = {row["tier"]: row["n"] for row in rows} + with self.lock: + fused_total = self.conn.execute( + "SELECT COUNT(*) AS n FROM ip_fused").fetchone()["n"] + fused_both = self.conn.execute( + "SELECT COUNT(*) AS n FROM ip_fused WHERE sources='both'" + ).fetchone()["n"] + fused_behavior = self.conn.execute( + "SELECT COUNT(*) AS n FROM ip_fused WHERE sources='behavior'" + ).fetchone()["n"] + fused_new = self.conn.execute( + "SELECT COUNT(*) AS n FROM ip_fused WHERE first_banned_at >= ?", + (time.strftime("%Y-%m-%dT%H:%M:%SZ", + time.gmtime(time.time() - 86400)),) + ).fetchone()["n"] + pushed = self.conn.execute( + "SELECT COUNT(*) AS n FROM ip_fused WHERE last_pushed_at IS NOT NULL" + ).fetchone()["n"] + return { + "total": sum(counts.values()), + "core": core, + "permanent": permanent, + "by_tier": counts, + "events": events, + "updated_at": updated, + "fused_total": fused_total, + "fused_new_24h": fused_new, + "fused_both": fused_both, + "fused_behavior_only": fused_behavior, + "fused_pushed": pushed, + } + + def update_first_seen(self, ip, timestamp): + with self.lock, self.conn: + cursor = self.conn.execute( + "UPDATE ip_stats SET first_seen = ? " + "WHERE ip = ? AND (first_seen IS NULL OR first_seen > ?)", + (timestamp, ip, timestamp), + ) + return cursor.rowcount + + def update_asn(self, ip, info): + with self.lock, self.conn: + self.conn.execute( + "UPDATE ip_stats SET asn=?, asn_org=?, asn_brand=?, asn_type=?, " + "asn_icon=? WHERE ip=?", + (info.get("asn"), info.get("org"), info.get("brand"), + info.get("type"), info.get("icon"), ip)) + + def update_geo(self, ip, geo): + with self.lock, self.conn: + self.conn.execute( + "UPDATE ip_stats SET country_code=?, country_name=?, region=?, " + "city=? WHERE ip=?", (geo.get("country_code"), geo.get("country_name"), + geo.get("region"), geo.get("city"), ip)) + + def update_times(self, ip, last_seen, last_malicious_at=None): + with self.lock, self.conn: + self.conn.execute( + "UPDATE ip_stats SET last_seen = ?, " + "last_malicious_at = COALESCE(?, last_malicious_at) WHERE ip = ?", + (last_seen, last_malicious_at, ip), + ) + + def get_evidence(self, ip): + with self.lock: + row = self.conn.execute( + "SELECT evidence FROM ip_stats WHERE ip = ?", (ip,) + ).fetchone() + return row["evidence"] if row else None + + def set_evidence(self, ip, payload, now): + with self.lock, self.conn: + self.conn.execute( + "UPDATE ip_stats SET evidence = ?, evidence_at = ? WHERE ip = ?", + (json.dumps(payload, ensure_ascii=False), now, ip), + ) + + def forget(self, older_than_seconds, now_epoch): + cutoff = now_epoch - older_than_seconds + with self.lock, self.conn: + self.conn.execute( + "DELETE FROM ip_stats WHERE last_seen < ? " + "AND COALESCE(ever_malicious, 0) = 0", + (_iso(datetime.fromtimestamp(cutoff, timezone.utc)),), + ) + + def close(self): + self.conn.close() + + +def _iso(moment): + return moment.strftime("%Y-%m-%dT%H:%M:%SZ") + + +def now_iso(): + return _iso(datetime.now(timezone.utc)) + + +# ------------------------------------------------------------------ detector +class Detector: + def __init__(self, client, store, fields, window=900, size=200, + min_requests=3, verify_limit=8, forget=7200, allowlist=None, + filter_str=None, min_observe_score=10.0, warm_limit=5, geo_db=None, + asn_db=None): + self.client = client + self.store = store + self.fields = fields + self.window = window + self.size = size + self.min_requests = min_requests + self.verify_limit = verify_limit + self.forget = forget + self.allowlist = allowlist + self.filter_str = filter_str + self.min_observe_score = float(min_observe_score) + self.warm_limit = int(warm_limit) + self.asn = None + if asn_db: + try: + self.asn = AsnDB(asn_db) + except (FileNotFoundError, sqlite3.Error): + self.asn = None + self.geo = None + if geo_db: + try: + from .geo import GeoIP + self.geo = GeoIP(geo_db) + except (FileNotFoundError, sqlite3.Error): + self.geo = None + self.rules = [ + {"name": rule["name"], "title": rule["title"], + "window_cap": rule.get("window_cap"), + "query": rule["query"](fields)} + for rule in RULE_DEFS + ] + self.last_run = None + self.last_success = None + self.last_error = None + self.runs = 0 + + def _terms(self, query, window_cap=None): + window = self.window + if window_cap: + window = min(window, window_cap) + data = self.client.terms( + field=self.fields["client_ip"], query=query, size=self.size, + range_seconds=window, filter_str=self.filter_str, + ) + return {row["term"]: int(row["count"] or 0) + for row in normalize_terms(data) + if row["term"] not in (None, "")} + + def _batch_totals(self, ips, chunk=40): + """One terms query can return totals for many IPs at once.""" + field = self.fields["client_ip"] + result = {} + for start in range(0, len(ips), chunk): + part = ips[start:start + chunk] + query = "%s:(%s)" % (field, " OR ".join('"%s"' % ip for ip in part)) + try: + result.update(self._terms(query)) + except GraylogError as exc: + self.last_error = "batch totals: %s" % exc + return result + + def collect(self): + hits = {} + for rule in self.rules: + try: + hits[rule["name"]] = self._terms( + rule["query"], window_cap=rule.get("window_cap")) + except GraylogError as exc: + hits[rule["name"]] = {} + self.last_error = "%s: %s" % (rule["name"], exc) + + # Low-volume scanners never make the "top requests" list, so the + # candidate pool must be the union of every rule hit plus the volume + # top. Totals for the missing ones are fetched in batches. + totals = self._terms("*") + candidates = set(totals) + for mapping in hits.values(): + candidates.update(mapping) + missing = [ip for ip in candidates if ip not in totals] + if len(missing) > 1200: # keep the load bounded + missing = missing[:1200] + if missing: + totals.update(self._batch_totals(missing)) + return totals, hits + + def _verify(self, records): + """Downgrade single-endpoint automation (broken token, not an attacker).""" + limit = self.verify_limit + for record in records[:limit]: + if record["tier"] not in (TIER_MALICIOUS, TIER_SUSPICIOUS): + continue + try: + data = self.client.terms( + field=self.fields["uri"], + query='%s:"%s"' % (self.fields["client_ip"], record["ip"]), + size=8, range_seconds=self.window, filter_str=self.filter_str, + ) + except GraylogError as exc: + record["reasons"].append("核验失败: %s" % exc.message) + continue + rows = [(row["term"], row["count"]) for row in normalize_terms(data) + if row["term"] not in (None, "")] + total = sum(count for _, count in rows) + if not rows or not total: + continue + top_uri, top_count = rows[0] + ratio = top_count / float(total) + login_like = any(hint in str(top_uri).lower() for hint in AUTH_PATH_HINTS) + if ratio >= 0.9 and len(rows) <= 2 and not login_like: + record["tier"] = TIER_AUTOMATION + record["score"] = min(record["score"], 45.0) + record["rules"]["automation"] = { + "top_uri": top_uri, "ratio": round(ratio, 4)} + record["reasons"].append( + "流量集中在单一接口 %s（%.0f%%），疑似业务客户端" % (top_uri, ratio * 100)) + + def first_attack_at(self, ip, window=None): + """Earliest message for this IP inside the window (probe-only first).""" + if getattr(self, "client", None) is None: + return None + window = int(window or self.window) + base = '%s:"%s"' % (self.fields["client_ip"], ip) + probe = _probe_query(self, None, window) + queries = [] + if probe: + queries.append("(%s) AND %s" % (probe, base)) + queries.append(base) + for query in queries: + try: + messages = self.client.search_views_messages( + query=query, range_seconds=window, limit=1, + filter_str=self.filter_str, sort_order="ASC") + except GraylogError: + continue + if messages and messages[0].get("timestamp"): + return messages[0]["timestamp"] + return None + + def last_message_at(self, ip, window=None): + if getattr(self, "client", None) is None: + return None + window = int(window or self.window) + base = '%s:"%s"' % (self.fields["client_ip"], ip) + try: + messages = self.client.search_views_messages( + query=base, range_seconds=window, limit=1, + filter_str=self.filter_str, sort_order="DESC") + except GraylogError: + return None + return messages[0].get("timestamp") if messages else None + + def last_attack_at(self, ip, window=None): + if getattr(self, "client", None) is None: + return None + window = int(window or self.window) + base = '%s:"%s"' % (self.fields["client_ip"], ip) + probe = _probe_query(self, None, window) + if probe: + try: + messages = self.client.search_views_messages( + query="(%s) AND %s" % (probe, base), + range_seconds=window, limit=1, + filter_str=self.filter_str, sort_order="DESC") + except GraylogError: + messages = [] + if messages and messages[0].get("timestamp"): + return messages[0]["timestamp"] + return self.last_message_at(ip, window) + + def backfill_asn(self, limit=5000): + if self.asn is None: + return 0 + done = 0 + for row in self.store.list(limit=limit, scope=None): + if row.get("asn"): + continue + info = self.asn.lookup(row["ip"]) + if info: + self.store.update_asn(row["ip"], info) + done += 1 + return done + + def backfill_geo(self, limit=5000): + if self.geo is None: + return 0 + done = 0 + for row in self.store.list(limit=limit, scope=None): + if row.get("country_code"): + continue + geo = self.geo.lookup(row["ip"]) + if geo: + self.store.update_geo(row["ip"], geo) + done += 1 + return done + + def backfill_times(self, limit=500): + """Re-derive first-attack / last-attack / last-seen from the logs.""" + if getattr(self, "client", None) is None: + return 0 + rows = self.store.list(limit=20000, scope=None) + done = 0 + for row in rows: + if limit and done >= limit: + break + window = max(row.get("window") or self.window, BACKFILL_WINDOW) + first = self.first_attack_at(row["ip"], window) + last = self.last_message_at(row["ip"], window) + attack = (self.last_attack_at(row["ip"], window) + if row.get("ever_malicious") else None) + if first: + self.store.update_first_seen(row["ip"], first) + if last: + self.store.update_times(row["ip"], last, attack) + done += 1 + return done + + # backwards-compatible alias + backfill_first_seen = backfill_times + + def warm_snapshots(self, limit=10): + """Pre-generate compact evidence for listed malicious IPs.""" + if getattr(self, "client", None) is None: + return 0 + rows = self.store.list(limit=10000, tier=TIER_MALICIOUS) + done = 0 + for row in rows: + if limit and done >= limit: + break + if row.get("evidence_at"): + continue + try: + evidence = collect_evidence(self, row["ip"], row.get("rules"), + row.get("window") or self.window) + except Exception: + continue + if evidence.get("available"): + self.store.set_evidence(row["ip"], _compact_evidence(evidence), + now_iso()) + done += 1 + return done + + def run_once(self): + started = time.time() + totals, hits = self.collect() + records = [] + for ip, total in totals.items(): + if total < self.min_requests or not is_public(ip): + continue + if self.allowlist and self.allowlist(ip): + continue + rule_hits = {name: mapping.get(ip, 0) for name, mapping in hits.items()} + tier, score, reasons = classify( + total, rule_hits, window_scale=max(1.0, self.window / 900.0)) + + risk = None + if tier == TIER_OBSERVE and (not reasons or score < self.min_observe_score): + continue + records.append({ + "ip": ip, + "tier": tier, + "score": score, + "total": total, + "rules": {name: count for name, count in rule_hits.items() if count}, + "reasons": reasons, + "risk": risk, + "window": self.window, + }) + + records.sort(key=lambda row: (-row["score"], -row["total"])) + for record in records: + if self.geo is not None: + record["geo"] = self.geo.lookup(record["ip"]) + if self.asn is not None: + record["asn"] = self.asn.lookup(record["ip"]) + existing = self.store.get(record["ip"]) + if existing is None: + window = max(self.window, BACKFILL_WINDOW) + first = self.first_attack_at(record["ip"], window) + last = self.last_message_at(record["ip"], window) + if first: + record["first_seen"] = first + if last: + record["last_seen"] = last + if record["tier"] == TIER_MALICIOUS: + attack = self.last_attack_at(record["ip"], window) + if attack: + record["last_malicious_at"] = attack + elif last: + record["last_malicious_at"] = last + continue + changed = (existing.get("total") != record["total"] + or (existing.get("rules") or {}) != record["rules"]) + if not changed: + # Same evidence as last round: do not move any timestamp. + continue + last = self.last_message_at(record["ip"], self.window) + if last: + record["last_seen"] = last + if record["tier"] == TIER_MALICIOUS: + attack = self.last_attack_at(record["ip"], self.window) + if attack: + record["last_malicious_at"] = attack + elif last: + record["last_malicious_at"] = last + self._verify(records) + + stamp = now_iso() + # Notifications are sent only by the fused pipeline (notify_fused) so a + # new IP produces exactly one message, never one per detection source. + for record in records: + self.store.upsert(record, stamp) + self.store.forget(self.forget, time.time()) + + try: + self.warm_snapshots(int(getattr(self, "warm_limit", 0) or 0)) + except Exception: + pass + + self.last_run = stamp + self.last_success = time.time() + self.runs += 1 + return { + "window": self.window, + "request_seconds": round(time.time() - started, 2), + "candidates": len(records), + "malicious": sum(1 for row in records if row["tier"] == TIER_MALICIOUS), + "suspicious": sum(1 for row in records if row["tier"] == TIER_SUSPICIOUS), + "automation": sum(1 for row in records if row["tier"] == TIER_AUTOMATION), + } + + +def _csv_text(items): + buffer = io.StringIO() + writer = csv.writer(buffer) + writer.writerow(["ip", "tier", "score", "total", "first_seen", + "last_malicious_at", "risk", "reasons", + "country_code", "country_name", "region", "city", + "asn", "asn_org", "asn_brand", "asn_type"]) + for item in items: + writer.writerow([ + item.get("ip", ""), + item.get("tier", ""), + item.get("score", ""), + item.get("total", ""), + item.get("first_seen") or "", + item.get("last_malicious_at") or "", + item.get("risk") or "", + "; ".join(item.get("reasons") or []), + item.get("country_code") or "", + item.get("country_name") or "", + item.get("region") or "", + item.get("city") or "", + item.get("asn") or "", + item.get("asn_org") or "", + item.get("asn_brand") or "", + item.get("asn_type") or "", + ]) + # BOM so Excel opens the UTF-8 file without mojibake + return "\ufeff" + buffer.getvalue() + + +def export_blocklist(store, path): + """Write every IP ever convicted as malicious.""" + rows = store.list(limit=20000, scope="core", sort="first_seen", order="asc") + chosen = [row["ip"] for row in rows] + directory = os.path.dirname(os.path.abspath(path)) + if directory: + os.makedirs(directory, exist_ok=True) + with open(path, "w", encoding="utf-8") as handle: + handle.write("\n".join(chosen) + ("\n" if chosen else "")) + return chosen + + +# --------------------------------------------------------------------- web +EVIDENCE_CACHE = {} +EVIDENCE_TTL = 600 + +SAMPLE_KEYS = ("timestamp", "client_ip", "client_port", "server_ip", + "server_port", "host", "bigip", "vs", "method", "uri", "query", + "status", "ua", "referer", "req_len", "rt", + "client_ip_country_code", "client_ip_city_name", + "client_ip_geolocation", "event_type", "source", + "gl2_remote_ip", "message") + + +HEADER_FIELDS = {"method", "status", "uri", "host", "client_ip", "server_ip", + "client_port", "server_port", "vs", "bigip", "event_type", + "source", "gl2_remote_ip"} + + +def _clean_text(value, limit=280): + """Strip control chars / replacement chars that source logs sometimes carry.""" + out = [] + for char in str(value): + code = ord(char) + if char in "\n\r\t": + out.append(" ") + elif code < 32 or code == 127 or char == "\ufffd": + out.append("·") + else: + out.append(char) + text = re.sub(r"·{2,}", "·", "".join(out)) + return _clip(text, limit) + + +def _scrub_binary(text): + """Header-ish fields must be ASCII; a high non-ASCII ratio means garbage.""" + if not text: + return text + non_ascii = sum(1 for char in text if ord(char) > 127) + if non_ascii / max(1, len(text)) > 0.2: + return "[非 UTF-8 数据]" + return text + + +def _clip(value, limit=280): + text = str(value) + return text[:limit] + ("…" if len(text) > limit else "") + + +def _sample(message): + payload = {} + for key in SAMPLE_KEYS: + value = message.get(key) + if value in (None, "", "N/A"): + continue + text = _clean_text(value, 4000 if key == "message" else 600) + if key in HEADER_FIELDS: + text = _scrub_binary(text) + payload[key] = text + return payload + + +def _compact_evidence(evidence): + """Small summary persisted in SQLite: survives Graylog index rotation.""" + return { + "compact": True, + "window": evidence.get("window"), + "captured_at": now_iso(), + "uris": (evidence.get("uris") or [])[:10], + "statuses": (evidence.get("statuses") or [])[:6], + "methods": (evidence.get("methods") or [])[:6], + "hosts": (evidence.get("hosts") or [])[:6], + "server_ips": (evidence.get("server_ips") or [])[:6], + "vips": (evidence.get("vips") or [])[:6], + "uas": (evidence.get("uas") or [])[:3], + "peak": evidence.get("peak"), + "interval": evidence.get("interval"), + "timeline": (evidence.get("timeline") or [])[-72:], + "sample_count": len(evidence.get("samples") or []), + } + + +def _probe_query(detector, item_rules=None, window=None): + """Sample only the probe families that actually fired for this IP. + + Combining all seven families (plus a regex) over a 24h window made the + Elasticsearch node drop the connection, so keep the sample query narrow and + skip rules whose window cap is smaller than the evidence window. + """ + names = {rule["name"] for rule in detector.rules if rule["name"] in PROBE_RULES} + if item_rules: + fired = sorted(((name, item_rules.get(name) or 0) for name in names + if item_rules.get(name)), key=lambda pair: -pair[1]) + if fired: + # Only the strongest two families: keeps the sample query cheap for + # IPs that fired six or seven rules at once. + names = {name for name, _ in fired[:2]} + queries = [] + window = window or detector.window + for rule in detector.rules: + if rule["name"] not in names or not rule.get("query"): + continue + cap = rule.get("window_cap") + if cap and window > cap: + continue + queries.append(rule["query"]) + if not queries: + return None + return " OR ".join("(%s)" % query for query in queries) + + +def collect_evidence(detector, ip, item_rules=None, window=None): + """Fetch a handful of typical requests for one IP (never the whole set). + + All Graylog calls run concurrently and the result is persisted by the + caller, so opening the same IP a second time is instant. + """ + if detector is None or getattr(detector, "client", None) is None: + return {"available": False, "reason": "未连接 Graylog，仅展示入库统计"} + + window = int(window or detector.window) + cache_key = (ip, window) + cached = EVIDENCE_CACHE.get(cache_key) + if cached and time.time() - cached[0] < EVIDENCE_TTL: + return cached[1] + + from concurrent.futures import ThreadPoolExecutor + + fields = detector.fields + client = detector.client + query = '%s:"%s"' % (fields["client_ip"], ip) + result = {"available": True, "window": window, "samples": [], + "uris": [], "uas": [], "statuses": [], "methods": [], "hosts": [], + "timeline": [], + "rule_titles": {rule["name"]: rule.get("title") or rule["name"] + for rule in detector.rules}} + + def terms_call(field, size): + data = client.terms(field=field, query=query, size=size, + range_seconds=window, filter_str=detector.filter_str) + rows = [{"term": _scrub_binary(_clean_text(row["term"], 240)), + "count": row["count"]} + for row in normalize_terms(data) if row["term"] not in (None, "")] + if field == fields.get("status"): + # "200 OK" and "200" are the same code; merge them for the chart. + merged = {} + for row in rows: + match = re.match(r"\s*(\d{3})", str(row["term"])) + key = match.group(1) if match else str(row["term"]) + merged[key] = merged.get(key, 0) + row["count"] + rows = [{"term": key, "count": count} + for key, count in sorted(merged.items(), + key=lambda pair: (-pair[1], pair[0]))] + return rows + + def samples_call(): + """Latest 3 messages per fired rule, so each attack type keeps evidence.""" + fired = [] + for rule in detector.rules: + name = rule["name"] + if not item_rules or not int(item_rules.get(name) or 0): + continue + if not rule.get("query"): + continue + cap = rule.get("window_cap") + rule_window = min(window, int(cap)) if cap else window + fired.append((name, rule["query"], rule_window)) + if not fired: + try: + messages = client.search_views_messages( + query=query, range_seconds=window, limit=5, + filter_str=detector.filter_str) + except GraylogError: + messages = [] + return [(None, messages)] + + def fetch(job): + rule_name, rule_query, rule_window = job + try: + messages = client.search_views_messages( + query="(%s) AND %s" % (rule_query, query), + range_seconds=rule_window, limit=3, + filter_str=detector.filter_str) + except GraylogError: + messages = [] + return rule_name, messages + + results = [] + with ThreadPoolExecutor(max_workers=6) as pool: + for item in pool.map(fetch, fired): + results.append(item) + return results + + def timeline_call(): + interval = "5m" if window <= 3600 else "1h" + history = client.histogram(query=query, range_seconds=window, + interval=interval, filter_str=detector.filter_str) + return interval, normalize_histogram(history) + + dimensions = (("uris", "uri", 10), ("uas", "ua", 5), ("statuses", "status", 6), + ("methods", "method", 6), ("hosts", "host", 6), + ("server_ips", "server_ip", 6), ("vips", "bigip", 6)) + + with ThreadPoolExecutor(max_workers=8) as pool: + futures = {} + for name, field_key, size in dimensions: + field = fields.get(field_key) + if field: + futures[name] = pool.submit(terms_call, field, size) + futures["samples"] = pool.submit(samples_call) + futures["timeline"] = pool.submit(timeline_call) + + for name, future in futures.items(): + try: + value = future.result() + except GraylogError as exc: + result[name + "_error"] = str(exc) + continue + if name == "samples": + merged = [] + seen = {} + for rule_name, messages in value: + for message in messages: + # second precision: the same request is often logged + # twice (two client ports) within the same second. + fingerprint = ((message.get("timestamp") or "")[:19], + message.get("uri"), message.get("status"), + message.get("method")) + if fingerprint in seen: + entry = seen[fingerprint] + if rule_name and rule_name not in entry["rules"]: + entry["rules"].append(rule_name) + continue + payload = _sample(message) + payload["rules"] = [rule_name] if rule_name else [] + seen[fingerprint] = payload + merged.append(payload) + merged.sort(key=lambda entry: entry.get("timestamp") or "", + reverse=True) + result["samples"] = merged[:24] + elif name == "timeline": + interval, timeline = value + result["timeline"] = timeline + result["interval"] = interval + if timeline: + peak = max(timeline, key=lambda row: row["count"]) + result["peak"] = {"timestamp": peak["timestamp"], + "count": peak["count"]} + result["active_buckets"] = sum(1 for row in timeline if row["count"]) + else: + result[name] = value + + result["distinct_uris_shown"] = len(result.get("uris") or []) + if not result.get("samples") and not result.get("samples_error"): + result["samples_error"] = "该窗口内没有可抽取的样本" + if not result.get("uris") and not result.get("samples"): + result["available"] = False + if not result.get("error") and result.get("uris_error"): + result["error"] = result["uris_error"] + + EVIDENCE_CACHE[cache_key] = (time.time(), result) + return result + + +DASHBOARD = r""" + + +恶意 IP 清单 + + +
+
+ +

恶意 IP 清单

特征 + 行为融合检测 · Graylog · F5 访问日志
+
+
+ +
加载中…
+
+
+ +
+
+
+
+ + +
+
+ + +
+
+ + +
+
+
+
+ + + + + + +
IP 地址位置 / 归属来源评分请求数频率发现时间最近活动判定理由
+ +
+
+
+ + +
+ +
+ + + +""" + + +def make_handler(store, window, interval, detector=None, path_prefix=""): + class Handler(BaseHTTPRequestHandler): + def log_message(self, *args): + pass + + def _send(self, code, payload, content_type="application/json; charset=utf-8"): + if isinstance(payload, (dict, list)): + payload = json.dumps(payload, ensure_ascii=False) + raw = payload.encode("utf-8") + self.send_response(code) + self.send_header("Content-Type", content_type) + self.send_header("Content-Length", str(len(raw))) + self.send_header("Cache-Control", "no-store") + self.end_headers() + self.wfile.write(raw) + + def _send_raw(self, code, text, content_type, filename=None): + raw = text.encode("utf-8") + self.send_response(code) + self.send_header("Content-Type", content_type) + self.send_header("Content-Length", str(len(raw))) + if filename: + self.send_header("Content-Disposition", + 'attachment; filename="%s"' % filename) + self.send_header("Cache-Control", "no-store") + self.end_headers() + self.wfile.write(raw) + + def do_GET(self): + parsed = urlparse(self.path) + path = parsed.path + query = {key: values[0] for key, values in parse_qs(parsed.query).items()} + if path_prefix: + if path == path_prefix: + self.send_response(302) + self.send_header("Location", path_prefix + "/") + self.end_headers() + return + if path.startswith(path_prefix + "/"): + path = path[len(path_prefix):] or "/" + elif path != "/healthz": + return self._send_raw(404, "Not Found", + "text/plain; charset=utf-8") + if path == "/": + return self._send(200, DASHBOARD.replace("__BASE__", path_prefix), + "text/html; charset=utf-8") + if path == "/healthz": + return self._send(200, {"ok": True}) + if path == "/api/stats": + return self._send(200, store.stats()) + if path == "/api/status": + status = {"graylog": "unknown", "last_run": None, + "last_error": None, "age_seconds": None} + if detector is not None: + age = (time.time() - detector.last_success + if getattr(detector, "last_success", None) else None) + status["last_run"] = detector.last_run + status["last_error"] = detector.last_error + status["age_seconds"] = round(age, 1) if age is not None else None + if detector.last_error and (age is None + or age > interval * 2): + status["graylog"] = "error" + elif age is not None and age <= max(120, interval * 2.5): + status["graylog"] = "ok" + elif age is not None: + status["graylog"] = "stale" + return self._send(200, status) + + if path == "/api/ips": + limit = min(2000, int(query.get("limit", 500))) + fused = store.fused_items(limit) + window = detector.window if detector is not None else 3600 + items = [] + for row in fused: + try: + rules = json.loads(row.get("s_rules") or "{}") + except ValueError: + rules = {} + items.append({ + "ip": row["ip"], + "sources": row["sources"], + "tier": row.get("tier") or "behavior", + "score": max(float(row.get("feature_score") or 0), + float(row.get("behavior_score") or 0)), + "feature_score": row.get("feature_score"), + "behavior_score": row.get("behavior_score"), + "total": row.get("total") or 0, + "rules": rules, + "reasons": [row.get("fused_reason") or ""], + "first_seen": row.get("first_banned_at"), + "last_seen": row.get("last_activity_at"), + "last_malicious_at": row.get("last_activity_at"), + "window": window, + "country_code": row.get("country_code") or "", + "country_name": row.get("country_name") or "", + "region": row.get("region") or "", + "city": row.get("city") or "", + "asn": row.get("asn") or "", + "asn_org": row.get("asn_org") or "", + "asn_brand": row.get("asn_brand") or "", + "asn_type": row.get("asn_type") or "", + "asn_icon": row.get("asn_icon") or "", + }) + return self._send(200, {"count": len(items), + "updated_at": store.stats().get("updated_at"), + "items": items}) + if path in ("/api/export.csv", "/api/export.txt"): + scope = query.get("scope", "core") + if scope not in ("core", "all"): + scope = "core" + sort = query.get("sort", "first_seen") + if sort not in Store.SORT_COLUMNS: + sort = "first_seen" + order = query.get("order", "desc") + if order not in ("asc", "desc"): + order = "desc" + items = store.list(limit=20000, + scope=None if scope == "all" else scope, + sort=sort, order=order) + if path.endswith(".csv"): + body = _csv_text(items) + ctype = "text/csv; charset=utf-8" + filename = "malicious-ips.csv" + else: + body = "\n".join(item["ip"] for item in items) + ("\n" if items else "") + ctype = "text/plain; charset=utf-8" + filename = "malicious-ips.txt" + return self._send_raw(200, body, ctype, filename) + + if path.startswith("/api/ip/"): + ip = path[len("/api/ip/"):] + item = store.get(ip) + fused = store.fused_row(ip) + behavior = store.behavior_profile(ip) + if item is None and fused is None: + return self._send(404, {"error": "not found", "ip": ip}) + if item is None: + item = { + "ip": ip, "tier": "behavior", + "score": (behavior or {}).get("score") or 0, + "total": (behavior or {}).get("total") or 0, + "rules": {}, + "reasons": [(fused or {}).get("fused_reason") + or (behavior or {}).get("reason_text") or ""], + "first_seen": (fused or {}).get("first_banned_at"), + "last_seen": (fused or {}).get("last_activity_at"), + "last_malicious_at": (fused or {}).get("last_activity_at"), + "window": 86400, + "country_code": (behavior or {}).get("country_code", ""), + "country_name": (behavior or {}).get("country_name", ""), + "region": (behavior or {}).get("region", ""), + "city": (behavior or {}).get("city", ""), + "asn": (behavior or {}).get("asn", ""), + "asn_org": (behavior or {}).get("asn_org", ""), + "asn_brand": (behavior or {}).get("asn_brand", ""), + "asn_type": (behavior or {}).get("asn_type", ""), + "asn_icon": (behavior or {}).get("asn_icon", ""), + } + item["sources"] = (fused or {}).get("sources") or "feature" + item["behavior"] = behavior + item["fused"] = fused + if item.get("window") is None and detector is not None: + item["window"] = detector.window + + # Evidence is fetched live from Graylog on click (memory + # cached) so the database stays small; a compact snapshot is + # kept as a fallback once the index has rotated away. + stored_raw = item.pop("evidence", None) + try: + evidence = collect_evidence( + detector, ip, item.get("rules"), + item.get("window") or (detector.window if detector else None)) + except Exception as exc: # never kill the connection + evidence = {"available": False, "error": str(exc)} + + live_ok = bool(evidence.get("available") + and (evidence.get("uris") or evidence.get("samples"))) + if live_ok: + try: + stamp = now_iso() + store.set_evidence(ip, _compact_evidence(evidence), stamp) + item["evidence_at"] = stamp + except Exception: + pass + evidence["stale"] = False + else: + snapshot = None + if stored_raw: + try: + snapshot = json.loads(stored_raw) + except ValueError: + snapshot = None + if snapshot: + snapshot["available"] = True + snapshot["stale"] = True + snapshot["samples"] = [] + snapshot["reason"] = "Graylog 中该时段日志已过期，以下为判定时的快照" + snapshot.setdefault("window", item.get("window")) + evidence = snapshot + return self._send(200, {"item": item, "evidence": evidence}) + return self._send(404, {"error": "not found", "path": path}) + + return Handler + + +# --------------------------------------------------------------------- entry +def run(ctx, args): + fields = dict(DEFAULT_FIELDS) + profile_path = getattr(args, "profile", None) + if profile_path: + with open(profile_path, "r", encoding="utf-8") as handle: + profile = json.load(handle) + for key, value in (profile.get("fields") or {}).items(): + if value and key in fields: + fields[key] = value + for attribute, key in (("field", "client_ip"), ("status_field", "status"), + ("uri_field", "uri"), ("ua_field", "ua")): + value = getattr(args, attribute, None) + if value: + fields[key] = value + + allow_entries = [] + for path in (getattr(args, "allow", None) or []): + with open(path, "r", encoding="utf-8") as handle: + text = handle.read() + for line in text.splitlines(): + line = line.split("#", 1)[0].strip() + if not line: + continue + try: + allow_entries.append(ipaddress.ip_network(line, strict=False)) + except ValueError: + raise SystemExit("白名单 %s 不是合法 IP/CIDR: %r" % (path, line)) + + def allowlisted(ip): + try: + address = ipaddress.ip_address(ip) + except ValueError: + return False + return any(address.version == net.version and address in net + for net in allow_entries) + + notifier = None + notify_path = getattr(args, "notify_config", "") or "" + if notify_path and os.path.exists(notify_path): + try: + notifier = load_notifier(notify_path) + print("推送已启用: %s" % notifier.kind, flush=True) + except Exception as exc: + print("推送配置无效: %s" % exc, flush=True) + if notifier is None: + try: + notifier = notifier_from_env() + if notifier is not None: + print("推送已启用（环境变量）: %s" % notifier.kind, flush=True) + except Exception as exc: + print("推送环境变量无效: %s" % exc, flush=True) + + if getattr(args, "notify_test", False): + if notifier is None: + print("未配置推送：请创建 %s（参考 notify.example.json）" + % (notify_path or "notify.json"), flush=True) + return 1 + try: + reply = notifier.send_test() + except Exception as exc: + print("测试推送失败: %s" % exc, flush=True) + return 1 + print("测试推送成功: %s" % str(reply)[:300], flush=True) + return 0 + + store = Store(args.db) + detector = Detector( + ctx.client, store, fields, + window=int(args.window), size=int(args.size), + min_requests=int(args.min_requests), verify_limit=int(args.verify), + forget=int(args.forget), allowlist=allowlisted, + min_observe_score=float(args.min_observe_score), + warm_limit=int(getattr(args, "warm_limit", 5) or 0), + filter_str=("streams:" + ctx.stream_id) if ctx.stream_id else None, + geo_db=getattr(args, "geo_db", "") or None, + asn_db=getattr(args, "asn_db", "") or None, + ) + if getattr(detector, "geo", None) is not None: + filled = detector.backfill_geo(3000) + if filled: + print("补充地理信息 %d 个 IP" % filled, flush=True) + if getattr(detector, "asn", None) is not None: + filled = detector.backfill_asn(3000) + if filled: + print("补充 ASN 信息 %d 个 IP" % filled, flush=True) + + # Fused block list: existing feature detections first (historical rows keep + # their original discovery time), then the behaviour thread keeps adding. + store.sync_fused() + behavior_analyzer = None + if not getattr(args, "no_behavior", False): + behavior_analyzer = BehaviorAnalyzer( + ctx.client, store, fields, + filter_str=("streams:" + ctx.stream_id) if ctx.stream_id else None, + root=os.path.dirname(os.path.dirname(os.path.abspath(__file__))), + interval=int(getattr(args, "behavior_interval", 900) or 900), + notifier=notifier, + ) + threading.Thread(target=behavior_analyzer.run_forever, daemon=True).start() + print("行为检测线程已启动（每 %d 秒一次）" % behavior_analyzer.interval, + flush=True) + + def report(result): + line = ("[%s] 窗口 %ds: %d 个候选（恶意 %d / 可疑 %d / 自动化 %d），耗时 %.1fs" + % (now_iso(), result["window"], result["candidates"], + result["malicious"], result["suspicious"], result["automation"], + result["request_seconds"])) + print(line, flush=True) + + if getattr(args, "fix_first_seen", False): + count = detector.backfill_first_seen(int(getattr(args, "warm_limit", 500) or 500)) + print("已修正 %d 个 IP 的首次发现时间" % count, flush=True) + store.close() + return 0 + + if getattr(args, "warm_only", False): + count = detector.warm_snapshots(int(getattr(args, "warm_limit", 50) or 50)) + print("已预热 %d 个 IP 的证据快照" % count, flush=True) + store.close() + return 0 + + if getattr(args, "once", False): + report(detector.run_once()) + if getattr(args, "export", None): + chosen = export_blocklist(store, args.export) + print("已导出 %d 个恶意 IP -> %s" % (len(chosen), args.export)) + for row in store.list(limit=int(args.top)): + print("%-16s %-11s %5.1f %6d %s" % ( + row["ip"], TIER_LABELS.get(row["tier"], row["tier"]), + row["score"], row["total"], "；".join(row["reasons"][:3]))) + store.close() + return 0 + + prefix = (getattr(args, "path_prefix", "") or "").strip() + if prefix: + prefix = "/" + prefix.strip("/") + server = ThreadingHTTPServer((args.host, int(args.port)), + make_handler(store, int(args.window), + int(args.interval), detector, prefix)) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + print("面板: http://%s:%s%s/ (Ctrl+C 退出)" + % (args.host, args.port, prefix), flush=True) + + if getattr(args, "serve_only", False): + try: + thread.join() + except KeyboardInterrupt: + print("", flush=True) + finally: + server.shutdown() + store.close() + return 0 + + try: + while True: + try: + report(detector.run_once()) + store.sync_fused() + if notifier is not None: + notify_fused(store, notifier) + if getattr(args, "export", None): + chosen = export_blocklist(store, args.export) + print(" 封禁名单: %d 个 -> %s" % (len(chosen), args.export), flush=True) + except GraylogError as exc: + detector.last_error = str(exc) + print("[%s] 检测失败: %s" % (now_iso(), exc), flush=True) + except Exception as exc: + detector.last_error = str(exc) + print("[%s] 检测异常: %s" % (now_iso(), exc), flush=True) + time.sleep(max(5, int(args.interval))) + except KeyboardInterrupt: + print("\n停止。数据库: %s" % os.path.abspath(args.db), flush=True) + finally: + server.shutdown() + store.close() + return 0 diff --git a/notify.example.json b/notify.example.json new file mode 100644 index 0000000..b9e5683 --- /dev/null +++ b/notify.example.json @@ -0,0 +1,7 @@ +{ + "type": "dingtalk", + "url": "https://oapi.dingtalk.com/robot/send?access_token=REPLACE_WITH_TOKEN", + "secret": "SEC_REPLACE_WITH_SIGN_SECRET", + "link_prefix": "http://179.253.234.124:12139/s/301b0525d852feacedcbf813a422f6e5/", + "title": "新增恶意 IP" +} diff --git a/profiles/f5.json b/profiles/f5.json new file mode 100644 index 0000000..c99f548 --- /dev/null +++ b/profiles/f5.json @@ -0,0 +1,19 @@ +{ + "fields": { + "client_ip": "client_ip", + "status": "status", + "uri": "uri", + "ua": "ua", + "method": "method", + "host": "host" + }, + "signals": { + "bad_method": { + "enabled": false + } + }, + "thresholds": { + "min_requests": 30, + "min_score": 40.0 + } +} \ No newline at end of file diff --git a/tests/demo.py b/tests/demo.py new file mode 100644 index 0000000..d905bae --- /dev/null +++ b/tests/demo.py @@ -0,0 +1,50 @@ +"""Preview the CLI output without a real Graylog instance. + +Runs a handful of commands against the bundled fake Graylog 4.0 server so you +can see the exact shape of the tables/JSON before pointing the tool at +edu-master. + + python3 tests/demo.py +""" + +import io +import os +import sys +from contextlib import redirect_stdout + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.dirname(HERE)) +sys.path.insert(0, HERE) + +from fake_graylog import JUDGE_MESSAGES, FakeGraylog # noqa: E402 +from graylog_ops import cli # noqa: E402 + +COMMANDS = ( + ["probe"], + ["streams"], + ["discover", "--range", "3600"], + ["top-ips", "--field", "client_ip", "--query", "status:401", "--size", "5"], + ["judge", "--range", "300", "--explain", "3"], +) + + +def main(): + server = FakeGraylog(messages=JUDGE_MESSAGES).start() + base = ["--url", server.url, "--token", "testtoken"] + try: + for command in COMMANDS: + print("=" * 78) + print("$ graylog-ops " + " ".join(command)) + print("-" * 78) + captured = io.StringIO() + with redirect_stdout(captured): + cli.main(base + command) + print(captured.getvalue().rstrip()) + print() + finally: + server.stop() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/fake_graylog.py b/tests/fake_graylog.py new file mode 100644 index 0000000..f80ca4a --- /dev/null +++ b/tests/fake_graylog.py @@ -0,0 +1,468 @@ +"""A tiny fake Graylog 4.0 server used by the end-to-end tests. + +It intentionally mimics the response envelopes of the real 4.0 API (including +the ``{"messages": [{"message": {...}}]}`` shape of the legacy search API) so +the client code is exercised the same way it will be against edu-master. +""" + +import base64 +import json +import re +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from urllib.parse import parse_qs, urlparse + +TOKEN = "testtoken" +EXPECTED_AUTH = "Basic " + base64.b64encode( + ("%s:token" % TOKEN).encode("utf-8") +).decode("ascii") + +VERSION = "4.0.8+6b8c55d" + +MESSAGES = [ + { + "timestamp": "2026-10-17T15:00:00.000Z", + "source": "10.180.254.17", + "client_ip": "203.0.113.5", + "status": 401, + "request_method": "POST", + "request_uri": "/api/login", + "http_user_agent": "python-requests/2.31", + "response_time": 0.011, + "message": '203.0.113.5 - - "POST /api/login HTTP/1.1" 401 0.011', + }, + { + "timestamp": "2026-10-17T15:00:05.000Z", + "source": "10.180.254.17", + "client_ip": "203.0.113.5", + "status": 401, + "request_method": "POST", + "request_uri": "/api/login", + "http_user_agent": "python-requests/2.31", + "response_time": 0.009, + "message": '203.0.113.5 - - "POST /api/login HTTP/1.1" 401 0.009', + }, + { + "timestamp": "2026-10-17T15:00:10.000Z", + "source": "10.180.254.17", + "client_ip": "203.0.113.5", + "status": 401, + "request_method": "POST", + "request_uri": "/api/login", + "http_user_agent": "python-requests/2.31", + "response_time": 0.010, + "message": '203.0.113.5 - - "POST /api/login HTTP/1.1" 401 0.010', + }, + { + "timestamp": "2026-10-17T15:00:15.000Z", + "source": "10.180.254.17", + "client_ip": "198.51.100.7", + "status": 401, + "request_method": "GET", + "request_uri": "/admin", + "http_user_agent": "Mozilla/5.0", + "response_time": 0.021, + "message": '198.51.100.7 - - "GET /admin HTTP/1.1" 401 0.021', + }, + { + "timestamp": "2026-10-17T15:00:20.000Z", + "source": "10.180.254.17", + "client_ip": "198.51.100.7", + "status": 403, + "request_method": "GET", + "request_uri": "/admin", + "http_user_agent": "Mozilla/5.0", + "response_time": 0.018, + "message": '198.51.100.7 - - "GET /admin HTTP/1.1" 403 0.018', + }, + { + "timestamp": "2026-10-17T15:00:25.000Z", + "source": "10.180.254.17", + "client_ip": "192.0.2.44", + "status": 404, + "request_method": "GET", + "request_uri": "/wp-login.php", + "http_user_agent": "sqlmap/1.7", + "response_time": 0.006, + "message": '192.0.2.44 - - "GET /wp-login.php HTTP/1.1" 404 0.006', + }, + { + "timestamp": "2026-10-17T15:00:30.000Z", + "source": "10.180.254.17", + "client_ip": "192.0.2.44", + "status": 404, + "request_method": "GET", + "request_uri": "/.env", + "http_user_agent": "sqlmap/1.7", + "response_time": 0.005, + "message": '192.0.2.44 - - "GET /.env HTTP/1.1" 404 0.005', + }, + { + "timestamp": "2026-10-17T15:00:35.000Z", + "source": "10.180.254.17", + "client_ip": "10.180.2.9", + "status": 200, + "request_method": "GET", + "request_uri": "/health", + "http_user_agent": "kube-probe/1.25", + "response_time": 0.002, + "message": '10.180.2.9 - - "GET /health HTTP/1.1" 200 0.002', + }, +] + + +def _build_judge_messages(): + """Synthetic traffic for the judgement tests (brute force / scanner / legit).""" + rows = [] + + def add(ip, count, status, uri, agent, method="GET"): + for index in range(count): + rows.append({ + "timestamp": "2026-10-17T15:%02d:%02d.000Z" % (index // 60, index % 60), + "source": "10.180.254.17", + "client_ip": ip, + "status": status, + "request_method": method, + "request_uri": uri, + "http_user_agent": agent, + "message": '%s - "%s %s HTTP/1.1" %d' % (ip, method, uri, status), + }) + + add("203.0.113.66", 40, 401, "/api/login", "python-requests/2.31", "POST") + add("192.0.2.99", 25, 404, "/wp-login.php", "sqlmap/1.7") + add("198.51.100.200", 50, 200, "/index", "Mozilla/5.0") + add("10.180.5.5", 30, 401, "/api/login", "Mozilla/5.0", "POST") + return rows + + +JUDGE_MESSAGES = _build_judge_messages() + +CLAUSE = re.compile(r'^([A-Za-z_][A-Za-z0-9_.]*):(\([^)]*\)|"[^"]*"|[^\s]+)$') + + +def _clause_values(raw): + raw = raw.strip() + if raw.startswith("(") and raw.endswith(")"): + return [part.strip().strip('"') for part in raw[1:-1].split(" OR ") if part.strip()] + return [raw.strip('"')] + + +def matches(message, query): + """Very small subset of Lucene, enough to exercise the judge signal queries.""" + query = (query or "*").strip() + if query in ("", "*"): + return True + clause = CLAUSE.match(query) + if clause is None: + return True # unsupported syntax: do not filter (test double only) + field, raw = clause.groups() + value = message.get(field) + if value is None: + return False + text = str(value).lower() + for candidate in _clause_values(raw): + # Mirror Lucene: `foo*` is a prefix match, `\/` is an escaped slash. + needle = candidate.lower().replace("\\/", "/") + if needle.endswith("*"): + if needle[:-1] and text.startswith(needle[:-1]): + return True + continue + if needle and needle in text: + return True + return False + + +def aggregate(messages, body): + """Minimal views-pivot implementation (one query, one search type).""" + query = (body.get("queries") or [{}])[0] + query_string = (query.get("query") or {}).get("query_string") or "*" + filtered = [message for message in messages if matches(message, query_string)] + + stream_filter = query.get("filter") + stream_ids = set() + if isinstance(stream_filter, dict): + for item in stream_filter.get("filters") or []: + if isinstance(item, dict) and item.get("id"): + stream_ids.add(item["id"]) + if stream_ids: + filtered = [message for message in filtered + if set(message.get("streams") or []) & stream_ids] + + search_type = (query.get("search_types") or [{}])[0] + series_id = ((search_type.get("series") or [{}])[0]).get("id", "c1") + row_groups = search_type.get("row_groups") or [] + column_groups = search_type.get("column_groups") or [] + + rows = [] + total = 0 + if row_groups: + field = row_groups[0].get("field") + limit = int(row_groups[0].get("limit") or 10) + counts = {} + for message in filtered: + value = message.get(field) + if value is None: + continue + counts[str(value)] = counts.get(str(value), 0) + 1 + total = sum(counts.values()) + ordered = sorted(counts.items(), key=lambda item: (-item[1], item[0]))[:limit] + for key, count in ordered: + rows.append({"key": [key], "source": "leaf", + "values": [{"key": [series_id], "value": count, + "rollup": True, "source": "row-leaf"}]}) + rows.append({"key": [], "source": "rollup", + "values": [{"key": [series_id], "value": total, + "rollup": True, "source": "row-leaf"}]}) + elif column_groups: + buckets = {} + for message in filtered: + stamp = str(message.get("timestamp", "")) + key = stamp[:16] + ":00.000Z" + buckets[key] = buckets.get(key, 0) + 1 + total = sum(buckets.values()) + for key in sorted(buckets): + rows.append({"key": [], "source": "leaf", + "values": [{"key": [key, series_id], "value": buckets[key], + "rollup": False, "source": "col-leaf"}]}) + return {"id": search_type.get("id", "st1"), "type": "pivot", + "total": total, "rows": rows} + + +class FakeGraylog: + def __init__(self, require_auth=True, messages=None, views_enabled=True): + self.require_auth = require_auth + self.messages = list(MESSAGES if messages is None else messages) + self.views_enabled = views_enabled + self.terms_size_param = "size" + self.requests = [] + owner = self + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *args): # silence test output + pass + + def _send(self, code, payload): + body = json.dumps(payload).encode("utf-8") + self.send_response(code) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_POST(self): + parsed = urlparse(self.path) + path = parsed.path + query = {key: values[0] for key, values in parse_qs(parsed.query).items()} + length = int(self.headers.get("Content-Length") or 0) + raw_body = self.rfile.read(length) if length else b"" + owner.requests.append((path, query)) + + if owner.require_auth and self.headers.get("Authorization") != EXPECTED_AUTH: + return self._send(401, {"message": "Unauthorized"}) + if not owner.views_enabled: + return self._send(404, {"message": "Sorry, Page Not Found"}) + if path == "/api/views/search/sync": + try: + body = json.loads(raw_body or b"{}") + except ValueError: + return self._send(400, {"message": "invalid json"}) + result = aggregate(owner.messages, body) + return self._send(200, { + "execution": {"done": True, "cancelled": False, + "completed_exceptionally": False}, + "results": {"q1": {"errors": [], "search_types": {"st1": result}}}, + "search_id": "mock-search", + }) + return self._send(404, {"message": "not found: %s" % path}) + + def do_GET(self): + parsed = urlparse(self.path) + path = parsed.path + query = {key: values[0] for key, values in parse_qs(parsed.query).items()} + owner.requests.append((path, query)) + + if owner.require_auth and self.headers.get("Authorization") != EXPECTED_AUTH: + return self._send(401, {"message": "Unauthorized"}) + + if path == "/api/system": + return self._send(200, { + "version": VERSION, + "cluster_id": "6b8c55d0-dead-beef-0000-000000000001", + "hostname": "edu-master", + "is_processing": True, + "timezone": "Asia/Shanghai", + "locale": "zh_CN", + }) + if path == "/api/system/cluster/nodes": + return self._send(200, {"total": 1, "nodes": [{ + "node_id": "node-1", + "hostname": "edu-master", + "version": VERSION, + "lifecycle": "RUNNING", + "is_leader": True, + "lb_status": "ALIVE", + }]}) + if path == "/api/system/indexer/cluster/health": + return self._send(200, { + "status": "green", + "cluster_name": "graylog", + "number_of_nodes": 1, + }) + if path == "/api/system/deflector": + return self._send(200, {"index": "graylog_0", "is_up": True}) + if path == "/api/system/fields": + return self._send(200, {"fields": [ + {"name": "client_ip", "type": "ip"}, + {"name": "status", "type": "long"}, + {"name": "request_uri", "type": "string"}, + ]}) + if path == "/api/system/license": + return self._send(200, {"license": {"type": "opensource"}}) + if path == "/api/system/inputs": + return self._send(200, {"total": 1, "inputs": [{ + "id": "5f0000000000000000000001", + "title": "GELF UDP", + "type": "org.graylog2.inputs.gelf.udp.GELFUDPInput", + "global": True, + "configuration": {"bind_address": "0.0.0.0", "port": 12201}, + }]}) + if path == "/api/streams": + return self._send(200, {"total": 2, "streams": [ + { + "id": "5f00000000000000000000aa", + "title": "业务访问日志", + "description": "nginx access log", + "disabled": False, + "rules": [], + }, + { + "id": "000000000000000000000001", + "title": "Default Stream", + "description": "default", + "disabled": False, + "rules": [], + }, + ]}) + if path == "/api/system/indices/index_sets": + return self._send(200, {"total": 1, "index_sets": [{ + "id": "5f00000000000000000000bb", + "title": "Default index set", + "index_prefix": "graylog", + "shards": 4, + "replicas": 0, + "rotation_strategy": "time", + "rotation_strategy_configuration": {"type": "time", "period": "P1D"}, + "retention_strategy": "delete", + "retention_strategy_configuration": {"max_number_of_indices": 10}, + }]}) + if path == "/api/system/indexer/indices": + return self._send(200, {"indices": [{ + "index_name": "graylog_0", + "index_set_id": "5f00000000000000000000bb", + "size": 1048576, + "begin": "2026-10-17T00:00:00.000Z", + "end": "2026-10-18T00:00:00.000Z", + }]}) + if path == "/api/events/definitions": + return self._send(200, { + "total": 1, "count": 1, "page": 1, "per_page": 50, + "event_definitions": [{ + "id": "5f00000000000000000000cc", + "title": "401 风暴", + "priority": 3, + "disabled": False, + "config": {"type": "aggregation-v1"}, + "notifications": [], + }], + }) + if path == "/api/events/notifications": + return self._send(200, {"total": 1, "notifications": [{ + "id": "5f00000000000000000000dd", + "title": "ban webhook", + "config": {"type": "http-notification-v1", + "url": "http://127.0.0.1:8080/ban"}, + }]}) + if path == "/api/events": + return self._send(200, {"total_events": 0, "events": []}) + if path == "/api/sidecars": + return self._send(200, {"total": 0, "sidecars": []}) + if path in ("/api/system/pipelines/pipeline", "/api/system/pipelines/rule"): + return self._send(200, []) + if path == "/api/views": + return self._send(200, {"total": 0, "views": [], "page": 1, "per_page": 50}) + if path == "/api/search/universal/relative": + try: + limit = int(query.get("limit", 100)) + except ValueError: + limit = 100 + try: + offset = int(query.get("offset", 0)) + except ValueError: + offset = 0 + matched = [item for item in owner.messages + if matches(item, query.get("query", "*"))] + page = matched[offset:offset + limit] + return self._send(200, { + "messages": [{"message": dict(item), "index": "graylog_0"} for item in page], + "total_results": len(matched), + "time": 3, + "built_query": "mock", + }) + if path == "/api/search/universal/terms": + if owner.terms_size_param == "limit" and "size" in query: + return self._send(400, {"message": "Parameter 'size' is not allowed"}) + field = query.get("field", "source") + param = "size" if "size" in query else ("limit" if "limit" in query else None) + try: + size = int(query.get(param, 50)) if param else 50 + except ValueError: + size = 50 + counts = {} + query_string = query.get("query", "*") + for message in owner.messages: + if not matches(message, query_string): + continue + value = message.get(field) + if value is None: + continue + counts[str(value)] = counts.get(str(value), 0) + 1 + terms = sorted( + ({"term": key, "count": value} for key, value in counts.items()), + key=lambda row: (-row["count"], row["term"]), + )[:size] + return self._send(200, { + "field": field, + "terms": terms, + "missing": 0, + "other": 0, + "total": len(terms), + }) + if path == "/api/search/universal/histogram": + return self._send(200, { + "results": { + "2026-10-17T15:00:00.000Z": {"count": 3}, + "2026-10-17T15:01:00.000Z": {"count": 5}, + }, + "interval": "1m", + "time": 2, + }) + return self._send(404, {"message": "not found: %s" % path}) + + self.server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + self.port = self.server.server_address[1] + self.thread = None + + @property + def url(self): + return "http://127.0.0.1:%d" % self.port + + def start(self): + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + self.thread.start() + return self + + def stop(self): + self.server.shutdown() + self.server.server_close() + if self.thread is not None: + self.thread.join(timeout=5) diff --git a/tests/test_asn.py b/tests/test_asn.py new file mode 100644 index 0000000..a6536be --- /dev/null +++ b/tests/test_asn.py @@ -0,0 +1,49 @@ +import os +import sys +import tempfile +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.dirname(HERE)) + +from graylog_ops.asn import AsnDB, build_asn_db, classify_org # noqa: E402 + + +class AsnTests(unittest.TestCase): + def test_build_and_lookup(self): + with tempfile.TemporaryDirectory() as directory: + csv_path = os.path.join(directory, "asn.csv") + with open(csv_path, "w", encoding="utf-8") as handle: + handle.write('1.0.1.0,1.0.3.255,4134,"Chinanet"\n') + handle.write('8.8.8.0,8.8.8.255,15169,"Google LLC"\n') + handle.write('47.0.0.0,47.0.0.255,37963,' + '"Hangzhou Alibaba Advertising Co.,Ltd."\n') + handle.write('2001:db8::,2001:db8::ffff,15169,"Google LLC"\n') + db_path = os.path.join(directory, "asn.db") + count = build_asn_db(csv_path, db_path) + self.assertEqual(count, 3) # IPv6 skipped + + asn = AsnDB(db_path) + try: + china = asn.lookup("1.0.2.1") + self.assertEqual(china["brand"], "中国电信") + self.assertEqual(china["type"], "broadband") + aliyun = asn.lookup("47.0.0.1") + self.assertEqual(aliyun["brand"], "阿里云") + self.assertEqual(aliyun["type"], "cloud") + self.assertEqual(aliyun["icon"], "☁️") + self.assertIsNone(asn.lookup("9.9.9.9")) + finally: + asn.close() + + def test_classify_org(self): + self.assertEqual(classify_org("China Mobile Communications Corporation")[:2], + ("中国移动", "mobile")) + self.assertEqual(classify_org("China Education and Research Network Center")[:2], + ("教育网", "education")) + self.assertEqual(classify_org("Wowrack.com")[:2], ("IDC 机房", "idc")) + self.assertEqual(classify_org("Some Unknown ISP")[:2], ("未知", "unknown")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_behavior.py b/tests/test_behavior.py new file mode 100644 index 0000000..9ffbb27 --- /dev/null +++ b/tests/test_behavior.py @@ -0,0 +1,103 @@ +import os +import sys +import tempfile +import time +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.dirname(HERE)) + +from graylog_ops import watch # noqa: E402 + +NOW = "2026-10-17T16:00:00Z" + + +def _behavior(ip, verdict="建议封禁", score=75): + return { + "ip": ip, "score": score, "verdict": verdict, + "signals": ["高失败率"], "reason_text": "24 小时内扫描了大量路径。", + "share_note": "", "behavior_type": "目录扫描", + "asn": "4538", "asn_org": "教育网", "shared_type": "education", + "total": 50, "pct4": 60.0, "uri_card": 20, "uri_templates": 15, + "ua_card": 1, "host_card": 2, "coverage": 100.0, "peak_share": 50.0, + "top_uri": "/.env", "top_ua": "curl/8.0", + "country_code": "CN", "country_name": "中国", "region": "北京", + "city": "Beijing", + } + + +class FusedTests(unittest.TestCase): + def test_feature_and_behavior_merge(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + try: + store.upsert({"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS, + "score": 91.0, "total": 120, "window": 3600, + "rules": {"probe_java": 10}, + "reasons": ["探测 10 次"]}, NOW) + store.sync_fused(NOW) + fused = store.fused_items() + self.assertEqual(len(fused), 1) + self.assertEqual(fused[0]["sources"], "feature") + self.assertEqual(fused[0]["first_banned_at"], NOW) + self.assertIn("特征命中", fused[0]["fused_reason"]) + + # behaviour-only ban is merged + store.replace_behavior_profiles([_behavior("198.51.100.9")]) + store.sync_fused("2026-10-17T16:10:00Z") + merged = {item["ip"]: item for item in store.fused_items()} + self.assertIn("198.51.100.9", merged) + self.assertEqual(merged["198.51.100.9"]["sources"], "behavior") + self.assertEqual(merged["203.0.113.9"]["sources"], "feature") + + # same IP on both sides becomes "both" + store.upsert({"ip": "198.51.100.9", "tier": watch.TIER_MALICIOUS, + "score": 80.0, "total": 50, "window": 3600, + "rules": {"probe_java": 6}, + "reasons": ["探测 6 次"]}, "2026-10-17T16:11:00Z") + store.sync_fused("2026-10-17T16:12:00Z") + merged = {item["ip"]: item for item in store.fused_items()} + self.assertEqual(merged["198.51.100.9"]["sources"], "both") + + # watch-only behaviour must not enter the block list + store.replace_behavior_profiles([ + _behavior("192.0.2.44", verdict="仅告警（共享出口）")]) + store.sync_fused("2026-10-17T16:20:00Z") + ips = {item["ip"] for item in store.fused_items()} + self.assertNotIn("192.0.2.44", ips) + self.assertIn("203.0.113.9", ips) # feature rows stay forever + finally: + store.close() + + def test_push_throttle_24h(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + try: + now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + store.replace_behavior_profiles([_behavior("198.51.100.7")]) + store.sync_fused(now) + # first sync suppresses historical rows; a fresh detection is due + with store.lock, store.conn: + store.conn.execute( + "UPDATE ip_fused SET last_pushed_at=NULL WHERE ip=?", + ("198.51.100.7",)) + due = store.due_pushes(active_hours=24, cooldown_hours=24) + self.assertEqual([row["ip"] for row in due], ["198.51.100.7"]) + + store.mark_pushed(["198.51.100.7"], now) + self.assertEqual(store.due_pushes(24, 24), []) # throttled + + old = time.strftime("%Y-%m-%dT%H:%M:%SZ", + time.gmtime(time.time() - 25 * 3600)) + with store.lock, store.conn: + store.conn.execute( + "UPDATE ip_fused SET last_pushed_at=? WHERE ip=?", + (old, "198.51.100.7")) + due = store.due_pushes(active_hours=24, cooldown_hours=24) + self.assertEqual([row["ip"] for row in due], ["198.51.100.7"]) + finally: + store.close() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_detect.py b/tests/test_detect.py new file mode 100644 index 0000000..905b061 --- /dev/null +++ b/tests/test_detect.py @@ -0,0 +1,127 @@ +"""Unit tests for the client-IP field heuristics.""" + +import os +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from graylog_ops import detect # noqa: E402 + + +def message(client_ip, source="10.180.254.17", **extra): + payload = { + "timestamp": "2026-10-17T15:00:00.000Z", + "source": source, + "client_ip": client_ip, + "status": 401, + "request_uri": "/api/login", + "message": '%s - - "POST /api/login HTTP/1.1" 401 0.01' % client_ip, + } + payload.update(extra) + return payload + + +class AsIpTests(unittest.TestCase): + def test_ipv4(self): + self.assertEqual(str(detect.as_ip("203.0.113.5")), "203.0.113.5") + + def test_ipv6(self): + self.assertEqual(str(detect.as_ip("2001:db8::1")), "2001:db8::1") + + def test_bracketed_ipv6_with_port(self): + self.assertEqual(str(detect.as_ip("[2001:db8::1]:443")), "2001:db8::1") + + def test_ipv4_with_port(self): + self.assertEqual(str(detect.as_ip("192.0.2.44:8080")), "192.0.2.44") + + def test_rejects_cidr(self): + self.assertIsNone(detect.as_ip("10.0.0.0/8")) + + def test_rejects_hostname(self): + self.assertIsNone(detect.as_ip("edu-master")) + + def test_rejects_bad_octet(self): + self.assertIsNone(detect.as_ip("999.1.1.1")) + + def test_rejects_empty_and_number(self): + self.assertIsNone(detect.as_ip("")) + self.assertIsNone(detect.as_ip(12345)) + + +class AnalyseTests(unittest.TestCase): + def test_empty_sample(self): + report = detect.analyse_messages([]) + self.assertEqual(report["sampled"], 0) + self.assertIsNone(report["recommendation"]["field"]) + self.assertEqual(report["recommendation"]["confidence"], "none") + + def test_recommends_client_ip_over_source(self): + messages = [message(ip) for ip in ( + "203.0.113.5", "203.0.113.5", "203.0.113.5", + "198.51.100.7", "198.51.100.7", "192.0.2.44", "192.0.2.44", + "10.180.2.9", + )] + report = detect.analyse_messages(messages) + self.assertEqual(report["recommendation"]["field"], "client_ip") + self.assertEqual(report["recommendation"]["confidence"], "high") + + candidates = {row["field"]: row for row in report["candidates"]} + self.assertIn("source", candidates) + self.assertNotIn("message", candidates) + self.assertLess(candidates["source"]["score"], candidates["client_ip"]["score"]) + # The constant `source` must not outrank the varying client address. + self.assertEqual(candidates["source"]["distinct_ips"], 1) + + def test_warns_when_source_is_a_proxy(self): + messages = [message("203.0.113.%d" % index) for index in range(1, 7)] + report = detect.analyse_messages(messages) + joined = " ".join(report["recommendation"]["warnings"]) + self.assertIn("source", joined) + + def test_private_ratio_note(self): + messages = [message("10.180.2.%d" % index) for index in range(1, 7)] + report = detect.analyse_messages(messages) + best = report["recommendation"]["field"] + self.assertEqual(best, "client_ip") + joined = " ".join(report["recommendation"]["warnings"]) + self.assertIn("内网", joined) + + def test_no_ip_field_at_all(self): + messages = [{"source": "edu-master", "message": "no address here"}] * 3 + report = detect.analyse_messages(messages) + self.assertIsNone(report["recommendation"]["field"]) + self.assertEqual(report["candidates"], []) + + def test_http_fields_detected(self): + report = detect.analyse_messages([message("203.0.113.5")]) + self.assertIn("status", report["http_fields"]) + self.assertIn("uri", report["http_fields"]) + + +class InternalAddressTests(unittest.TestCase): + """`is_internal` must not use ipaddress.is_private: that flag also covers + documentation ranges such as 203.0.113.0/24.""" + + def test_rfc1918(self): + self.assertTrue(detect.is_internal(detect.as_ip("10.1.2.3"))) + self.assertTrue(detect.is_internal(detect.as_ip("172.16.5.5"))) + self.assertTrue(detect.is_internal(detect.as_ip("192.168.1.1"))) + + def test_documentation_range_is_not_internal(self): + self.assertFalse(detect.is_internal(detect.as_ip("203.0.113.5"))) + self.assertFalse(detect.is_internal(detect.as_ip("2001:db8::1"))) + + def test_loopback_link_local_and_cgnat(self): + self.assertTrue(detect.is_internal(detect.as_ip("127.0.0.1"))) + self.assertTrue(detect.is_internal(detect.as_ip("169.254.1.1"))) + self.assertTrue(detect.is_internal(detect.as_ip("100.64.1.1"))) + self.assertTrue(detect.is_internal(detect.as_ip("::1"))) + + def test_ipv6_unique_local(self): + self.assertTrue(detect.is_internal(detect.as_ip("fd00::1"))) + self.assertFalse(detect.is_internal(detect.as_ip("2400:cb00::1"))) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_e2e.py b/tests/test_e2e.py new file mode 100644 index 0000000..70e6e15 --- /dev/null +++ b/tests/test_e2e.py @@ -0,0 +1,212 @@ +"""End-to-end tests: run the real CLI against the fake Graylog server.""" + +import io +import json +import os +import sys +import tempfile +import unittest +from contextlib import redirect_stderr, redirect_stdout + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(HERE) +sys.path.insert(0, ROOT) +sys.path.insert(0, HERE) + +from fake_graylog import MESSAGES, FakeGraylog # noqa: E402 +from graylog_ops import cli # noqa: E402 + + +class CliEndToEndTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.server = FakeGraylog().start() + + @classmethod + def tearDownClass(cls): + cls.server.stop() + + def setUp(self): + self.server.terms_size_param = "size" + self.server.requests.clear() + + def run_cli(self, *argv, json_mode=True): + args = ["--url", self.server.url, "--token", "testtoken"] + if json_mode: + args.append("--json") + args.extend(argv) + stdout, stderr = io.StringIO(), io.StringIO() + with redirect_stdout(stdout), redirect_stderr(stderr): + code = cli.main(args) + return code, stdout.getvalue(), stderr.getvalue() + + # ------------------------------------------------------------- probe + def test_probe_json(self): + code, out, err = self.run_cli("probe") + self.assertEqual(code, 0, err) + data = json.loads(out) + self.assertIn("4.0.8", data["system"]["version"]) + self.assertEqual(data["indexer"]["status"], "green") + failed = [check["name"] for check in data["checks"] if not check["ok"]] + self.assertEqual(failed, [], "unexpected failing endpoints: %s" % failed) + + def test_probe_human_output(self): + code, out, err = self.run_cli("probe", json_mode=False) + self.assertEqual(code, 0, err) + self.assertIn("检查项", out) + self.assertIn("4.0.8", out) + self.assertIn("全部", out) + + # ---------------------------------------------------------- discover + def test_discover_finds_client_ip(self): + code, out, err = self.run_cli("discover", "--range", "3600") + self.assertEqual(code, 0, err) + data = json.loads(out) + self.assertEqual(data["sampled"], len(MESSAGES)) + self.assertEqual(data["recommendation"]["field"], "client_ip") + self.assertEqual(data["recommendation"]["confidence"], "high") + self.assertNotIn("message", [row["field"] for row in data["candidates"]]) + self.assertIn("status", data["http_fields"]) + self.assertIn("uri", data["http_fields"]) + + def test_discover_stream_title_resolution(self): + code, out, err = self.run_cli("discover", "--stream", "业务访问") + self.assertEqual(code, 0, err) + data = json.loads(out) + self.assertEqual(data["stream_id"], "5f00000000000000000000aa") + + def test_discover_unknown_stream_fails_cleanly(self): + stdout, stderr = io.StringIO(), io.StringIO() + with redirect_stdout(stdout), redirect_stderr(stderr): + with self.assertRaises(SystemExit) as ctx: + cli.main(["--url", self.server.url, "--token", "testtoken", + "discover", "--stream", "does-not-exist"]) + self.assertNotEqual(ctx.exception.code, 0) + + # ------------------------------------------------------------- terms + def test_top_ips(self): + code, out, err = self.run_cli("top-ips", "--field", "client_ip", "--size", "10") + self.assertEqual(code, 0, err) + data = json.loads(out) + self.assertEqual(data["field"], "client_ip") + self.assertEqual(len(data["terms"]), 4) + self.assertEqual(data["terms"][0]["term"], "203.0.113.5") + self.assertEqual(data["terms"][0]["count"], 3) + + def test_terms_use_views_backend(self): + """4.x removed the legacy terms endpoint, so aggregation must POST.""" + code, out, err = self.run_cli("top-ips", "--field", "client_ip") + self.assertEqual(code, 0, err) + self.assertTrue(json.loads(out)["terms"]) + posts = [path for path, _ in self.server.requests + if path == "/api/views/search/sync"] + self.assertTrue(posts, "expected a views search POST") + + def test_legacy_fallback_when_views_missing(self): + """Graylog 3.x has the legacy terms endpoint but no views API.""" + server = FakeGraylog(views_enabled=False).start() + server.terms_size_param = "limit" + try: + stdout, stderr = io.StringIO(), io.StringIO() + with redirect_stdout(stdout), redirect_stderr(stderr): + code = cli.main(["--url", server.url, "--token", "testtoken", "--json", + "terms", "--field", "client_ip"]) + self.assertEqual(code, 0, stderr.getvalue()) + self.assertTrue(json.loads(stdout.getvalue())["terms"]) + term_calls = [query for path, query in server.requests + if path.endswith("/universal/terms")] + self.assertEqual(len(term_calls), 2) + self.assertIn("size", term_calls[0]) + self.assertIn("limit", term_calls[1]) + self.assertNotIn("size", term_calls[1]) + finally: + server.stop() + + # ----------------------------------------------------------- snapshot + def test_snapshot(self): + with tempfile.TemporaryDirectory() as directory: + code, out, err = self.run_cli("snapshot", "--dir", directory, "--sample", "5") + self.assertEqual(code, 0, err) + data = json.loads(out) + failed = [row for row in data["resources"] if row["status"] != "ok"] + self.assertEqual(failed, []) + self.assertTrue(os.path.exists(os.path.join(directory, "manifest.json"))) + self.assertTrue(os.path.exists(os.path.join(directory, "system.json"))) + self.assertEqual(data["sampled_messages"], 5) + messages_path = os.path.join(directory, "messages.jsonl") + with open(messages_path, "r", encoding="utf-8") as handle: + lines = [line for line in handle if line.strip()] + self.assertEqual(len(lines), 5) + json.loads(lines[0]) + + # -------------------------------------------------------------- raw + def test_raw_endpoint(self): + code, out, err = self.run_cli("raw", "/api/system/inputs") + self.assertEqual(code, 0, err) + data = json.loads(out) + self.assertIn("inputs", data) + + # ---------------------------------------------------------- messages + def test_messages_jsonl(self): + code, out, err = self.run_cli("messages", "--limit", "3", "--jsonl") + self.assertEqual(code, 0, err) + lines = [line for line in out.splitlines() if line.strip()] + self.assertEqual(len(lines), 3) + for line in lines: + json.loads(line) + + # --------------------------------------------------------------- auth + def test_bad_token_returns_1(self): + stdout, stderr = io.StringIO(), io.StringIO() + with redirect_stdout(stdout), redirect_stderr(stderr): + code = cli.main(["--url", self.server.url, "--token", "wrong", "probe"]) + self.assertEqual(code, 1) + self.assertIn("401", stderr.getvalue()) + + def test_missing_config_returns_2(self): + saved = {key: os.environ.pop(key, None) for key in ( + "GRAYLOG_URL", "GRAYLOG_TOKEN", "GRAYLOG_USER", "GRAYLOG_PASSWORD")} + original_cwd = os.getcwd() + try: + # Run from an empty directory: a real ./graylog-ops.json in the + # project root must not leak into this test. + with tempfile.TemporaryDirectory() as directory: + os.chdir(directory) + stderr = io.StringIO() + with redirect_stderr(stderr): + code = cli.main(["system"]) + self.assertEqual(code, 2) + self.assertIn("配置错误", stderr.getvalue()) + finally: + os.chdir(original_cwd) + for key, value in saved.items(): + if value is not None: + os.environ[key] = value + + # --------------------------------------------------------------- collect + def test_collect_bundle(self): + with tempfile.TemporaryDirectory() as directory: + code, out, err = self.run_cli("collect", "--dir", directory, "--range", "3600") + self.assertEqual(code, 0, err) + for name in ("probe.json", "discover.json", "judge.json", + "profile.json", "SUMMARY.txt"): + self.assertTrue(os.path.exists(os.path.join(directory, name)), name) + self.assertTrue(os.path.isdir(os.path.join(directory, "snapshot"))) + data = json.loads(out) + self.assertTrue(all(step["ok"] for step in data["steps"]), data["steps"]) + self.assertEqual(data["recommendation"]["field"], "client_ip") + + # ---------------------------------------------------------------- out + def test_out_file(self): + with tempfile.TemporaryDirectory() as directory: + path = os.path.join(directory, "ips.json") + code, out, err = self.run_cli("top-ips", "--field", "client_ip", "--out", path) + self.assertEqual(code, 0, err) + self.assertTrue(os.path.exists(path)) + with open(path, "r", encoding="utf-8") as handle: + payload = json.load(handle) + self.assertEqual(payload["field"], "client_ip") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_geo.py b/tests/test_geo.py new file mode 100644 index 0000000..d1a8bf1 --- /dev/null +++ b/tests/test_geo.py @@ -0,0 +1,48 @@ +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from graylog_ops.geo import GeoIP, build_geo_db, country_name, flag, region_name + + +class GeoIPTests(unittest.TestCase): + def test_build_and_lookup(self): + with tempfile.TemporaryDirectory() as directory: + csv_path = os.path.join(directory, "city.csv") + with open(csv_path, "w", encoding="utf-8") as fh: + fh.write("1.0.0.0,1.0.0.255,OC,AU,Queensland,\"South Brisbane\",0,0\n") + fh.write("1.0.1.0,1.0.3.255,AS,CN,Fujian,Wenquan,0,0\n") + fh.write("8.8.8.0,8.8.8.255,NA,US,California,\"Mountain View\",0,0\n") + fh.write("2001:db8::,2001:db8::ffff,NA,US,California,X,0,0\n") + db_path = os.path.join(directory, "geo.db") + count = build_geo_db(csv_path, db_path) + self.assertEqual(count, 3) # IPv6 range skipped + + geo = GeoIP(db_path) + try: + cn = geo.lookup("1.0.2.1") + self.assertEqual(cn["country_code"], "CN") + self.assertEqual(cn["country_name"], "中国") + self.assertEqual(cn["region"], "福建") + self.assertEqual(geo.lookup("8.8.8.8")["country_name"], "美国") + self.assertIsNone(geo.lookup("9.9.9.9")) + self.assertIsNone(geo.lookup("not-an-ip")) + finally: + geo.close() + + def test_helpers(self): + self.assertEqual(flag("CN"), "\U0001F1E8\U0001F1F3") + self.assertEqual(flag("ZZ"), "") + self.assertEqual(flag(""), "") + self.assertEqual(country_name("CN"), "中国") + self.assertEqual(country_name("US"), "美国") + self.assertEqual(country_name("XX"), "XX") + self.assertEqual(region_name("CN", "Guangdong"), "广东") + self.assertEqual(region_name("US", "California"), "California") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_judge.py b/tests/test_judge.py new file mode 100644 index 0000000..23c603b --- /dev/null +++ b/tests/test_judge.py @@ -0,0 +1,168 @@ +"""Tests for the judgement layer (ban candidate scoring).""" + +import io +import json +import os +import sys +import tempfile +import unittest +from contextlib import redirect_stderr, redirect_stdout + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(HERE) +sys.path.insert(0, ROOT) +sys.path.insert(0, HERE) + +from fake_graylog import JUDGE_MESSAGES, FakeGraylog # noqa: E402 +from graylog_ops import cli # noqa: E402 + + +class JudgeTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.server = FakeGraylog(messages=JUDGE_MESSAGES).start() + + @classmethod + def tearDownClass(cls): + cls.server.stop() + + def run_cli(self, *argv, json_mode=True): + args = ["--url", self.server.url, "--token", "testtoken"] + if json_mode: + args.append("--json") + args.extend(argv) + stdout, stderr = io.StringIO(), io.StringIO() + with redirect_stdout(stdout), redirect_stderr(stderr): + code = cli.main(args) + return code, stdout.getvalue(), stderr.getvalue() + + def judged(self, *argv): + code, out, err = self.run_cli("judge", "--range", "300", *argv) + self.assertEqual(code, 0, err) + return json.loads(out) + + # ------------------------------------------------------------ behaviour + def test_bruteforce_client_is_recommended_for_ban(self): + data = self.judged() + rows = {row["ip"]: row for row in data["candidates"]} + self.assertIn("203.0.113.66", rows) + row = rows["203.0.113.66"] + self.assertEqual(row["action"], "ban_1h") + self.assertGreaterEqual(row["score"], 75) + self.assertIn("auth_fail", row["signals"]) + self.assertTrue(any("失败请求" in reason for reason in row["reasons"])) + + def test_scanner_client_is_recommended_for_ban(self): + data = self.judged() + row = {item["ip"]: item for item in data["candidates"]}["192.0.2.99"] + self.assertEqual(row["action"], "ban_1h") + self.assertIn("scanner_ua", row["signals"]) + self.assertIn("sensitive_path", row["signals"]) + + def test_busy_but_legitimate_client_is_not_banned(self): + data = self.judged("--all") + row = {item["ip"]: item for item in data["candidates"]}["198.51.100.200"] + self.assertIn(row["action"], ("ignore", "observe")) + self.assertFalse(row["evidence"]) + + def test_internal_addresses_are_excluded_by_default(self): + data = self.judged() + reasons = {row["ip"]: row["reason"] for row in data["excluded"]} + self.assertIn("10.180.5.5", reasons) + self.assertIn("内部", reasons["10.180.5.5"]) + self.assertNotIn("10.180.5.5", {row["ip"] for row in data["candidates"]}) + + def test_include_internal_flag(self): + data = self.judged("--include-internal", "--all") + ips = {row["ip"] for row in data["candidates"]} + self.assertIn("10.180.5.5", ips) + + def test_allowlist_file(self): + with tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False) as handle: + handle.write("# 深信服 AD / VIP\n10.180.254.17\n203.0.113.66\n") + path = handle.name + try: + data = self.judged("--allow", path) + excluded = {row["ip"]: row["reason"] for row in data["excluded"]} + self.assertIn("203.0.113.66", excluded) + self.assertIn("白名单", excluded["203.0.113.66"]) + self.assertNotIn("203.0.113.66", {row["ip"] for row in data["candidates"]}) + finally: + os.unlink(path) + + def test_explain_attaches_samples(self): + data = self.judged("--explain", "2") + sampled = [row for row in data["candidates"] if row.get("samples")] + self.assertTrue(sampled) + self.assertLessEqual(len(sampled[0]["samples"]), 3) + + def test_verify_downgrades_automation_client(self): + """A steady single-endpoint poller with 403s must not be auto-banned.""" + automation = [{ + "timestamp": "2026-10-17T15:%02d:%02d.000Z" % (index // 60, index % 60), + "source": "10.180.254.17", + "client_ip": "203.0.113.77", + "status": 403, + "request_method": "POST", + "request_uri": "/prod-api/biz/liveCourseChapter/msgSubscription", + "http_user_agent": "okhttp/4.9", + "message": '203.0.113.77 - "POST /prod-api/biz/x" 403', + } for index in range(40)] + server = FakeGraylog(messages=list(JUDGE_MESSAGES) + automation).start() + try: + stdout, stderr = io.StringIO(), io.StringIO() + with redirect_stdout(stdout), redirect_stderr(stderr): + code = cli.main(["--url", server.url, "--token", "testtoken", + "--json", "judge", "--range", "300", + "--verify", "10", "--all"]) + self.assertEqual(code, 0, stderr.getvalue()) + data = json.loads(stdout.getvalue()) + row = {item["ip"]: item for item in data["candidates"]}["203.0.113.77"] + self.assertEqual(row["action"], "observe") + self.assertTrue(row["verify"]["automation_suspected"]) + self.assertFalse(row["verify"]["login_like"]) + finally: + server.stop() + + def test_verify_keeps_login_bruteforce(self): + """Failures against a login path stay bannable even if concentrated.""" + data = self.judged("--verify", "5") + row = {item["ip"]: item for item in data["candidates"]}["203.0.113.66"] + self.assertEqual(row["action"], "ban_1h") + self.assertTrue(row["verify"]["login_like"]) + + def test_profile_roundtrip(self): + with tempfile.TemporaryDirectory() as directory: + profile = os.path.join(directory, "profile.json") + code, _, err = self.run_cli("discover", "--range", "3600", + "--save-profile", profile) + self.assertEqual(code, 0, err) + with open(profile, "r", encoding="utf-8") as handle: + saved = json.load(handle) + self.assertEqual(saved["fields"]["client_ip"], "client_ip") + self.assertEqual(saved["fields"]["status"], "status") + self.assertEqual(saved["fields"]["uri"], "request_uri") + + data = self.judged("--profile", profile) + self.assertTrue(data["candidates"]) + self.assertEqual(data["field"], "client_ip") + + # -------------------------------------------------------------- output + def test_human_output(self): + code, out, err = self.run_cli("judge", json_mode=False) + self.assertEqual(code, 0, err) + self.assertIn("信号命中", out) + self.assertIn("本命令只读", out) + self.assertIn("ban_1h", out) + + def test_signal_queries_are_reported(self): + data = self.judged() + names = {stat["name"]: stat for stat in data["signal_stats"]} + self.assertIn("auth_fail", names) + self.assertGreater(names["auth_fail"]["ips_matched"], 0) + self.assertGreater(names["scanner_ua"]["ips_matched"], 0) + self.assertIn("query", names["auth_fail"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_notify.py b/tests/test_notify.py new file mode 100644 index 0000000..2dd568c --- /dev/null +++ b/tests/test_notify.py @@ -0,0 +1,209 @@ +import json +import os +import sys +import tempfile +import threading +import unittest +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(HERE) +for path in (ROOT, HERE): + if path not in sys.path: + sys.path.insert(0, path) + +from graylog_ops import watch # noqa: E402 +from graylog_ops.behavior import notify_fused # noqa: E402 +from graylog_ops.notify import Notifier, load_notifier # noqa: E402 +from test_watch import _StubClient # noqa: E402 + + +class _CaptureNotifier: + def __init__(self): + self.batches = [] + self.posts = [] + + def send(self, entries): + self.batches.append(entries) + + def link_for(self, ip): + return "" + + def post(self, text): + self.posts.append(text) + + +class _Receiver(BaseHTTPRequestHandler): + received = [] + + def log_message(self, *args): + pass + + def do_POST(self): + length = int(self.headers.get("Content-Length") or 0) + body = self.rfile.read(length).decode("utf-8") + _Receiver.received.append(body) + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b'{"errcode":0}') + + +class NotifierTests(unittest.TestCase): + def _config(self, kind="generic", url="http://127.0.0.1:1/hook"): + return {"type": kind, "url": url, "link_prefix": "http://example/panel/"} + + def test_payload_per_kind(self): + text = "**新增恶意 IP**\n\n1.2.3.4" + self.assertEqual( + Notifier(self._config("wecom")).payload(text), + {"msgtype": "markdown", "markdown": {"content": text}}) + self.assertEqual( + Notifier(self._config("dingtalk")).payload(text)["msgtype"], "markdown") + self.assertEqual( + Notifier(self._config("feishu")).payload(text)["msg_type"], "text") + self.assertEqual(Notifier(self._config("slack")).payload(text), {"text": text}) + generic = Notifier(self._config("generic")).payload(text) + self.assertEqual(generic["text"], text) + self.assertEqual(generic["title"], "新增恶意 IP") + + def test_render_includes_geo_rules_and_link(self): + notifier = Notifier(self._config()) + text = notifier.render([{ + "ip": "203.0.113.5", "country_name": "美国", "region": "California", + "score": 91.0, "total": 120, "first_seen": "2026-10-17T16:00:00Z", + "rules_text": "Spring/中间件探测 4 次", + }]) + self.assertIn("203.0.113.5", text) + self.assertIn("美国 · California", text) + self.assertIn("Spring/中间件探测", text) + self.assertIn("http://example/panel/#/ip/203.0.113.5", text) + + def test_send_to_http_endpoint(self): + _Receiver.received = [] + server = ThreadingHTTPServer(("127.0.0.1", 0), _Receiver) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + url = "http://127.0.0.1:%d/hook" % server.server_address[1] + notifier = Notifier(self._config("generic", url)) + notifier.send_test() + self.assertEqual(len(_Receiver.received), 1) + payload = json.loads(_Receiver.received[0]) + self.assertIn("测试推送", payload["text"]) + finally: + server.shutdown() + server.server_close() + + def test_dingtalk_signing(self): + import base64 + import hashlib + import hmac + import urllib.parse + secret = "SECtest" + url = "https://oapi.dingtalk.com/robot/send?access_token=abc" + notifier = Notifier({"type": "dingtalk", "url": url, "secret": secret}) + stamp = 1700000000000 + expected = base64.b64encode(hmac.new( + secret.encode(), ("%d\n%s" % (stamp, secret)).encode(), + hashlib.sha256).digest()).decode() + signed = notifier.signed_url(stamp) + self.assertIn("timestamp=1700000000000", signed) + self.assertIn("sign=" + urllib.parse.quote_plus(expected), signed) + self.assertTrue(signed.startswith(url + "&")) + + def test_dingtalk_without_secret_keeps_url(self): + url = "https://oapi.dingtalk.com/robot/send?access_token=abc" + self.assertEqual(Notifier({"type": "dingtalk", "url": url}).signed_url(1), url) + + def test_dingtalk_signature(self): + import base64 + import hashlib + import hmac + import urllib.parse + + notifier = Notifier({ + "type": "dingtalk", + "url": "https://oapi.dingtalk.com/robot/send?access_token=abc", + "secret": "SECtest-secret", + }) + url = notifier.signed_url() + self.assertIn("timestamp=", url) + self.assertIn("sign=", url) + query = urllib.parse.parse_qs(urllib.parse.urlparse(url).query) + timestamp = query["timestamp"][0] + expected = base64.b64encode(hmac.new( + b"SECtest-secret", + ("%s\nSECtest-secret" % timestamp).encode("utf-8"), + hashlib.sha256).digest()).decode("ascii") + self.assertEqual(query["sign"][0], expected) + + unsigned = Notifier({"type": "dingtalk", + "url": "https://example/hook?access_token=abc"}) + self.assertEqual(unsigned.signed_url(), + "https://example/hook?access_token=abc") + + def test_load_notifier(self): + with tempfile.TemporaryDirectory() as directory: + path = os.path.join(directory, "notify.json") + json.dump(self._config("wecom"), open(path, "w", encoding="utf-8")) + notifier = load_notifier(path) + self.assertEqual(notifier.kind, "wecom") + + +class UpsertStatusTests(unittest.TestCase): + def test_upsert_returns_new_upgraded_unchanged(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + try: + base = {"ip": "203.0.113.9", "total": 10, "rules": {}, + "reasons": []} + self.assertEqual(store.upsert(dict(base, tier=watch.TIER_OBSERVE, + score=5.0), + "2026-10-17T16:00:00Z"), "new") + self.assertEqual(store.upsert(dict(base, tier=watch.TIER_OBSERVE, + score=5.0), + "2026-10-17T16:01:00Z"), "unchanged") + self.assertEqual(store.upsert(dict(base, tier=watch.TIER_MALICIOUS, + score=80.0), + "2026-10-17T16:02:00Z"), "upgraded") + self.assertEqual(store.upsert(dict(base, tier=watch.TIER_MALICIOUS, + score=80.0), + "2026-10-17T16:03:00Z"), "unchanged") + finally: + store.close() + + +class DetectorNotificationTests(unittest.TestCase): + def test_new_malicious_notifies_once(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + try: + probe = watch.Detector(None, store, dict(watch.DEFAULT_FIELDS)) + queries = {rule["name"]: rule["query"] for rule in probe.rules} + by_query = {"*": [("203.0.113.5", 40)], + queries["probe_java"]: [("203.0.113.5", 40)]} + for name, query in queries.items(): + by_query.setdefault(query, []) + capture = _CaptureNotifier() + detector = watch.Detector(_StubClient(by_query), store, + dict(watch.DEFAULT_FIELDS), window=300) + detector.run_once() + store.sync_fused() + # first sync suppresses historical rows; make it a fresh detection + with store.lock, store.conn: + store.conn.execute( + "UPDATE ip_fused SET last_pushed_at=NULL") + notify_fused(store, capture) + self.assertEqual(len(capture.posts), 1) + self.assertIn("203.0.113.5", capture.posts[0]) + detector.run_once() + store.sync_fused() + notify_fused(store, capture) + self.assertEqual(len(capture.posts), 1) # 24h cooldown + finally: + store.close() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_watch.py b/tests/test_watch.py new file mode 100644 index 0000000..a7f874f --- /dev/null +++ b/tests/test_watch.py @@ -0,0 +1,436 @@ +"""Tests for the continuous detector, the SQLite store and the dashboard.""" + +import io +import json +import os +import sys +import tempfile +import threading +import time +import unittest +import urllib.error +import urllib.request +from http.server import ThreadingHTTPServer + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(HERE) +sys.path.insert(0, ROOT) + +from graylog_ops import watch # noqa: E402 + + +class ClassifyTests(unittest.TestCase): + def test_sensitive_probe_is_malicious(self): + tier, score, reasons = watch.classify(44, {"probe_java": 44}) + self.assertEqual(tier, watch.TIER_MALICIOUS) + self.assertGreaterEqual(score, 60) + self.assertTrue(any("探测" in reason for reason in reasons)) + + def test_single_sensitive_hit_is_enough(self): + tier, score, _ = watch.classify(18, {"probe_java": 18}) + self.assertEqual(tier, watch.TIER_MALICIOUS) + + def test_dotbot_crawler_is_malicious(self): + tier, score, reasons = watch.classify(352, {"bot_ua": 352, "not_found": 176}) + self.assertEqual(tier, watch.TIER_MALICIOUS) + self.assertTrue(any("404" in reason for reason in reasons)) + + def test_login_burst_alone_is_only_observed(self): + """Students logging in repeatedly must not look suspicious.""" + tier, score, _ = watch.classify(2253, {"login_burst": 660}) + self.assertEqual(tier, watch.TIER_OBSERVE) + + def test_login_attack_plus_burst_is_malicious(self): + tier, score, _ = watch.classify(666, {"login_attack": 333, "login_burst": 666}) + self.assertEqual(tier, watch.TIER_MALICIOUS) + + def test_busy_clean_ip_scores_zero(self): + tier, score, reasons = watch.classify(5000, {}) + self.assertEqual(score, 0.0) + self.assertEqual(reasons, []) + + def test_404_ratio_needs_volume(self): + _, score, _ = watch.classify(10, {"not_found": 5}) + self.assertEqual(score, 0.0) + + +class _StubClient: + """Returns canned counts keyed by the exact query string.""" + + def __init__(self, by_query, totals=None): + self.by_query = by_query + self.totals = totals or {} + + def terms(self, field, query, size, range_seconds, filter_str=None): + if query.startswith("client_ip:("): + ips = [part.strip().strip('"') + for part in query[len("client_ip:("):-1].split(" OR ")] + items = [(ip, self.totals[ip]) for ip in ips if ip in self.totals] + else: + items = self.by_query.get(query, []) + return {"terms": [{"term": ip, "count": count} for ip, count in items], + "total": sum(count for _, count in items)} + + def search_views_messages(self, query="*", range_seconds=300, limit=5, offset=0, + filter_str=None, from_ts=None, to_ts=None, + timeout_ms=60000, sort_order="DESC"): + return [] + + +class StoreTests(unittest.TestCase): + def test_upsert_list_forget(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + try: + record = {"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS, + "score": 88.0, "total": 120, + "rules": {"sensitive_probe": 4}, "reasons": ["敏感路径探测 4 次"]} + store.upsert(record, "2026-10-17T16:00:00Z") + store.upsert(dict(record, score=90.0), "2026-10-17T16:01:00Z") + rows = store.list() + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0]["score"], 90.0) + self.assertEqual(rows[0]["hits"], 2) + self.assertEqual(rows[0]["rules"]["sensitive_probe"], 4) + + self.assertEqual(store.stats()["by_tier"][watch.TIER_MALICIOUS], 1) + self.assertEqual(store.stats()["events"], 1) + + store.upsert({"ip": "198.51.100.7", "tier": watch.TIER_SUSPICIOUS, + "score": 40.0, "total": 10, "rules": {}, "reasons": []}, + "2026-10-17T16:02:00Z") + self.assertEqual(len(store.list(tier=watch.TIER_SUSPICIOUS)), 1) + finally: + store.close() + + +class PermanentRetentionTests(unittest.TestCase): + def _store(self, directory): + return watch.Store(os.path.join(directory, "ips.db")) + + def test_ever_malicious_survives_downgrade(self): + with tempfile.TemporaryDirectory() as directory: + store = self._store(directory) + try: + base = {"ip": "203.0.113.9", "total": 10, "rules": {}, "reasons": []} + store.upsert(dict(base, tier=watch.TIER_MALICIOUS, score=80.0), + "2026-10-17T16:00:00Z") + store.upsert(dict(base, tier=watch.TIER_OBSERVE, score=5.0), + "2026-10-17T16:01:00Z") + row = store.get("203.0.113.9") + self.assertEqual(row["tier"], watch.TIER_OBSERVE) + self.assertEqual(row["ever_malicious"], 1) + self.assertEqual(row["last_malicious_at"], "2026-10-17T16:00:00Z") + core = store.list(scope="core") + self.assertEqual([item["ip"] for item in core], ["203.0.113.9"]) + finally: + store.close() + + def test_forget_keeps_convicted_ips(self): + with tempfile.TemporaryDirectory() as directory: + store = self._store(directory) + try: + store.upsert({"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS, + "score": 80.0, "total": 10, "rules": {}, + "reasons": []}, "2026-10-01T00:00:00Z") + store.upsert({"ip": "198.51.100.7", "tier": watch.TIER_OBSERVE, + "score": 5.0, "total": 10, "rules": {}, + "reasons": []}, "2026-10-01T00:00:00Z") + store.forget(3600, time.time()) + self.assertIsNotNone(store.get("203.0.113.9")) + self.assertIsNone(store.get("198.51.100.7")) + finally: + store.close() + + def test_sorting_modes(self): + with tempfile.TemporaryDirectory() as directory: + store = self._store(directory) + try: + store.upsert({"ip": "203.0.113.2", "tier": watch.TIER_MALICIOUS, + "score": 70.0, "total": 10, "rules": {}, + "reasons": []}, "2026-10-17T16:00:00Z") + store.upsert({"ip": "203.0.113.1", "tier": watch.TIER_MALICIOUS, + "score": 90.0, "total": 5, "rules": {}, + "reasons": []}, "2026-10-17T17:00:00Z") + first = [row["ip"] for row in store.list(sort="first_seen", order="asc")] + self.assertEqual(first, ["203.0.113.2", "203.0.113.1"]) + recent = [row["ip"] for row in store.list(sort="last_malicious", + order="desc")] + self.assertEqual(recent, ["203.0.113.1", "203.0.113.2"]) + score = [row["ip"] for row in store.list(sort="score", order="desc")] + self.assertEqual(score, ["203.0.113.1", "203.0.113.2"]) + finally: + store.close() + + +class TextSanitisingTests(unittest.TestCase): + def test_strips_control_and_replacement_chars(self): + self.assertEqual(watch._clean_text("a\x00b\ufffdc"), "a·b·c") + + def test_scrubs_binary_header_field(self): + self.assertEqual(watch._scrub_binary("\ufffd\ufffdk\ufffdU\ufffd\ufffd"), + "[非 UTF-8 数据]") + self.assertEqual(watch._scrub_binary("GET"), "GET") + + +class DetectorTests(unittest.TestCase): + def _detector(self, by_query, store): + fields = dict(watch.DEFAULT_FIELDS) + probe = watch.Detector(None, store, fields) + queries = {rule["name"]: rule["query"] for rule in probe.rules} + mapping = dict(by_query) + mapping.setdefault(queries["not_found"], []) + return watch.Detector(_StubClient(mapping), store, fields, window=300) + + def test_automation_is_downgraded(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + try: + probe = watch.Detector(None, store, dict(watch.DEFAULT_FIELDS)) + q = {rule["name"]: rule["query"] for rule in probe.rules} + by_query = { + "*": [("124.70.2.111", 666)], + q["login_attack"]: [("124.70.2.111", 333)], + q["login_burst"]: [("124.70.2.111", 666)], + q["not_found"]: [], + q["probe_java"]: [], + q["bad_ua"]: [], + q["bot_ua"]: [], + q["probe_config"]: [], + q["probe_cms"]: [], + q["probe_iot"]: [], + q["probe_devops"]: [], + q["probe_webshell"]: [], + q["exploit_payload"]: [], + 'client_ip:"124.70.2.111"': [ + ("/prod-api/biz/liveCourseChapter/msgSubscription", 666)], + } + detector = watch.Detector(_StubClient(by_query), store, + dict(watch.DEFAULT_FIELDS), window=300) + result = detector.run_once() + self.assertEqual(result["automation"], 1) + row = store.get("124.70.2.111") + self.assertEqual(row["tier"], watch.TIER_AUTOMATION) + self.assertTrue(any("单一接口" in r for r in row["reasons"])) + finally: + store.close() + + def test_low_volume_scanner_is_still_caught(self): + """A scanner with 18 requests must not need a top-volume slot.""" + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + try: + probe = watch.Detector(None, store, dict(watch.DEFAULT_FIELDS)) + q = {rule["name"]: rule["query"] for rule in probe.rules} + by_query = { + "*": [("10.185.9.41", 5000), ("202.201.16.87", 4000)], + q["probe_java"]: [("5.188.86.234", 18)], + } + detector = watch.Detector( + _StubClient(by_query, totals={"5.188.86.234": 18}), store, + dict(watch.DEFAULT_FIELDS), window=300) + result = detector.run_once() + self.assertEqual(result["malicious"], 1) + self.assertEqual(store.get("5.188.86.234")["tier"], watch.TIER_MALICIOUS) + finally: + store.close() + + def test_internal_and_allowlisted_are_skipped(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + try: + probe = watch.Detector(None, store, dict(watch.DEFAULT_FIELDS)) + q = {rule["name"]: rule["query"] for rule in probe.rules} + by_query = { + "*": [("10.185.9.41", 5000), ("203.0.113.5", 30), ("198.51.100.9", 30)], + q["probe_java"]: [("203.0.113.5", 30), ("198.51.100.9", 30)], + } + detector = watch.Detector( + _StubClient(by_query), store, dict(watch.DEFAULT_FIELDS), + window=300, allowlist=lambda ip: ip == "198.51.100.9") + detector.run_once() + self.assertIsNone(store.get("10.185.9.41")) + self.assertIsNone(store.get("198.51.100.9")) + self.assertEqual(store.get("203.0.113.5")["tier"], watch.TIER_MALICIOUS) + finally: + store.close() + + +class SampleTests(unittest.TestCase): + def test_sample_keeps_all_source_fields(self): + message = { + "timestamp": "2026-10-17T05:08:58.046Z", + "client_ip": "51.158.145.102", "client_port": 46990, + "server_ip": "10.185.9.90", "server_port": 80, + "host": "61.178.20.46:443", "bigip": "www.gsedu.edu.cn_2_4600", + "vs": "/Common/SmartEDU_https", "method": "GET", + "uri": "/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php", + "status": "403", "ua": "libredtail-http", + "client_ip_country_code": "FR", "rt": "0.031", + "message": 'event_type="req",client_ip="51.158.145.102"', + } + sample = watch._sample(message) + for key in ("client_port", "server_port", "bigip", "vs", "rt", + "client_ip_country_code", "message"): + self.assertIn(key, sample) + self.assertEqual(sample["client_port"], "46990") + + def test_sample_drops_empty_and_na_values(self): + sample = watch._sample({"client_ip": "1.2.3.4", + "client_ip_city_name": "N/A", + "referer": ""}) + self.assertEqual(sample, {"client_ip": "1.2.3.4"}) + + +class SnapshotTests(unittest.TestCase): + def test_compact_evidence_is_bounded(self): + evidence = { + "window": 89600, + "uris": [{"term": "/probe/%d" % i, "count": i} for i in range(30)], + "statuses": [{"term": str(code), "count": 1} for code in range(10)], + "methods": [{"term": "GET", "count": 1}], + "hosts": [{"term": "h", "count": 1}], + "server_ips": [{"term": "10.0.0.1", "count": 1}], + "vips": [{"term": "vip", "count": 1}], + "uas": [{"term": "ua", "count": 1}], + "samples": [{} for _ in range(5)], + "timeline": [{"timestamp": "t%d" % i, "count": 1} for i in range(200)], + "peak": {"timestamp": "t", "count": 3}, + } + compact = watch._compact_evidence(evidence) + self.assertTrue(compact["compact"]) + self.assertLessEqual(len(compact["uris"]), 10) + self.assertLessEqual(len(compact["timeline"]), 72) + self.assertEqual(compact["sample_count"], 5) + self.assertEqual(compact["window"], 89600) + + +class PathPrefixTests(unittest.TestCase): + def test_root_is_hidden_and_prefix_works(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + store.upsert({"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS, + "score": 91.0, "total": 120, "window": 3600, + "rules": {"probe_java": 4}, + "reasons": ["探测 4 次"]}, "2026-10-17T16:00:00Z") + store.sync_fused() + server = ThreadingHTTPServer( + ("127.0.0.1", 0), + watch.make_handler(store, 3600, 60, None, "/s/secret123")) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + base = "http://127.0.0.1:%d" % server.server_address[1] + try: + with self.assertRaises(urllib.error.HTTPError) as ctx: + urllib.request.urlopen(base + "/", timeout=5) + self.assertEqual(ctx.exception.code, 404) + + with urllib.request.urlopen(base + "/s/secret123/", timeout=5) as response: + page = response.read().decode("utf-8") + self.assertIn('const BASE = "/s/secret123"', page) + + with urllib.request.urlopen(base + "/s/secret123/api/ips", timeout=5) as response: + payload = json.loads(response.read().decode("utf-8")) + self.assertEqual(payload["count"], 1) + + with self.assertRaises(urllib.error.HTTPError): + urllib.request.urlopen(base + "/api/ips", timeout=5) + finally: + server.shutdown() + server.server_close() + store.close() + + +class StatusIndicatorTests(unittest.TestCase): + def test_status_endpoint(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + + class DetectorStub: + last_success = time.time() - 5 + last_error = None + last_run = "2026-10-17T16:00:00Z" + + server = ThreadingHTTPServer( + ("127.0.0.1", 0), + watch.make_handler(store, 3600, 60, DetectorStub())) + threading.Thread(target=server.serve_forever, daemon=True).start() + base = "http://127.0.0.1:%d" % server.server_address[1] + try: + with urllib.request.urlopen(base + "/api/status", timeout=5) as response: + payload = json.loads(response.read().decode("utf-8")) + self.assertEqual(payload["graylog"], "ok") + + DetectorStub.last_error = "connection refused" + DetectorStub.last_success = None + with urllib.request.urlopen(base + "/api/status", timeout=5) as response: + payload = json.loads(response.read().decode("utf-8")) + self.assertEqual(payload["graylog"], "error") + finally: + server.shutdown() + server.server_close() + store.close() + + +class DashboardTests(unittest.TestCase): + def test_api_and_html(self): + with tempfile.TemporaryDirectory() as directory: + store = watch.Store(os.path.join(directory, "ips.db")) + store.upsert({"ip": "203.0.113.9", "tier": watch.TIER_MALICIOUS, + "score": 91.0, "total": 120, "window": 3600, + "rules": {"sensitive_probe": 4}, + "reasons": ["敏感路径探测 4 次"]}, "2026-10-17T16:00:00Z") + store.upsert({"ip": "198.51.100.77", "tier": watch.TIER_MALICIOUS, + "score": 70.0, "total": 653344, "window": 3600, + "risk": "shared_egress", + "rules": {"sensitive_probe": 4}, + "reasons": ["疑似共享出口"]}, "2026-10-17T16:00:00Z") + store.sync_fused() + server = ThreadingHTTPServer(("127.0.0.1", 0), + watch.make_handler(store, 3600, 60)) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + base = "http://127.0.0.1:%d" % server.server_address[1] + try: + with urllib.request.urlopen(base + "/", timeout=5) as response: + page = response.read().decode("utf-8") + self.assertIn("恶意 IP 清单", page) + self.assertIn("/api/ips", page) + + with urllib.request.urlopen(base + "/api/ips", timeout=5) as response: + payload = json.loads(response.read().decode("utf-8")) + self.assertEqual(payload["count"], 2) + self.assertEqual({item["ip"] for item in payload["items"]}, + {"203.0.113.9", "198.51.100.77"}) + self.assertTrue(all(item["sources"] == "feature" + for item in payload["items"])) + + with urllib.request.urlopen(base + "/api/stats", timeout=5) as response: + stats = json.loads(response.read().decode("utf-8")) + self.assertEqual(stats["core"], 2) + self.assertNotIn("review", stats) + + with urllib.request.urlopen(base + "/api/ip/203.0.113.9", timeout=5) as response: + detail = json.loads(response.read().decode("utf-8")) + self.assertEqual(detail["item"]["tier"], watch.TIER_MALICIOUS) + self.assertFalse(detail["evidence"]["available"]) + + with urllib.request.urlopen(base + "/api/export.csv", timeout=5) as response: + body = response.read().decode("utf-8") + self.assertIn("text/csv", response.headers.get("Content-Type", "")) + self.assertIn("ip,tier,score", body) + self.assertIn("203.0.113.9", body) + self.assertIn("198.51.100.77", body) + + with urllib.request.urlopen(base + "/api/export.txt", timeout=5) as response: + text = response.read().decode("utf-8") + self.assertEqual(set(text.split()), + {"203.0.113.9", "198.51.100.77"}) + finally: + server.shutdown() + server.server_close() + store.close() + + +if __name__ == "__main__": + unittest.main()