Sensitive Data Scanner (SDS)

▸ How it works

▸ What this lets you do

• ✓ 170 built-in patterns spanning credentials, PII, financial, and infra
• ✓ Stop credentials, PII, and PHI from being persisted to your observability store
• ✓ Validator-aware redaction — false-positive 16-digit order IDs aren't touched
• ✓ NER for names + US addresses; pluggable ML adapter for true-ML recall
• ✓ Custom regex rules with ReDoS validation at config time, runtime timeout per call
• ✓ Live regex tester previews matches against pasted text before you enable a rule
• ✓ Cluster-wide kill switch flips in milliseconds via PubSub
• ✓ Per-project CPU budget keeps noisy regexes from starving other projects
• ✓ Findings durable: snapshots of name + pattern + action survive rule deletion
• ✓ GraphQL + REST APIs + live subscriptions for SIEM/SOAR integration
• ✓ Compliant by default — redaction happens before write, not after

▸ Get it running

1. 1 Enabled by default for every project (tier-bound)
2. 2 Open `SDS` → click `+ Seed built-in rules` to seed 170 patterns at once (high-noise ones default to disabled)
3. 3 Use the **Live tester** panel to validate regex patterns against pasted text before saving
4. 4 Add a custom regex or NER rule via `+ Custom rule` — pattern safety + scope toggles in the form
5. 5 Hits land in `sds_findings` AND fire on `/v1/findings` for unified review
6. 6 High/critical findings fan out to configured webhook destinations (Slack, PagerDuty, custom SIEM)
7. 7 Subscribe to `findingCreated` over GraphQL for live streaming to security dashboards

▸ Code examples

# The GitHub PAT will be matched by the `github_pat` rule.
# Default action is `redact`, so the stored log message reads
#   "leaked token: [REDACTED:github_pat]"
# and an `sds.finding.created` event fires to your webhooks.
curl -X POST https://funnel.example.com/v1/logs \
  -H "Authorization: Bearer st_YOUR_KEY" \
  -d '{
    "logs": [{
      "severity":     "error",
      "service_name": "api",
      "message":      "leaked token: ghp_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
    }]
  }'

▸ What we detect 170 patterns across 25 categories

Patterns are precompiled at module load · validators (Luhn / IBAN / ABA / entropy / JWT) filter false positives · NER catches free-form PII the regex layer misses · custom regex rules pass through ReDoS validation before they ever run.