λgitsite

~ober/jerboa-gitsafe

Imported from ~/mine/jerboa-gitsafe

download snapshot

about

# gitsafe

A fast, zero-dependency git hook that blocks commits and pushes containing leaked secrets, API keys, tokens, and credentials. Compiles to a single static binary — no runtime dependencies on the target machine.

## Security And Release Status

This repo is preview security tooling until the release checklist is complete. Local release gates:

```bash
make verify
make release-evidence
```

Security notes are in `SECURITY.md`, `docs/threat-model.md`, `docs/security-hardening.md`, `docs/corpus-policy.md`, and `docs/release-evidence.md`.

## Quick Start

```bash
git clone https://git.jerboa.sh/ober/jerboa-gitsafe
cd jerboa-gitsafe
make binary install
```

This builds a native binary and installs it to `~/.local/bin/gitsafe`, then configures global git hooks so every new repo is protected automatically. No Jerboa checkout needed: if `jerbuild` isn't already installed, the build downloads the prebuilt Jerboa toolchain for your os/arch into `.jerboa/bin` (macOS arm64, Linux amd64/arm64, FreeBSD amd64). The only requirement is a C compiler.

## Install

```bash
make install
```

This builds the binary (fetching the Jerboa toolchain on first run if needed) and then:
- Copies the binary to `~/.local/bin/gitsafe`
- Creates pre-commit and pre-push hooks in `~/.git-templates/hooks/`
- Sets `git config --global init.templateDir ~/.git-templates`

If a global template hook already exists and does not contain the
`Installed by gitsafe` marker, `make install` refuses to overwrite it. Back up or
merge that hook manually, or use per-repo `gitsafe install`.

Make sure `~/.local/bin` is on your `PATH`:

```bash
# Add to ~/.bashrc or ~/.zshrc if not already present
export PATH="$HOME/.local/bin:$PATH"
```

### Build only (no install)

```bash
make binary                # Native build → ./gitsafe-bin
make linux-amd64           # Linux x86_64 musl static → ./gitsafe-linux-amd64
make linux-arm64           # Linux arm64 musl static → ./gitsafe-linux-arm64
make freebsd-amd64         # FreeBSD amd64 dynamic → ./gitsafe-freebsd-amd64
make verify                # Security checks, tests, and binary build
make release-evidence      # Capture release evidence under dist/release-evidence/
```

To pin a toolchain explicitly: `make binary JERBUILD=/path/to/jerbuild` or `JERBOA_VERSION=<tag>`.
Cross targets use `JERBOA_HOME` for Chez cross prefixes and still run `ensure-jerboa-tools`, so a missing `jerbuild` is bootstrapped the same way as native builds.
Production support also requires a reviewed `JSAFE_TARGET_PROOF_FILE` with the
markers documented in `docs/release-evidence.md`. `JSAFE_REQUIRE_TARGET_PROOF=1`
fails closed when that target proof is missing, incomplete, or contains secret
material.

## Setting Up Existing Repos

`make install` configures git's global template directory, which applies to all **new** repos (`git init` / `git clone`). To add gitsafe hooks to repos that already exist, you have two options:

### Option 1: Re-initialize (safe, non-destructive)

Run `git init` inside existing repos. This copies the template hooks without touching your code, branches, or history:

```bash
# Single repo
cd ~/projects/my-repo && git init

# All repos under a directory
find ~/projects -name .git -type d -execdir git init \;
```

### Option 2: Per-repo install

```bash
cd ~/projects/my-repo
gitsafe install
```

This writes gitsafe hooks directly into `.git/hooks/` for that repo.

### Option 3: Per-repo uninstall

```bash
cd ~/projects/my-repo
gitsafe uninstall
```

Removes only hooks that were installed by gitsafe. Leaves other hooks untouched.

## Global Configuration

The global git template is set during `make install`:

```
~/.git-templates/hooks/pre-commit   → exec gitsafe pre-commit
~/.git-templates/hooks/pre-push     → exec gitsafe pre-push
```

Both generated hooks include an `Installed by gitsafe` marker so future gitsafe
installs can update them without overwriting unrelated hook content.

You can verify this is active:

```bash
git config --global init.templateDir
# → /home/you/.git-templates
```

## Per-Repo Configuration

Place a `.gitsafe.json` in any repo root to customize behavior for that project:

```json
{
  "severity": "medium",
  "entropy": true,
  "patterns": {
    "disabled": ["high-entropy-hex"],
    "custom": []
  },
  "exclude": [
    "*.lock",
    "go.sum",
    "*.md",
    "vendor/**",
    "node_modules/**",
    "*.min.js",
    "*.min.css",
    "test/fixtures/**"
  ],
  "allowlist": {
    "files": [],
    "patterns": [
      "EXAMPLE_KEY",
      "YOUR_API_KEY_HERE"
    ]
  },
  "max_decode_depth": 0,
  "baseline": "gitsafe-baseline.json"
}
```

### Config Fields

| Field | Default | Description |
|---|---|---|
| `severity` | `"medium"` | Minimum severity to report: `low`, `medium`, `high`, `critical` |
| `entropy` | `true` | Enable Shannon entropy analysis for detecting random-looking strings |
| `patterns.disabled` | `[]` | Pattern IDs to skip (e.g. `["high-entropy-hex", "jwt"]`) |
| `patterns.custom` | `[]` | Custom pattern definitions with optional Gitleaks-style metadata |
| `exclude` | Lock files, docs, vendor dirs | Glob patterns for paths to skip |
| `allowlist.files` | `[]` | File paths or globs to skip entirely |
| `allowlist.patterns` | `[]` | Known-safe strings to ignore (e.g. placeholder keys) |
| `max_decode_depth` | `0` | Recursively scan decoded percent, unicode, hex, and base64 candidates |
| `baseline` | unset | Previous JSON or SARIF report whose fingerprints should be suppressed |

Custom patterns use this shape:

```json
{
  "id": "internal-service-key",
  "name": "Internal Service Key",
  "severity": "high",
  "pattern": "service_key\\s*=\\s*\"(ISK_[A-Za-z0-9]{32})\"",
  "secretGroup": 1,
  "entropy": 3.5,
  "path": "^src/",
  "keywords": ["service_key"],
  "tags": ["internal", "service"],
  "allowlists": [
    {
      "stopwords": ["ISK_EXAMPLE"],
      "regexTarget": "secret"
    }
  ],
  "description": "Internal service authentication key"
}
```

`regex` is accepted as an alias for `pattern`. `path`/`paths` are regular
expressions against the file path; `pathGlobs`/`path_globs` are glob matches.
Rule allowlists support `paths`, `regexes`, `stopwords`, `regexTarget`
(`secret`, `match`, or `line`), and `condition` (`OR` or `AND`).

### Default Excludes

Without a `.gitsafe.json`, these paths are excluded automatically:
`*.lock`, `go.sum`, `*.md`, `vendor/**`, `node_modules/**`, `*.min.js`, `*.min.css`

## How It Works

gitsafe uses a multi-layered detection pipeline to catch leaked secrets with high accuracy and low false positives. Each line of scanned content passes through several stages before a finding is reported.

### Detection Pipeline

```
  line of code
       |
  [1. Pattern Match]     -- regex against 28 known secret formats
       |
  [2. Rule Metadata]     -- path and keyword prefilters
       |
  [3. Capture Extract]   -- pull the secret value from capture groups
       |
  [4. Validator]         -- pattern-specific checks (length, prefix, entropy)
       |
  [5. Placeholder Filter] -- reject example/dummy values
       |
  [6. Allowlist Check]   -- skip known-safe strings from config or rule metadata
       |
  [7. Inline/Baseline Suppression]
       |
    FINDING
```

### 1. Pattern Matching

gitsafe ships with 28 built-in regex patterns organized by severity. Each pattern targets a specific secret format (e.g. `AKIA` prefix for AWS keys, `ghp_` for GitHub PATs, `eyJ` for JWTs). Patterns are compiled once at startup and matched against every line of scanned content.

When a pattern has capture groups, gitsafe extracts the last non-empty capture group as the matched value. This ensures validators receive just the secret (e.g. `AKIAIOSFODNN7EXAMPLE`) rather than the full match with surrounding context characters.

Custom patterns can override capture extraction with `secretGroup`, apply a
per-rule `entropy` threshold, restrict matches by `path`/`paths` or
`pathGlobs`, prefilter by `keywords`, attach `tags`, and define rule-local
`allowlists`.

### 2. Validators

Many patterns include a validator function that runs additional checks on the extracted match:

- **Length checks** -- AWS access keys must be exactly 20 characters, Google API keys exactly 39
- **Prefix checks** -- Anthropic keys must start with `sk-ant-`, GitHub PATs with `gh`
- **Marker checks** -- OpenAI keys must contain the `T3BlbkFJ` marker
- **Entropy checks** -- Generic patterns (API key assignments, bearer tokens) require the matched value to exceed a Shannon entropy threshold, rejecting low-randomness values that are unlikely to be real secrets
- **Placeholder rejection** -- URL credential patterns reject common test values like `user:password@localhost`

### 3. Shannon Entropy Analysis

For patterns that match broad formats (generic key assignments, high-entropy hex/base64), gitsafe computes the [Shannon entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)) of the matched string -- a measure of randomness in bits per character. Real secrets tend to have high entropy; configuration values and identifiers tend to have low entropy.

Thresholds vary by character set:

| Character Set | Threshold | Example |
|---|---|---|
| Hex (`0-9a-f`) | 3.0 bits | `deadbeef1234...` |
| Base64 (`A-Za-z0-9+/=`) | 4.0 bits | `SGVsbG8gV29y...` |
| Generic (validator-level) | 3.0--3.5 bits | Used by bearer tokens, API key assignments |

A string like `"aaaaaaaaaaaaaaaa"` has 0.0 bits of entropy and is ignored. A string like `"xK9mP2nQrT5vWy8zA3bCdEfGhJk"` has >4.0 bits and triggers a finding.

### 4. Placeholder Filtering

The `not-placeholder?` filter rejects common test and example values that structurally match secret formats but aren't real credentials. It catches:

- Repeated single characters (`XXXXXXXX`, `aaaaaaaa`)
- Strings shorter than 8 characters
- Placeholder words at word boundaries: `example`, `placeholder`, `dummy`, `sample`, `your-api-key`, `replace-me`, `change-me`, `insert-here`

Word-boundary matching prevents false negatives -- a real secret like `testX9mP2nQrT5vWy8zA3b` won't be rejected just because it contains the substring "test".

### 5. Scan Modes

gitsafe scans different content depending on how it's invoked:

- **Pre-commit** (`gitsafe pre-commit`): Reads the git index via `git diff --cached`. For modified files, only added lines in diff hunks are scanned. For entirely new files, the full staged content is scanned. This means gitsafe only flags secrets you're about to commit, not existing content.
- **Pre-push** (`gitsafe pre-push`): Computes `git rev-list` for the commit range being pushed and scans one unified diff across all changed files in that range.
- **Manual scan** (`gitsafe scan PATH...`): Reads and scans entire file contents, recursing into directories.
- **Stdin scan** (`gitsafe stdin`): Reads content from standard input and scans it as `<stdin>`.

In all modes, binary files (images, archives, compiled objects, `.lock` files) are skipped based on file extension. Paths matching exclude globs from `.gitsafe.json` are also skipped.

### 6. False Positive Reduction

Several layers work together to minimize noise:

- **Severity filtering**: Only patterns at or above the configured severity level are active (default: `medium`). Set to `critical` to only catch the most dangerous leaks.
- **Env-var detection**: Lines like `os.getenv('API_KEY')` or `os.environ.get('TOKEN')` match generic patterns but the extracted values (`API_KEY'`) fail entropy checks.
- **Long line skip**: Lines over 2000 characters (minified JS, generated code) are skipped entirely.
- **Allowlist strings**: Known-safe values (e.g. `EXAMPLE_KEY`) in `.gitsafe.json` are ignored when found in any match.
- **Rule allowlists**: Custom rules can suppress matches by path, regex, or stopword.
- **`.gitsafeignore`**: A gitignore-style file for excluding paths, or exact finding fingerprints copied from gitsafe output.
- **Baselines**: `--baseline PATH` or `.gitsafe.json` `baseline` suppresses fingerprints from previous JSON or SARIF reports.
- **Inline suppression**: `# gitsafe:ignore` or `// gitsafe:ignore=pattern-id` on a line suppresses that finding.

## What It Detects

### Critical
- AWS Access Key IDs (`AKIA...`) and Secret Access Keys
- GitHub Personal Access Tokens (classic `ghp_` and fine-grained `github_pat_`)
- OpenAI API Keys (standard, project-scoped `sk-proj-`, and service account `sk-svcacct-`)
- Anthropic API Keys (`sk-ant-`)
- Stripe Secret and Restricted Keys (`sk_live_`, `rk_live_`)
- PEM-encoded Private Keys (RSA, DSA, EC, OpenSSH, PGP, encrypted)
- PuTTY Private Keys (PPK format)

### High
- Generic API key/secret/token/password assignments (with entropy validation)
- Bearer tokens
- Slack tokens (`xoxb-`, `xoxp-`, etc.) and webhook URLs
- Google API keys (`AIza...`)
- Twilio, SendGrid, Mailgun API keys
- NPM and PyPI tokens
- JWTs (`eyJ...` three-part base64url)
- Credentials embedded in URLs (with placeholder rejection)

### Medium
- Database connection strings with credentials
- High-entropy hex strings (40+ chars)
- High-entropy base64 strings (40+ chars)

## Usage

### As git hooks (automatic)

Once installed, gitsafe runs automatically on `git commit` and `git push`. If secrets are found, the operation is blocked and findings are printed.

### Manual scan

```bash
gitsafe scan path/to/file.json
gitsafe scan src/
cat config.env | gitsafe stdin
gitsafe scan src/ --format sarif
```

### CLI Options

```
gitsafe [MODE] [OPTIONS]

Modes:
  pre-commit         Scan staged files (default)
  pre-push           Scan commits being pushed
  scan PATH...       Scan specific files or directories
  stdin              Scan content from stdin
  install            Install git hooks in .git/hooks/
  uninstall          Remove gitsafe-installed hooks

Options:
  --config PATH      Path to .gitsafe.json (default: .gitsafe.json)
  --format text|json|sarif
                     Output format (default: text)
  --severity LEVEL   Minimum severity: low|medium|high|critical
  --no-entropy       Disable entropy analysis
  --max-decode-depth N
                     Recursively scan decoded percent/unicode/hex/base64 text
  --baseline PATH    Suppress findings from a previous JSON or SARIF report
  --verbose          Show scan statistics
  --version          Print version
  --help             Print this help
```

## Suppression

- **Inline:** append `# gitsafe:ignore` to a line
- **Per-pattern:** `# gitsafe:ignore=aws-access-key`
- **Per-file:** add paths to `exclude` in `.gitsafe.json`
- **Per-string:** add known-safe values to `allowlist.patterns` in `.gitsafe.json`
- **Per-finding:** add a reported fingerprint like `config.env:generic-secret:12` to `.gitsafeignore`
- **Baseline:** run with `--baseline previous-report.json` to suppress already-known fingerprints

## Make Targets

```
make binary                 Native build → ./gitsafe-bin (auto-downloads jerbuild)
make linux-amd64            Cross-build Linux x86_64 musl static binary
make linux-arm64            Cross-build Linux arm64 musl static binary
make freebsd-amd64          Cross-build FreeBSD amd64 dynamic binary
make install                Build + install to ~/.local/bin + global hooks
make run ARGS='...'         Build + run ./gitsafe-bin
make test                   Run test suite
make clean                  Remove all build artifacts
```

recent commits