Implement Gitleaks-inspired scanning features
ober
bfe01eaf0f3a1b2444779ffefb7e5dbe7bd4e091
new file mode 100644 --- /dev/null +++ b/GITLEAKS_IMPLEMENTATION_NOTES.md @@ -0,0 +1,80 @@ +# Gitleaks Implementation Notes + +This repo and `../gitleaks` solve the same problem at different scales. Gitsafe +should keep the fast, static hook shape, but several Gitleaks design choices are +worth adopting incrementally. + +## Implemented In This Pass + +- Wire parsed config fields into the scanner: + - `patterns.custom` now creates runtime patterns. + - `allowlist.files` now skips configured files or globs. + - `entropy: false` now disables entropy-dependent rules. +- Make pre-push hooks pass commit SHAs, not ref names. The scanner now handles + deleted refs and new branches conservatively. +- Add stable finding fingerprints in the form `file:rule-id:line`. +- Let `.gitsafeignore` ignore exact fingerprints as well as file globs. +- Let manual `gitsafe scan PATH...` recurse into directories. +- Add Gitleaks-style custom rule metadata: + - `secretGroup` selects an explicit capture group. + - `entropy` applies a per-rule entropy threshold. + - `path`/`paths` and `pathGlobs`/`path_globs` restrict rules by file path. + - `keywords` prefilter lines before regex matching. + - `tags` flow into text, JSON, and SARIF output. + - Rule `allowlists` suppress matches by paths, regexes, or stopwords. +- Add a scan-fragment abstraction shared by files, diff hunks, stdin, and + decoded candidates. +- Parse staged and push diffs with one git process per scan instead of + per-file `git diff` calls. +- Add optional recursive decoding for percent, unicode, hex, and base64 encoded + candidates via `max_decode_depth` / `--max-decode-depth`. +- Add `gitsafe stdin` for pipeline use. +- Add SARIF output for CI code-scanning integrations (`--format sarif`). +- Add baseline suppression from previous JSON or SARIF reports via + `.gitsafe.json` `baseline` or `--baseline`. + +## Future Hardening + +- Archive extraction and chunked large-file reads, following Gitleaks' + `sources/file.go`. +- Built-in rule metadata tables for keyword prefiltering on first-party rules. +- Support for Gitleaks `RequiredRules` semantics if a use case appears. + +## Gitleaks References + +- `config/rule.go`: rule metadata (`SecretGroup`, `Entropy`, `Keywords`, + `Path`, `Allowlists`, `RequiredRules`). +- `detect/detect.go`: keyword prefiltering, decoding loop, allowlist checks, + fingerprint suppression, baseline suppression. +- `sources/source.go` and `sources/fragment.go`: source/fragment interface. +- `sources/git.go`: streaming git diff/log source. +- `sources/file.go`: recursive file scanning, archive handling, binary sniffing, + and chunked reads. +- `report/sarif.go`: SARIF report shape. + +## Custom Pattern Schema + +`patterns.custom` accepts objects with this shape: + +```json +{ + "id": "internal-service-key", + "name": "Internal Service Key", + "severity": "high", + "pattern": "service_key\\s*=\\s*\"(ISK_[A-Za-z0-9]{32})\"", + "secretGroup": 1, + "entropy": 3.5, + "path": "^src/", + "keywords": ["service_key"], + "tags": ["internal", "service"], + "allowlists": [ + { + "stopwords": ["ISK_EXAMPLE"], + "regexTarget": "secret" + } + ], + "description": "Internal service authentication key" +} +``` + +`regex` is also accepted as an alias for `pattern`. --- a/Makefile +++ b/Makefile @@ -68,7 +68,7 @@ install: binary mkdir -p $(HOOK_DIR) printf '#!/bin/sh\nexec gitsafe pre-commit\n' > $(HOOK_DIR)/pre-commit chmod +x $(HOOK_DIR)/pre-commit - printf '#!/bin/sh\nwhile read local_ref local_sha remote_ref remote_sha; do\n gitsafe pre-push --local-ref "$$local_ref" --remote-ref "$$remote_ref" || exit $$?\ndone\n' > $(HOOK_DIR)/pre-push + printf '#!/bin/sh\nwhile read local_ref local_sha remote_ref remote_sha; do\n gitsafe pre-push --local-ref "$$local_sha" --remote-ref "$$remote_sha" || exit $$?\ndone\n' > $(HOOK_DIR)/pre-push chmod +x $(HOOK_DIR)/pre-push git config --global init.templateDir $(TEMPLATE_DIR) @echo "Installed gitsafe to $(BIN_DIR)/gitsafe + global git hooks." --- a/README.md +++ b/README.md @@ -120,7 +120,9 @@ Place a `.gitsafe.json` in any repo root to customize behavior for that project: "EXAMPLE_KEY", "YOUR_API_KEY_HERE" ] - } + }, + "max_decode_depth": 0, + "baseline": "gitsafe-baseline.json" } ``` @@ -131,10 +133,40 @@ Place a `.gitsafe.json` in any repo root to customize behavior for that project: | `severity` | `"medium"` | Minimum severity to report: `low`, `medium`, `high`, `critical` | | `entropy` | `true` | Enable Shannon entropy analysis for detecting random-looking strings | | `patterns.disabled` | `[]` | Pattern IDs to skip (e.g. `["high-entropy-hex", "jwt"]`) | -| `patterns.custom` | `[]` | Custom pattern definitions | +| `patterns.custom` | `[]` | Custom pattern definitions with optional Gitleaks-style metadata | | `exclude` | Lock files, docs, vendor dirs | Glob patterns for paths to skip | -| `allowlist.files` | `[]` | File paths to skip entirely | +| `allowlist.files` | `[]` | File paths or globs to skip entirely | | `allowlist.patterns` | `[]` | Known-safe strings to ignore (e.g. placeholder keys) | +| `max_decode_depth` | `0` | Recursively scan decoded percent, unicode, hex, and base64 candidates | +| `baseline` | unset | Previous JSON or SARIF report whose fingerprints should be suppressed | + +Custom patterns use this shape: + +```json +{ + "id": "internal-service-key", + "name": "Internal Service Key", + "severity": "high", + "pattern": "service_key\\s*=\\s*\"(ISK_[A-Za-z0-9]{32})\"", + "secretGroup": 1, + "entropy": 3.5, + "path": "^src/", + "keywords": ["service_key"], + "tags": ["internal", "service"], + "allowlists": [ + { + "stopwords": ["ISK_EXAMPLE"], + "regexTarget": "secret" + } + ], + "description": "Internal service authentication key" +} +``` + +`regex` is accepted as an alias for `pattern`. `path`/`paths` are regular +expressions against the file path; `pathGlobs`/`path_globs` are glob matches. +Rule allowlists support `paths`, `regexes`, `stopwords`, `regexTarget` +(`secret`, `match`, or `line`), and `condition` (`OR` or `AND`). ### Default Excludes @@ -152,15 +184,17 @@ gitsafe uses a multi-layered detection pipeline to catch leaked secrets with hig | [1. Pattern Match] -- regex against 28 known secret formats | - [2. Capture Extract] -- pull the secret value from capture groups + [2. Rule Metadata] -- path and keyword prefilters | - [3. Validator] -- pattern-specific checks (length, prefix, entropy) + [3. Capture Extract] -- pull the secret value from capture groups | - [4. Placeholder Filter] -- reject example/dummy values + [4. Validator] -- pattern-specific checks (length, prefix, entropy) | - [5. Allowlist Check] -- skip known-safe strings from config + [5. Placeholder Filter] -- reject example/dummy values | - [6. Inline Suppression] -- honor gitsafe:ignore comments + [6. Allowlist Check] -- skip known-safe strings from config or rule metadata + | + [7. Inline/Baseline Suppression] | FINDING ``` @@ -171,6 +205,11 @@ gitsafe ships with 28 built-in regex patterns organized by severity. Each patter When a pattern has capture groups, gitsafe extracts the last non-empty capture group as the matched value. This ensures validators receive just the secret (e.g. `AKIAIOSFODNN7EXAMPLE`) rather than the full match with surrounding context characters. +Custom patterns can override capture extraction with `secretGroup`, apply a +per-rule `entropy` threshold, restrict matches by `path`/`paths` or +`pathGlobs`, prefilter by `keywords`, attach `tags`, and define rule-local +`allowlists`. + ### 2. Validators Many patterns include a validator function that runs additional checks on the extracted match: @@ -210,8 +249,9 @@ Word-boundary matching prevents false negatives -- a real secret like `testX9mP2 gitsafe scans different content depending on how it's invoked: - **Pre-commit** (`gitsafe pre-commit`): Reads the git index via `git diff --cached`. For modified files, only added lines in diff hunks are scanned. For entirely new files, the full staged content is scanned. This means gitsafe only flags secrets you're about to commit, not existing content. -- **Pre-push** (`gitsafe pre-push`): Computes `git rev-list` for the commit range being pushed and scans the unified diff across all changed files in that range. -- **Manual scan** (`gitsafe scan PATH...`): Reads and scans entire file contents. +- **Pre-push** (`gitsafe pre-push`): Computes `git rev-list` for the commit range being pushed and scans one unified diff across all changed files in that range. +- **Manual scan** (`gitsafe scan PATH...`): Reads and scans entire file contents, recursing into directories. +- **Stdin scan** (`gitsafe stdin`): Reads content from standard input and scans it as `<stdin>`. In all modes, binary files (images, archives, compiled objects, `.lock` files) are skipped based on file extension. Paths matching exclude globs from `.gitsafe.json` are also skipped. @@ -223,7 +263,9 @@ Several layers work together to minimize noise: - **Env-var detection**: Lines like `os.getenv('API_KEY')` or `os.environ.get('TOKEN')` match generic patterns but the extracted values (`API_KEY'`) fail entropy checks. - **Long line skip**: Lines over 2000 characters (minified JS, generated code) are skipped entirely. - **Allowlist strings**: Known-safe values (e.g. `EXAMPLE_KEY`) in `.gitsafe.json` are ignored when found in any match. -- **`.gitsafeignore`**: A gitignore-style file for excluding paths from scanning. +- **Rule allowlists**: Custom rules can suppress matches by path, regex, or stopword. +- **`.gitsafeignore`**: A gitignore-style file for excluding paths, or exact finding fingerprints copied from gitsafe output. +- **Baselines**: `--baseline PATH` or `.gitsafe.json` `baseline` suppresses fingerprints from previous JSON or SARIF reports. - **Inline suppression**: `# gitsafe:ignore` or `// gitsafe:ignore=pattern-id` on a line suppresses that finding. ## What It Detects @@ -263,6 +305,8 @@ Once installed, gitsafe runs automatically on `git commit` and `git push`. If se ```bash gitsafe scan path/to/file.json gitsafe scan src/ +cat config.env | gitsafe stdin +gitsafe scan src/ --format sarif ``` ### CLI Options @@ -274,14 +318,19 @@ Modes: pre-commit Scan staged files (default) pre-push Scan commits being pushed scan PATH... Scan specific files or directories + stdin Scan content from stdin install Install git hooks in .git/hooks/ uninstall Remove gitsafe-installed hooks Options: --config PATH Path to .gitsafe.json (default: .gitsafe.json) - --format text|json Output format (default: text) + --format text|json|sarif + Output format (default: text) --severity LEVEL Minimum severity: low|medium|high|critical --no-entropy Disable entropy analysis + --max-decode-depth N + Recursively scan decoded percent/unicode/hex/base64 text + --baseline PATH Suppress findings from a previous JSON or SARIF report --verbose Show scan statistics --version Print version --help Print this help @@ -293,6 +342,8 @@ Options: - **Per-pattern:** `# gitsafe:ignore=aws-access-key` - **Per-file:** add paths to `exclude` in `.gitsafe.json` - **Per-string:** add known-safe values to `allowlist.patterns` in `.gitsafe.json` +- **Per-finding:** add a reported fingerprint like `config.env:generic-secret:12` to `.gitsafeignore` +- **Baseline:** run with `--baseline previous-report.json` to suppress already-known fingerprints ## Make Targets --- a/build-common.ss +++ b/build-common.ss @@ -75,6 +75,8 @@ "gitsafe/allowlist" "gitsafe/patterns" "gitsafe/git" + "gitsafe/source" + "gitsafe/decoder" "gitsafe/scanner" "gitsafe/output")) --- a/gitsafe/allowlist.ss +++ b/gitsafe/allowlist.ss @@ -3,7 +3,8 @@ (export line-suppressed? allowlisted? load-ignorefile - ignored-file?) + ignored-file? + ignored-fingerprint?) (import (except (chezscheme) make-hash-table hash-table? sort sort! @@ -56,14 +57,23 @@ (def (load-ignorefile (path ".gitsafeignore")) (if (not (file-exists? path)) '() - (filter (lambda (line) - (and (not (string-empty? line)) - (not (string-prefix? "#" line)))) - (read-file-lines path)))) + (filter-map + (lambda (line) + (let ([trimmed (string-trim line)]) + (if (or (string-empty? trimmed) + (string-prefix? "#" trimmed)) + #f + trimmed))) + (read-file-lines path)))) ;; --- Check if a file path matches any ignorefile pattern --- (def (ignored-file? path ignore-patterns) (any (lambda (glob) (glob-match? glob path)) ignore-patterns)) + ;; --- Check if a finding fingerprint is ignored exactly --- + (def (ignored-fingerprint? fingerprint ignore-patterns) + (any (lambda (entry) (string=? entry fingerprint)) + ignore-patterns)) + ) ;; end library --- a/gitsafe/config.ss +++ b/gitsafe/config.ss @@ -11,8 +11,11 @@ gitsafe-config-allowlist-strings gitsafe-config-max-file-size-mb gitsafe-config-ml-data-detection + gitsafe-config-max-decode-depth + gitsafe-config-baseline-fingerprints default-config load-config + load-baseline-fingerprints config-excluded? glob-match?) (import (except (chezscheme) @@ -40,6 +43,8 @@ allowlist-strings ;; list of literal strings known safe max-file-size-mb ;; integer: skip files larger than this (0 = no limit) ml-data-detection ;; boolean: drop entropy patterns on ML-data shapes + max-decode-depth ;; integer: recursive decoding depth (0 = disabled) + baseline-fingerprints ;; list of fingerprints from a previous report )) ;; --- Default configuration --- @@ -56,6 +61,8 @@ '() ;; allowlist-strings 10 ;; max-file-size-mb #t ;; ml-data-detection + 0 ;; max-decode-depth + '() ;; baseline-fingerprints )) ;; --- Glob matching --- @@ -123,6 +130,72 @@ [(list? v) (filter string? v)] [else '()])) + (def (json->list v) + (cond + [(vector? v) (vector->list v)] + [(list? v) v] + [else '()])) + + (def (json-ref obj key default) + (if (and (hash-table? obj) (hash-key? obj key)) + (hash-ref obj key default) + default)) + + (def (parse-nonnegative-int v default) + (cond + [(and (integer? v) (>= v 0)) v] + [(and (real? v) (>= v 0)) (exact (round v))] + [else default])) + + ;; --- Load baseline fingerprints from a previous JSON/SARIF report --- + ;; The native JSON report shape is `{ "findings": [...] }`; for SARIF we + ;; read `partialFingerprints.gitsafeFingerprint` from `runs[].results[]`. + (def (json-finding-fingerprint f) + (and (hash-table? f) + (let ([fp (json-ref f "fingerprint" #f)]) + (and (string? fp) fp)))) + + (def (sarif-result-fingerprint r) + (and (hash-table? r) + (let* ([partials (json-ref r "partialFingerprints" #f)] + [fp (json-ref partials "gitsafeFingerprint" #f)]) + (and (string? fp) fp)))) + + (def (native-report-fingerprints obj) + (cond + [(and (hash-table? obj) (hash-key? obj "findings")) + (filter-map json-finding-fingerprint + (json->list (hash-ref obj "findings" (vector))))] + [(or (vector? obj) (list? obj)) + (filter-map json-finding-fingerprint (json->list obj))] + [else '()])) + + (def (sarif-report-fingerprints obj) + (if (and (hash-table? obj) (hash-key? obj "runs")) + (let ([groups + (map (lambda (run) + (if (hash-table? run) + (filter-map sarif-result-fingerprint + (json->list (json-ref run "results" (vector)))) + '())) + (json->list (hash-ref obj "runs" (vector))))]) + (if (null? groups) + '() + (apply append groups))) + '())) + + (def (load-baseline-fingerprints path) + (if (or (not path) (not (file-exists? path))) + '() + (try + (let* ([obj (string->json-object (read-file-string path))] + [fps (append (native-report-fingerprints obj) + (sarif-report-fingerprints obj))]) + (unique fps)) + (catch (e) + (displayln "gitsafe: warning: could not parse baseline report, ignoring it") + '())))) + ;; --- Load config from file --- (def (load-config (path ".gitsafe.json")) (if (not (file-exists? path)) @@ -159,17 +232,22 @@ (json->string-list (hash-ref allowlist-obj "patterns" (vector))) '())] - [max-mb (let ([v (hash-ref obj "max_file_size_mb" 10)]) - (cond - [(and (integer? v) (>= v 0)) v] - [(and (real? v) (>= v 0)) (exact (round v))] - [else 10]))] + [max-mb (parse-nonnegative-int + (hash-ref obj "max_file_size_mb" 10) + 10)] [ml-detect (let ([v (hash-ref obj "detect_ml_data" #t)]) - (if (boolean? v) v #t))]) + (if (boolean? v) v #t))] + [decode-depth (parse-nonnegative-int + (hash-ref obj "max_decode_depth" 0) + 0)] + [baseline-path (hash-ref obj "baseline" #f)] + [baseline-fps (if (string? baseline-path) + (load-baseline-fingerprints baseline-path) + '())]) (make-gitsafe-config severity entropy disabled custom excludes al-files al-strs - max-mb ml-detect)) + max-mb ml-detect decode-depth baseline-fps)) (catch (e) (displayln "gitsafe: warning: could not parse .gitsafe.json, using defaults") (default-config))))) new file mode 100644 --- /dev/null +++ b/gitsafe/decoder.ss @@ -0,0 +1,196 @@ +#!chezscheme +(library (gitsafe decoder) + (export decode-once + decoded-variants) + (import (except (chezscheme) + make-hash-table hash-table? + sort sort! + printf fprintf + path-extension path-absolute? + with-input-from-string with-output-to-string + iota 1+ 1- + partition + make-date make-time) + (except (jerboa prelude) meta atom?) + (std regex) + (std misc string) + (only (std text base64) base64-string->u8vector) + (only (std text hex) hex-string->u8vector)) + + (def *base64-token-re* (re "[A-Za-z0-9+/_=-]{16,}")) + (def *hex-token-re* (re "[0-9A-Fa-f]{32,}")) + (def *percent-token-re* (re "(?:%[0-9A-Fa-f]{2}){2,}")) + (def *unicode-token-re* (re "(?:\\\\u[0-9A-Fa-f]{4}){2,}")) + + (def *max-decoded-chars* 200000) + + (def (regex-matches rx text) + (let ([len (string-length text)]) + (let loop ([start 0] [acc '()]) + (if (>= start len) + (reverse acc) + (let ([m (re-search rx text start)]) + (if (not m) + (reverse acc) + (let ([end (re-match-end m)]) + (loop (if (> end start) end (+ start 1)) + (cons (re-match-full m) acc))))))))) + + (def (printable-char? ch) + (or (char=? ch #\newline) + (char=? ch #\return) + (char=? ch #\tab) + (and (char>=? ch #\space) + (char<=? ch #\~)))) + + (def (mostly-printable? s) + (let ([len (string-length s)]) + (and (> len 0) + (<= len *max-decoded-chars*) + (let loop ([i 0] [printable 0]) + (cond + [(>= i len) + (>= (/ (inexact printable) (inexact len)) 0.85)] + [(printable-char? (string-ref s i)) + (loop (+ i 1) (+ printable 1))] + [else + (loop (+ i 1) printable)]))))) + + (def (decoded-ok? s original) + (and s + (string? s) + (not (string-empty? s)) + (not (string=? s original)) + (mostly-printable? s))) + + (def (bytevector->decoded-string bv original) + (try + (let ([s (utf8->string bv)]) + (and (decoded-ok? s original) s)) + (catch (e) #f))) + + (def (base64-normalize s) + (let ([out (open-output-string)]) + (let loop ([i 0]) + (when (< i (string-length s)) + (let ([ch (string-ref s i)]) + (put-char out + (cond + [(char=? ch #\-) #\+] + [(char=? ch #\_) #\/] + [else ch])) + (loop (+ i 1))))) + (let* ([base (get-output-string out)] + [rem (modulo (string-length base) 4)]) + (cond + [(= rem 0) base] + [(= rem 2) (string-append base "==")] + [(= rem 3) (string-append base "=")] + [else base])))) + + (def (decode-base64-token token) + (try + (bytevector->decoded-string + (base64-string->u8vector (base64-normalize token)) + token) + (catch (e) #f))) + + (def (decode-hex-token token) + (try + (and (even? (string-length token)) + (bytevector->decoded-string + (hex-string->u8vector token) + token)) + (catch (e) #f))) + + (def (hex-value ch) + (cond + [(and (char>=? ch #\0) (char<=? ch #\9)) + (- (char->integer ch) (char->integer #\0))] + [(and (char>=? ch #\a) (char<=? ch #\f)) + (+ 10 (- (char->integer ch) (char->integer #\a)))] + [(and (char>=? ch #\A) (char<=? ch #\F)) + (+ 10 (- (char->integer ch) (char->integer #\A)))] + [else #f])) + + (def (decode-percent-token token) + (try + (let ([out (open-output-string)] + [len (string-length token)]) + (let loop ([i 0]) + (when (< i len) + (if (and (<= (+ i 2) (- len 1)) + (char=? (string-ref token i) #\%)) + (let ([hi (hex-value (string-ref token (+ i 1)))] + [lo (hex-value (string-ref token (+ i 2)))]) + (if (and hi lo) + (begin + (put-char out + (integer->char + (+ (* hi 16) lo))) + (loop (+ i 3))) + (begin + (put-char out (string-ref token i)) + (loop (+ i 1))))) + (begin + (put-char out (string-ref token i)) + (loop (+ i 1)))))) + (let ([s (get-output-string out)]) + (and (decoded-ok? s token) s))) + (catch (e) #f))) + + (def (decode-unicode-token token) + (try + (let ([out (open-output-string)] + [len (string-length token)]) + (let loop ([i 0]) + (when (< i len) + (if (and (<= (+ i 5) (- len 1)) + (char=? (string-ref token i) #\\) + (char=? (string-ref token (+ i 1)) #\u)) + (let ([h0 (hex-value (string-ref token (+ i 2)))] + [h1 (hex-value (string-ref token (+ i 3)))] + [h2 (hex-value (string-ref token (+ i 4)))] + [h3 (hex-value (string-ref token (+ i 5)))]) + (if (and h0 h1 h2 h3) + (begin + (put-char out + (integer->char + (+ (* h0 4096) (* h1 256) (* h2 16) h3))) + (loop (+ i 6))) + (begin + (put-char out (string-ref token i)) + (loop (+ i 1))))) + (begin + (put-char out (string-ref token i)) + (loop (+ i 1)))))) + (let ([s (get-output-string out)]) + (and (decoded-ok? s token) s))) + (catch (e) #f))) + + (def (decode-once content) + (unique + (filter (lambda (s) (and s (string? s) (mostly-printable? s))) + (append + (filter-map decode-percent-token (regex-matches *percent-token-re* content)) + (filter-map decode-unicode-token (regex-matches *unicode-token-re* content)) + (filter-map decode-hex-token (regex-matches *hex-token-re* content)) + (filter-map decode-base64-token (regex-matches *base64-token-re* content)))))) + + (def (decoded-variants content depth) + (let loop ([remaining depth] + [frontier (list content)] + [seen (list content)] + [acc '()]) + (if (<= remaining 0) + (reverse acc) + (let* ([decoded (unique (apply append (map decode-once frontier)))] + [fresh (filter (lambda (s) (not (member s seen))) decoded)]) + (if (null? fresh) + (reverse acc) + (loop (- remaining 1) + fresh + (append fresh seen) + (append fresh acc))))))) + +) ;; end library --- a/gitsafe/git.ss +++ b/gitsafe/git.ss @@ -2,14 +2,18 @@ (library (gitsafe git) (export staged-files staged-diff + staged-diff-all staged-content staged-blob-size push-commits changed-files-in-range range-diff + range-diff-all range-blob-size git-repo? git-root + parse-unified-diff + parse-unified-diff-all make-diff-hunk diff-hunk? diff-hunk-file @@ -129,6 +133,72 @@ [else (loop rest hunks cur-hunk new-line-no)]))))) + (def (flush-hunk hunks cur-hunk) + (if (and cur-hunk + (not (null? (diff-hunk-lines cur-hunk)))) + (cons cur-hunk hunks) + hunks)) + + (def *diff-new-file-re* + (re "^\\+\\+\\+ b/(.*)$")) + + (def (parse-unified-diff-all diff-text) + (let loop ([lines (string-split diff-text #\newline)] + [current-file #f] + [hunks '()] + [cur-hunk #f] + [new-line-no 0]) + (if (null? lines) + (reverse (flush-hunk hunks cur-hunk)) + (let ([line (car lines)] + [rest (cdr lines)]) + (cond + [(re-search *diff-new-file-re* line) + => + (lambda (m) + (loop rest + (re-match-group m 1) + (flush-hunk hunks cur-hunk) + #f + 0))] + [(string=? line "+++ /dev/null") + (loop rest #f (flush-hunk hunks cur-hunk) #f 0)] + [(and current-file (re-search *hunk-header-re* line)) + => + (lambda (m) + (let ([new-start (string->number (re-match-group m 1))]) + (loop rest + current-file + (flush-hunk hunks cur-hunk) + (make-diff-hunk current-file 0 new-start '()) + new-start)))] + [(and cur-hunk + (> (string-length line) 0) + (char=? (string-ref line 0) #\+) + (not (string-prefix? "+++" line))) + (let ([content (substring line 1 (string-length line))] + [ln new-line-no]) + (loop rest + current-file + hunks + (make-diff-hunk + (diff-hunk-file cur-hunk) + (diff-hunk-old-start cur-hunk) + (diff-hunk-new-start cur-hunk) + (append (diff-hunk-lines cur-hunk) + (list (cons ln content)))) + (+ new-line-no 1)))] + [(and cur-hunk + (> (string-length line) 0) + (char=? (string-ref line 0) #\space)) + (loop rest current-file hunks cur-hunk (+ new-line-no 1))] + [(and cur-hunk + (> (string-length line) 0) + (char=? (string-ref line 0) #\-)) + (loop rest current-file hunks cur-hunk new-line-no)] + [else + (loop rest current-file hunks cur-hunk new-line-no)]))))) + ;; --- Public API --- ;; Returns #t if inside a git repository. @@ -161,6 +231,14 @@ '() (parse-unified-diff diff-text path)))) + ;; Returns all staged added-line hunks using one git process. + (def (staged-diff-all) + (let ([diff-text (git-output + '("git" "diff" "--cached" "-U0" "--diff-filter=ACMR"))]) + (if (string-empty? diff-text) + '() + (parse-unified-diff-all diff-text)))) + ;; Returns the full staged (index) content of a file. ;; This is what would be committed, not the working copy. (def (staged-content path) @@ -194,6 +272,15 @@ (def (range-diff from-ref to-ref path) (git-output (list "git" "diff" "-U0" from-ref to-ref "--" path))) + ;; Returns all added-line hunks in a commit range using one git process. + (def (range-diff-all from-ref to-ref) + (let ([diff-text (git-output + (list "git" "diff" "-U0" "--diff-filter=ACMR" + from-ref to-ref))]) + (if (string-empty? diff-text) + '() + (parse-unified-diff-all diff-text)))) + ;; Returns the size of the blob at to-ref for a file, or #f if unknown. (def (range-blob-size to-ref path) (let ([out (git-output (list "git" "cat-file" "-s" --- a/gitsafe/main-binary.ss +++ b/gitsafe/main-binary.ss @@ -60,7 +60,7 @@ (make-executable! (string-append hooks-dir "/pre-commit")) (install-hook! (string-append hooks-dir "/pre-push") - "#!/bin/sh\n# Installed by gitsafe\nwhile read local_ref local_sha remote_ref remote_sha; do\n gitsafe pre-push --local-ref \"$local_ref\" --remote-ref \"$remote_ref\" || exit $?\ndone\n") + "#!/bin/sh\n# Installed by gitsafe\nwhile read local_ref local_sha remote_ref remote_sha; do\n gitsafe pre-push --local-ref \"$local_sha\" --remote-ref \"$remote_sha\" || exit $?\ndone\n") (make-executable! (string-append hooks-dir "/pre-push")) (displayln "Done.")))) @@ -88,7 +88,8 @@ (def (run-scan findings format verbose?) (if (null? findings) (begin - (when verbose? (display-findings findings format verbose?)) + (when (or verbose? (not (string=? format "text"))) + (display-findings findings format verbose?)) (display-summary findings) (exit 0)) (begin @@ -111,6 +112,12 @@ (begin (displayln "gitsafe: error: no paths specified") (exit 2)) (run-scan (scan-files paths config) format verbose?))) +(def (cmd-stdin config format verbose?) + (run-scan + (scan-content "<stdin>" (get-string-all (current-input-port)) config) + format + verbose?)) + ;; --- Argument parsing (identical to main.ss) --- (def (parse-args args) @@ -123,44 +130,57 @@ [entropy #t] [verbose #f] [local-ref #f] - [remote-ref #f]) + [remote-ref #f] + [decode-depth #f] + [baseline-path #f]) (cond [(null? args) - (list mode paths cfg-path format severity entropy verbose local-ref remote-ref)] + (list mode paths cfg-path format severity entropy verbose local-ref remote-ref + decode-depth baseline-path)] - [(member (car args) '("pre-commit" "pre-push" "scan" "install" "uninstall")) - (loop (cdr args) (car args) paths cfg-path format severity entropy verbose local-ref remote-ref)] + [(member (car args) '("pre-commit" "pre-push" "scan" "stdin" "install" "uninstall")) + (loop (cdr args) (car args) paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path)] [(string=? (car args) "--config") (if (pair? (cdr args)) - (loop (cddr args) mode paths (cadr args) format severity entropy verbose local-ref remote-ref) - (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref))] + (loop (cddr args) mode paths (cadr args) format severity entropy verbose local-ref remote-ref decode-depth baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] [(string=? (car args) "--format") (if (pair? (cdr args)) - (loop (cddr args) mode paths cfg-path (cadr args) severity entropy verbose local-ref remote-ref) - (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref))] + (loop (cddr args) mode paths cfg-path (cadr args) severity entropy verbose local-ref remote-ref decode-depth baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] [(string=? (car args) "--severity") (if (pair? (cdr args)) - (loop (cddr args) mode paths cfg-path format (cadr args) entropy verbose local-ref remote-ref) - (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref))] + (loop (cddr args) mode paths cfg-path format (cadr args) entropy verbose local-ref remote-ref decode-depth baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] [(string=? (car args) "--no-entropy") - (loop (cdr args) mode paths cfg-path format severity #f verbose local-ref remote-ref)] + (loop (cdr args) mode paths cfg-path format severity #f verbose local-ref remote-ref decode-depth baseline-path)] [(string=? (car args) "--verbose") - (loop (cdr args) mode paths cfg-path format severity entropy #t local-ref remote-ref)] + (loop (cdr args) mode paths cfg-path format severity entropy #t local-ref remote-ref decode-depth baseline-path)] [(string=? (car args) "--local-ref") (if (pair? (cdr args)) - (loop (cddr args) mode paths cfg-path format severity entropy verbose (cadr args) remote-ref) - (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref))] + (loop (cddr args) mode paths cfg-path format severity entropy verbose (cadr args) remote-ref decode-depth baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] [(string=? (car args) "--remote-ref") (if (pair? (cdr args)) - (loop (cddr args) mode paths cfg-path format severity entropy verbose local-ref (cadr args)) - (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref))] + (loop (cddr args) mode paths cfg-path format severity entropy verbose local-ref (cadr args) decode-depth baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] + + [(string=? (car args) "--max-decode-depth") + (if (pair? (cdr args)) + (loop (cddr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref (cadr args) baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] + + [(string=? (car args) "--baseline") + (if (pair? (cdr args)) + (loop (cddr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth (cadr args)) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] [(string=? (car args) "--version") (displayln "gitsafe " *gitsafe-version*) @@ -174,14 +194,19 @@ Modes: pre-commit Scan staged files (default) pre-push Scan commits being pushed scan PATH... Scan specific files or directories + stdin Scan content from stdin install Install git hooks in .git/hooks/ uninstall Remove git hooks Options: --config PATH Path to .gitsafe.json (default: .gitsafe.json) - --format text|json Output format (default: text) + --format text|json|sarif + Output format (default: text) --severity LEVEL Minimum severity: low|medium|high|critical (default: medium) --no-entropy Disable entropy analysis + --max-decode-depth N + Recursively scan decoded percent/unicode/hex/base64 text + --baseline PATH Suppress findings from a previous JSON or SARIF report --verbose Show scan statistics --version Print version --help Print this help @@ -190,7 +215,13 @@ Options: [else (loop (cdr args) mode (append paths (list (car args))) - cfg-path format severity entropy verbose local-ref remote-ref)]))) + cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path)]))) + +(def (parse-decode-depth s default) + (let ([n (and s (string->number s))]) + (if (and n (integer? n) (>= n 0)) + n + default))) ;; --- Entry point --- @@ -205,6 +236,8 @@ Options: [verbose (list-ref parsed 6)] [local-ref (list-ref parsed 7)] [remote-ref (list-ref parsed 8)] + [decode-depth (list-ref parsed 9)] + [baseline-path (list-ref parsed 10)] [config (let ([c (load-config cfg-path)]) (make-gitsafe-config (if severity @@ -224,7 +257,15 @@ Options: (gitsafe-config-allowlist-files c) (gitsafe-config-allowlist-strings c) (gitsafe-config-max-file-size-mb c) - (gitsafe-config-ml-data-detection c)))]) + (gitsafe-config-ml-data-detection c) + (parse-decode-depth decode-depth + (gitsafe-config-max-decode-depth c)) + (unique + (append + (gitsafe-config-baseline-fingerprints c) + (if baseline-path + (load-baseline-fingerprints baseline-path) + '())))))]) (match mode ["install" (cmd-install)] ["uninstall" (cmd-uninstall)] @@ -235,6 +276,8 @@ Options: (cmd-pre-push lr rr config format verbose))] ["scan" (cmd-scan paths config format verbose)] + ["stdin" + (cmd-stdin config format verbose)] [_ (displayln "gitsafe: unknown mode: " mode) (exit 2)])) --- a/gitsafe/main.ss +++ b/gitsafe/main.ss @@ -78,7 +78,7 @@ (make-executable! (string-append hooks-dir "/pre-commit")) (install-hook! (string-append hooks-dir "/pre-push") - "#!/bin/sh\n# Installed by gitsafe\nwhile read local_ref local_sha remote_ref remote_sha; do\n gitsafe pre-push --local-ref \"$local_ref\" --remote-ref \"$remote_ref\" || exit $?\ndone\n") + "#!/bin/sh\n# Installed by gitsafe\nwhile read local_ref local_sha remote_ref remote_sha; do\n gitsafe pre-push --local-ref \"$local_sha\" --remote-ref \"$remote_sha\" || exit $?\ndone\n") (make-executable! (string-append hooks-dir "/pre-push")) (displayln "Done.")))) @@ -106,7 +106,8 @@ (def (run-scan findings format verbose?) (if (null? findings) (begin - (when verbose? (display-findings findings format verbose?)) + (when (or verbose? (not (string=? format "text"))) + (display-findings findings format verbose?)) (display-summary findings) (exit 0)) (begin @@ -129,6 +130,12 @@ (begin (displayln "gitsafe: error: no paths specified") (exit 2)) (run-scan (scan-files paths config) format verbose?))) +(def (cmd-stdin config format verbose?) + (run-scan + (scan-content "<stdin>" (get-string-all (current-input-port)) config) + format + verbose?)) + ;; --- Argument parsing --- (def (parse-args args) @@ -141,44 +148,57 @@ [entropy #t] [verbose #f] [local-ref #f] - [remote-ref #f]) + [remote-ref #f] + [decode-depth #f] + [baseline-path #f]) (cond [(null? args) - (list mode paths cfg-path format severity entropy verbose local-ref remote-ref)] + (list mode paths cfg-path format severity entropy verbose local-ref remote-ref + decode-depth baseline-path)] - [(member (car args) '("pre-commit" "pre-push" "scan" "install" "uninstall")) - (loop (cdr args) (car args) paths cfg-path format severity entropy verbose local-ref remote-ref)] + [(member (car args) '("pre-commit" "pre-push" "scan" "stdin" "install" "uninstall")) + (loop (cdr args) (car args) paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path)] [(string=? (car args) "--config") (if (pair? (cdr args)) - (loop (cddr args) mode paths (cadr args) format severity entropy verbose local-ref remote-ref) - (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref))] + (loop (cddr args) mode paths (cadr args) format severity entropy verbose local-ref remote-ref decode-depth baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] [(string=? (car args) "--format") (if (pair? (cdr args)) - (loop (cddr args) mode paths cfg-path (cadr args) severity entropy verbose local-ref remote-ref) - (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref))] + (loop (cddr args) mode paths cfg-path (cadr args) severity entropy verbose local-ref remote-ref decode-depth baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))] [(string=? (car args) "--severity") (if (pair? (cdr args)) - (loop (cddr args) mode paths cfg-path format (cadr args) entropy verbose local-ref remote-ref) - (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref))] + (loop (cddr args) mode paths cfg-path format (cadr args) entropy verbose local-ref remote-ref decode-depth baseline-path) + (loop (cdr args) mode paths cfg-path format severity entropy verbose local-ref remote-ref decode-depth baseline-path))]