Commit Jerboa knowledge data updates
ober
fea36172de83df2d1be6b772584e8c21b6f94434
--- a/data/anti-patterns.sexp +++ b/data/anti-patterns.sexp @@ -6046,4 +6046,88 @@ "jerboa_eval" "jerboa_verify" "jerboa_module_exports" - "jerboa_error_fix_lookup"))) + "jerboa_error_fix_lookup")) + (("advice" + . + "In jerboa-sinatra routes, the return value of the route body is interpreted by interpret-result. A (http-respond ...) value is a th:response-raw struct which is NOT one of the recognized response kinds, so interpret-result returns #f for the body and the response is silently dropped (client sees an empty 200 or a dropped connection). Always write the response through the current sinatra response: (status! n), (header! k v), (body! bytes/string), (halt).") + ("avoid" + . + "Returning (http-respond ...) (a th:response-raw) from a sinatra GET/POST route handler and expecting it to be sent to the client.") + ("id" . "sinatra-route-returns-http-respond-dropped") + ("kinds" "code") + ("pattern" . "http-respond|th:response-raw") + ("severity" . "high") + ("tags" "sinatra" "route" "http-respond" "response-raw" + "dropped-response" "interpret-result") + ("title" + . + "Sinatra route returning http-respond is silently dropped") + ("tools" + "jerboa_howto" + "jerboa_balanced_replace" + "jerboa_verify")) + (("advice" + . + "When re-match? returns #f: (1) diff each literal in the pattern against the input; (2) strip the pattern to the failing prefix and re-test; (3) run the same input+pattern twice to confirm stable output before blaming the engine. Only after the pattern is verified correct AND the result is genuinely inconsistent across identical calls should you switch to a char-by-char validator, and record the exact failing pattern.") + ("avoid" + . + "Do not conclude that re-match? / (std regex) is broken, flaky, or non-deterministic when a hand-written pattern returns #f. The usual cause is a stray literal character in the pattern (e.g. a '/' sitting between a quote and a group) or a quote mismatch, not the engine.") + ("id" . "regex-false-failure-blamed-on-engine") + ("kinds" "parsing" "verification" "debugging" "script") + ("pattern" . "re-match?.*#f") ("severity" . "medium") + ("tags" "regex" "re-match?" "std-regex" "debugging" + "false-diagnosis") + ("title" + . + "Blame the regex engine before checking the pattern string") + ("tools" "jerboa_eval" "jerboa_check_syntax")) + (("advice" + . + "Define a non-halting redirect in your app: (def (redirect path) (status! 302) (header! \"Location\" path) (void)). Then import (except (sinatra) redirect) so route handlers use your version and session cookies survive login/registration redirects.") + ("avoid" + . + "Using (redirect \"/path\") in a route that relies on sinatra's built-in sessions; the redirect helper calls (halt), which jumps past the save-session step so the Set-Cookie header is never written.") + ("id" . "sinatra-redirect-halts-session-save") + ("kinds" "code" "security") + ("pattern" . "(redirect .*)\n.*session-set!|save-session") + ("severity" . "high") + ("tags" "sinatra" "redirect" "halt" "session" "cookie" + "save-session") + ("title" . "Sinatra redirect halts and skips session save") + ("tools" + "jerboa_howto" + "jerboa_balanced_replace" + "jerboa_verify")) + (("advice" + . + "Shadow param in your app: import (except (sinatra) param), then define (def (param name) (or (hash-get (current-params) name) (if (memq (sinatra-request-method (request)) '(POST PUT PATCH DELETE)) (hash-get (sinatra-request-body-params (request)) name) #f))). Also consider shadowing redirect if you use sessions.") + ("avoid" + . + "Using (param \"name\") in POST routes and expecting it to return form fields. The vendored jerboa-sinatra seeds current-params with query params only; body params are never merged.") + ("id" . "sinatra-param-omits-body-params") ("kinds" "code") + ("pattern" . "(param \"") ("severity" . "high") + ("tags" "sinatra" "param" "body-params" "form" "POST" + "current-params") + ("title" + . + "Sinatra param only reads query params, not POST body") + ("tools" + "jerboa_howto" + "jerboa_balanced_replace" + "jerboa_verify")) + (("advice" + . + "Treat jsqlite as single-process. Run add-user/migrate while serve/buildd are stopped. For out-of-process workers, coordinate through the filesystem (e.g., rename(2) spool files) instead of the database. Document this limitation in the README.") + ("avoid" + . + "Running a CLI command (add-user, migrate) against a gitsite.db while the server or buildd is using it, or running two server processes against the same db-path. jsqlite keeps the database image in memory and writes the file on commit; a separate process opening the same file sees its own stale snapshot.") + ("id" . "jsqlite-multi-process-db-write") + ("kinds" "code" "operations") + ("pattern" . "db-init!|sqlite-open|gitsite.db") + ("severity" . "high") + ("tags" "jsqlite" "sqlite" "single-process" "db" "gitsite" + "spool") + ("title" + . + "jsqlite databases cannot be shared across processes") + ("tools" "jerboa_howto" "jerboa_anti_pattern_lookup"))) --- a/data/cookbooks.sexp +++ b/data/cookbooks.sexp @@ -8946,4 +8946,69 @@ ("tags" "mcp" "llm" "codegen" "verify" "prompt-to-code") ("title" . - "Generate verified Jerboa code with a local LLM endpoint"))) + "Generate verified Jerboa code with a local LLM endpoint")) + (("code" + . + "(import (sinatra context)\n (sinatra helpers)\n (sinatra request))\n\n;; From a sinatra route, get the RAW httpd request so you can read\n;; headers/method/query/body yourself (for a binary protocol like git):\n(def (raw-req) (sinatra-request-raw (request)))\n\n;; Register the route AFTER defining the handler; both GET and POST are\n;; needed for git smart-HTTP (info/refs GET, git-upload-pack POST).\n;; The '*' splat captures the suffix like /info/refs or /git-upload-pack.\n(def (git-route-handler)\n (def owner (param \"owner\"))\n (def name (strip-git-suffix (param \"name\")))\n (def suffix (let ([s (splat)]) (if (null? s) \"/\" (car s))))\n (git-bridge (current-user) (sinatra-request-raw (request)) owner name suffix))\n\n(GET \"/git/~:owner/:name*\" (git-route-handler))\n(POST \"/git/~:owner/:name*\" (git-route-handler))\n\n;; Inside git-bridge, write the response through sinatra's response API --\n;; DO NOT return (http-respond ...): that builds a th:response-raw object\n;; which sinatra's interpret-result silently DROPS (route returns void).\n(status! 200)\n(header! \"Content-Type\" \"application/x-git-upload-pack-result\")\n(body! bytevector)\n(halt)\n") ("id" . "sinatra-route-binary-cgi-bridge") + ("imports" "(jerboa prelude)" "(sinatra)" + "(sinatra context)" "(sinatra helpers)" "(sinatra request)") + ("notes" + . + "CRITICAL: a route body must be its own (lambda () ...), so internal defs must come FIRST in the body or be hoisted to top level (invalid context for definition otherwise). git-bridge must write via status!/header!/body!/halt into the current sinatra response; returning http-respond values is silently dropped by interpret-result, so the client sees an empty 200. read-body of thread-httpd is UTF-8-decoded string (binary-unsafe, 4MiB cap) so POST git bodies are buffered via temp files, and HTTPS push is rejected 403. sinatra-request-raw is exported from (sinatra request) but NOT re-exported by (sinatra) -- import it directly.") + ("tags" "sinatra" "git" "smart-http" "route" "bytevector" + "http-respond" "git-bridge" "cgi") + ("title" + . + "Wire a binary CGI bridge (git smart-HTTP) through a sinatra route")) + (("code" + . + "(import (only (std net httpsd) httpsd-start))\n\n;; In-process TLS termination for a sinatra app. When both tls-cert and\n;; tls-key are configured in the sexp config, start httpsd-start with the\n;; sinatra handler directly; otherwise RUN! plain HTTP.\n(let ([cert (cfg-tls-cert)]\n [key (cfg-tls-key)])\n (if (and cert key)\n (begin\n (set-option! 'force-headers\n '((\"Strict-Transport-Security\" . \"max-age=31536000; includeSubDomains\")))\n (set-option! 'session-secure #t)\n (httpsd-start (cfg-port) (sinatra-handler default-app) cert key))\n (begin\n (set-option! 'session-secure #f)\n (RUN!))))\n") ("id" . "sinatra-in-process-tls-httpsd") + ("imports" + "(jerboa prelude)" + "(sinatra)" + "(only (std net httpsd) httpsd-start)") + ("notes" + . + "httpsd-start takes (port handler cert key) -- 4+ args. Both httpsd-start and httpd-start-https exist; httpsd-start is verified working with (sinatra-handler default-app). Set session-secure #t when TLS (Secure cookies) and #f for plain HTTP (else browsers refuse the cookie and login breaks). sinatra-request-secure? only checks X-Forwarded-Proto when trust-proxy is on, NOT in-process TLS, so 'auto session-secure does not detect httpsd. Verified: openssl s_client handshake, HTTPS git clone, HSTS header present, Secure flag on cookie over TLS and absent over HTTP.") + ("tags" "sinatra" "tls" "httpsd-start" "https" "hsts" + "session-secure" "cert" "in-process") + ("title" . "In-process TLS for sinatra via httpsd-start")) + (("code" + . + "(import (jerboa prelude)\n (only (chezscheme) rename-file directory-list))\n\n;; Admission: write to <id>.sexp.tmp then rename, so workers never see a\n;; partially-written spec. The .tmp suffix also stops claim from grabbing it.\n(def (spool-enqueue! dir id spec)\n (let* ([name (str (number->string id) \".sexp\")]\n [tmp (path-join dir (str name \".tmp\"))]\n [final (path-join dir name)])\n (call-with-output-file tmp (lambda (p) (write spec p)))\n (rename-file tmp final)))\n\n;; Claim: rename(2) is atomic — exactly one worker wins; losers get an\n;; exception, so catch and move on. Lock-free across threads AND processes.\n(def (claim-spool-job! new-dir run-dir)\n (let loop ([names (directory-list new-dir)])\n (cond\n [(null? names) #f]\n [(not (string-suffix? \".sexp\" (car names))) (loop (cdr names))]\n [else\n (let ([name (car names)])\n (try\n (begin\n (rename-file (path-join new-dir name) (path-join run-dir name))\n (substring name 0 (- (string-length name) 5)))\n (catch (e) (loop (cdr names)))))])))\n\n;; Complete: move run/ -> done/, update any out-of-band state.\n(def (spool-done! run-dir done-dir stem)\n (try (rename-file (path-join run-dir (str stem \".sexp\"))\n (path-join done-dir (str stem \".sexp\")))\n (catch (e) (void))))") ("id" . "lockfree-file-spool-job-queue") + ("imports" + "(jerboa prelude)" + "(only (chezscheme) rename-file directory-list)") + ("notes" + . + "Use (only (chezscheme) rename-file directory-list) — a subset import avoids the multiple-definitions conflicts that raw (chezscheme) causes next to (jerboa prelude). rename-file and directory-list are NOT exported by the prelude. directory-list returns entry name strings. Chez rename-file raises on a missing source (good for claim) and replaces an existing target (use with care). Put the .tmp admission file in the SAME directory as the destination so the final rename stays on one filesystem. This is the jerboa-smtp queue-style pattern adapted for a buildd: DB rows stay the display layer while the filesystem is the coordination channel, which sidesteps single-process sqlite visibility limits. Seen in jerboa-gitsite builds.ss + worker.ss.") + ("tags" "spool" "queue" "rename-file" "atomic" "worker" + "concurrency") + ("title" + . + "Lock-free file-spool job queue via rename(2) atomic claim")) + (("code" + . + ";; Run a streaming git service (upload-pack/receive-pack) with inherited stdio\n;; and forward the child's real exit code. No shell, no captured output.\n(import (jerboa prelude))\n(import (only (std os fd) spawn-process process-wait process-exit-code))\n\n(def (run-git-service op rp)\n (def proc (spawn-process (list op rp))) ;; argv list, execvp-style\n (process-wait proc)\n (exit (or (process-exit-code proc) 1)))\n\n(run-git-service \"/usr/bin/git-upload-pack\" \"/srv/git/alice/repo.git\")\n") ("id" . "std-os-fd-spawn-process-git-services") + ("imports" "(std os fd)") + ("notes" + . + "(std misc process) run-process/run-process/batch CAPTURE stdout/stderr and return (values out err code) with stdin closed — wrong for git's binary protocol and useless for interactive/streaming children. process-wait/process-exit-code/process-exited? live in (std os fd), NOT (std misc process). spawn-process takes a flat argv list of strings, passes each element literally to execvp (shell metachars are data — no injection), and inherits stdin/stdout/stderr by default so the git pkt-line protocol flows unmangled. Requires libjerboa_native (aproc-native-spawn-available? -> #t). process-wait returns the raw wait status; read the exit code via (process-exit-code proc), then (exit code) to propagate it. If the child was killed by a signal, process-exit-code returns 128+N; use process-signaled?/process-signal to distinguish.") + ("tags" "subprocess" "spawn-process" "process-wait" "git" + "exec" "std-os-fd") + ("title" + . + "Run git services with spawn-process/process-wait from (std os fd)")) + (("code" + . + "(import (jerboa prelude))\n\n;; Chez's ~x format directive emits UPPERCASE hex digits:\n(format \"~2,'0x\" 254) ;; => \"FE\" (NOT \"fe\")\n(format \"~2,'0X\" 254) ;; => \"FE\" (uppercase too)\n\n;; So a hand-rolled bytes->hex from Python's hashlib.hexdigest() (lowercase)\n;; will NOT string=? match the same bytes hashed in Chez:\n(def (bytes->hex bv)\n (apply string-append\n (map (lambda (b) (format \"~2,'0x\" b))\n (bytevector->u8-list bv))))\n\n(def (sha256-hex bv)\n (string-downcase\n (apply string-append\n (map (lambda (b) (format \"~2,'0x\" b))\n (bytevector->u8-list (sha256-bytevector bv))))))\n\n;; Compare: sha256 of #vu8(0 97 115 109 1 0 0 0)\n;; bytes->hex (no downcase) => \"93A44BBB...\" uppercase\n;; sha256-hex (downcase) => \"93a44bbb...\" lowercase, matches Python\n(sha256-hex #vu8(0 97 115 109 1 0 0 0)) ;; => \"93a44bbb96c751218e4c00d479e4c14358122a389acca16205b1e4d0dc5f9476\"") ("id" . "chez-format-hex-uppercase") + ("imports" + . + "(jerboa prelude), (only (chezscheme) sha256-bytevector)") + ("notes" + . + "sha256-bytevector is available from (jerboa prelude) in scripts but NOT exported by the prelude library module — libraries must import it explicitly: (only (chezscheme) sha256-bytevector). Direct (chezscheme) imports are blocked at runtime in safe mode but DO work in WPO/unsafe-prelude build context, so a bad import can pass the build and fail at runtime. Always string-downcase (or string-upcase both sides) before comparing hex digests generated by different toolchains.") + ("tags" "format" "hex" "sha256" "string" "chezscheme") + ("title" + . + "Chez format ~x hex is UPPERCASE — downcase before comparing to Python hexdigest"))) --- a/data/error-fixes.sexp +++ b/data/error-fixes.sexp @@ -2283,9 +2283,7 @@ . "export form outside of a module or library .*src/jsh/ffi\\.ss.*while verifying.*jerboa-src/src/jsh/.*\\.ss") ("type" . "module-resolution")) - (("code_example" - . - "make chez\njerboa tests/test-core.ss") + (("code_example" . "make chez\njerboa tests/test-core.ss") ("explanation" . "MCP verifier tools use the repo-local Chez runtime internally. In a fresh checkout or client environment, that runtime may not have been bootstrapped yet.") @@ -3426,4 +3424,31 @@ ("pattern" . "native path component is group/world-writable") - ("type" . "runtime"))) + ("type" . "runtime")) + (("code_example" + . + "(def (setup-routes!)\n ;; WRONG: def after an expression\n (GET \"/x\" \"ok\")\n (def (helper) ...) ;; invalid context\n\n ;; RIGHT: hoist helper to top level\n (def (helper) ...) ;; top-level, above setup-routes!\n (def (setup-routes!)\n (GET \"/x\" \"ok\")\n (GET \"/git/~:owner/:name*\" (helper))))") + ("explanation" + . + "sinatra GET/POST route macros wrap the body in (lambda () ...). In a body, internal (def ...) must come before expressions; putting a def after a route registration like (GET ...) makes it appear mid-body and triggers invalid-context-for-definition.") + ("fix" + . + "Hoist the helper def to top level (or the top of the setup-routes! body, before any route registration). Route bodies that need multiple defs should define them all before any expression.") + ("id" . "sinatra-route-body-def-invalid-context") + ("pattern" + . + "invalid context for definition.*git-route-handler|invalid context for definition")) + (("code_example" + . + "(import (only (std os fd) spawn-process process-wait process-exit-code))\n(def p (spawn-process '(\"chmod\" \"600\" tmp))) ; NOT (system \"chmod 600 ...\")\n(process-wait p)\n(if (not (= (process-exit-code p) 0)) (error 'chmod \"failed\" tmp))") + ("explanation" + . + "LLMs reach for R7RS `system` out of habit. The prelude deliberately omits it; the shell-free argv path is the supported exec mechanism and avoids shell-injection findings from the security scanner.") + ("fix" + . + "`system` is not exported by (jerboa prelude) (it is R7RS, not Chez). Use a native argv exec: (import (only (std os fd) spawn-process process-wait process-exit-code)) then (process-wait (spawn-process (list \"cmd\" \"arg\"))) — shell metachars are treated as data. If you need captured output use (std misc process) run-process/run-process/batch. Use (std os shell) safe-system + shell-quote only when you genuinely need a shell string.") + ("id" . "system-unbound-in-prelude") + ("pattern" + . + "attempt to reference unbound identifier system") + ("type" . "Unbound Variable")))