data: add cookbook recipes + feature note from session work
ober
caf48a998155c43e2696117ea05f49692752d601
--- a/data/cookbooks.sexp +++ b/data/cookbooks.sexp @@ -4850,4 +4850,51 @@ "prelude" "guard") ("title" . - "match: bind a predicate-checked value with (and (? pred) var)"))) + "match: bind a predicate-checked value with (and (? pred) var)")) + (("code" + . + ";; Extending an immutable map: copying a hashtable per extension is O(n) and\n;; allocates a whole new table each time. For SMALL maps extended many times\n;; (query bindings, interpreter environments), an alist is far cheaper:\n;; O(1) cons to extend, assq to look up.\n\n;; SLOW (whole-map copy per extension):\n;; (define (m-set m k v) (let ([h (hashtable-copy m #t)]) (hashtable-set! h k v) h))\n\n;; FAST:\n(define (m-empty) '())\n(define (m-ref m k) (let ([p (assq k m)]) (and p (cdr p))))\n(define (m-set m k v) (cons (cons k v) m)) ; O(1), structural sharing") ("id" . "alist-vs-hashtable-short-lived-maps") ("imports") + ("notes" + . + "Discovered building a Datalog join engine: switching per-result bindings from copied hashtables to alists roughly HALVED query time on deep joins (the hashtable-copy per extension dominated, ~655k copies). assq is O(keys) but query/env maps have few keys, so it wins. Only add NEW keys (no shadowing) to keep the alist dup-free; dedup when merging two maps. Cross back to a real hashtable when maps get large (dozens+ keys) or are read far more often than extended.") + ("tags" "performance" "alist" "hashtable" "bindings" + "environment" "allocation") + ("title" + . + "Use alists, not copied hashtables, for many short-lived incrementally-extended maps")) + (("code" + . + ";; Store a column unboxed when homogeneous: flvector for flonums, fxvector for\n;; fixnums, boxed vector otherwise. Scans/reductions avoid per-element boxing.\n(define (make-column vals)\n (let* ([v (list->vector vals)] [n (vector-length v)])\n (let scan ([i 0] [all-fx #t] [all-fl #t])\n (cond\n [(< i n) (let ([x (vector-ref v i)])\n (scan (+ i 1) (and all-fx (fixnum? x)) (and all-fl (flonum? x))))]\n [(and all-fl (> n 0))\n (let ([c (make-flvector n 0.0)])\n (do ([i 0 (+ i 1)]) ((= i n) (cons 'double c)) (flvector-set! c i (vector-ref v i))))]\n [(and all-fx (> n 0))\n (let ([c (make-fxvector n 0)])\n (do ([i 0 (+ i 1)]) ((= i n) (cons 'long c)) (fxvector-set! c i (vector-ref v i))))]\n [else (cons 'mixed v)]))))\n\n(define (col-sum col)\n (case (car col)\n [(double) (let ([v (cdr col)]) (let loop ([i 0] [a 0.0])\n (if (= i (flvector-length v)) a (loop (+ i 1) (fl+ a (flvector-ref v i))))))]\n [(long) (let ([v (cdr col)]) (let loop ([i 0] [a 0])\n (if (= i (fxvector-length v)) a (loop (+ i 1) (+ a (fxvector-ref v i))))))]\n [else (error 'col-sum \"non-numeric column\")]))") ("id" . "columnar-typed-columns-fxvector-flvector") + ("imports" "(chezscheme)") + ("notes" + . + "make-flvector/make-fxvector + flvector-set!/fxvector-set! + fl+ are Chez primitives. flvector holds unboxed flonums and fxvector unboxed fixnums, so a tight reduction loop avoids per-element allocation (~2-3x faster than a boxed vector of the same numbers). Fall back to a boxed vector for heterogeneous/string columns. Measured: a 500k-row double column summed ~2.4x faster unboxed than via boxed record access.") + ("tags" "columnar" "performance" "fxvector" "flvector" + "unboxed" "storage") + ("title" + . + "Type-specialized columnar storage with unboxed fxvector/flvector")) + (("code" + . + ";; Write/read a bytevector to a file. call-with-port closes the port on normal\n;; return AND on escape/error -> no fd leak (prefer over manual open + close).\n(define (write-bytes path bv)\n (call-with-port (open-file-output-port path) (lambda (p) (put-bytevector p bv))))\n(define (read-bytes path)\n (call-with-port (open-file-input-port path) (lambda (p) (get-bytevector-all p))))\n\n;; Content-addressed blob store: filename = hash of the bytes -> idempotent\n;; writes + automatic dedup. (hash-bytes returns a small bytevector id.)\n(define (hex bv)\n (let ([d \"0123456789abcdef\"] [n (bytevector-length bv)])\n (let ([o (make-string (* 2 n))])\n (do ([i 0 (+ i 1)]) ((= i n) o)\n (let ([b (bytevector-u8-ref bv i)])\n (string-set! o (* 2 i) (string-ref d (quotient b 16)))\n (string-set! o (+ (* 2 i) 1) (string-ref d (remainder b 16))))))))\n(define (blob-put! dir bv)\n (unless (file-exists? dir) (mkdir dir))\n (let* ([id (hash-bytes bv)] [p (string-append dir \"/\" (hex id) \".blob\")])\n (unless (file-exists? p) (write-bytes p bv))\n id))\n(define (blob-get dir id)\n (let ([p (string-append dir \"/\" (hex id) \".blob\")])\n (and (file-exists? p) (read-bytes p))))\n;; GC: (for-each delete-file unreferenced) over (directory-list dir)") ("id" . "binary-file-content-addressed-store") + ("imports" "(chezscheme)") + ("notes" + . + "Chez primitives: open-file-output-port/open-file-input-port (binary), put-bytevector, get-bytevector-all, call-with-port, mkdir, file-exists?, directory-list, delete-file. call-with-port is the resource-safe pattern (closes on unwind). For content addressing, prefer a CRYPTOGRAPHIC hash (sha-256) for hash-bytes if inputs are adversarial -- a non-crypto hash (FNV/CRC) can collide and silently alias two different blobs to the same file.") + ("tags" "file" "binary" "bytevector" "content-addressed" + "blob" "persistence") + ("title" + . + "Binary file I/O and a content-addressed blob store")) + (("code" + . + ";; A define-record-type record may NOT round-trip via fasl across processes\n;; (generative rtd), and fasl-read on untrusted input can deserialize arbitrary\n;; procedures (RCE). Decompose records to plain data, recompose on read, and\n;; serialize with write/read (safe + portable).\n(define-record-type point (fields x y))\n\n(define (point->data p) (list (point-x p) (point-y p))) ; plain list\n(define (data->point d) (apply make-point d))\n\n(define (save-points ps) ; -> string\n (let-values ([(port get) (open-string-output-port)])\n (write (map point->data ps) port)\n (get)))\n(define (load-points s) ; string ->\n (map data->point (read (open-string-input-port s))))") ("id" . "persist-decompose-records-to-plain-data") + ("imports" "(chezscheme)") + ("notes" + . + "Decompose records to lists/vectors/numbers/strings/bytevectors (all portable), then serialize with write + read (open-string-output-port / open-string-input-port, R6RS). fasl-write/fasl-read are faster but: (1) records can fail to round-trip across processes (generative rtds), and (2) fasl-read is unsafe on untrusted input -- the Jerboa security scanner flags `unsafe-fasl-deserialize` as critical (pickle-style RCE). Reserve fasl for trusted, same-process caches; use decompose + write/read for on-disk or cross-process persistence.") + ("tags" "persistence" "serialize" "record" "fasl" "portable" + "security") + ("title" + . + "Persist records portably by decomposing to plain data (avoid fasl pitfalls)"))) --- a/data/features.sexp +++ b/data/features.sexp @@ -1141,4 +1141,22 @@ ("use_case" . "Discovering cookbook recipes by natural keyword queries, and ensuring recipes just saved via jerboa_howto_add are actually findable by search rather than only by exact id.") + ("votes" . 0)) + (("description" + . + "compile_check, check_syntax, run_tests, and eval require passing extra_libdirs (plus project_path and jerboa_home) on essentially every call. For a project with a stable layout, the tools could auto-detect libdirs from a project marker -- e.g. a Makefile's LIBDIRS= line, a .jerboa config file, or a sibling jerboa checkout -- given just project_path (or by walking up from file_path). Callers would stop repeating the identical extra_libdirs array dozens of times per session.") + ("estimated_token_reduction" + . + "~60-80 tokens per compile/run call; ~1000+ tokens per iterative session") + ("example_scenario" + . + "In one session I called jerboa_compile_check ~15 times, each repeating extra_libdirs=[\"<proj>/lib\",\"<jerboa>/lib\"], project_path, and jerboa_home -- byte-identical every time -- because the Makefile already declares LIBDIRS=lib:$(JERBOA_DIR)/lib. The tool could read that once.") + ("id" . "auto-detect-project-libdirs") ("impact" . "medium") + ("tags" "compile-check" "libdirs" "project" "dx" "tokens") + ("title" + . + "Auto-detect project libdirs for compile_check / run_tests / eval") + ("use_case" + . + "Iteratively compile-checking and running many files in one project across a long session.") ("votes" . 0)))