Preserve Typed Jerboa declaration sources
ober
064516b6d5c0dbc98f06c3ccefdf262708c329fe
--- a/docs/jerboa-to-rust.md +++ b/docs/jerboa-to-rust.md @@ -53,6 +53,45 @@ typed Jerboa source The Rust backend should consume a lowered IR, not surface syntax. That keeps the language independent from Rust and makes the later LLVM backend practical. +Current implementation note: the MVP still emits Rust directly from checked +surface datums. That is acceptable for the first split-tree slice, but the next +backend maintainer should introduce a typed core IR before adding import +resolution, optimization, LLVM, or richer source maps. + +## Current Backend State + +Landed: + +- Deterministic single-file Cargo crate generation under `build/typed/rust`. +- Safe Rust text for primitive values, `let`, `if`, arithmetic, numeric + comparisons, boolean operators, same-type `equal?`, `debug-string`, + `string-length`, `string-append`, and `bytevector-length`. +- Rust `struct`/`enum` generation for Typed Jerboa records and variants, + including `Clone`, `Debug`, and `PartialEq` derives. +- Recursive variant field boxing and match rebinding for owned child values. +- Option/Result constructor lowering: `option-some`, `option-none`, + `result-ok`, and `result-err`. +- `String`/`Bytes` ABI support via Rust-owned return buffers and generated + destructor calls. +- Opaque handle ABI for same-module records, variants, Option values, and + Result values. +- Generated Scheme wrappers that validate dynamic arguments, reject stale or + wrong-type handles, expose explicit handle drops, and use a Chez guardian + reaper as a cleanup safety net. + +Still open: + +- Typed core IR. This is the most important backend architecture gap. +- Import resolution between typed modules. +- Structured Rust-to-Scheme error returns instead of conservative panic + defaults. +- Direct Scheme conversions for Option/Result instead of opaque handles. +- Resource `#:close` lowering and Rust `Drop` integration. +- Borrow-aware lowering. Current generated Rust clones liberally. +- Source-map plumbing from parser spans to checker diagnostics and generated + Rust comments. +- Direct `rustc`/static integration polish and eventual LLVM parity tests. + ## Output Layout Candidate layout: --- a/docs/typed-jerboa.md +++ b/docs/typed-jerboa.md @@ -136,8 +136,10 @@ The compiler can later accept `.tss` as a convenience. Current landing: - `(jerboa typed parser)` parses and validates `(typed-library ...)` datums - into a first AST layer. It strips Jerboa reader source annotations so file - based `typecheck` and backend generation use the same AST path as unit tests. + into a first AST layer. It preserves reader source locations on the module + and declaration-level AST records while stripping annotations from semantic + names, type expressions, and function bodies so file based `typecheck` and + backend generation use the same AST path as unit tests. - `(jerboa typed checker)` performs the first backend-neutral validation pass: duplicate names, export resolution, type-reference resolution, compound type arities, duplicate fields, duplicate params, duplicate variant cases, and a @@ -198,7 +200,10 @@ Current landing: immutable and `mut` record fields, `resource` declarations with optional `#:close` hooks, variants including nullary cases, and `def` forms with typed parameters, `:` or `->` return markers, and optional `#:effects (...)` - annotations. + annotations. Parser source accessors currently include `typed-module-source`, + `typed-type-decl-source`, `typed-field-source`, `typed-record-source`, + `typed-resource-source`, `typed-variant-source`, + `typed-variant-case-source`, `typed-param-source`, and `typed-def-source`. - This is still a front-end milestone. Function-body checking currently covers literals, variables, `begin`, simple `let`, `if`, arithmetic primitives, numeric comparisons, same-type `equal?`, boolean primitives, calls to typed @@ -225,6 +230,74 @@ Current landing: than one helper. This is deliberately simple and correct for the first split-tree slice; borrow-aware lowering remains future work. +## Handoff Snapshot + +This document is the primary implementation plan for the Typed Jerboa work. +`docs/jerboa-to-rust.md` is the backend-specific companion plan, and +`docs/gerbil-contracts.md` explains why contracts complement but do not replace +Typed Jerboa. The current branch is advancing the Typed Jerboa route: typed +source stays in `.ss` files, the parser/checker own the source of truth, and +Rust is generated as a disposable native backend artifact. + +Code that has landed: + +- Parser: datum parser for `(typed-library ...)`; records, variants, resources, + type declarations, typed defs, `#:effects`, `Option`, `Result`, + `Owned`/`Borrow`/`MutBorrow`; module and declaration-level source locations. +- Checker: export/type/reference validation, duplicate detection, same-module + typed calls, records/variants, exhaustive variant `match`, effects coverage, + straight-line owned resource move checks, Option/Result constructors, + same-type `equal?`, and `debug-string`. +- Rust backend: deterministic Rust text, Cargo crate generation, primitive + expressions, records/enums, recursive variant boxing, same-module + record/variant calls, exhaustive variant match, `String`/`Bytes` helpers, + Option/Result constructors, `equal?` to `==`, and `debug-string` to Rust + Debug formatting. +- Wrapper backend: generated `.ss` wrappers for scalar, `Char`, `String`, + `Bytes`, same-module record/variant handles, Option/Result opaque handles, + explicit handle drops, stale/double-drop rejection, and guardian-backed + reaping through `%typed-rust-reap-handles!`. +- Fixtures/smokes: `tests/fixtures/typed/rust-basic.ss`, + `tests/fixtures/typed/valid-split-tree.ss`, `make typed-wrapper-smoke`, and + `make typed-split-tree-smoke`. + +The next model should continue in small commits with tests and docs per step. +Highest-value next steps: + +1. Add expression-level source spans and thread them into `typed-check-error` + so diagnostics can report file/line/column for bad calls, mismatched branch + types, and ownership errors. Declaration-level spans now exist; body spans + do not. +2. Introduce an explicit typed core IR between parser/checker and backends. + Today the checker and Rust emitter both walk surface datums, which keeps the + MVP simple but will make imports, source maps, and optimization harder. +3. Resolve imports between typed modules. The checker currently handles calls + within one typed module only; imported calls are intentionally unsupported. +4. Improve boundary semantics for Option/Result. They currently cross the FFI + boundary as opaque handles; direct conversion to idiomatic Scheme result + values is still open. +5. Make resource lowering borrow-aware. The checker has straight-line owned + move rules, but the Rust backend still uses a conservative Clone-heavy model + and does not use resource `#:close` hooks for generated `Drop`. +6. Add structured error returns. Rust ABI wrappers catch panics and return + conservative defaults; there is not yet a typed error/result ABI for wrapper + failures. +7. Add fuzz/property tests for parser/checker and broader differential tests + against dynamic Jerboa implementations before moving to LLVM or static + integration. + +Expected verification before each commit: + +- `jerboa_compile_check` on any changed `.ss`/`.sls` implementation files. +- `jerboa_security_scan` for changed compiler/wrapper/FFI code. +- Focused test file with `jerboa_run_tests`. +- `make typed-test` +- `make typed-build` +- `make typed-wrapper-smoke` when wrapper or ABI code changes. +- `make typed-split-tree-smoke` when backend, wrapper, or fixture behavior + changes. +- `make build` after any Jerboa source change. + ## Surface Syntax A typed module should look familiar to Jerboa users: @@ -842,8 +915,10 @@ Minimum excluded features: `(jerboa typed parser)`. - Preserve source spans. Initial module-level source preservation has landed: `typed-module-source` stores the reader source location for annotated - `typed-library` forms. Declaration and expression spans are still future - work. + `typed-library` forms, and declaration-level AST records now retain source + accessors for type declarations, records, fields, resources, variants, + variant cases, params, and defs. Function body/expression spans are still + future work. - Parse type expressions. Initial symbolic and compound type parsing landed. - Parse records, variants, and function definitions. Initial AST records landed. - Reject unsupported forms clearly. --- a/lib/jerboa/typed/parser.ss +++ b/lib/jerboa/typed/parser.ss @@ -18,43 +18,45 @@ typed-module-source typed-type-decl? make-typed-type-decl - typed-type-decl-name + typed-type-decl-name typed-type-decl-source typed-field? make-typed-field typed-field-name typed-field-type typed-field-mutable? + typed-field-source typed-record? make-typed-record - typed-record-name typed-record-fields + typed-record-name typed-record-fields typed-record-source typed-resource? make-typed-resource - typed-resource-name typed-resource-close + typed-resource-name typed-resource-close typed-resource-source typed-variant? make-typed-variant - typed-variant-name typed-variant-cases + typed-variant-name typed-variant-cases typed-variant-source typed-variant-case? make-typed-variant-case typed-variant-case-name typed-variant-case-fields + typed-variant-case-source typed-param? make-typed-param - typed-param-name typed-param-type + typed-param-name typed-param-type typed-param-source typed-def? make-typed-def typed-def-name typed-def-params typed-def-return-type - typed-def-effects typed-def-body) + typed-def-effects typed-def-body typed-def-source) (import (chezscheme) ; jerboa-security: suppress direct-chezscheme-import-user-code -- trusted typed compiler front-end module (only (jerboa core) def defstruct) (jerboa reader)) (defstruct typed-module (name exports declarations source)) - (defstruct typed-type-decl (name)) - (defstruct typed-field (name type mutable?)) - (defstruct typed-record (name fields)) - (defstruct typed-resource (name close)) - (defstruct typed-variant (name cases)) - (defstruct typed-variant-case (name fields)) - (defstruct typed-param (name type)) - (defstruct typed-def (name params return-type effects body)) + (defstruct typed-type-decl (name source)) + (defstruct typed-field (name type mutable? source)) + (defstruct typed-record (name fields source)) + (defstruct typed-resource (name close source)) + (defstruct typed-variant (name cases source)) + (defstruct typed-variant-case (name fields source)) + (defstruct typed-param (name type source)) + (defstruct typed-def (name params return-type effects body source)) (def (strip-source-annotations datum) (cond @@ -72,6 +74,11 @@ (and (annotated-datum? datum) (annotated-datum-source datum))) + (def (datum-value datum) + (if (annotated-datum? datum) + (annotated-datum-value datum) + datum)) + (def (proper-list? x) (cond [(null? x) #t] @@ -124,8 +131,9 @@ (eq? (car form) 'typed-library)))) (def (parse-typed-library form) - (let ([source (datum-source form)] - [form (strip-source-annotations form)]) + (let* ([source (datum-source form)] + [raw-form (datum-value form)] + [form (strip-source-annotations form)]) (expect-proper-list 'parse-typed-library form) (unless (typed-library-form? form) (error 'parse-typed-library "expected typed-library form" form)) @@ -135,7 +143,7 @@ form)) (let ([name (parse-module-name (cadr form))] [exports (parse-export-form (caddr form))] - [declarations (map parse-typed-declaration (cdddr form))]) + [declarations (map parse-typed-declaration (cdddr raw-form))]) (make-typed-module name exports declarations source)))) (def (parse-typed-type type) @@ -150,68 +158,90 @@ type)]))) (def (parse-field form) - (expect-proper-list 'parse-typed-field form) - (case (length form) - [(3) - (unless (eq? (cadr form) ':) - (error 'parse-typed-field - "expected (name : Type)" - form)) - (make-typed-field - (expect-symbol 'parse-typed-field (car form) form) - (parse-typed-type (caddr form)) - #f)] - [(4) - (unless (and (eq? (car form) 'mut) - (eq? (caddr form) ':)) + (let ([source (datum-source form)] + [form (strip-source-annotations form)]) + (expect-proper-list 'parse-typed-field form) + (case (length form) + [(3) + (unless (eq? (cadr form) ':) + (error 'parse-typed-field + "expected (name : Type)" + form)) + (make-typed-field + (expect-symbol 'parse-typed-field (car form) form) + (parse-typed-type (caddr form)) + #f + source)] + [(4) + (unless (and (eq? (car form) 'mut) + (eq? (caddr form) ':)) + (error 'parse-typed-field + "expected (mut name : Type)" + form)) + (make-typed-field + (expect-symbol 'parse-typed-field (cadr form) form) + (parse-typed-type (cadddr form)) + #t + source)] + [else (error 'parse-typed-field - "expected (mut name : Type)" - form)) - (make-typed-field - (expect-symbol 'parse-typed-field (cadr form) form) - (parse-typed-type (cadddr form)) - #t)] - [else - (error 'parse-typed-field - "expected (name : Type) or (mut name : Type)" - form)])) + "expected (name : Type) or (mut name : Type)" + form)]))) (def (parse-param form) - (expect-length 'parse-typed-param form 3) - (unless (eq? (cadr form) ':) - (error 'parse-typed-param - "expected (name : Type)" - form)) - (make-typed-param - (expect-symbol 'parse-typed-param (car form) form) - (parse-typed-type (caddr form)))) + (let ([source (datum-source form)] + [form (strip-source-annotations form)]) + (expect-length 'parse-typed-param form 3) + (unless (eq? (cadr form) ':) + (error 'parse-typed-param + "expected (name : Type)" + form)) + (make-typed-param + (expect-symbol 'parse-typed-param (car form) form) + (parse-typed-type (caddr form)) + source))) (def (parse-record form) - (expect-length 'parse-typed-record form 3) - (let ([name (expect-symbol 'parse-typed-record (cadr form) form)] - [fields-form (caddr form)]) - (expect-proper-list 'parse-typed-record fields-form) - (make-typed-record name (map parse-field fields-form)))) + (let* ([source (datum-source form)] + [raw-form (datum-value form)] + [form (strip-source-annotations form)]) + (expect-length 'parse-typed-record form 3) + (let ([name (expect-symbol 'parse-typed-record (cadr form) form)] + [fields-form (caddr form)] + [raw-fields-form (caddr raw-form)]) + (expect-proper-list 'parse-typed-record fields-form) + (make-typed-record + name + (map parse-field (datum-value raw-fields-form)) + source)))) (def (parse-variant-case form) - (expect-proper-list 'parse-typed-variant-case form) - (unless (and (pair? form) (symbol? (car form))) - (error 'parse-typed-variant-case - "expected (Case field ...)" - form)) - (make-typed-variant-case - (car form) - (map parse-field (cdr form)))) + (let* ([source (datum-source form)] + [raw-form (datum-value form)] + [form (strip-source-annotations form)]) + (expect-proper-list 'parse-typed-variant-case form) + (unless (and (pair? form) (symbol? (car form))) + (error 'parse-typed-variant-case + "expected (Case field ...)" + form)) + (make-typed-variant-case + (car form) + (map parse-field (cdr raw-form)) + source))) (def (parse-variant form) - (expect-proper-list 'parse-typed-variant form) - (unless (>= (length form) 3) - (error 'parse-typed-variant - "expected (variant Name case ...)" - form)) - (make-typed-variant - (expect-symbol 'parse-typed-variant (cadr form) form) - (map parse-variant-case (cddr form)))) + (let* ([source (datum-source form)] + [raw-form (datum-value form)] + [form (strip-source-annotations form)]) + (expect-proper-list 'parse-typed-variant form) + (unless (>= (length form) 3) + (error 'parse-typed-variant + "expected (variant Name case ...)" + form)) + (make-typed-variant + (expect-symbol 'parse-typed-variant (cadr form) form) + (map parse-variant-case (cddr raw-form)) + source))) (def (keyword-marker? value name) (and (symbol? value) @@ -222,32 +252,38 @@ (keyword-marker? value "close")) (def (parse-resource form) - (expect-proper-list 'parse-typed-resource form) - (case (length form) - [(2) - (make-typed-resource - (expect-symbol 'parse-typed-resource (cadr form) form) - #f)] - [(4) - (unless (close-marker? (caddr form)) + (let ([source (datum-source form)] + [form (strip-source-annotations form)]) + (expect-proper-list 'parse-typed-resource form) + (case (length form) + [(2) + (make-typed-resource + (expect-symbol 'parse-typed-resource (cadr form) form) + #f + source)] + [(4) + (unless (close-marker? (caddr form)) + (error 'parse-typed-resource + "expected #:close close-function" + form)) + (make-typed-resource + (expect-symbol 'parse-typed-resource (cadr form) form) + (expect-symbol 'parse-typed-resource (cadddr form) form) + source)] + [else (error 'parse-typed-resource - "expected #:close close-function" - form)) - (make-typed-resource - (expect-symbol 'parse-typed-resource (cadr form) form) - (expect-symbol 'parse-typed-resource (cadddr form) form))] - [else - (error 'parse-typed-resource - "expected (resource Name) or (resource Name #:close close-function)" - form)])) + "expected (resource Name) or (resource Name #:close close-function)" + form)]))) (def (parse-def-head head) - (expect-proper-list 'parse-typed-def head) - (unless (and (pair? head) (symbol? (car head))) - (error 'parse-typed-def - "expected function head (name (arg : Type) ...)" - head)) - (values (car head) (map parse-param (cdr head)))) + (let ([raw-head (datum-value head)] + [head (strip-source-annotations head)]) + (expect-proper-list 'parse-typed-def head) + (unless (and (pair? head) (symbol? (car head))) + (error 'parse-typed-def + "expected function head (name (arg : Type) ...)" + head)) + (values (car head) (map parse-param (cdr raw-head))))) (def (effects-marker? value) (keyword-marker? value "effects")) @@ -277,40 +313,47 @@ (values '() body))) (def (parse-def form) - (expect-proper-list 'parse-typed-def form) - (unless (>= (length form) 5) - (error 'parse-typed-def - "expected (def (name (arg : Type) ...) : Return body ...)" - form)) - (let ([head (cadr form)] - [return-marker (caddr form)] - [return-type (cadddr form)] - [body (cddddr form)]) - (unless (memq return-marker '(: ->)) + (let* ([source (datum-source form)] + [raw-form (datum-value form)] + [form (strip-source-annotations form)]) + (expect-proper-list 'parse-typed-def form) + (unless (>= (length form) 5) (error 'parse-typed-def - "expected : or -> before return type" + "expected (def (name (arg : Type) ...) : Return body ...)" form)) - (let-values ([(name params) (parse-def-head head)]) - (let-values ([(effects parsed-body) - (parse-def-effects-and-body body)]) - (make-typed-def - name - params - (parse-typed-type return-type) - effects - parsed-body))))) + (let ([head (cadr raw-form)] + [return-marker (caddr form)] + [return-type (cadddr form)] + [body (cddddr form)]) + (unless (memq return-marker '(: ->)) + (error 'parse-typed-def + "expected : or -> before return type" + form)) + (let-values ([(name params) (parse-def-head head)]) + (let-values ([(effects parsed-body) + (parse-def-effects-and-body body)]) + (make-typed-def + name + params + (parse-typed-type return-type) + effects + parsed-body + source)))))) (def (parse-type-decl form) - (expect-length 'parse-typed-type-decl form 2) - (make-typed-type-decl - (expect-symbol 'parse-typed-type-decl (cadr form) form))) + (let ([source (datum-source form)] + [form (strip-source-annotations form)]) + (expect-length 'parse-typed-type-decl form 2) + (make-typed-type-decl + (expect-symbol 'parse-typed-type-decl (cadr form) form) + source))) (def (parse-typed-declaration form) - (let ([form (strip-source-annotations form)]) - (expect-proper-list 'parse-typed-declaration form) - (unless (pair? form) - (error 'parse-typed-declaration "empty declaration" form)) - (case (car form) + (let ([stripped-form (strip-source-annotations form)]) + (expect-proper-list 'parse-typed-declaration stripped-form) + (unless (pair? stripped-form) + (error 'parse-typed-declaration "empty declaration" stripped-form)) + (case (car stripped-form) [(type) (parse-type-decl form)] [(record) (parse-record form)] [(resource) (parse-resource form)] @@ -319,6 +362,6 @@ [else (error 'parse-typed-declaration "unsupported typed declaration" - form)]))) + stripped-form)]))) ) ;; end library --- a/tests/test-typed-parser.ss +++ b/tests/test-typed-parser.ss @@ -54,6 +54,12 @@ (define parsed (parse-typed-library sample)) (define annotated-sample (car (jerboa-read-file "tests/fixtures/typed/rust-basic.ss"))) +(define annotated-parsed (parse-typed-library annotated-sample)) +(define annotated-declarations (typed-module-declarations annotated-parsed)) +(define annotated-record-decl (list-ref annotated-declarations 0)) +(define annotated-variant-decl (list-ref annotated-declarations 1)) +(define annotated-zero-def (list-ref annotated-declarations 2)) +(define annotated-add-one-def (list-ref annotated-declarations 3)) (define declarations (typed-module-declarations parsed)) (define type-decl (list-ref declarations 0)) (define record-decl (list-ref declarations 1)) @@ -85,7 +91,7 @@ #t) (test "parse reader-annotated typed library" - (typed-module-name (parse-typed-library annotated-sample)) + (typed-module-name annotated-parsed) '(sample typed rust-basic)) (test "unannotated typed library source is false" @@ -94,7 +100,35 @@ (test "parse reader-annotated typed library source path" (source-location-path - (typed-module-source (parse-typed-library annotated-sample))) + (typed-module-source annotated-parsed)) + "tests/fixtures/typed/rust-basic.ss") + +(test "parse reader-annotated record source path" + (source-location-path (typed-record-source annotated-record-decl)) + "tests/fixtures/typed/rust-basic.ss") + +(test "parse reader-annotated field source path" + (source-location-path + (typed-field-source (car (typed-record-fields annotated-record-decl)))) + "tests/fixtures/typed/rust-basic.ss") + +(test "parse reader-annotated variant source path" + (source-location-path (typed-variant-source annotated-variant-decl)) + "tests/fixtures/typed/rust-basic.ss") + +(test "parse reader-annotated variant case source path" + (source-location-path + (typed-variant-case-source + (car (typed-variant-cases annotated-variant-decl)))) + "tests/fixtures/typed/rust-basic.ss") + +(test "parse reader-annotated def source path" + (source-location-path (typed-def-source annotated-zero-def)) + "tests/fixtures/typed/rust-basic.ss") + +(test "parse reader-annotated param source path" + (source-location-path + (typed-param-source (car (typed-def-params annotated-add-one-def)))) "tests/fixtures/typed/rust-basic.ss") (test "module name"