v0.19 native nullable types #1496

vyzo · 2026-09-13T18:16:08Z

vyzo commented

2026-09-13 18:16:08 +00:00

This has been pending for a long while, ever since the introduction of the :? type annotation sigils coz it was quite tricky and hairy... but not for astra. It is now a reality, the compiler now natively understands nullable types and does not do unsafe shit because you forgot to check a nullable reference, and you can use (Maybe T) to declare a nullable type return.

also updates the gambit pin which was broken and includes unwind-protect! and fixes a type bug, cherry picked ,from my wip branch on ensembles.

Implemented by astra with guidance and oversight from the gerbilosaurus rex.

This has been pending for a long while, ever since the introduction of the :? type annotation sigils coz it was quite tricky and hairy... but not for astra. It is now a reality, the compiler now natively understands nullable types and does not do unsafe shit because you forgot to check a nullable reference, and you can use (Maybe T) to declare a nullable type return. also updates the gambit pin which was broken and includes unwind-protect! and fixes a type bug, cherry picked ,from my wip branch on ensembles. Implemented by astra with guidance and oversight from the gerbilosaurus rex.

vyzo added 9 commits

2026-09-13 18:16:08 +00:00

fix condition variable type predicate 51970b865c

add exception-only unwind protection 95b624771a

support unwind-protect! in gerbil-mode a5c0aff64b

fix Gambit submodule and configure revision e6fb3b7b60

implement nullable type metadata and path refinement 73369b65f2

test predicate and cast folding in optimized nullable code d6ebcf1a3c

add Maybe return type declarations 0b500ab839

declare nullable stdlib results with Maybe 03dcbfff4b

bootstrap 67cac2746a

vyzo requested reviews from fare, jay

2026-09-13 18:16:16 +00:00

fare commented

2026-09-13 20:51:04 +00:00

I'll review manually, but in the mean time, here's what Sol says:

I would request changes, for one substantive correctness hole. Overall the nullable-type machinery itself looks well thought out: a distinct optimizer type, covariance/subtyping rules, SSXI preservation, conservative treatment of mutable bindings and pretty extensive tests. (cons.io git)

Blocking: nullable intermediate dotted slots lose their nullability. The new code correctly annotates a terminal dotted slot read such as holder.item as (nullable ...) when the slot contract contains :?. But when the same slot is an intermediate component, %%ref-dotted simply resolves its underlying type and recurses on #'(accessor object); %%app-dotted and %%set-dotted! do the same thing. (cons.io git) Worse, get-slot-accessor is deliberately an unchecked accessor and can expand all the way down to ##unchecked-structure-ref. (cons.io git)

So I think this defeats the central safety property of the PR. Consider:
```
(defclass B ((x : :fixnum)))
(defclass A ((b :? B)))

(def (f (a : A))
  a.b.x)
```
a.b may be #f, but while a.b alone receives nullable metadata, in a.b.x the b contract is forgotten and x is fetched from the result as though it were definitely a B. The tests cover terminal nullable slots (holder.item) but I don't see a nested nullable receiver case. (cons.io git)

There is already syntax capable of making the intended nil check explicit (a.b.?x, by the expansion logic). So I don't have a strong opinion whether a.b.x should be rejected or automatically checked. But it absolutely should not silently turn :? B back into B and perform an unchecked access. The same regression tests should cover read, method/application and set! through an intermediate nullable slot.
Minor: optimizer-current-types throws away the base of a nullable type. Its generic !type? case returns (!type-id t), which for every !nullable-type is simply 'nullable. Thus (nullable Foo::t) and (nullable Bar::t) become indistinguishable in this introspection output. You already have optimizer-type-expression, which preserves exactly this structure. An !nullable-type? clause before the generic !type? clause seems appropriate. (cons.io git) I don't see this affecting optimization correctness, so I'd call it a diagnostic/debugging bug, not a blocker.
The committed nullable-types.md needs a serious haircut. It contains useful design documentation, but also /tmp/opencode/... logs, a /home/vyzo/... path, stash hashes, interrupted-work recovery history, statements such as “the user approved committing this follow-up”, whether a branch was pushed and other session archaeology. (cons.io git) I would retain the semantics, invariants, examples, test commands and perhaps concise verification results, and remove the AI/worktree diary. That stuff will be meaningless noise six months from now.
One test-coverage note: the new kqueue nullable declaration/test was not actually run or native-compiled because the work was performed on Linux; the PR itself records this. (cons.io git) Since this is mostly metadata rather than an implementation change I wouldn't necessarily block on it, but ideally BSD CI should exercise it before release.

I found no problem with the exact-Maybe signature comparison I initially suspected: that path is reconciling inherited interface declarations, not checking covariant implementation returns. Likewise, the nullable interface adaptation and the @type.return machinery appear deliberately conservative rather than unsound.

So my verdict is: good underlying design, but I would not merge until the intermediate-dotted-access case is resolved. That one lands squarely on the PR's advertised guarantee that forgetting to check a nullable reference must not lead to unsafe code. (cons.io git)

I'll review manually, but in the mean time, here's what Sol says: I would **request changes**, for one substantive correctness hole. Overall the nullable-type machinery itself looks well thought out: a distinct optimizer type, covariance/subtyping rules, SSXI preservation, conservative treatment of mutable bindings and pretty extensive tests. ([cons.io git][1]) 1. **Blocking: nullable intermediate dotted slots lose their nullability.** The new code correctly annotates a *terminal* dotted slot read such as `holder.item` as `(nullable ...)` when the slot contract contains `:?`. But when the same slot is an intermediate component, `%%ref-dotted` simply resolves its underlying type and recurses on `#'(accessor object)`; `%%app-dotted` and `%%set-dotted!` do the same thing. ([cons.io git][2]) Worse, `get-slot-accessor` is deliberately an unchecked accessor and can expand all the way down to `##unchecked-structure-ref`. ([cons.io git][2]) So I think this defeats the central safety property of the PR. Consider: ```scheme (defclass B ((x : :fixnum))) (defclass A ((b :? B))) (def (f (a : A)) a.b.x) ``` `a.b` may be `#f`, but while `a.b` alone receives nullable metadata, in `a.b.x` the `b` contract is forgotten and `x` is fetched from the result as though it were definitely a `B`. The tests cover terminal nullable slots (`holder.item`) but I don't see a nested nullable receiver case. ([cons.io git][1]) There is already syntax capable of making the intended nil check explicit (`a.b.?x`, by the expansion logic). So I don't have a strong opinion whether `a.b.x` should be rejected or automatically checked. But it absolutely should not silently turn `:? B` back into `B` and perform an unchecked access. The same regression tests should cover **read, method/application and set!** through an intermediate nullable slot. 2. **Minor: `optimizer-current-types` throws away the base of a nullable type.** Its generic `!type?` case returns `(!type-id t)`, which for every `!nullable-type` is simply `'nullable`. Thus `(nullable Foo::t)` and `(nullable Bar::t)` become indistinguishable in this introspection output. You already have `optimizer-type-expression`, which preserves exactly this structure. An `!nullable-type?` clause before the generic `!type?` clause seems appropriate. ([cons.io git][3]) I don't see this affecting optimization correctness, so I'd call it a diagnostic/debugging bug, not a blocker. 3. **The committed `nullable-types.md` needs a serious haircut.** It contains useful design documentation, but also `/tmp/opencode/...` logs, a `/home/vyzo/...` path, stash hashes, interrupted-work recovery history, statements such as “the user approved committing this follow-up”, whether a branch was pushed and other session archaeology. ([cons.io git][1]) I would retain the semantics, invariants, examples, test commands and perhaps concise verification results, and remove the AI/worktree diary. That stuff will be meaningless noise six months from now. 4. **One test-coverage note:** the new `kqueue` nullable declaration/test was not actually run or native-compiled because the work was performed on Linux; the PR itself records this. ([cons.io git][4]) Since this is mostly metadata rather than an implementation change I wouldn't necessarily block on it, but ideally BSD CI should exercise it before release. I found **no problem** with the exact-`Maybe` signature comparison I initially suspected: that path is reconciling inherited interface declarations, not checking covariant implementation returns. Likewise, the nullable interface adaptation and the `@type.return` machinery appear deliberately conservative rather than unsound. So my verdict is: **good underlying design, but I would not merge until the intermediate-dotted-access case is resolved.** That one lands squarely on the PR's advertised guarantee that forgetting to check a nullable reference must not lead to unsafe code. ([cons.io git][5]) [1]: https://git.cons.io/mighty-gerbils/gerbil/pulls/1496/commits/73369b65f271b59a7dbf61e105a022326cb18932 "#1496 - v0.19 native nullable types - mighty-gerbils/gerbil - cons.io git" [2]: https://git.cons.io/mighty-gerbils/gerbil/src/commit/0b500ab8390272c912f6098fbf88f401f41a361f/src/gerbil/core/contract.ss "gerbil/src/gerbil/core/contract.ss at 0b500ab8390272c912f6098fbf88f401f41a361f - mighty-gerbils/gerbil - cons.io git" [3]: https://git.cons.io/mighty-gerbils/gerbil/src/commit/73369b65f271b59a7dbf61e105a022326cb18932/src/gerbil/compiler/optimize-base.ss "gerbil/src/gerbil/compiler/optimize-base.ss at 73369b65f271b59a7dbf61e105a022326cb18932 - mighty-gerbils/gerbil - cons.io git" [4]: https://git.cons.io/mighty-gerbils/gerbil/pulls/1496/commits/03dcbfff4bf6356a45b7ca49b390a7f96f03c8d4 "#1496 - v0.19 native nullable types - mighty-gerbils/gerbil - cons.io git" [5]: https://git.cons.io/mighty-gerbils/gerbil/pulls/1496 "#1496 - v0.19 native nullable types - mighty-gerbils/gerbil - cons.io git"

vyzo commented

2026-09-13 21:29:46 +00:00

great review by sol. For 1, we discussed with astra, and decided to emit the checked accessor for chains that include a nullable dereference.

vyzo added 1 commit

2026-09-13 22:44:28 +00:00

check nullable intermediate dotted receivers 8dd21144a8

vyzo commented

2026-09-13 22:45:49 +00:00

addressed review findings 1-3, 4 is out of scope as i don't have a bsd box.

fare commented

2026-09-13 23:03:46 +00:00

Maybe the clanker can cook an automatic job with qemu or something to check BSD once in a while?

fare commented

2026-09-13 23:08:48 +00:00

Sol comments:

Yes. The update at 8dd21144a8 fixes the issues I raised in the first pass, and the intermediate nullable receiver fix is real, not cosmetic. But I think there is still one blocking hole, closely related to the one just fixed.

The good news first: h.item.x, h.item.proc, setters through h.item, deeper chains, structs, inherited slots and the system-class offset case now propagate the nullable status to the next receiver and select checked accessors/mutators. The added test is substantial and exercises exactly those cases. (cons.io git) The optimizer-current-types problem is also fixed, and nullable-types.md has been cleaned up into useful design documentation rather than a work-session transcript. (cons.io git)

However, a nullable binding itself still seems to lose its nullability in the dotted macro environment.

type-env has only (var type checked? super). with-class knows maybe? and correctly puts (nullable Type::t) into the optimizer annotation, but when it constructs the type-env, it stores only the ordinary class plus checked?; maybe? disappears. Note that checked? cannot substitute for it because both : and :? bindings are checked. (cons.io git)

Then all three dotted walkers initialize the receiver as non-nullable:

;; %%ref-dotted
(checked-receiver? #f)

;; %%app-dotted
(checked-receiver? #f)

;; %%set-dotted!
(checked-receiver? #f)

They only set it to true after traversing a nullable slot. (cons.io git)

So I believe this is still unsafe:

(defclass Item ((x : :fixnum) proc raw))

(def (read-base (item :? Item))
  item.x)

(def (call-base (item :? Item))
  (item.proc 1))

(def (set-base! (item :? Item) x)
  (set! item.raw x))

Calling any of those with #f passes the :? Item entry contract, as intended, but the first dotted hop sees checked-receiver? = #f and asks get-slot-accessor for the unchecked accessor. For offset-based classes that literally produces ##unchecked-structure-ref; ordinary unchecked accessors likewise stay unchecked. (cons.io git)

And the optimizer doesn't repair this downstream: an unchecked !accessor skips argument checking, and optimized final/struct accesses lower to %#struct-unchecked-ref; the generic fallback is unchecked-slot-ref. (cons.io git)

The new tests don't catch this. They comprehensively test nullable slots used as receivers, e.g. h.item.x. The closest direct nullable local is:

(using (item h.item :? Item)
  (and item (begin (set! h.item #f) item.x)))

but item.x is reached only after (and item ...) has refined the stable local to non-null, so that is exactly the safe case. (cons.io git)

So my revised verdict is: the original blocker is fixed correctly, but I would still request changes for the root-binding analogue of the same bug.

I suspect the clean fix is to put a nullable?/maybe? bit into type-env and initialize checked-receiver? from that, then add direct-base tests for read/call/set and #f. That also makes the invariant pleasantly simple:

Every time the dotted walker knows its current receiver has nullable type, the next dereference uses a checked receiver.

After that, I think this PR is looking quite solid.

Sol comments: Yes. The update at `8dd21144a8` fixes the issues I raised in the first pass, and the **intermediate nullable receiver fix is real**, not cosmetic. But I think there is still **one blocking hole**, closely related to the one just fixed. The good news first: `h.item.x`, `h.item.proc`, setters through `h.item`, deeper chains, structs, inherited slots and the system-class offset case now propagate the nullable status to the *next* receiver and select checked accessors/mutators. The added test is substantial and exercises exactly those cases. ([cons.io git][1]) The `optimizer-current-types` problem is also fixed, and `nullable-types.md` has been cleaned up into useful design documentation rather than a work-session transcript. ([cons.io git][2]) However, **a nullable binding itself still seems to lose its nullability in the dotted macro environment**. `type-env` has only `(var type checked? super)`. `with-class` knows `maybe?` and correctly puts `(nullable Type::t)` into the optimizer annotation, but when it constructs the `type-env`, it stores only the ordinary class plus `checked?`; `maybe?` disappears. Note that `checked?` cannot substitute for it because both `:` and `:?` bindings are checked. ([cons.io git][1]) Then all three dotted walkers initialize the receiver as non-nullable: ```scheme ;; %%ref-dotted (checked-receiver? #f) ;; %%app-dotted (checked-receiver? #f) ;; %%set-dotted! (checked-receiver? #f) ``` They only set it to true **after traversing a nullable slot**. ([cons.io git][1]) So I believe this is still unsafe: ```scheme (defclass Item ((x : :fixnum) proc raw)) (def (read-base (item :? Item)) item.x) (def (call-base (item :? Item)) (item.proc 1)) (def (set-base! (item :? Item) x) (set! item.raw x)) ``` Calling any of those with `#f` passes the `:? Item` entry contract, as intended, but the first dotted hop sees `checked-receiver? = #f` and asks `get-slot-accessor` for the **unchecked accessor**. For offset-based classes that literally produces `##unchecked-structure-ref`; ordinary unchecked accessors likewise stay unchecked. ([cons.io git][1]) And the optimizer doesn't repair this downstream: an unchecked `!accessor` skips argument checking, and optimized final/struct accesses lower to `%#struct-unchecked-ref`; the generic fallback is `unchecked-slot-ref`. ([cons.io git][3]) The new tests don't catch this. They comprehensively test **nullable slots used as receivers**, e.g. `h.item.x`. The closest direct nullable local is: ```scheme (using (item h.item :? Item) (and item (begin (set! h.item #f) item.x))) ``` but `item.x` is reached only after `(and item ...)` has refined the stable local to non-null, so that is exactly the safe case. ([cons.io git][4]) So my revised verdict is: **the original blocker is fixed correctly, but I would still request changes for the root-binding analogue of the same bug.** I suspect the clean fix is to put a `nullable?`/`maybe?` bit into `type-env` and initialize `checked-receiver?` from that, then add direct-base tests for read/call/set and `#f`. That also makes the invariant pleasantly simple: > Every time the dotted walker knows its current receiver has nullable type, the next dereference uses a checked receiver. After that, I think this PR is looking quite solid. [1]: https://git.cons.io/mighty-gerbils/gerbil/src/commit/8dd21144a8841f1c6f97d2b02ca344e540d78578/src/gerbil/core/contract.ss "gerbil/src/gerbil/core/contract.ss at 8dd21144a8841f1c6f97d2b02ca344e540d78578 - mighty-gerbils/gerbil - cons.io git" [2]: https://git.cons.io/mighty-gerbils/gerbil/src/commit/8dd21144a8841f1c6f97d2b02ca344e540d78578/src/gerbil/compiler/nullable-types.md "gerbil/src/gerbil/compiler/nullable-types.md at 8dd21144a8841f1c6f97d2b02ca344e540d78578 - mighty-gerbils/gerbil - cons.io git" [3]: https://git.cons.io/mighty-gerbils/gerbil/src/commit/73369b65f271b59a7dbf61e105a022326cb18932/src/gerbil/compiler/optimize-call.ss "gerbil/src/gerbil/compiler/optimize-call.ss at 73369b65f271b59a7dbf61e105a022326cb18932 - mighty-gerbils/gerbil - cons.io git" [4]: https://git.cons.io/mighty-gerbils/gerbil/commit/8dd21144a8841f1c6f97d2b02ca344e540d78578 "check nullable intermediate dotted receivers · 8dd21144a8 - mighty-gerbils/gerbil - cons.io git"

vyzo added 2 commits

2026-09-14 08:53:12 +00:00

check nullable root dotted receivers 0d5009d780

bootstrap 7d057c8a7b

vyzo commented

2026-09-14 08:54:10 +00:00

addressed the review issue and rebootstrapped.