Investigate JSON/LL1 import closure and empty std reexports in separate linking #1516
Labels
No labels
UX
active development
backlog
blocker
bootstrap
bounty
bug
dependencies
discussion
documentation
duplicate
enhancement
flaky test
help wanted
invalid
javascript
question
release
tendentious
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
mighty-gerbils/gerbil#1516
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Two isolated findings from profiling fresh-process imports of
gerbil-poo::std/iofacade and load an otherwise unnecessarysocket/tempfile/crypto dependency closure for these consumers.
std/requests an object file thatis not produced for the empty static Scheme unit. This blocks a normal
compiled consumer of some narrowed standard-library interfaces.
No compiler/package source patch has been installed or submitted. This issue
records reproducible findings and a candidate direction, not release acceptance.
Results actually obtained after applying the isolated patches
The before/after results below are measured outcomes after applying the patches
shown in this issue. They are not projections from image counts or the Lean
model. The patches remain isolated; these are not results from an adopted
compiler release or the unchanged POO PR's CI.
Dependency patch: complete fresh-process wall time and tail both improved in
each measured order/profile. Positive percentages mean less elapsed time;
compilation is excluded. All timings use the same installed compiler, while
only the source-built JSON/LL1 dependency arm changes.
The first real operation was included as a separate anti-deferral control:
Within those processes, the paired median of the inner import-through-first-JSON
interval fell by 342 ms forward and 412 ms inverse. The JSON operation itself
stayed around 0.3 ms. The gain is import/startup work, with the complete MOP
retained; it is not parsing throughput or work deferred until first use.
Separate-link patch: a reproducible build failure becomes a working linked
consumer. This patch has a correctness result, not a separately measured
runtime speedup:
std/import-checkpure-reexport executablestd__import-check__public.o(also reproduced with installed2591dcd)REEXPORT-LINKED-OK, external module search disabledSTD-JSON-LINKED-OKPOO-JSON-LINKED-OKThe four linked JSON consumers perform actual nested false/null round trips.
Seven generated public interfaces retain their exports/bindings. Full compiler
CI, complete POO release acceptance, resource qualification and arbitrary
initializer-effect equivalence are still outstanding as described below.
Environment and observation boundary
1cfb032, Darwin provider patch618d76f9eb72, GCC 16. The separate-link failure also reproduces with theinstalled
2591dcdprovider (f5cedd8168cb). It is not claimed here for everyunpatched platform or compiler revision.
excluded from runtime comparisons. Standard JSON/LL1 sources were rebuilt
in both arms. Original installed files were not edited.
alternating AB/BA plus inverse arm assignment and A/A/B/B controls.
separately from the inner import interval. The P90 below is nearest-rank
for this small local sample, not a portable latency guarantee.
1. Dependency-closure finding
Candidate source changes use existing standard-library owners:
std/encoding/json/reader.ss:std/io:std/io/interfacestd/encoding/json/writer.ss:std/io:std/io/interfacestd/encoding/json/io.ss:std/io:std/io/interfacestd/encoding/json/util.ss:std/io(only-in :std/io/util with-buffered-reader with-buffered-writer)std/text/parser/ll1.ss:std/io:std/io/bio/api,(only-in :std/io/interface/bio BufferedReader),(only-in :std/io/util with-buffered-reader)Narrowing JSON alone did not remove the optional closure: LL1 still reached the
same facade. With both routes narrowed, the combined consumer's registry
observation fell from 157 to 129 phase-expansion images and 123 to 94 evaluation
images, with no added images in those observations. These phase observations
are not a count of distinct physical files across the whole process lifetime.
Validation so far:
38 cases each, via
gerbil test, including explicit success/error-markerchecks rather than relying on process exit alone.
.ssiinterfaces compare equal after omitting only theirdirect import declarations. Headers, exports, runtime/syntax binding
declarations and loader calls are retained. The public JSON facade, API and
environment interfaces are byte-identical.
dependency graph. A small Lean model establishes reachability containment
and optional-region exclusion under those premises. It does not prove
arbitrary initializer-effect equivalence, compiler correctness or wall time.
with the untraced pairs. Earlier POO-only import/C4/load-path candidates
without stable whole-process/tail gains were rejected.
2. Minimal separate-link reproduction
Use a fresh ordinary package directory and output prefix. The
std/prefix isessential to this reproduction: the same source under an ordinary user-package
prefix links successfully.
gerbil.pkg:value.ss:public.ss:main.ss:build.ss:Make
build.ssexecutable and invoke standardgerbil buildwith a freshGERBIL_PATHand the establishedGERBIL_BUILD_CORES/2 GiB settings.Observed linker error:
The compiler's separate-link path maps all
libgerbil-scmentries to C/objectpaths. The user-library path already filters empty Scheme units with the
existing
not-file-empty?helper. The isolated candidate applies that samefilter after
fold-libgerbil-runtime-scm, retainingdepsand the logicalbuiltin-modulesinventory:With the isolated rebuilt compiler, the minimal executable prints
REEXPORT-LINKED-OK. Both source-built JSON arms and both source-built POO JSONconsumers also link normally and perform a nested false/null roundtrip with
external module search disabled: four successful executable controls. The
38-case official suites were rerun successfully with that compiler as well.
Exact patches used in the isolated experiments
The dependency-only paired measurements above use this five-file standard-library
patch. It changes the dependency owner imported; it adds no loader or cache:
The linked correctness controls additionally use this driver patch against the
published Darwin-provider source baseline. The logical builtin-module inventory
stays unchanged; only empty physical Scheme units are omitted from link inputs:
Remaining acceptance and limits
release remain outstanding; the current POO CI does not validate this patch.
one C frontend exceeded 2 GiB RSS. It is not evidence of a whole-build 2 GiB
process-RSS qualification.
every initializer effect of every removed module is unobservable in arbitrary
external programs. Review the dependency/API boundary before adoption.
No system safety settings, priorities, loader, cache layer or MOP split was
introduced. Raw local logs, failed controls, source identities and the exact
five-file candidate patch are retained outside tracked sources; no benchmark
archives or generated bulk data are proposed for Git.
The builds and regressions use the existing
standard build template and
gerbil test; no alternate compiler or test runner is proposed.PS: Thanks a lot for contributing!
Also, Gerbil-POO itself was never designed for performance. But if you're interested in Prototype OO and performance, we should be talking, because I have plans.
@fare wrote in #1516 (comment):
The patch set I mentioned is for Darwin. I haven’t tested it on Linux yet. In my measurements, Gerbil builds on Darwin take roughly 5–10 times as long as comparable Linux builds. Macro expansion is a significant cost on Darwin and has had much less impact in my Linux CI runs. My current patched build appears to be -2–3 times faster on Darwin, but the underlying build bottleneck remains.
Gerbil-POO is one of the projects where I’ve run into this. I use it extensively, and building it on Darwin exposes a Gerbil bottleneck I want to investigate here. Macro expansion is particularly expensive on Darwin and contributes substantially to the slow build times I’m seeing. In my Linux CI runs, it has had much less impact. I haven’t tested the Darwin patch set on Linux yet. I recently moved from the US to China and don’t have a Linux machine available for local testing.
I’m working on two separate efforts. One focuses on improving the Darwin build experience. The other explores a deeper redesign of SMP, especially the shared heap.
I’ve included some initial reproduction notes in the issue. I’ll run more detailed Linux benchmarks in GitHub Actions to see whether I can reproduce the problem there and establish a comparable baseline. The Darwin slowdown is already visible in my GitHub Actions runs. My agent core will rely heavily on the Gerbil VM, so I want people to be able to build, develop, and run Gerbil efficiently on both Linux and Darwin.
@fare wrote in #1516 (comment):
sure, I developed(most with LLM assistant) a full feature of Gerbil-POO -> https://github.com/tao3k/lean-poo . Also inspired by doom and nixosModules, I designed modularPOO
I’d like Gerbil to have a community ecosystem for the AI era, inspired by Doom Emacs and NixOS modules. People could contribute their own modules and combine them to suit the way they work. I’ve already added Nix support and successfully deployed OpenTofu.
Guix has given the Guile community a shared project to build around, and I think that’s helped Guile thrive in recent years. I’d love to see Gerbil build that same spirit of contribution and go even further.
I’ve already built an agent implementation that can serve as an alternative to tools like Codex and OpenCode. Getting that working has given me confidence that I can help build this ecosystem.
By the way, What do you think if we reintroduce the racket like quick-check to gerbil?