v0.19: Add std/markup/djot #1509

jay · 2026-09-25T19:21:48Z

jay commented

2026-09-25 19:21:48 +00:00

Adds a native Djot parser and renderer under :std/text/markup/djot.

Typed syntax tree with optional source locations and :std/io input/output.
SXML and HTML rendering, including tables, footnotes, attributes, math, and raw content. Output is not sanitized.
Typed parser access and interface-based rendering.
Documentation, regression tests, and a native Gerbil benchmark. Also fixes SXML printing of literal symbol attribute names.

Benchmarks

Complete parsing into syntax trees, source locations disabled, no rendering. Gerbil compiled with gxc -O -exe, measured on an AMD Ryzen 9 7940HS at commit be13dea6.

Median milliseconds per parse across five timed batches. File reading and process startup are excluded; garbage collection during parsing is included.

Input	Gerbil
README	0.325
Syntax reference	1.026
Tutorial	0.118
Large repeated README (~1 MB)	26.646
Plain text	2.022
Malformed markup	3.365

Warning

LLM-assisted with gpt-6-astra and gerbil-mcp.

Adds a native Djot parser and renderer under `:std/text/markup/djot`. - Typed syntax tree with optional source locations and `:std/io` input/output. - SXML and HTML rendering, including tables, footnotes, attributes, math, and raw content. Output is not sanitized. - Typed parser access and interface-based rendering. - Documentation, regression tests, and a native Gerbil benchmark. Also fixes SXML printing of literal symbol attribute names. ### Benchmarks Complete parsing into syntax trees, source locations disabled, no rendering. Gerbil compiled with `gxc -O -exe`, measured on an AMD Ryzen 9 7940HS at commit `be13dea6`. Median milliseconds per parse across five timed batches. File reading and process startup are excluded; garbage collection during parsing is included. | Input | Gerbil | |---|---:| | README | 0.325 | | Syntax reference | 1.026 | | Tutorial | 0.118 | | Large repeated README (~1 MB) | 26.646 | | Plain text | 2.022 | | Malformed markup | 3.365 | > [!WARNING] > LLM-assisted with `gpt-6-astra` and `gerbil-mcp`.

jay added 7 commits

2026-09-25 19:21:48 +00:00

sxml: Preserve literal symbol attribute names de2fd941bd

Emit symbol names rather than their Scheme write representation, which
can add bars around numeric or colon-ending attribute names. This
matches string attribute keys and existing element-name emission.

djot: Add the native reader and typed syntax tree 86c06f8887

djot: Add SXML and HTML rendering 5f3384fa02

djot: Add conformance and regression coverage 99d55ba3c5

Keep all 302 pinned functional fixtures byte-for-byte unchanged. Adapt
only the three checkbox expectations to the existing HTML printer's
void-element spelling, without general normalization or skipped cases.

djot: Document parsing, rendering, and limits d4959153c9

desloppify 5d8fd13e63

formatting deaa582f48

jay added 1 commit

2026-09-25 19:22:30 +00:00

Merge branch 'v0.19-staging' into v0.19-djot 2f21716bc2

vyzo reviewed

2026-09-26 18:42:30 +00:00

vyzo left a comment

just a readability comment: can you ask astra to make the code sparser so that it is easier to read?

jay added 3 commits

2026-09-26 19:15:45 +00:00

djot: Reduce parser allocation and scanning costs 0b7b92b15b

Use compact slice maps and demand-grown token storage to reduce
allocation and repeated scanning. Preserve eager AST construction,
source coordinates, and resource checks.

Source-range lookup trades constant-time indexing for binary search
over slices; a located multiline recovery workload regresses 5.7%.

djot: Reduce parser hot-path overhead 9be0a029fe

djot: Fix attribute attachment and polish APIs d429d2af75

jay added 1 commit

2026-09-26 20:08:19 +00:00

more formatting 1d42ed0192

jay commented

2026-09-26 20:09:37 +00:00

@vyzo wrote in #1509 (comment):

just a readability comment: can you ask astra to make the code sparser so that it is easier to read?

sure thing, done. still need to do some benchmarking and write up a description. you are welcome to look over the code though, it's basically done

@vyzo wrote in https://git.cons.io/mighty-gerbils/gerbil/pulls/1509#issuecomment-2960: > just a readability comment: can you ask astra to make the code sparser so that it is easier to read? sure thing, done. still need to do some benchmarking and write up a description. you are welcome to look over the code though, it's basically done

jay changed title from ~~WIP: v0.19: Add std/markup/djot~~ to v0.19: Add std/markup/djot

2026-09-27 20:31:19 +00:00

jay requested review from contributors

2026-09-27 20:31:26 +00:00

jay commented

2026-09-28 00:16:27 +00:00

actually, now working on some more optimizations, i found this https://github.com/dcampbell24/djot-implementations/
and I see we are quite a bit behind the C parser in performance. unacceptable!!

Input	Gerbil parse	JS parse	Gerbil → HTML	C → HTML
README	0.378 ms	0.318 ms	2.558 ms	0.059 ms
Pandoc manual	11.909 ms	14.010 ms	85.021 ms	1.561 ms
Tartan	127.874 ms	107.001 ms	421.235 ms	11.613 ms

actually, now working on some more optimizations, i found this https://github.com/dcampbell24/djot-implementations/ and I see we are quite a bit behind the C parser in performance. unacceptable!! | Input | Gerbil parse | JS parse | Gerbil → HTML | C → HTML | |---|---:|---:|---:|---:| | README | 0.378 ms | 0.318 ms | 2.558 ms | 0.059 ms | | Pandoc manual | 11.909 ms | 14.010 ms | 85.021 ms | 1.561 ms | | Tartan | 127.874 ms | 107.001 ms | 421.235 ms | 11.613 ms |

vyzo commented

2026-09-28 11:45:53 +00:00

ok, let me review and i can probably give some optimization advice. the gap to C is pretty big, but let's not throw safety out the window to close it. Also it provides a good benchmark we can use for the type-gerbil optimizer work.

vyzo commented

2026-09-28 11:48:51 +00:00

so is the benchmark checked in? one obvious potential pitfall is using a port for the read; you should use directly a Reader, you can get one with call-with-file-reader from std/io.

so is the benchmark checked in? one obvious potential pitfall is using a port for the read; you should use directly a Reader, you can get one with `call-with-file-reader` from std/io.

vyzo commented

2026-09-28 11:50:48 +00:00

we probably need an optimized writer for sxml, using a BufferedWriter instead of a port. maybe a bit out of scope for this pr, but definitely relevant to benchmarking.

fare commented

2026-09-28 12:59:55 +00:00

S-expressions are not a very efficient data representation for HTML, either, if what you're looking for is performance.

vyzo reviewed

2026-09-28 15:25:51 +00:00

vyzo left a comment

first pass review, mainly looking at how we can improve the code and its performance:

use dotted notation for all annotated types, and use using liberally. this results in compiler checkable and eliminable contracts and unchecked accessors.
avoid big linear scans on types with cond; use an interface and dispatch directly.
big linear case checks could benefit from using hash tables, but this is something to measure, it might not.

first pass review, mainly looking at how we can improve the code and its performance: - use dotted notation for all annotated types, and use using liberally. this results in compiler checkable and eliminable contracts and unchecked accessors. - avoid big linear scans on types with cond; use an interface and dispatch directly. - big linear case checks could benefit from using hash tables, but this is something to measure, it might not.

src/std/text/markup/djot/ast.ss

					
				@ -0,0 +96,4 @@

				   (footnotes       : :list))

				  transparent: #t)

				(defstruct (djot-paragraph djot-container)

you should mark the leaves of the AST with final: #t. results in faster predicate and accessors/mutators.

you should mark the leaves of the AST with `final: #t`. results in faster predicate and accessors/mutators.

also note that transparent: #t is unnecessary, it is the default.

src/std/text/markup/djot/attributes.ss Outdated

					
				@ -0,0 +109,4 @@

				    (if (or (= p end) (memq (attribute-parser-state parser) '(done fail))) p

				      (let ((c (string-ref input p))

				            (state (attribute-parser-state parser)))

				        (case state

this could benefit from a hash table.

src/std/text/markup/djot/html.ss Outdated

					
				@ -0,0 +154,4 @@

				                (extra    : :list := [])) => :list

				    (render-element tag node extra (children) newlines))

				  (cond

this is a big linear scan, i think it could benefit from an interface and dispatching the prerequisite methods. probably something to measure.

src/std/text/markup/djot/inline.ss

					
				@ -0,0 +620,4 @@

				(def (inline-container (kind     : :symbol)

				                       (source   :? djot-source-range)

				                       (children : :list)) => djot-container

				  (case kind

this could probably benefit from a hash table.

src/std/text/markup/djot/parse.ss Outdated

					
				@ -0,0 +7,4 @@

				(export string->djot)

				;;; Public Gambit Unicode character sets have no Gerbil wrapper in v0.19.

				(extern namespace: #f char-set:punctuation char-set-contains?)

this should be added to the runtime module in the prelude.

src/std/text/markup/djot/parse.ss

					
				@ -0,0 +114,4 @@

				                  candidate)))))))

				    (def (visit (node : djot-node)) => :void

				      (let (identifier (node-identifier node))

you have a type annotation, use dotted notation!

vyzo commented

2026-09-28 15:28:24 +00:00

i agree with fare; SSXML is ancient and not the fastest of cats. We could make a properly typed xml module, to replace sxml and html could be implemented using that. probably not in this pr though, it is already large, but it could be follow up work where we can measure improvements.

now, for immediate performance improvements see my comments, and also use a Reader instead of a port directly for reading the file to benchmark, this will make an immediate (but maybe small) difference.

i agree with fare; SSXML is ancient and not the fastest of cats. We could make a properly typed xml module, to replace sxml and html could be implemented using that. probably not in this pr though, it is already large, but it could be follow up work where we can measure improvements. now, for immediate performance improvements see my comments, and also use a Reader instead of a port directly for reading the file to benchmark, this will make an immediate (but maybe small) difference.

jay added 1 commit

2026-09-28 18:56:22 +00:00

Preserve SXML rendering and optimize typed Djot dispatch af09a22646

jay changed title from ~~v0.19: Add std/markup/djot~~ to WIP: v0.19: Add std/markup/djot

2026-09-28 18:59:58 +00:00

jay added 2 commits

2026-09-28 19:16:31 +00:00

Use a native Gerbil benchmark for Djot 49e5ee1a69

Keep Djot benchmark guidance in source comments 583c19e97a

jay added 1 commit

2026-09-28 19:27:59 +00:00

djot: Simplify internal type refinements be13dea6e8

jay added 1 commit

2026-09-28 20:14:19 +00:00

djot: Handle large text and empty footnote labels 60f7f514c7

jay changed title from ~~WIP: v0.19: Add std/markup/djot~~ to v0.19: Add std/markup/djot

2026-09-28 20:14:35 +00:00

jay commented

2026-09-28 20:29:24 +00:00

@vyzo addressed your feedback, i kept the SXML intermediary for now
there is definitely a lot of meat left on the bones in terms of performance gains, though

@vyzo addressed your feedback, i kept the SXML intermediary for now there is definitely a lot of meat left on the bones in terms of performance gains, though

vyzo commented

2026-09-28 20:35:36 +00:00

yeah, but at least my quick feedback already produced gains; i looked in the benchmark.md and there seems to be quite an improvement -- clearly ahead of js now.

I think maybe we could improve performance by stopping parsing strings (and prereading a giant string) and just reading bytes/utf8 characters straight from the reader. Also it is likely that location tracking is expensive, we could have a flag to turn that off. It is useful for reporting errors, but it does cost, potentially quite a bit. Can you check with astra how hard it would be to make this transition to parsing from a buffered reader? Probably not so hard. This will also substantially improve memory usage as well, those big strings are heavy and memory hungry.

next we should figure out what to do with sxml; i think std/markup/xml and std/markup/html packages with properly structured xml and html are the way to go and now we have a nice benchmark we can use to measure things. But as I said this is for another pr, you want to take it?

yeah, but at least my quick feedback already produced gains; i looked in the benchmark.md and there seems to be quite an improvement -- clearly ahead of js now. I think maybe we could improve performance by stopping parsing strings (and prereading a giant string) and just reading bytes/utf8 characters straight from the reader. Also it is likely that location tracking is expensive, we could have a flag to turn that off. It is useful for reporting errors, but it does cost, potentially quite a bit. Can you check with astra how hard it would be to make this transition to parsing from a buffered reader? Probably not so hard. This will also substantially improve memory usage as well, those big strings are heavy and memory hungry. next we should figure out what to do with sxml; i think std/markup/xml and std/markup/html packages with properly structured xml and html are the way to go and now we have a nice benchmark we can use to measure things. But as I said this is for another pr, you want to take it?

jay commented

2026-09-28 21:17:28 +00:00

@vyzo yeah I can take it, and you were thinking to remove sxml yes? like fully replace with std/markup/xml?

vyzo commented

2026-09-29 08:16:15 +00:00

yea, sxml is ancient and not exactly a joy to work with.

vyzo commented

2026-09-29 08:29:46 +00:00

@jay what do you think of trying the parser working directly with a (buffered) reader instead of parsing strings? We need to measure its performance, and we have benchmarks already.

jay added 1 commit

2026-10-01 02:44:13 +00:00

Merge branch 'v0.19-staging' into v0.19-djot b7e6682f25

jay commented

2026-10-01 03:09:22 +00:00

@vyzo I'm working on addressing your feedback. I had astra try switching to buffered reader but did not see performance gains (~15–50% slower, although lower peak memory), but looking to see if there are additional optimizations

vyzo commented

2026-10-01 08:28:00 +00:00

Ok, it was worth a try.

in this case for the benchmark just read the whole file at once with a buffer sized at the file size, read-all-from-reader does many small reads and then concatenates.

Ok, it was worth a try. in this case for the benchmark just read the whole file at once with a buffer sized at the file size, read-all-from-reader does many small reads and then concatenates.

vyzo commented

2026-10-01 08:29:00 +00:00

you can probably just use read-file-u8vector.

jay added 1 commit

2026-10-03 19:44:00 +00:00

read benchmark files in one sized buffer ec64835363

jay commented

2026-10-03 19:48:25 +00:00

@vyzo done, benchmark now uses read-file-u8vector for all file loading

maybe we can pursue optimizing this more later, but the public API will be unchanged so this can unlock working on the documentation generator now. are you good with that?

@vyzo done, benchmark now uses `read-file-u8vector` for all file loading maybe we can pursue optimizing this more later, but the public API will be unchanged so this can unlock working on the documentation generator now. are you good with that?

vyzo commented

2026-10-05 08:37:15 +00:00

yeah totally, lets make progress.
I'll do another pass, and @fare can point the clanker for the comprehensive review.

yeah totally, lets make progress. I'll do another pass, and @fare can point the clanker for the comprehensive review.

fare commented

2026-10-05 10:57:44 +00:00

GPT 6.1 Sol Light says:

Reviewed head ec648353. I found one parser regression and two renderer issues. The latter two also occur in pinned djot.js 0.3.2.

[P2] Completing a reference link clears an enclosing destination.
inline.ss:393
```
[x](a[b][c])
```
When [b][c] completes, finish-inline-link! unconditionally sets parser.destination? to #f. The outer ) consequently cannot complete the original link.

Upstream produces:
```
<p><a href="a[b][c]">x</a></p>
```
Preserve destination state when completing a reference link. Add this case and its image variant as regressions.
[P2] Generated and explicit attributes can produce duplicate HTML attributes.
html.ss:59–70
```
<https://example.org>{href="/override"}
```
render-attributes appends both generated and explicit attributes, only special-casing class. This produces two href attributes; browsers retain the first, ignoring the explicit override. Footnote references similarly permit duplicate id, href and role.

Ordinary links/images already filter these collisions in render-reference. Apply a consistent precedence rule in the common attribute renderer and assert uniqueness in SXML as well as HTML.
Unused footnotes can cause other unused footnotes to appear.
html.ss:621–625
```
text[^used]

[^unused]: hidden[^extra]

[^used]: used

[^extra]: leaked
```
Rendering every definition discovers extra through the unused note, so “leaked” appears as footnote 2. Its backlink targets fnref2, which occurs only in discarded output.

Render definitions reachable from visible references, extending a queue as nested references are discovered. Mark notes before traversing them to handle cycles. If retaining upstream behavior is intentional, I’d track this as a follow-up.

The test coverage is substantial, and keeping the SXML replacement outside this PR seems sensible. I’d fix item 1 before merging. I ran the examples against upstream 0.3.2; the Gerbil findings are from source inspection because this environment has no Gerbil executable.

GPT 6.1 Sol Light says: Reviewed head `ec648353`. **I found one parser regression and two renderer issues.** The latter two also occur in pinned djot.js 0.3.2. 1. **[P2] Completing a reference link clears an enclosing destination.** [inline.ss:393](https://git.cons.io/jay/gerbil/src/commit/ec648353630029e8e9cdf63b9796fbe82d01433c/src/std/text/markup/djot/inline.ss#L393) ```djot [x](a[b][c]) ``` When `[b][c]` completes, `finish-inline-link!` unconditionally sets `parser.destination?` to `#f`. The outer `)` consequently cannot complete the original link. Upstream produces: ```html <p><a href="a[b][c]">x</a></p> ``` Preserve destination state when completing a **reference** link. Add this case and its image variant as regressions. 2. **[P2] Generated and explicit attributes can produce duplicate HTML attributes.** [html.ss:59–70](https://git.cons.io/jay/gerbil/src/commit/ec648353630029e8e9cdf63b9796fbe82d01433c/src/std/text/markup/djot/html.ss#L59) ```djot <https://example.org>{href="/override"} ``` `render-attributes` appends both generated and explicit attributes, only special-casing `class`. This produces two `href` attributes; browsers retain the first, ignoring the explicit override. Footnote references similarly permit duplicate `id`, `href` and `role`. Ordinary links/images already filter these collisions in `render-reference`. Apply a consistent precedence rule in the common attribute renderer and assert uniqueness in SXML as well as HTML. 3. **Unused footnotes can cause other unused footnotes to appear.** [html.ss:621–625](https://git.cons.io/jay/gerbil/src/commit/ec648353630029e8e9cdf63b9796fbe82d01433c/src/std/text/markup/djot/html.ss#L621) ```djot text[^used] [^unused]: hidden[^extra] [^used]: used [^extra]: leaked ``` Rendering every definition discovers `extra` through the unused note, so “leaked” appears as footnote 2. Its backlink targets `fnref2`, which occurs only in discarded output. Render definitions reachable from visible references, extending a queue as nested references are discovered. Mark notes before traversing them to handle cycles. If retaining upstream behavior is intentional, I’d track this as a follow-up. The test coverage is substantial, and keeping the SXML replacement outside this PR seems sensible. **I’d fix item 1 before merging.** I ran the examples against upstream 0.3.2; the Gerbil findings are from source inspection because this environment has no Gerbil executable.

jay added 1 commit

2026-10-05 22:27:51 +00:00

preserve destinations across reference links 82cd7eec0b

jay commented

2026-10-05 22:29:23 +00:00

@fare I pushed a fix for 1

for 2 and 3 I am hesitant to fix those since they would be diverging from the "official" djot.js parser, but they do indeed look like bugs. I'll see about opening an issue on that repo, and if it does need fixing we can do that here later

@fare I pushed a fix for 1 for 2 and 3 I am hesitant to fix those since they would be diverging from the "official" djot.js parser, but they do indeed look like bugs. I'll see about opening an issue on that repo, and if it does need fixing we can do that here later

fare commented

2026-10-06 04:36:08 +00:00

I don't have strong opinions for what the "correct" behavior should be for the other issues, especially if upstream doesn't have their shit together. However, IF that is not too hard, it is better to issue an error when finding a badly defined situation than making up a bad answer, even one identical to upstream. i.e. I'd rather we err on the side of being too strict than that of spewing likely unintended nonsense as output.

I don't have strong opinions for what the "correct" behavior should be for the other issues, especially if upstream doesn't have their shit together. However, *IF* that is not too hard, it is better to issue an error when finding a badly defined situation than making up a bad answer, even one identical to upstream. i.e. I'd rather we err on the side of being too strict than that of spewing likely unintended nonsense as output.

jay added 2 commits

2026-10-07 20:07:29 +00:00

djot: Avoid duplicate HTML attributes 82dde1ff16

Keep the first attribute value to match existing browser behavior.
Compare names using ASCII case folding and preserve class merging.

djot: Render only reachable footnotes 8d4b124850

Avoid discovering references in unused definitions, which created
extra notes with backlinks into discarded output.

jay commented

2026-10-07 20:09:07 +00:00

@fare you were right, these were renderer bugs, not incorrect specs from the Djot spec. I compared djot.js with cdjot and Jotdown, and neither has those bugs. Both issues are fixed now.

@fare you were right, these were renderer bugs, not incorrect specs from the [Djot spec](https://github.com/jgm/djot/blob/main/doc/syntax.md). I compared `djot.js` with `cdjot` and `Jotdown`, and neither has those bugs. Both issues are fixed now.

fare commented

2026-10-07 20:52:12 +00:00

LGT Sol 6.1 Light, that adds:

One documentation follow-up: update the Compatibility paragraph in djot.md to mention attribute deduplication and reachable-only footnote rendering, since both intentionally differ from djot.js 0.3.2.

LGT Sol 6.1 Light, that adds: One documentation follow-up: update the Compatibility paragraph in djot.md to mention attribute deduplication and reachable-only footnote rendering, since both intentionally differ from djot.js 0.3.2.

fare commented

2026-10-07 20:54:10 +00:00

semi-relatedly, what's the plan for extracting running tests from documentation examples?