v0.19: Add std/markup/djot #1509

jay · 2026-09-25T19:21:48Z

jay commented

2026-09-25 19:21:48 +00:00

Adds a native Djot parser and renderer under :std/text/markup/djot.

Typed syntax tree with optional source locations and :std/io input/output.
SXML and HTML rendering, including tables, footnotes, attributes, math, and raw content. Output is not sanitized.
Typed parser access and interface-based rendering.
Documentation, regression tests, and a native Gerbil benchmark. Also fixes SXML printing of literal symbol attribute names.

Benchmarks

Complete parsing into syntax trees, source locations disabled, no rendering. Gerbil compiled with gxc -O -exe, measured on an AMD Ryzen 9 7940HS at commit be13dea6.

Median milliseconds per parse across five timed batches. File reading and process startup are excluded; garbage collection during parsing is included.

Input	Gerbil
README	0.381
Syntax reference	1.156
Tutorial	0.140
Large repeated README (~1 MB)	31.555
Plain text	3.029
Malformed markup	3.858

Warning

LLM-assisted with gpt-6-astra and gerbil-mcp.

Adds a native Djot parser and renderer under `:std/text/markup/djot`. - Typed syntax tree with optional source locations and `:std/io` input/output. - SXML and HTML rendering, including tables, footnotes, attributes, math, and raw content. Output is not sanitized. - Typed parser access and interface-based rendering. - Documentation, regression tests, and a native Gerbil benchmark. Also fixes SXML printing of literal symbol attribute names. ### Benchmarks Complete parsing into syntax trees, source locations disabled, no rendering. Gerbil compiled with `gxc -O -exe`, measured on an AMD Ryzen 9 7940HS at commit `be13dea6`. Median milliseconds per parse across five timed batches. File reading and process startup are excluded; garbage collection during parsing is included. | Input | Gerbil | |---|---:| | README | 0.381 | | Syntax reference | 1.156 | | Tutorial | 0.140 | | Large repeated README (~1 MB) | 31.555 | | Plain text | 3.029 | | Malformed markup | 3.858 | > [!WARNING] > LLM-assisted with `gpt-6-astra` and `gerbil-mcp`.

jay added 7 commits

2026-09-25 19:21:48 +00:00

sxml: Preserve literal symbol attribute names de2fd941bd

Emit symbol names rather than their Scheme write representation, which
can add bars around numeric or colon-ending attribute names. This
matches string attribute keys and existing element-name emission.

djot: Add the native reader and typed syntax tree 86c06f8887

djot: Add SXML and HTML rendering 5f3384fa02

djot: Add conformance and regression coverage 99d55ba3c5

Keep all 302 pinned functional fixtures byte-for-byte unchanged. Adapt
only the three checkbox expectations to the existing HTML printer's
void-element spelling, without general normalization or skipped cases.

djot: Document parsing, rendering, and limits d4959153c9

desloppify 5d8fd13e63

formatting deaa582f48

jay added 1 commit

2026-09-25 19:22:30 +00:00

Merge branch 'v0.19-staging' into v0.19-djot 2f21716bc2

vyzo reviewed

2026-09-26 18:42:30 +00:00

vyzo left a comment

just a readability comment: can you ask astra to make the code sparser so that it is easier to read?

jay added 3 commits

2026-09-26 19:15:45 +00:00

djot: Reduce parser allocation and scanning costs 0b7b92b15b

Use compact slice maps and demand-grown token storage to reduce
allocation and repeated scanning. Preserve eager AST construction,
source coordinates, and resource checks.

Source-range lookup trades constant-time indexing for binary search
over slices; a located multiline recovery workload regresses 5.7%.

djot: Reduce parser hot-path overhead 9be0a029fe

djot: Fix attribute attachment and polish APIs d429d2af75

jay added 1 commit

2026-09-26 20:08:19 +00:00

more formatting 1d42ed0192

jay commented

2026-09-26 20:09:37 +00:00

@vyzo wrote in #1509 (comment):

just a readability comment: can you ask astra to make the code sparser so that it is easier to read?

sure thing, done. still need to do some benchmarking and write up a description. you are welcome to look over the code though, it's basically done

@vyzo wrote in https://git.cons.io/mighty-gerbils/gerbil/pulls/1509#issuecomment-2960: > just a readability comment: can you ask astra to make the code sparser so that it is easier to read? sure thing, done. still need to do some benchmarking and write up a description. you are welcome to look over the code though, it's basically done

jay changed title from ~~WIP: v0.19: Add std/markup/djot~~ to v0.19: Add std/markup/djot

2026-09-27 20:31:19 +00:00

jay requested review from contributors

2026-09-27 20:31:26 +00:00

jay commented

2026-09-28 00:16:27 +00:00

actually, now working on some more optimizations, i found this https://github.com/dcampbell24/djot-implementations/
and I see we are quite a bit behind the C parser in performance. unacceptable!!

Input	Gerbil parse	JS parse	Gerbil → HTML	C → HTML
README	0.378 ms	0.318 ms	2.558 ms	0.059 ms
Pandoc manual	11.909 ms	14.010 ms	85.021 ms	1.561 ms
Tartan	127.874 ms	107.001 ms	421.235 ms	11.613 ms

actually, now working on some more optimizations, i found this https://github.com/dcampbell24/djot-implementations/ and I see we are quite a bit behind the C parser in performance. unacceptable!! | Input | Gerbil parse | JS parse | Gerbil → HTML | C → HTML | |---|---:|---:|---:|---:| | README | 0.378 ms | 0.318 ms | 2.558 ms | 0.059 ms | | Pandoc manual | 11.909 ms | 14.010 ms | 85.021 ms | 1.561 ms | | Tartan | 127.874 ms | 107.001 ms | 421.235 ms | 11.613 ms |

vyzo commented

2026-09-28 11:45:53 +00:00

ok, let me review and i can probably give some optimization advice. the gap to C is pretty big, but let's not throw safety out the window to close it. Also it provides a good benchmark we can use for the type-gerbil optimizer work.

vyzo commented

2026-09-28 11:48:51 +00:00

so is the benchmark checked in? one obvious potential pitfall is using a port for the read; you should use directly a Reader, you can get one with call-with-file-reader from std/io.

so is the benchmark checked in? one obvious potential pitfall is using a port for the read; you should use directly a Reader, you can get one with `call-with-file-reader` from std/io.

vyzo commented

2026-09-28 11:50:48 +00:00

we probably need an optimized writer for sxml, using a BufferedWriter instead of a port. maybe a bit out of scope for this pr, but definitely relevant to benchmarking.

fare commented

2026-09-28 12:59:55 +00:00

S-expressions are not a very efficient data representation for HTML, either, if what you're looking for is performance.

vyzo reviewed

2026-09-28 15:25:51 +00:00

vyzo left a comment

first pass review, mainly looking at how we can improve the code and its performance:

use dotted notation for all annotated types, and use using liberally. this results in compiler checkable and eliminable contracts and unchecked accessors.
avoid big linear scans on types with cond; use an interface and dispatch directly.
big linear case checks could benefit from using hash tables, but this is something to measure, it might not.

first pass review, mainly looking at how we can improve the code and its performance: - use dotted notation for all annotated types, and use using liberally. this results in compiler checkable and eliminable contracts and unchecked accessors. - avoid big linear scans on types with cond; use an interface and dispatch directly. - big linear case checks could benefit from using hash tables, but this is something to measure, it might not.

src/std/text/markup/djot/ast.ss

					
				@ -0,0 +96,4 @@

				   (footnotes       : :list))

				  transparent: #t)

				(defstruct (djot-paragraph djot-container)

you should mark the leaves of the AST with final: #t. results in faster predicate and accessors/mutators.

you should mark the leaves of the AST with `final: #t`. results in faster predicate and accessors/mutators.

also note that transparent: #t is unnecessary, it is the default.

src/std/text/markup/djot/attributes.ss Outdated

					
				@ -0,0 +109,4 @@

				    (if (or (= p end) (memq (attribute-parser-state parser) '(done fail))) p

				      (let ((c (string-ref input p))

				            (state (attribute-parser-state parser)))

				        (case state

this could benefit from a hash table.

src/std/text/markup/djot/html.ss Outdated

					
				@ -0,0 +154,4 @@

				                (extra    : :list := [])) => :list

				    (render-element tag node extra (children) newlines))

				  (cond

this is a big linear scan, i think it could benefit from an interface and dispatching the prerequisite methods. probably something to measure.

src/std/text/markup/djot/inline.ss

					
				@ -0,0 +620,4 @@

				(def (inline-container (kind     : :symbol)

				                       (source   :? djot-source-range)

				                       (children : :list)) => djot-container

				  (case kind

this could probably benefit from a hash table.

src/std/text/markup/djot/parse.ss Outdated

					
				@ -0,0 +7,4 @@

				(export string->djot)

				;;; Public Gambit Unicode character sets have no Gerbil wrapper in v0.19.

				(extern namespace: #f char-set:punctuation char-set-contains?)

this should be added to the runtime module in the prelude.

src/std/text/markup/djot/parse.ss

					
				@ -0,0 +114,4 @@

				                  candidate)))))))

				    (def (visit (node : djot-node)) => :void

				      (let (identifier (node-identifier node))

you have a type annotation, use dotted notation!

vyzo commented

2026-09-28 15:28:24 +00:00

i agree with fare; SSXML is ancient and not the fastest of cats. We could make a properly typed xml module, to replace sxml and html could be implemented using that. probably not in this pr though, it is already large, but it could be follow up work where we can measure improvements.

now, for immediate performance improvements see my comments, and also use a Reader instead of a port directly for reading the file to benchmark, this will make an immediate (but maybe small) difference.

i agree with fare; SSXML is ancient and not the fastest of cats. We could make a properly typed xml module, to replace sxml and html could be implemented using that. probably not in this pr though, it is already large, but it could be follow up work where we can measure improvements. now, for immediate performance improvements see my comments, and also use a Reader instead of a port directly for reading the file to benchmark, this will make an immediate (but maybe small) difference.

jay added 1 commit

2026-09-28 18:56:22 +00:00

Preserve SXML rendering and optimize typed Djot dispatch af09a22646

jay changed title from ~~v0.19: Add std/markup/djot~~ to WIP: v0.19: Add std/markup/djot

2026-09-28 18:59:58 +00:00

jay added 2 commits

2026-09-28 19:16:31 +00:00

Use a native Gerbil benchmark for Djot 49e5ee1a69

Keep Djot benchmark guidance in source comments 583c19e97a

jay added 1 commit

2026-09-28 19:27:59 +00:00

djot: Simplify internal type refinements be13dea6e8

jay added 1 commit

2026-09-28 20:14:19 +00:00

djot: Handle large text and empty footnote labels 60f7f514c7

jay changed title from ~~WIP: v0.19: Add std/markup/djot~~ to v0.19: Add std/markup/djot

2026-09-28 20:14:35 +00:00

jay commented

2026-09-28 20:29:24 +00:00

@vyzo addressed your feedback, i kept the SXML intermediary for now
there is definitely a lot of meat left on the bones in terms of performance gains, though

@vyzo addressed your feedback, i kept the SXML intermediary for now there is definitely a lot of meat left on the bones in terms of performance gains, though

vyzo commented

2026-09-28 20:35:36 +00:00

yeah, but at least my quick feedback already produced gains; i looked in the benchmark.md and there seems to be quite an improvement -- clearly ahead of js now.

I think maybe we could improve performance by stopping parsing strings (and prereading a giant string) and just reading bytes/utf8 characters straight from the reader. Also it is likely that location tracking is expensive, we could have a flag to turn that off. It is useful for reporting errors, but it does cost, potentially quite a bit. Can you check with astra how hard it would be to make this transition to parsing from a buffered reader? Probably not so hard. This will also substantially improve memory usage as well, those big strings are heavy and memory hungry.

next we should figure out what to do with sxml; i think std/markup/xml and std/markup/html packages with properly structured xml and html are the way to go and now we have a nice benchmark we can use to measure things. But as I said this is for another pr, you want to take it?

yeah, but at least my quick feedback already produced gains; i looked in the benchmark.md and there seems to be quite an improvement -- clearly ahead of js now. I think maybe we could improve performance by stopping parsing strings (and prereading a giant string) and just reading bytes/utf8 characters straight from the reader. Also it is likely that location tracking is expensive, we could have a flag to turn that off. It is useful for reporting errors, but it does cost, potentially quite a bit. Can you check with astra how hard it would be to make this transition to parsing from a buffered reader? Probably not so hard. This will also substantially improve memory usage as well, those big strings are heavy and memory hungry. next we should figure out what to do with sxml; i think std/markup/xml and std/markup/html packages with properly structured xml and html are the way to go and now we have a nice benchmark we can use to measure things. But as I said this is for another pr, you want to take it?

jay commented

2026-09-28 21:17:28 +00:00

@vyzo yeah I can take it, and you were thinking to remove sxml yes? like fully replace with std/markup/xml?

vyzo commented

2026-09-29 08:16:15 +00:00

yea, sxml is ancient and not exactly a joy to work with.

vyzo commented

2026-09-29 08:29:46 +00:00

@jay what do you think of trying the parser working directly with a (buffered) reader instead of parsing strings? We need to measure its performance, and we have benchmarks already.

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u v0.19-djot:jay-v0.19-djot

git switch jay-v0.19-djot

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch v0.19-staging

git merge --no-ff jay-v0.19-djot

git switch jay-v0.19-djot

git rebase v0.19-staging

git switch v0.19-staging

git merge --ff-only jay-v0.19-djot

git switch jay-v0.19-djot

git rebase v0.19-staging

git switch v0.19-staging

git merge --no-ff jay-v0.19-djot

git switch v0.19-staging

git merge --squash jay-v0.19-djot