STAR: settling the details
On the path to stabilizing STreaming ARchives
fig (aka:[phil])Oct 27, 2026
These are the aspects of STAR which I feel haven't been fully settled yet, and a summary of all other feedback to date. Thoughts and pushback welcome! I'll keep an eye out for follow-ups on any of the linked places below, and on responses to this post.
The current draft for the STAR spec in DASL is in PR: https://tangled.org/dasl.ing/dasl.ing/pulls/16/round/2 -- it's a rewrite since the last microcosm STAR version, any inconsistency should defer to the DASL PR.
Main unsettled questions [section still TODO]
Q1: Format determinism
- varint: SHOULD vs MUST for readers -- stdlib varint decoders are typically not strict, so it might be nice to make readers checking be a SHOULD.
Q2: Metadata $type authority
- authority: should we recommend using domain authority (like NSID)
- with optional
#frag(bryan newbold -- can you say more about this? do we have a syntax in atproto for bare nsid with a fragment suffix already?)
- with optional
- my current view is that there is no official authority of STAR
$types. It's useful for content negotiation. producers and consumers usually know who eachother are and why they are talking to eachother, so when i'm processing a$type: atproto-space-repo, it's not an random STAR from the universe. If someone makes an unrelated application profile and uses the same$type, it's only a problem if they start getting collected together. ...- is this naive? we already almost could have been at risk of a collision like that, since I was going to make an independent STAR format for atproto spaces before spaces got on board with adopting STAR. if the adoption had happened later, the "official" application profile might have needed to pick a worse
$typestring to avoid a collision with my independent one, or reuse it "in the same universe" and have more of a risk of actual $type confusion. - i hope that some basic guidance in creating application profiles (prefix it with a name you feel authoritative over, recommend nsid, whatever) can mostly avoid this?
- is this naive? we already almost could have been at risk of a collision like that, since I was going to make an independent STAR format for atproto spaces before spaces got on board with adopting STAR. if the adoption had happened later, the "official" application profile might have needed to pick a worse
Q3: Data control bytes
- Right now there is no explicit end-of-archive signal -- an archive truncated at the boundary of any data pair is a valid archive, at least at the format level. So, unless applications add their own check that covers truncation, they can silently miss data when reading truncated archives. Currently, that will just be an Application Profile guidance: put a "number of records" or "MST root hash" or lthash over the set or checksum of keys or whatever, in the
meta.- I don't think there's a really nice way to do an end-of-archive signal with the current concatenated (len | key | len | val) format. anything in-band would be unfortunate... i suppose we could reserve <empty key | empty val> or something but i don't like it. The only other thing I can see is using an invalid
len... which means a non-canonical len. This seems like a very bad idea, since it ties an internal varint detail to the format, and might lead to unsafe failures with decoders that permit non-minimalvarints on read. I don't like it.
- I don't think there's a really nice way to do an end-of-archive signal with the current concatenated (len | key | len | val) format. anything in-band would be unfortunate... i suppose we could reserve <empty key | empty val> or something but i don't like it. The only other thing I can see is using an invalid
- A control byte between each data pair would provide an opportunity for signalling like end-of-archive. Like,
0xAAmeans "another keypair" (endless scream);0xEEmeans "end of archive" (eeee stop!), etc.- I don't think we'd want any other control bytes, but if we ever did, there'd be a place for that.
- It's not much overhead except in applications with very small keys and values, and/but would be near-free in size when archives are compressed.
- My biggest gripe is aesthetic. It feels like it adds noisiness to the currently-very-simple description. I can get over it if we need it. Do we need it? CAR gets by without, but we're aiming for stricter-than-CAR.
Q4: Format-level maximum sizes
- I dropped them in the DASL draft, since CAR's specs don't list any. The numbers I picked (64MiB meta and keys, 64 GiB values) felt very arbitrary. Application profiles should require smaller limits that suit their needs, and reader implementations should provide interfaces to enforce lower limits before attempting to read any content anyway... so what does a spec-level limit add? I think values should be allowed to be pretty big (readers can even offer chunked value-streaming interfaces), which makes it hard to pick a number that feels like a meaningful backstop for memory exhaustion. But, having no limits specified seems like playing loose with it. Maybe it's fine?
Initial dasl.ing issue comments
- Actually there was not really any pushback in this thread, mostly very nice words of support :)))
STAR in atproto spaces PR comments
- Nick Gerakines:
$typeshould be be an NSID like all other lexicons so it can be resolved to a lexicon schema to be described, documented, etc. (See Q2) - daniel holmgren 🫠: option example: DASL group maintains a list of STAR profiles (See Q2)
- bryan newbold: we might recommend folks defining new STAR profiles use NSID in
$type. It might not be ideal to blur or create ambiguity over the namespace of a "protocol-level string" (See Q2) - Matthieu 🦋: NSID makes makes publication of a new version of the schema easier (though
com.atprotolexicons are atproto spec). alternatively could use a different string than$typeto avoid confusion. (See Q2)
Review/Feedback (2026-10-16) from bryan newbold
- Rename this just STAR, from STAR-lite
- Security section TODOs
$typepreference (see Q2)- Termination indicator (see Q3)
- Per-block integrity checks, more integrity checks generally
- Bijou64 discouraged
- Encountering "EOF" (earlier proposal's poor wording choice)
Initial dasl.ing PR comments
- Mosh Lee: "parse errors should be considered fatal" -> "must"
- Mosh Lee: dasl spec should have some links out while in HNS status
- Volker Mische: clarification of canonical varint prefix wording
- Volker Mische: ordered pairs (sorted vs ordered). suggestion to call it "sequenced" instead of "ordered", other suggestions. Do we need this to be spelled out as a requirement? (or, is it even required?)
Implementation feedback (2026-10-24) from bryan newbold
- Note for clarity that the overall meta object can not be empty (
$typerequired) $type: note empty string allowed (or require it be non-empty)- must canonical varint be a MUST for readers or can it be SHOULD? stdlib impls tend to accept non-canonical/encodings (see Q1)
- Semantics of a pair with empty key and empty value are kind of weird, should it be disallowed?
Intended changes
TODO: meta strictness
be explicit that readers should reject non-canonical meta, or else note that whether an archive is deterministic depends on that.
TODO: other $type rules
explicitly note that metadata must be present, which is implied since $type is always required. explicitly note that empty-string $type is allowed.
Write a note…Log in