Should STAR detect early truncation?
Weird varints? Control bytes?? And we are not too preoccupied to stop and ask whether or not we should.
CAR files 🚘 can be truncated at a block boundary, and the format doesn't give you any hint that more blocks are missing:
|--- Header --| |----------------- CAR Body -----------------|
[ int | DRISL ] [ int | CID | data ] [ int | CID | data ] …
you can CHOP here ~~~~~~~~~~~~~~~~~^
[ int | DRISL ] [ int | CID | data ] # <- still a valid CARSTAR inherits this:
|------ header ------| |------------------ STAR data pairs ------------------|
[ magic | len | meta ] [ len | key | len | val ] [ len | key | len | val ] …
you can CHOP here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
[ magic | len | meta ] [ len | key | len | val ] # <- still a valid STAR?!This is what Q3 in Settling the details is about, which has gotten enough discussion that I'm making this whole post about it.
Option 1: just accept it.
CARs work, this isn't a regression on strictness. Apps can mitigate this by putting a checksum or expected number-of-keys in the header meta to prevent a truncated archive from being silently accepted.
But we're aiming to be stricter and safer than CAR. It would be nice to do better. But, we'd be okay if we don't here.
Option 2: signalling bytes
What if:
|--- header ---| |---------------- STAR data pairs ---------------|
[magic|len|meta] [ 'D'|len|key|len|val ] [ 'D'|len|key|len|val ] … 'F'
a 'D' before each ~^ pair ~~~~~~~~~~~~~~~~~^ and then 'F' to end ~~^
a valid archive with zero data pairs becomes:
[magic|len|meta] 'F'
explicit end ~~~~^D ("data"?) and F ("fin"?) are whatever single-byte flags we want to bikeshed. A reader gets a signal before reading data, D means a data-pair follows, and F means we're at the end. If you don't see an F at the end, the archive was probably truncated.
I personally feel like this is the best option technically, but ugly aesthetically. I had to compress the spaces around the | in the ascii diagram above to make space for the extra byte markers, and it looks noisy compared to the clean, marker-free version before it. Bleh.
So far, nobody seems to like this option, or at least literally no one has said so when I've raised it. I kind of have a hard time arguing (with myself) why we shouldn't do this, though.
Option 3: special varint
If we break with ipfs/ipld/libp2p tradition and use a signed varint, then we can
|------ header ------| |----------- STAR data pairs ------------|
[ magic | len | meta ] [ len|key|len|val ] [ len|key|len|val ] … -1len prefixes suddenly gain expressiveness through negative numbers (which cannot be valid length prefixes). Then we can make len = -1 the end-of-archive marker.
I personally feel like this is the second-best option technically, and pretty acceptable aesthetically, and somewhat off-putting morally.
A variation of this: a non-canonically-encoded varint (like 0x8000) could be used as the end signal. I actually kind of like this but it seems likely to get people into trouble since most varint decoders are not strict and would just call that "zero".1
Option 4: sentinel data
If we reserve a special key (like the empty key), or even a special key-value pair (like empty-empty, 0x0000), then we can use that to mark the end of the archive.
Personally I'm both technically and morally opposed to this. I'll give it a fine aesthetically, and I'm willing to hear a case made for it.
Opinions pleasseeee 🥺
Now is the time to share your feelings about which option is best (or another one that hasn't come up yet!). Reply here, on the other post, on any of the tangled threads, on the community discourse, the discord, or on bluesky! I'll find your feedback! What option should we pick here??