Thoughts on the Word Spec in Rust

or - if feasible - extend the existing crate instead of creating a new one.

Thanks for the kind words, and I have given it serious thought.

It's definitely not impossible in the future.

I just don't think there is enough interest right now in contributing to the underlying tech without generalizing it so much that it basically becomes an inferior LibreOffice.

Instead, the business model for Tritium is to give away the niche legal product for free to the community, but charge commercial users who need more granular control over its network activity, etc. This gives smaller start-ups, law offices and in-house shops a chance to benefit from the niche features while reserving for more demanding organizations to express an interest in and benefit from advanced features.

This is the solution for that particular issue that Tritium uses.

[NOTE: one dragon would be the memory consumption alluded to in the article.]

Yes, it does render PDFs.

There's actually an example PDF in the bundle if you click "Fetch Example" from the web preview at: https://tritium.legal/preview.

Under the hood, Tritium is using PDFium[1]. That's the same library used by Chrome, for example. The PDF spec is another animal that will be tackled in due course, but most legal users only need to view and comment on PDFs.

Try and find a binding to PDFium from your language of choice and start at that layer. PDFs are complex beasts, most of which complexity it may not be necessary to try to tackle in the first instance.

[1] https://pdfium.googlesource.com/pdfium/

You can absolutely save data like that, it's just that it's a terrible idea. There are obvious portability concerns issues: little-endian vs. big endian, 32-bit vs. 64-bit, struct padding, etc.

Essentially, this system works great if you know the exact hardware and compiler toolchain, and you never expect to upgrade it with things that might break memory layout. Obviously this does not hold for Word: it was written originally in a 32-bit world and now we live in a 64-bit one, MSVC has been upgraded many times, etc. There's also address space concern: if you embed your pointers, are you SURE that you're always going to be able to load them in the same place in the address space?

The overhead of deserialization is very small with a properly written file format, it's nowhere near worth the sacrifice in portability. This is not why Word is slow.

Your mileage may be different. I didn't work on Word (though I talked to those guys about their format) but I worked on two other apps that used the same strategy in the same era. One, on load you had to fix up runtime data that landed in that part of the data segment. Two, the in memory representation was actually somewhat sparse. This meant that a serializer actually read and wrote less to disk than mapping the file. So documents were smaller and there was actually less i/o and faster loads.

The reason I hated it though was because it was very hard to version. I know the Word team had that problem, especially when the mandate came down for older versions to be able to read newer versions. Hard enough to organize the disk format so old versions can ignore stuff, but now you're putting the same requirements on the in-memory representation. Maybe Word did it better.

There's all kinds of discussions of recovering text from corrupted files that just kind of went away when they moved over to the explicit serialization in docx.