Hold Off on Litestream 0.5.0

Thanks to mtlynch and everyone else who has submitted bug reports. We're squashing issues and working to get everything stable as quickly as possible.

That alone should hint everyone it was a big leap.

What's in object storage is not an SQLite file, but enough information to restore your SQLite file at various points in time. The way it's stored is optimized for frequent incremental updates and less frequent point restores.

The lightweight read replicas VFS allows you open an SQLite database read-only directly from the data in object storage, without having to first do a point restore, downloading all the data, and creating a local copy of the database.

You “stream” the data directly from object storage as needed, get a consistent view of it, and can query it while the “primary” is concurrently updating it.

It works, but SQLite kinda expects data to be local and low latency, which this might not be.

SQLite has a page cache, which you can use, but everytime there is a write, the entire cache is dropped.

Which means if you poll for updates regularly (which for various reasons you have to) you'll get frequent latency spikes.

One way to improve this is to add a page cache that has an understanding of which pages have changed and which have stayed the same. Prefetching would help to, but SQLite is really unhelpful there.

This is the piece I'm still missing. The rest mostly works already with Litestream v0.5.1.