Product Launch
anonymous
680 points
154 comments
Posted15 days agoActive2d ago
Garage
garagehq.deuxfleurs.frmvplibrarydevelopers, system administrators
Key Features
S3 compatibilityhigh reliabilityself-hosted object store
Tech Stack
Rust
Key Features
Tech Stack
They just completely swapped out the whole service from the stack and wrote one in Go because of how much better the concurrency management was, and Ceph's team and codebase C++ was too resistant to change.
Able/willing to expand on this at all? Just curious.
I'm not sure if it even has any sort of cluster consensus algorithm? I can't imagine it not eating committed writes in a multi-node deployment.
Garage and Ceph (well, radosgw) are the only open source S3-compatible object storage which have undergone serious durability/correctness testing. Anything else will most likely eat your data.
To clarify our architecture: RustFS is purpose-built for high-performance object storage. We intentionally avoid relying on general-purpose consensus algorithms like Raft in the data path, as they introduce unnecessary latency for large blobs.
Instead, we rely on Erasure Coding for durability and Quorum-based Strict Consistency for correctness. A write is strictly acknowledged only after the data has been safely persisted to the majority of drives. This means the concern about "eating committed writes" is addressed through strict read-after-write guarantees rather than a background consensus log.
While we avoid heavy consensus for data transfer, we utilize dsync—a custom, lightweight distributed locking mechanism—for coordination. This specific architectural strategy has been proven reliable in production environments at the EiB scale.
It's really hard to solve this problem without a consensus algorithm in a way that doesn't sacrifice something somewhere (usually correctness in edge cases/network partitions). Data availability is easy(ish), but keeping the metadata consistent requires some sort of consensus.
> Beijing Address: Area C, North Territory, Zhongguancun Dongsheng Science Park, No. 66 Xixiaokou Road, Haidian District, Beijing
> Beijing ICP Registration No. 2024061305-1
Otherwise, the built in admin on one-executable was nice, but parallel write performance was pretty unimpressive and started throwing strange errors (investigating of which led to the AI ticket discovery).
Why skipping SeaweedFS? It rank #1 on all benchmarks, and has a lot of features.
Not a concern for many use-cases, just something to be aware of as it's not a universal solution.
[1]: https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#st...
We know trust matters, especially for a newer project, and we try to earn it through transparency and external validation. we were excited to see RustFS recently added as an optional service in Laravel Sail’s official Docker environment (PR #822). Having our implementation reviewed and accepted by a major ecosystem like Laravel was an encouraging milestone for us.
If the “non-technical reasons” you mentioned are around licensing or governance, I’m happy to discuss our long-term Apache 2.0 commitment and path to a stable GA.
It's a really cool system for hyper converged architecture where storage requests can pull data from the local machine and only hit the network when needed.
Erasure coding is another debate, for now we have chosen not to implement it, but I would personally be open to have it supported by Garage if someone codes it up.
> For the metadata storage, Garage does not do checksumming and integrity verification on its own, so it is better to use a robust filesystem such as BTRFS or ZFS. Users have reported that when using the LMDB database engine (the default), database files have a tendency of becoming corrupted after an unclean shutdown (e.g. a power outage), so you should take regular snapshots to be able to recover from such a situation.
It seems like you can also use SQLite, but a default database that isn't robust against power failure or crashes seems suprising to me.
If you really live somewhere with frequent outages, buy an industrial drive that has a PLP rating. Or get a UPS, they tend to be cheaper.
As I understood it, the capacitors on datacenter-grade drives are to give it more flexibility, as it allows the drive to issue a successful write response for cached data: the capacitor guarantees that even with a power loss the write will still finish, so for all intents and purposes it has been persisted, so an fsync can return without having to wait on the actual flash itself, which greatly increases performance. Have I just completely misunderstood this?
Unfortunately they do: https://news.ycombinator.com/item?id=38371307
Yes, otherwise those drives wouldn't work at all and would have a 100% warranty return rate. The reason they get away with it is that the misbehavior is only a problem in a specific edge-case (forgetting data written shortly before a power loss).
https://documents.westerndigital.com/content/dam/doc-library...
That doesn't even help if fsync() doesn't do what developers expect: https://danluu.com/fsyncgate/
I think this was the blog post that had a bunch more stuff that can go wrong too: https://danluu.com/deconstruct-files/
But basically fsync itself (sometimes) has dubious behaviour, then OS on top of kernel handles it dubiously, and then even on top of that most databases can ignore fsync erroring (and lie that the data was written properly)
So... yes.
If you use WITHOUT ROWID, you traverse only the BLOB->data tree.
Looking up lexicographically similar keys gets a huge performance boost since sqlite can scan a B-Tree node and the data is contiguous. Your current implementation is chasing pointers to random locations in a different b-tree.
On disk size may be smaller or larger. The BLOB->data tree will have more nodes than the BLOB->rowid tree, so with big keys and big values it might end up larger on disk. With small-ish keys, on disk might be smaller.
[1]: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...
It's built specifically to run on object storage, currently relies on the `object_store` crate but we're consdering OpenDAL instead so if Garage works with those crates (I assume it does if its S3 compatible) it should just work OOTB.
Checksumming detects corruption after it happened. A database like Postgres will simply notice it was not cleanly shut down and put the DB into a consistent state by replaying the write ahead log on startup. So that is kind of my default expectation for any DB that handles data that isn't ephemeral or easily regenerated.
But I also likely have the wrong mental model of what Garage does with the metadata, as I wouldn't have expected that to be ever limited by Sqlite.
We do recommend SQLite in our quick-start guide to setup a single-node deployment for small/moderate workloads, and it works fine. The "real world deployment" guide recommends LMDB because it gives much better performance (with the current status of Garage, not to say that this couldn't be improved), and the risk of critical data loss is mitigated by the fact that such a deployment would use multi-node replication, meaning that the data can always be recovered from another replica if one node is corrupted and no snapshot is available. Maybe this should be worded better, I can see that the alarmist wording of the deployment guide is creating quite a debate so we probably need to make these facts clearer.
We are also experimenting Fjall as an alternate KV engine based on LSM, as it theoretically has good speed and crash resilience, which would make it the best option. We are just not recommending it by default yet, as we don't have much data to confirm that it works up to these expectations.
Whether or not it's semantically "correct" because of usage of hyphen vs slash is irrelevant to that point.
A used-car lot A value-added tax A key-based access system
When you have two exclusive options, two sides to a situation, or separate things; you separate them with a slash:
An on/off switch A win/win situation A master/slave arrangement
Therefore a key-value store and a key/value store are quite different.
It's true that key–value store shouldn't be written with a hyphen. It should be written with an en dash, which is used "to contrast values or illustrate a relationship between two things [...] e.g. Mother–daughter relationship"
https://en.wikipedia.org/wiki/Dash#En_dash
I just didn't want to bother with typography at that level of pedanticism.
It’s worth noting too that B+ tree databases are not a fantastic match for ZFS - they usually require extra tuning (block sizes, other stuff like how WAL commits work) to get performance comparable to XFS/ext4. LSMs on the other hand naturally fit ZFS’s CoW internals like a glove.
LMDB mode also runs with flush/syncing disabled
Standard filesystems such as ext4 and xfs don't have checksumming, so you'll have to rely on another layer to provide integrity. Regardless, that's not garage's job imo. It's good that they're keeping their design simple and focus their recources on implementing the S3 spec.
this is the reliability question no?
https://archive.fosdem.org/2024/schedule/event/fosdem-2024-3...
Slides are available here:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...
Conditionnal writes : no, we can't do it with CRDTs, which are the core of Garage's design.
https://dd.thekkedam.org/assets/documents/publications/Repor... http://www.bailis.org/papers/ramp-sigmod2014.pdf
Garage looks really nice: I've evaluated it with test code and benchmarks and it looks like a winner. Also, very straightforward deployment (self contained executable) and good docs.
But no tags on objects is a pretty big gap, and I shelved it. If Garage folk see this: please think on this.
I really, really appreciate that Garage accommodates running as a single node without work-arounds and special configuration to yield some kind of degraded state. Despite the single minded focus on distributed operation you no doubt hear endlessly (as seen in this thread,) there are, in fact, traditional use cases where someone will be attracted to Garage only for the API compatibility, and where they will achieve availability in production sufficient to their needs by means other than clustering.
Arbitrary name+value pairs can be applied to S3 objects and buckets. Metadata, basically. AWS has some tie-ins with permissions and other features, but they can be used for any purpose.
The assumption Garage makes, which is well-documented, is that of 3 replica nodes, only 1 will be in a crash-like situation at any time. With 1 crashed node, the cluster is still fully functional. With 2 crashed nodes, the cluster is unavailable until at least one additional node is recovered, but no data is lost.
In other words, Garage makes a very precise promise to its users, which is fully respected. Database corruption upon power loss enters in the definition of a "crash state", similarly to a node just being offline due to an internet connection loss. We recommend making metadata snapshots so that recovery of a crashed node is faster and simpler, but it's not required per se: Garage can always start over from an empty database and recover data from the remaining copies in the cluster.
To talk more about concrete scenarios: if you have 3 replicas in 3 different physical locations, the assumption of at-most one crashed node is pretty reasonable, it's quite unlikely that 2 of the 3 locations will be offline at the same time. Concerning data corruption on a power loss, the probability to lose power at 3 distant sites at the exact same time with the same data in the write buffers is extremely low, so I'd say in practice it's not a problem.
Of course, this all implies a Garage cluster running with 3-way replication, which everyone should do.
If it's just the write buffer at risk, that's fine. But the chance of simultaneous power loss across multiple sites isn't low enough to risk all the existing data.
It's downright stupid if you build a system that loses all existing data when all nodes go down uncleanly, not even simultaneously but just overlapping. What if you just happen to input a shutdown command the wrong way?
I really hope they meant to just say the write buffer gets lost.
Again, I'm not concerned for new writes, I'm concerned for all existing data from the previous months and years.
And it only takes one out of a wide outage or a bad push that takes down the cluster. Even if that's stupid, it's a common enough stupid that you should never risk your data on the certainty you won't make that mistake.
You can't protect against everything, but you should protect against unclean shutdown.
Also, garage gives you the possibility to automatically snapshot the metadata, advices on how to do the snapshotting at the filesystem level and to restore that.
- you lose whatever writes to s3 haven't finished yet, if any
- the local node will need to repair itself a bit after rebooting
- the local node is now trashed and will have to copy all data back over
- all the nodes are now trashed and it's restore from backup time
I've been kicking the tyres for a bit and I think it's the happy case in the above, but lots of software out there completely falls apart on crashes so it's not generally a safe assumption. I think the behaviour is sqlite on zfs doesn't care about pulling the power cable out, lmdb is a bit further down the list.
Syncthing will synchronize a full folder between an arbitrary number of machines, but you still have to access this folder one way or another.
Garage provides an HTTP API for your data, and handles internally the placement of this data among a set of possible replica nodes. But the data is not in the form of files on disk like the ones you upload to the API.
Syncthing is good for, e.g., synchronizing your documents or music collection between computers. Garage is good as a storage service for back-ups with e.g. Restic, for media files stored by a web application, for serving personal (static) web sites to the Internet. Of course, you can always run something like Nextcloud in front of Garage and get folder synchronization between computers somewhat like what you would get with Syncthing.
But to answer your question, yes, Garage only provides a S3-compatible API specifically.
We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes)
I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.
---
In addition:
Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2].
We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking.
[0]: https://news.ycombinator.com/item?id=46140342
[1]: https://rustfs.com/
[2]: https://rook.io/
My favorite thing about all of this is that I had just invested a ton of time in understanding MinIO and its Kubernetes operator and got everything into a state that I felt good about. I was nearly ready to deploy it to production when the announcement was released that they would not be supporting it.
I’m somewhat surprised that no one is forking it (or I haven’t heard about any organizations of consequence stepping up anyway) instead of all of these projects to rebuild it from scratch.
nothing prevents them from hiking pricing, so expectations are not clear.
> RustFS is a high-performance, distributed object storage software developed using Rust, the world's most popular memory-safe language.
I’m actually something of a Rust booster, and have used it professionally more than once (including working on a primarily Rust codebase for a while). But it’s hard to take a project’s docs seriously when it describes Rust as “the world’s most popular memory-safe language”. Java, JavaScript, Python, even C# - these all blow it out of the water in popularity and are unambiguously memory safe. I’ve had a lot more segfaults in Rust dependencies than I have in Java dependencies (though both are minuscule in comparison to e.g. C++ dependencies).
Then things like this appear:
https://www.phoronix.com/news/First-Linux-Rust-CVE
And I'm all warm and feeling schadenfreude.
To hear "yes, it's safer" and yet not "everyone on the planet not using rust is a moron!!!", is a nice change.
Frankly, the whole cargo side of rust has the same issues that node has, and that's silly beyond comprehension. Memory safe is almost a non-concern, compared to installing random, unvetted stuff. Cargo vet seems barely helpful here.
I'd want any language caring about security and code safety, to have a human audit every single diff, on every single package, and host those specific crates on locked down servers.
No, I don't care about "but that will slow down development and change!". Security needs to be first and front.
And until the Rust community addresses this, and its requirement for 234234 packages, it's a toy.
I've long desired this approach (backporting security fixes) to be commercialized instead of the always-up-to-date-even-if-incompatible push, and on top of Red Hat, Suse, Canonical (with LTS), nobody has been doing it for product teams until recently (Chainguard seems to be doing this).
But, if you ignore speed, you also fail: others will build less secure products and conquer the market, and your product has no future.
The real engineering trick is to be fast and build new things, which is why we need supply chain commoditized stewards (for a fee) that will solve this problem for you and others at scale!
which is a bit silly considering that if you want fast, most packages land in testing/unstable pretty quickly.
I believe the sweet spot is Debian-like stable as the base platform to build on top of, and then commercial-support in a similar way for any dependencies you must have more recent versions on top.
If you need latest packages, you have to do it anyway.
> I believe the sweet spot is Debian-like stable as the base platform to build on top of, and then commercial-support in a similar way for any dependencies you must have more recent versions on top.
That if the company can build packages properly. Also too old OS deps sometimes do throw wrench in the works.
Tho frankly "latest Debian Testing" have far smaller chance breaking something than "latest piece of software that couldn't figure out how to upstream to Debian"
The latter has a huge maintenance burden, the former is the, as I said already, sweet spot. (And let's not talk about combining stable/testing, any machine I tried that on got into an non-upgradeable mess quickly)
I am not saying it is easy, which is exactly why I think it should be a commercial service that you pay for for it to actually survive.
> supply chain commoditized stewards (for a fee)
I would love to see a software distribution model in which we could pay for vetted libraries, from bodies that we trust, which would become FOSS after a time period - even a month would be fine.
There are flaws in my argument, but it is a safer option than the current normal practices.
[0]
qed
Again the statement is probably still untrue and bad marketing, but I suspect this kind of reasoning was behind it
Of course Rust technically fails too since `unsafe` is a language featureBut it might be interesting to see where the time is spent. I suspect they may be doing fewer things in parallel than MinIO, but maybe it's something entirely different.
I wouldn't be surprised if this will be fixed sometime in the future.
One of the issues I had for a long time is that my development environment was tied to the staging instance -- it was easy to have a local DB instance with a pg-dump but it was useless without having an object store replica, since otherwise a DB tied to a stag/remote R2 object store would drift my stag DB.
It took a while to find an s3-compatible store that was simple enough to quickly set up and spin up but could handle a good chunk of data. With s5cmd, I can seed about 15 GB (~60,000 objects) in less than 60s and have a perfect replica of my stag environment locally for dev and test work.
I tried LocalStack S3 before this but apparently it just keeps the entire store in memory rather than on disk. MinIO was another one that came up but the project is now in "maintenance mode"? Other options I looked at were SeaweedFS and RustFS, but this one provided the best fit for performance + easy setup.
Not affiliated with Hacker News or Y Combinator. We simply enrich the public API with analytics.