Vector Database That Can Index 1b Vectors in 48m

Thankfully not months.

Yeah, the SI symbol for minutes is min, if you're going to abbreviate it in a technical context. Super funky using M.

Legend

I was starting to think this was impressive, if not impossible. 1B vectors in 48 MB of storage => < 1 bit per vector.

Maybe not impossible using shared/lossy storage if they were sparsely scattered over a large space ?

But anyways - minutes. Thanks.

Edit: Gemini suggested that this sort of (lossy) storage size could be achieved using "Product Quantization" (sub vectors, clustering, cluster indices), giving an example of 256 dimensional vectors being stored at an average of 6 bits per vector, with ANN being one application that might use this.

Thank you, title needs edited.

Seriously. The amount of lift a SaaS product needs to give me is insane for me to even bother evaluating it, and there's a near zero percent chance I'll use it in my core.

Nothing for you to see here. Surely you just aren't their target customer.

What do you think an alternative is for someone who:

1. Has a technical system they think could be worth a fortune to large enterprises, containing at least a few novel insights to the industry.

2. Knows that competitors and open source alternatives could copy/implement these in a year or so if the product starts off open source.

3. Has to put food on the table and doesn’t want to give massive corporations extremely valuable software for free.

Open source has its place, but it is IMO one of the ways to give monopolies massive value for free. There are plenty of open source alternatives around for vector DBs. Do we (developers) need to give everything away to the rich

Not that locked in - you can just move your vectors to another platform, no?

Vectroid co-founder here. We're huge fans of open source. My co-founder, Talip, made Hazelcast, which is open source.

It might make sense to open source all or part of Vectroid at some point in the future, but at the moment, we feel that would slow us down.

I hate vendor lock-in just as much as the next person. I believe data portability is the ACTUAL counter to vendor lock-in. If we have clean APIs to get your data in, get your data out, and the ability to bulk export your data (which we need to implement soon!), then there's less of a concern, in my opinion.

I also totally understand and respect that some people only want open source software. I'm certainly like that w/ my homelab setup! Except for Plex... Love Plex... Usually.

Hey! It's a great question. Co-founder of Vectroid here.

Today, the differences are going to be performance, price, accuracy, flexibility, and some intangible UI elegance.

Performance: We actually INITIALLY built Vectroid for the use-case of billions of vectors and near single digit millisecond latency. During the process of building and talking to users, we found that there are just not that many use-cases (yet!) that are at that scale and require that latency. We still believe the market will get there, but it's not there today. So we re-focused on building a general purpose vector search platform, but we stayed close to our high performance roots, and we're seeing better query performance than the other serverless, object storage backed vector DBs. We think we can get way faster too.

Price: We optimized the heck out of this thing with object storage, pre-emptible virtual machines, etc. We've driven our cost down, and we're passing this to the user, starting with a free tier of 100GB. Actual pricing beyond that coming soon.

Accuracy: With our initial testing, we see recall greater or equal to competitors out there, all while being faster.

Flexibility: We are going to have a self managed version for users who want to run on their own infra, but admittedly, we don't have that today. Still working on it.

Other Product Elegance: My co-founder, Talip, made Hazelcast, and I've always been impressed by how easy it is to use and how the end to end experience is so elegant. As we continue to develop Vectroid, that same level of polish and focus on the UX will be there. As an example, one neat thing we rolled out is direct import of data from Hugging Face. We have lots of other cool ideas.

Apologies for the long winded answer. Feel free to ping us with any additional questions.

No special hardware. Google Cloud vms. We use multiple of them during index building.

Show your math lol

1B x 4096 = 4T scalars.

That doesn't fit in anyone's video ram.

I guess for 2D vectors that would work?

For 1024 dimensions even with 8 bit quantization you are looking at a terrabyte of data. Lets make it binary vectors, it is still 128GB of VRAM.

WAT?

Lucene is tough to deal with. About 15 hours ago — right when this comment was posted — I was giving a talk at Databricks comparing the world’s most widely used search engines. I’ve never run into as many issues with any other similar tool as I did with Lucene. To be fair, it’s been around for ~26 years and has aged remarkably well... but it’s the last thing I’d choose today.

Aside from being serverless. This is like elasticsearch but with a kind of built in redis-like layer, I think.

co-founder of Vectroid: We forked Lucene. Lucene is awesome for search in general, filters, and obviously full-text search. Very mature and well supported by so many big names and amazing engineers. So we take advantage of that but we had to change a few things to make it work perfectly for Vector use-case. We basically think Vector should be the main data type as it is the most difficult one to deal with. For instance, we modified Lucene to use X number of CPU / threads to build a single segment index. As a result, if/when needed, we can utilize hundreds of CPUs to index quicker and generate less number of segments that will enable lower query latency. We also built a custom File System Directory for Lucene to work off of GCS directly (or S3 later on). It can by-pass the kernel, read from network and write directly into the memory... no SSD, no page-cache, no mmap involved. Perhaps I should not say more...

I would think that 1 bln refers to the row count, not to a vector's length.

Those numbers are for the case where you want all possible pairs of two vectors to have a corresponding query that returns those vectors as the top two results.

If you mostly just want to find a particular single vector if possible and don't care so much what the second-best result is, you can get away with much smaller embeddings.

And if you do want to cover all possible pairs, 6500 dimensions or so should be enough. (Their empirical results roughly fit a cubic polynomial.)

I think we have so many of those nice open source libraries but the problem is not the library or the algorithm (hsnw or ivf derivatives).. the problem is figuring out the right distributed architecture to balance cost, accuracy (recall) and speed (latency). I believe no single library will give you all that. For instance if you don't separate writes (indexing) from reads (queries) and scale them separately then your indexing will either suck or your indexing will destroy your read latency. You won't be able to scale as easily either. I believe that is why AWS created Aurora and Google Cloud created AlloyDB to scale relational databases (mysql/postgresql) by separating the reads/writes, implementing a scalable storage backend and by offloading a lot of shared works (replication, compaction, indexing) to cluster of machines.

We’re building vector indexes into Datafusion for search (starting with S3 vectors).

Open source at https://github.com/spiceai/spiceai

USearch is this type of library: https://github.com/unum-cloud/usearch

Used in ClickHouse and a few other DBMS.

why not use this? https://github.com/facebookresearch/faiss

Soo… usearch? Its literally one header file (of what use to be strict c++11). Funnily enough that is what is used in the official duckdb-vss extension.

Disclaimer: I wrote duckdb-vss

Might be all you need, except an open source licence:

> For production or managed service use, please contact SQLite Cloud, Inc for a commercial license.

There’s vss as duckdb extension too that builds a hnsw index.

https://github.com/duckdb/duckdb-vss

Since duckdb is already columnar, it goes brrrrr with single digit millisecond vector similarly lookups.