Vector search support

[penberg]

This pull request adds initial support for vector search in libSQL.

Highlights

• Vector column type for storing vectors in tables.
• Vector index creation that is automatically updated on table updates.
• Exact vector search with metadata filtering using plain SQL.
• Approximate vector search using the new vector_top_k() function that is backed by DiskANN-based vector index.

Usage

Creating a table with a vector column:

CREATE TABLE movies (
  title TEXT, 
  year INT, 
  embedding FLOAT32(3)
);

Inserting vector data:

INSERT INTO movies (title, year, embedding) 
VALUES 
  (
    'Napoleon', 
    2023, 
    vector('[1,2,3]')
  ), 
  (
    'Black Hawk Down', 
    2001, 
    vector('[10,11,12]')
  ), 
  (
    'Gladiator', 
    2000, 
    vector('[7,8,9]')
  ), 
  (
    'Blade Runner', 
    1982, 
    vector('[4,5,6]')
  );

Creating an index on vector column:

CREATE INDEX movies_idx USING vector_cosine_ops ON movies (embedding);

Finding top-k similar rows (exact):

SELECT title, year FROM movies ORDER BY vector_distance_cos(embedding, '[3,1,2]') LIMIT 3;

Finding top-k similar rows (approximate):

SELECT 
  title, 
  year 
FROM 
  vector_top_k('movies_idx', '[4,5,6]', 3) 
JOIN
  movies 
ON 
  movies.rowid = id;

Limitations

• Index key is always rowid, primary keys not supported.
• CREATE INDEX does not index rows that already exist in the base table.
• Vector index uses 32-bit per vector element, which causes redundant I/O and space amplification.

[pax-k]

Any status for this PR? We're hardly waiting for vector support 🙏🏻 Thanks for the work btw!

[penberg]

@pax-k I am actively working with folks to iron out some bugs and then get this merged.

[haaawk]

Is it ever allowed for vector index to have pUsing->nId != 1?

[haaawk]

I'm probably still not understanding something but it seems that return -1 in line 393 and 399 will leave shadow table present despite index creation failing. Is this a problem? Probably nothing bad happens despite some junk being kept around. Creation of the index with the same name will just work because of IF NOT EXISTIS part in the SQL.

[haaawk]

Is it possible to define FLOAT32(X) with X being bigger than MAX_INT? Would it make sense to check for overflow here and report user readable error explaining that they try to use too big number of dimensions in case of X > MAX_INT or even X > MAX_VECTOR_DIMS?

[penberg]

This work has been merged as part of the following PRs:

#1531

#1551

#1557

#1560

Therefore, closing this.