v0.16.0

Genji is now ChaiSQL ! 🫖

A name change that comes with a change in objectives: ChaiSQL strives to become a modern SQL embedded database.
The goal is to follow the SQL standard as much as possible, without sacrificing schema flexibility.

⚠️ The on-disk format has changed quite a bit and is not compatible with databases created prior v0.16.0.
ChaiSQL is under heavy development and breaking changes can still happen, though the on-disk format should be mostly stable from now on.

API changes

• Support for DESC composite primary keys and indices
• Add TIMESTAMP type
• The DOCUMENT type is now OBJECT

Other notable changes

• StructScan check interface parameter is a pointer to a struct by @abramlab in #487
• Add LOWER builtin function by @agonist in #510
• Add UPPER function by @agonist in #511
• add trim, ltrim, rtrim functions by @agonist in #512
• Add coalesce function by @yaziine in #513
• Add random function by @agonist in #514

New Contributors

• @agonist made their first contribution in #510
• @abramlab made their first contribution in #487

Full Changelog: v0.15.3...v0.16.0

v0.15.3

• Update Pebble to v1.0.0
• Remove encryption experiment

v0.15.2

This release contains one bug fix:

• StructScan: deep copy strings during scan #499

v0.15.1

This release contains a few bug fixes:

• orderby: fix evaluation of ORDER BY expr when used with projection #471
• errors: publish error functions to easily determine if an error is a NotFoundError or an AlreadyExists error. 492cdbe
• pebble: use custom Separator function. Without this, Pebble would sometimes generate an incompatible key that would cause a panic a35719a

v0.15.0

Genji v0.15.0 is out and comes a few big architectural changes 🚀

Goodbye Bolt and Badger, hello Pebble

Historically, Genji started as a layer on top of the amazing Bolt and Badger.
As features were being added, Genji is pretending to become more and more like a full featured SQL database, and thus needed more flexibility than what Bolt and Badger could offer.

Things like custom key comparators (to control how keys are organized in the tree), snapshots, concurrent read/writes (in Bolt only), were missing greatly. Both came with their own transaction mechanism which were a bit limiting for Genji (limited number of writes per transaction, no support for transaction promotion, etc.).
Also, having to maintain both data layers came with lots of challenges that we believe wasn't worth it.

That's why we decided to use a single backend: Pebble.

Pebble comes with lots of nice features, here are the ones we consider particularly interesting for Genji:

• No transactions (this is a good thing, we have our own now!)
• Batches
• Snapshots
• Can also be used in-memory only
• Full control on how the tree is organized

New transaction mechanism

Pebble doesn't come with its own transactions mechanism, so we had to implement our own, which comes will lots of advantages:

• Reads don't block writes, writes don't block reads: Read transactions always operate on the latest snapshot and don't require any locking. Write transactions use small in-memory batches which are atomically persisted alongside a rollback segment. The rollback segment contains undo changes to apply if the transaction is rolled back or if the database crashes.
• Fully serializable transactions: Genji supports multiple readers at a time and a single writer. This doesn't change from the previous releases, however we believe having a single writer should be more than enough in a embedded setting.
• Arbitrarily large transactions: Write transactions can contain a virtually unlimited number of changes while guaranteeing they won't be taking more than 10MB of ram (hardcoded value for this release). This allows to run Genji on small devices without worrying about the ram usage.
• Super fast writes: This release doesn't come with any official benchmark but our own biased ones showed that when inserting large documents, writes are twice as fast as the amazing SQLite (with WAL mode). Again, this is on a very specific benchmark, on a very specific device. But it is promising.

New CREATE TABLE API

The CREATE TABLE statement introduces new rules for table creation that gives much more control.

By default, a table now only inserts fields that have been declared. Any other field will be ignored and not inserted.

CREATE TABLE foo (a INTEGER, b TEXT);
INSERT INTO foo (a, b, c) VALUES (1, 'hello', 'not inserted');
INSERT INTO foo VALUES {a: 2, b: 'bye', c: 'not inserted'};
SELECT * FROM foo;

{
  "a": 1,
  "b": "hello"
}
{
  "a": 2,
  "b": "bye"
}

If you want to insert any field, and only care about declaring a few ones, you can use the ... notation.

CREATE TABLE foo (a INTEGER, b TEXT, ...);
INSERT INTO foo (a, b, c) VALUES (1, 'hello', 'inserted');
SELECT * FROM foo;

{
  "a": 1,
  "b": "hello"
  "c": "inserted"
}

Nested fields are now declared like this:

CREATE TABLE user (
    name TEXT,
    address (
        number INTEGER,
        street TEXT,
        zipcode TEXT NOT NULL
    )
);
INSERT INTO user (name, address) VALUES (
    "Ilias",
    {
        number: 10,
        street: "rue de la paix",
        zipcode: "12345"
    }
);

Drop support for WASM and TinyGo

Historically, Genji has invested a lot of efforts in supporting TinyGo in the hopes of being compiled in WASM and being used in the browser.
However, because of the lack of support for certain packages of the standard library (like reflect, fmt, encoding/jsom, etc.), it was very difficult for us to keep maintaining compatibility across different releases.
Even when we managed to compile, TinyGo would produce very large files (> 3MB), which defeats the purpose of using it.
We may reconsider this in the future as the support for Wasm in TinyGo becomes more mature.

Breaking changes and data format stability

⚠️ This release introduces a big refactoring and thus is not compatible with previous versions.
⚠️ It also only support Go 1.18+ as it relies on the new generics feature introduced in Go 1.18.

However, the database format is almost stable and future releases should not introduce any breaking changes.
If that happens, we will over communicate and provide ways to make it transparent as much as possible.

Our goal is to reach v1 very soon 🚀

Other notable changes

• index: treat all NULL values differently in UNIQUE indexes by @asdine in 2d4df65
• index: use index with BETWEEN operator by @asdine in a304e80
• index: prevent index creation on undeclared paths by @asdine in 3f32bcb
• cli: fix genji restore command with certains whitespace characters by @tzzed in #439
• cli: add .timer shell command by @tzzed in #445
• cli: ensure the inserts are run in the same transaction in genji insert by @tzzed in b4606f7
• sql: fix UPDATE behavior with primary keys by @asdine in f81f3f1
• sql: support params in LIMIT and OFFSET by @asdine in 6178d1a

Full Changelog: v0.14.0...v0.15.0

v0.14.1

• Fix JSON encoding of msgpack documents #438

v0.14.0

SQL

• Add more math functions by @jhchabran in #421
• Parse blob literal by @asdine in #423
• Add typeof function by @asdine in bba4c0a
• Handle integer conversion errors by @jhchabran in #425
• Add support for scientific notation for doubles by @asdine in d3b81ad
• Parse optional parentheses on DEFAULT clause by @asdine in 1654f35
• Add support for UNION by @asdine in 412d300
• Add support for CHECK by @asdine in #436
• Add support for composite primary keys by @asdine in 8983d68

CLI

• Add bench command by @asdine in 74603aa
• Fix .indexes command when transaction is active by @asdine in d5fd879

Other

• Improve failures readability when testing exprs by @jhchabran in #426
• Add support for transient databases by @asdine in a45d2c1
• Improved planner logic by @asdine in #428
• Add errors package with optional stacktrace capturing by @jhchabran in #431
• Refactor to handle errors with internal/errors by @jhchabran in #432
• Parse path fields with brackets by @asdine in cfdb784
• Fixed filling index types when loading indices from existing db by @KudinovKV in #434
• Use same tree structure for both tables and indexes by @asdine (⚠️ Breaking change) in a7309a7
• Fix goacc and CI by @asdine in #435

New Contributors

• @KudinovKV made their first contribution in #434

Full Changelog: v0.13.0...v0.14.0

v0.13.0

SQL

• Add concat operator
• Add NOT operator
• Add BETWEEN operator
• Add INSERT ... RETURNING
• Add ON CONFLICT
• Add UNION ALL #408 (@jhchabran)
• Add CREATE SEQUENCE
• Add DROP SEQUENCE
• Add NEXT VALUE FOR

Core

• Add document.NewFromCSV
• Add prepared statements
• Add new __genji_catalog table and drop __genji_table and __genji_indexes
• Remove engine.NextSequence

CLI

• Add .import command

This release also contains various bug fixes.

v0.12.0

SQL

• Added support for composite indexes #376 (⚠️Breaking change) @jhchabran
• Added support for UNIQUE table constraint #387
• Added support for PRIMARY KEY table constraint #333
• Added support for INSERT ... SELECT #385
• Indexes can be created with a generated name now #386

Core

• Fix indexed array comparisons with numbers #378
• Fix a bug preventing the use of time.Time with driver.Scanner #258

CLI

• Added a new way of writing integration tests: Examplar #380 @jhchabran
• Added .schema command #339 @tzzed

v0.11.0

SQL

• Add OFFSET, LIMIT and ORDER BY to DELETE statement #318 #319 #320 @jhchabran
• fix: query returning no result when using ORDER BY with sql/driver implementation #352 @tzzed

Core

• New Stream API #343
• Update Badger to v3 #350
• Integers are now compacted when encoded #351 (⚠️Breaking change) @jhchabran
• Infer constraints based on user-defined ones #358
• Queries using primary keys are much faster #310
• fix: Encoding of consecutive integers in arrays #362 @jhchabran
• fix: Incorrect index state with UPDATE queries on indexed fields #355 #368
• fix: Build index upon creation #371
• fix: BLOB to byte array decoding #305

CLI

• New genji dump command #315 @tzzed
• New genji restore command #321 @tzzed
• fix: .dump not outputting DEFAULT clause #329 @jhchabran
• fix: Deadlock when trying to open a locked database #365
• fix: Panic on genji insert when specified engine doesn't exist #331