Skip to content

Commit

Permalink
Merge pull request #153 from go-jet/develop
Browse files Browse the repository at this point in the history
Release 2.8.0
  • Loading branch information
go-jet authored May 17, 2022
2 parents 6babd43 + 4b720d6 commit 38b6caf
Show file tree
Hide file tree
Showing 63 changed files with 2,015 additions and 1,265 deletions.
23 changes: 21 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@ jobs:
MYSQL_USER: jet
MYSQL_PASSWORD: jet

- image: cockroachdb/cockroach-unstable:v22.1.0-beta.4
command: ['start-single-node', '--insecure']
environment:
COCKROACH_USER: jet
COCKROACH_PASSWORD: jet
COCKROACH_DATABASE: jetdb

environment: # environment variables for the build itself
TEST_RESULTS: /tmp/test-results # path to where test results will be saved

Expand Down Expand Up @@ -82,7 +89,18 @@ jobs:
echo -n .
sleep 1
done
echo Failed waiting for MySQL && exit 1
echo Failed waiting for MySQL && exit 1
- run:
name: Waiting for Cockroach to be ready
command: |
for i in `seq 1 10`;
do
nc -z localhost 26257 && echo Success && exit 0
echo -n .
sleep 1
done
echo Failed waiting for Cockroach && exit 1
- run:
name: Install MySQL CLI;
Expand Down Expand Up @@ -122,8 +140,9 @@ jobs:
-coverpkg=github.com/go-jet/jet/v2/postgres/...,github.com/go-jet/jet/v2/mysql/...,github.com/go-jet/jet/v2/sqlite/...,github.com/go-jet/jet/v2/qrm/...,github.com/go-jet/jet/v2/generator/...,github.com/go-jet/jet/v2/internal/... \
-coverprofile=cover.out 2>&1 | go-junit-report > $TEST_RESULTS/results.xml
# run mariaDB tests. No need to collect coverage, because coverage is already included with mysql tests
# run mariaDB and cockroachdb tests. No need to collect coverage, because coverage is already included with mysql and postgres tests
- run: MY_SQL_SOURCE=MariaDB go test -v ./tests/mysql/
- run: PG_SOURCE=COCKROACH_DB go test -v ./tests/postgres/

- save_cache:
key: go-mod-v4-{{ checksum "go.sum" }}
Expand Down
66 changes: 33 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

Jet is a complete solution for efficient and high performance database access, consisting of type-safe SQL builder
with code generation and automatic query result data mapping.
Jet currently supports `PostgreSQL`, `MySQL`, `MariaDB` and `SQLite`. Future releases will add support for additional databases.
Jet currently supports `PostgreSQL`, `MySQL`, `CockroachDB`, `MariaDB` and `SQLite`. Future releases will add support for additional databases.

![jet](https://github.com/go-jet/jet/wiki/image/jet.png)
Jet is the easiest, and the fastest way to write complex type-safe SQL queries as a Go code and map database query result
Expand Down Expand Up @@ -62,40 +62,37 @@ $ go get -u github.com/go-jet/jet/v2

Jet generator can be installed in one of the following ways:

1) (Go1.16+) Install jet generator using go install:
```sh
go install github.com/go-jet/jet/v2/cmd/jet@latest
```

2) Install jet generator to GOPATH/bin folder:
```sh
cd $GOPATH/src/ && GO111MODULE=off go get -u github.com/go-jet/jet/cmd/jet
```
- (Go1.16+) Install jet generator using go install:
```sh
go install github.com/go-jet/jet/v2/cmd/jet@latest
```
*Jet generator is installed to the directory named by the GOBIN environment variable,
which defaults to $GOPATH/bin or $HOME/go/bin if the GOPATH environment variable is not set.*

3) Install jet generator into specific folder:
```sh
git clone https://github.com/go-jet/jet.git
cd jet && go build -o dir_path ./cmd/jet
```
*Make sure that the destination folder is added to the PATH environment variable.*
- Install jet generator to specific folder:
```sh
git clone https://github.com/go-jet/jet.git
cd jet && go build -o dir_path ./cmd/jet
```
*Make sure `dir_path` folder is added to the PATH environment variable.*



### Quick Start
For this quick start example we will use PostgreSQL sample _'dvd rental'_ database. Full database dump can be found in
[./tests/testdata/init/postgres/dvds.sql](https://github.com/go-jet/jet-test-data/blob/master/init/postgres/dvds.sql).
Schema diagram of interest for example can be found [here](./examples/quick-start/diagram.png).
Schema diagram of interest can be found [here](./examples/quick-start/diagram.png).

#### Generate SQL Builder and Model types
To generate jet SQL Builder and Data Model types from postgres database, we need to call `jet` generator with postgres
connection parameters and root destination folder path for generated files.
To generate jet SQL Builder and Data Model types from running postgres database, we need to call `jet` generator with postgres
connection parameters and destination folder path.
Assuming we are running local postgres database, with user `user`, user password `pass`, database `jetdb` and
schema `dvds` we will use this command:
```sh
jet -dsn=postgresql://user:pass@localhost:5432/jetdb -schema=dvds -path=./.gen
jet -dsn=postgresql://user:pass@localhost:5432/jetdb?sslmode=disable -schema=dvds -path=./.gen
```
```sh
Connecting to postgres database: postgresql://user:pass@localhost:5432/jetdb
Connecting to postgres database: postgresql://user:pass@localhost:5432/jetdb?sslmode=disable
Retrieving schema information...
FOUND 15 table(s), 7 view(s), 1 enum(s)
Cleaning up destination directory...
Expand All @@ -107,9 +104,10 @@ Generating view model files...
Generating enum model files...
Done
```
Procedure is similar for MySQL, MariaDB and SQLite. For instance:
Procedure is similar for MySQL, CockroachDB, MariaDB and SQLite. For example:
```sh
jet -source=mysql -dsn="user:pass@tcp(localhost:3306)/dbname" -path=./gen
jet -dsn=postgres://user:pass@localhost:26257/jetdb?sslmode=disable -schema=dvds -path=./.gen #cockroachdb
jet -dsn="mariadb://user:pass@tcp(localhost:3306)/dvds" -path=./gen # source flag can be omitted if data source appears in dsn
jet -source=sqlite -dsn="/path/to/sqlite/database/file" -schema=dvds -path=./gen
jet -dsn="file:///path/to/sqlite/database/file" -schema=dvds -path=./gen # sqlite database assumed for 'file' data sources
Expand Down Expand Up @@ -168,7 +166,7 @@ and _film category_ is not 'Action'.
stmt := SELECT(
Actor.ActorID, Actor.FirstName, Actor.LastName, Actor.LastUpdate, // or just Actor.AllColumns
Film.AllColumns,
Language.AllColumns.Except(Language.LastUpdate),
Language.AllColumns.Except(Language.LastUpdate), // all language columns except last_update
Category.AllColumns,
).FROM(
Actor.
Expand All @@ -186,7 +184,7 @@ stmt := SELECT(
Film.FilmID.ASC(),
)
```
_Package(dot) import is used, so the statements would resemble as much as possible as native SQL._
_Package(dot) import is used, so the statements look as close as possible to the native SQL._
Note that every column has a type. String column `Language.Name` and `Category.Name` can be compared only with
string columns and expressions. `Actor.ActorID`, `FilmActor.ActorID`, `Film.Length` are integer columns
and can be compared only with integer columns and expressions.
Expand Down Expand Up @@ -245,7 +243,7 @@ __How to get debug SQL from statement?__
```go
debugSql := stmt.DebugSql()
```
debugSql - this query string can be copy-pasted to sql editor and executed. __It is not intended to be used in production, only for the purpose of debugging!!!__
debugSql - this query string can be copy-pasted to sql editor and executed. __It is not intended to be used in production. For debug purposes only!!!__

<details>
<summary>Click to see debug sql</summary>
Expand Down Expand Up @@ -295,8 +293,8 @@ First we have to create desired structure to store query result.
This is done be combining autogenerated model types, or it can be done
by combining custom model types(see [wiki](https://github.com/go-jet/jet/wiki/Query-Result-Mapping-(QRM)#custom-model-types) for more information).

It's possible to overwrite default jet generator behavior, and all the aspects of generated model and SQLBuilder types can be
tailor-made([wiki](https://github.com/go-jet/jet/wiki/Generator#generator-customization)).
_Note that it's possible to overwrite default jet generator behavior. All the aspects of generated model and SQLBuilder types can be
tailor-made([wiki](https://github.com/go-jet/jet/wiki/Generator#generator-customization))._

Let's say this is our desired structure made of autogenerated types:
```go
Expand All @@ -315,14 +313,14 @@ var dest []struct {
`Langauge` field is just a single model struct. `Film` can belong to multiple categories.
_*There is no limitation of how big or nested destination can be._

Now lets execute above statement on open database connection (or transaction) db and store result into `dest`.
Now let's execute above statement on open database connection (or transaction) db and store result into `dest`.

```go
err := stmt.Query(db, &dest)
handleError(err)
```

__And thats it.__
__And that's it.__

`dest` now contains the list of all actors(with list of films acted, where each film has information about language and list of belonging categories) that acted in films longer than 180 minutes, film language is 'English'
and film category is not 'Action'.
Expand Down Expand Up @@ -528,7 +526,7 @@ The biggest benefit is speed. Speed is being improved in 3 major areas:

##### Speed of development

Writing SQL queries is faster and easier as the developers have help of SQL code completion and SQL type safety directly from Go.
Writing SQL queries is faster and easier, as developers will have help of SQL code completion and SQL type safety directly from Go code.
Automatic scan to arbitrary structure removes a lot of headache and boilerplate code needed to structure database query result.

##### Speed of execution
Expand All @@ -539,14 +537,14 @@ Thus handler time lost on latency between server and database can be constant. H
only to the query complexity and the number of rows returned from database.

With Jet, it is even possible to join the whole database and store the whole structured result in one database call.
This is exactly what is being done in one of the tests: [TestJoinEverything](/tests/postgres/chinook_db_test.go#L40).
The whole test database is joined and query result(~10,000 rows) is stored in a structured variable in less than 0.7s.
This is exactly what is being done in one of the tests: [TestJoinEverything](https://github.com/go-jet/jet/blob/6706f4b228f51cf810129f57ba90bbdb60b85fe7/tests/postgres/chinook_db_test.go#L187).
The whole test database is joined and query result(~10,000 rows) is stored in a structured variable in less than 0.5s.

##### How quickly bugs are found

The most expensive bugs are the one discovered on the production, and the least expensive are those found during development.
With automatically generated type safe SQL, not only queries are written faster but bugs are found sooner.
Lets return to quick start example, and take closer look at a line:
Let's return to quick start example, and take closer look at a line:
```go
AND(Film.Length.GT(Int(180))),
```
Expand All @@ -573,6 +571,8 @@ To run the tests, additional dependencies are required:
- `github.com/stretchr/testify`
- `github.com/google/go-cmp`
- `github.com/jackc/pgx/v4`
- `github.com/shopspring/decimal`
- `github.com/volatiletech/null/v8`

## Versioning

Expand Down
82 changes: 46 additions & 36 deletions cmd/jet/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ package main
import (
"flag"
"fmt"
"os"
"strings"

"github.com/go-jet/jet/v2/generator/metadata"
sqlitegen "github.com/go-jet/jet/v2/generator/sqlite"
"github.com/go-jet/jet/v2/generator/template"
Expand All @@ -11,8 +14,6 @@ import (
"github.com/go-jet/jet/v2/mysql"
postgres2 "github.com/go-jet/jet/v2/postgres"
"github.com/go-jet/jet/v2/sqlite"
"os"
"strings"

mysqlgen "github.com/go-jet/jet/v2/generator/mysql"
postgresgen "github.com/go-jet/jet/v2/generator/postgres"
Expand Down Expand Up @@ -42,7 +43,7 @@ var (
)

func init() {
flag.StringVar(&source, "source", "", "Database system name (postgres, mysql, mariadb or sqlite)")
flag.StringVar(&source, "source", "", "Database system name (postgres, mysql, cockroachdb, mariadb or sqlite)")

flag.StringVar(&dsn, "dsn", "", `Data source name. Unified format for connecting to database.
PostgreSQL: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING
Expand All @@ -59,7 +60,7 @@ func init() {
flag.StringVar(&user, "user", "", "Database user. Used only if dsn is not set.")
flag.StringVar(&password, "password", "", "The user’s password. Used only if dsn is not set.")
flag.StringVar(&dbName, "dbname", "", "Database name. Used only if dsn is not set.")
flag.StringVar(&schemaName, "schema", "public", `Database schema name. Used only if dsn is not set. (default "public")(PostgreSQL only)`)
flag.StringVar(&schemaName, "schema", "public", `Database schema name. (default "public")(PostgreSQL only)`)
flag.StringVar(&params, "params", "", "Additional connection string parameters(optional). Used only if dsn is not set.")
flag.StringVar(&sslmode, "sslmode", "disable", `Whether or not to use SSL. Used only if dsn is not set. (optional)(default "disable")(PostgreSQL only)`)
flag.StringVar(&ignoreTables, "ignore-tables", "", `Comma-separated list of tables to ignore`)
Expand All @@ -70,33 +71,7 @@ func init() {
}

func main() {

flag.Usage = func() {
fmt.Println("Jet generator 2.7.0")
fmt.Println()
fmt.Println("Usage:")

order := []string{
"source", "dsn", "host", "port", "user", "password", "dbname", "schema", "params", "sslmode",
"path",
"ignore-tables", "ignore-views", "ignore-enums",
}
for _, name := range order {
flagEntry := flag.CommandLine.Lookup(name)
fmt.Printf(" -%s\n", flagEntry.Name)
fmt.Printf("\t%s\n", flagEntry.Usage)
}

fmt.Println()
fmt.Println(`Example command:
$ jet -dsn=postgresql://jet:jet@localhost:5432/jetdb -schema=dvds -path=./gen
$ jet -source=postgres -dsn="user=jet password=jet host=localhost port=5432 dbname=jetdb" -schema=dvds -path=./gen
$ jet -source=mysql -host=localhost -port=3306 -user=jet -password=jet -dbname=jetdb -path=./gen
$ jet -source=sqlite -dsn="file://path/to/sqlite/database/file" -path=./gen
`)
}

flag.Usage = usage
flag.Parse()

if dsn == "" && (source == "" || host == "" || port == 0 || user == "" || dbName == "") {
Expand All @@ -111,11 +86,14 @@ func main() {
var err error

switch source {
case "postgresql", "postgres":
case "postgresql", "postgres", "cockroachdb", "cockroach":
generatorTemplate := genTemplate(postgres2.Dialect, ignoreTablesList, ignoreViewsList, ignoreEnumsList)

if dsn != "" {
err = postgresgen.GenerateDSN(dsn, schemaName, destDir)
err = postgresgen.GenerateDSN(dsn, schemaName, destDir, generatorTemplate)
break
}

dbConn := postgresgen.DBConnection{
Host: host,
Port: port,
Expand All @@ -131,14 +109,17 @@ func main() {
err = postgresgen.Generate(
destDir,
dbConn,
genTemplate(postgres2.Dialect, ignoreTablesList, ignoreViewsList, ignoreEnumsList),
generatorTemplate,
)

case "mysql", "mysqlx", "mariadb":
generatorTemplate := genTemplate(mysql.Dialect, ignoreTablesList, ignoreViewsList, ignoreEnumsList)

if dsn != "" {
err = mysqlgen.GenerateDSN(dsn, destDir)
err = mysqlgen.GenerateDSN(dsn, destDir, generatorTemplate)
break
}

dbConn := mysqlgen.DBConnection{
Host: host,
Port: port,
Expand All @@ -151,12 +132,13 @@ func main() {
err = mysqlgen.Generate(
destDir,
dbConn,
genTemplate(mysql.Dialect, ignoreTablesList, ignoreViewsList, ignoreEnumsList),
generatorTemplate,
)
case "sqlite":
if dsn == "" {
printErrorAndExit("ERROR: required -dsn flag missing.")
}

err = sqlitegen.GenerateDSN(
dsn,
destDir,
Expand All @@ -176,6 +158,34 @@ func main() {
}
}

func usage() {
fmt.Println("Jet generator 2.8.0")
fmt.Println()
fmt.Println("Usage:")

order := []string{
"source", "dsn", "host", "port", "user", "password", "dbname", "schema", "params", "sslmode",
"path",
"ignore-tables", "ignore-views", "ignore-enums",
}

for _, name := range order {
flagEntry := flag.CommandLine.Lookup(name)
fmt.Printf(" -%s\n", flagEntry.Name)
fmt.Printf("\t%s\n", flagEntry.Usage)
}

fmt.Println()
fmt.Println(`Example command:
$ jet -dsn=postgresql://jet:jet@localhost:5432/jetdb?sslmode=disable -schema=dvds -path=./gen
$ jet -dsn=postgres://jet:jet@localhost:26257/jetdb?sslmode=disable -schema=dvds -path=./gen #cockroachdb
$ jet -source=postgres -dsn="user=jet password=jet host=localhost port=5432 dbname=jetdb" -schema=dvds -path=./gen
$ jet -source=mysql -host=localhost -port=3306 -user=jet -password=jet -dbname=jetdb -path=./gen
$ jet -source=sqlite -dsn="file://path/to/sqlite/database/file" -path=./gen
`)
}

func printErrorAndExit(error string) {
fmt.Println("\n", error)
fmt.Println()
Expand Down
10 changes: 10 additions & 0 deletions examples/quick-start/.gen/jetdb/dvds/table/actor.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 38b6caf

Please sign in to comment.