-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add query settings interface #46
Conversation
driver.go
Outdated
type Driver struct{} | ||
// ODPSDriver impls database/sql/driver.Driver | ||
type ODPSDriver struct { | ||
conn odpsConn |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if it is reasonable to maintain a conn
inside the driver.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe it's okay, we need to maintain the settings
struct in the DB object and the Driver
struct can be exposed when we implement the sql.database
.
driver.go
Outdated
// SetQuerySettings sets the global query settings. | ||
// TODO(Yancey1989): add the one-off query settings interface. | ||
func (o MaxcomputeDriver) SetQuerySettings(hints map[string]string) error { | ||
o.conn.hints = hints |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
deep copy or shallow copy?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Yancey1989 @weiguoz Please be aware that after this change, odpsConn
is no longer reentrant. Multiple goroutines can't share the same odpsConn
. For example,
db := sql.Open("maxcompute", ...)
go func() {
db.Driver().(*MaxcomputeDriver).SetQuerySettings(...)
}()
go func() {
db.Query(...) // Oops, the setting has been changed. Or even worse, panic due to the race on hints.
}()
I like the idea // TODO(Yancey1989): add the one-off query settings interface.
. I am wondering if the following implementation is possible.
package maxcompute
func QueryWithHint(db sql.DB, query string, hints map[string]string) (driver.Rows, error) {
conn, ok := db.Driver().(*MaxcomputeDriver)
if !ok {
return nil, fmt.Errorf(...)
}
newConn := odpsConn{conn.Client, conn.cfg, hints}
return newConn.Query(query, ...)
}
Then in another package, for example
package main
func main() {
db := Open("maxcompute", ...)
query := ...
hint := ...
rows, err := maxcompute.QueryWithHint(conn, query, hints)
}
Thanks for @tonyyang-svail 's reminding, in the >>> from odps import options
>>> options.sql.settings = {'odps.sql.mapper.split.size': 16}
>>> o.execute_sql('select * from pyodps_iris') # 会根据全局配置添加hints and the one-off one: o.execute_sql('select * from pyodps_iris', hints={'odps.sql.mapper.split.size': 16}) ref: https://pyodps.readthedocs.io/zh_CN/latest/base-sql.html#sql-hints Maybe |
driver_test.go
Outdated
a := assert.New(t) | ||
db, err := sql.Open("maxcompute", cfg4test.FormatDSN()) | ||
a.NoError(err) | ||
db.Driver().(*MaxcomputeDriver).SetQuerySettings(map[string]string{"odps.sql.mapper.split.size": "16"}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this test verify the hints is effective?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we also test odps.sql.mode = script
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this test verify the hints is effective?
No, it's not easy to test odps.sql.mapper.split.size
on the client-side. But maybe test UDF script is good way, will add this one.
The script mode of Maxcompute on Aliyun: https://help.aliyun.com/document_detail/90082.html?spm=a2c4g.11186623.6.692.5036612drr5apU |
@Yancey1989 btw, do we need to support logging on a job? |
@tonyyang-svail , I think we need that, the logs message is useful such as the job view link. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Hi @tonyyang-svail , I file an issue to talk about the one-off API #50, maybe we can continue to talk about the API design there. |
Hi @weiguoz @tonyyang-svail @typhoonzero
the URL query arguments key has prefix |
dsn.go
Outdated
for k, v := range querys { | ||
// The query args such as hints_odps.sql.mapper.split_size=16 | ||
// would be converted to the maxcompute query hints: {"odps.sql.mapper.split_size": "16"} | ||
if strings.HasPrefix(k, "hints_") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use odps.
as prefix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can't, some case like UDF prediction job would set the query hints using mst
as the prefix:
set mst.model.path=oss://<oss path>;
set mst.model.name=<model name>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Is this ready to merge? @Yancey1989 |
fixed #45