Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reimplement import-sql in python #376

Open
nyurik opened this issue Sep 28, 2021 · 2 comments · May be fixed by #378
Open

Reimplement import-sql in python #376

nyurik opened this issue Sep 28, 2021 · 2 comments · May be fixed by #378

Comments

@nyurik
Copy link
Member

nyurik commented Sep 28, 2021

The current import-sql is implemented using shell script + xargs to parallelize file import.

It currently has a number of limitations, mostly dealing with error handling -- if one of the SQL files fail, it is not trivial to find which file failed and where, plus in some cases the script (xargs) can exit without waiting for files that have not yet completed - thus confusing the output even further.

One simple solution would be to rewrite it in Python, allowing more control and helping readability.

See also #373

nyurik added a commit to nyurik/openmaptiles-tools that referenced this issue Oct 2, 2021
@nyurik nyurik linked a pull request Oct 2, 2021 that will close this issue
@Falke-Design
Copy link
Contributor

I just want to throw the following into the room: If a script runs on error and then the other scripts are still running until end, it can go, depending upon data size, very long until it is terminated.

@nyurik
Copy link
Member Author

nyurik commented Oct 3, 2021

I agree, I actually had some parameters in docs for that, but I was too lazy to implement sending SIGINT to all running processes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants