You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current import-sql is implemented using shell script + xargs to parallelize file import.
It currently has a number of limitations, mostly dealing with error handling -- if one of the SQL files fail, it is not trivial to find which file failed and where, plus in some cases the script (xargs) can exit without waiting for files that have not yet completed - thus confusing the output even further.
One simple solution would be to rewrite it in Python, allowing more control and helping readability.
I just want to throw the following into the room: If a script runs on error and then the other scripts are still running until end, it can go, depending upon data size, very long until it is terminated.
The current import-sql is implemented using shell script +
xargs
to parallelize file import.It currently has a number of limitations, mostly dealing with error handling -- if one of the SQL files fail, it is not trivial to find which file failed and where, plus in some cases the script (xargs) can exit without waiting for files that have not yet completed - thus confusing the output even further.
One simple solution would be to rewrite it in Python, allowing more control and helping readability.
See also #373
The text was updated successfully, but these errors were encountered: