Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvement on str(tokenlist) #281

Closed
wants to merge 1 commit into from

Conversation

righthandabacus
Copy link

No description provided.

@vmuriart
Copy link
Contributor

Thanks for the pr. The change you're suggesting has had the effect of changing the behavior of how the output is generated and as a consequence it changed the parsed output.
Multiple tests are failing on this commit.

@vmuriart
Copy link
Contributor

I'm currently going over the changes that would be needed to make the pr work, and its overall effects would net on a slower sqlparse as we would need to add multiple sections to update the value of the tokenlists since they are not updated after they are created.

@righthandabacus
Copy link
Author

Hi,

Thanks. I have a terribly long SQL (but simple, no nested queries, just a
thousand result columns and many joins) and found it takes many seconds to
parse it. Upon investigating by cProfile, I believe that is the hot spot.
My change works on my example and cut the time spent to only 1/4.

Please let me know if I can help on something. I am new to sqlparse, just
used a few minutes on reading the source code.

Sincerely,
Adrian (righthandabacus)

On Tue, Aug 9, 2016 at 9:18 PM, Vik [email protected] wrote:

I'm currently going over the changes that would be needed to make the pr
work, and its overall effects would net on a slower sqlparse as we would
need to add multiple sections to update the value of the tokenlists since
they are not updated after they are created.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#281 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALyScsZ3_9NuoWjeOXuWH5X29VG7XBvEks5qeSbigaJpZM4JgSrO
.

@vmuriart
Copy link
Contributor

Sorry for the very late follow-up.
Can you provide this example (or a similar one) to use as a benchmark.

Did you notice the delay during the parsing or when using any of the filters? (formatting, etc).

If its during the parsing phase, I think the issue is actually with this line that calls the function you listed multiple times. AFAIK TokenList.value is barely used, so we can probably replace it with a lazy evaluation and that may provide the speed-up you saw on your pull request.

ovr pushed a commit to cube-js/sqlparse that referenced this pull request Aug 24, 2021
@living180
Copy link
Contributor

I suspect the issue that this PR was attempting to be address was the same as reported in #621, and should be fully addressed by PR #710.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants