Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Directlake options for collecting metrics #118

Merged
merged 4 commits into from
Mar 22, 2024

Conversation

dgosbell
Copy link
Collaborator

Added 3 different modes for direct lake models so that we do not always force all columns to be transcoded into memory.

  • ResidentOnly - only run the distinctcount queries on columns that are current resident in memory
  • Referenced - run distinctcount queries on columns that are referenced by measures or relationships. This should provide the minimum information required for performance analysis of measures and for tools like DAX Optimizer.
  • Full - run distinctcount queries on all columns

I've added this option since the default behaviour at the moment is the same as Full and this can be detrimental to large models. It can take longer than necessary and potentially may still not produce the desired result since as we cycle through all the columns it could force some of the earlier columns to be paged out of memory.

I've set ResidentOnly as the default as it's the cheapest/fastest option.

Note that if an option other than ResidentOnly is chosen I am re-running the DMV collection after the stats collection has been run since the memory usage is usually changed as a result of running the distinctcount queries.

I did consider whether to add the selected Direct Lake analysis mode as a property to the vpax file, but I decided against his for the time being since it can be implied by checking the IsResident property on a column.

@albertospelta albertospelta linked an issue Mar 21, 2024 that may be closed by this pull request
src/Directory.Packages.props Outdated Show resolved Hide resolved
src/Dax.Model.Extractor/StatExtractor.cs Outdated Show resolved Hide resolved
@albertospelta albertospelta merged commit 1fa32d3 into sql-bi:master Mar 22, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

direct lake - columns into memory
2 participants