tvips memory usage on windows #15
Comments
|
I would like to add that I tried this same procedure with a blockfile: data = hs.load('BLO/' + fname +'.blo', lazy=True)
center = data.data[..., 128, 128].compute()And the whole file was not loaded to memory, only the computed array. If I call Again testing the TVIPS reader, I am not sure how the amount of memory is being allocated, for example: >>> from hyperspy import api as hs
>>> data = hs.load('TVIPS/' + fname +'.tvips', lazy=True)
>>> # select cropped region from data
>>> region = data.data[:, -400:, -400:]
>>> # get size in GB
>>> region.nbytes / 1e9
6.12576
>>> vals = region.compute()
This fills up my memory (at least 24GB for a 6GB array). Perhaps this is related to dask compute however, as if I load the whole file as: >>> from hyperspy import api as hs
>>> fname = 'rec_+60deg_20211004_150915_000'
>>> data = hs.load('TVIPS/' + fname +'.tvips', lazy=True)
>>>region = data.data[:, :, :]
>>>region.nbytes / 1e9
10.036445184
>>> vals = region.compute()This also fills my memory (at least 24 GB): Whereas if I don't use the lazy loader: >>> data = hs.load('TVIPS/' + fname +'.tvips', lazy=False)Then less memory is allocated: |
|
Maybe you could look into dask-memusage to figure out where things are going wrong? See https://blog.dask.org/2021/03/11/dask_memory_usage |
|
Good idea! So I edited the example script to copy the compute commands above: from time import sleep
import numpy as np
from dask.bag import from_sequence
from dask import compute
from dask.distributed import Client, LocalCluster
from dask_memusage import install # <-- IMPORT
from hyperspy import api as hs
fname = r'file.tvips'
def get_center():
data = hs.load(fname, lazy=True)
center = data.data[:, 256, 256]
return center
def main():
cluster = LocalCluster(n_workers=1, threads_per_worker=1,
memory_limit=None)
install(cluster.scheduler, "memusage.csv") # <-- INSTALL
client = Client(cluster)
compute(get_center())
if __name__ == '__main__':
main()and got the output shown below. Quite difficult to interpret. The max memory usage never appears to go above 10 GB (file size is ~9.8 GB). Monitoring task manager at the same time showed that at least 24 GB were being used as a result of the script. Quite likely that I am not using the library properly in this case. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From @harripj in hyperspy/hyperspy#2781:
Just a follow up on this. On my Windows machine I seem to be experiencing the same behaviour as before:
If I load one of our experimental datasets (size 9.9 GB) as
data = hs.load('dset.tvips', lazy=True)the file is not loaded to memory as expected. Withlazy=Falsethe peak memory usage is double the file size before settling back to ~file size:Using the lazy
loaderagain, if I just extract the center pixel from each frame as:The whole file is loaded to memory and persists:
Interestingly the memory presists even if I call
del center; del data.Dask version:
2022.03.0NumPy version :
1.21.5Hyperspy version:
1.7.0.dev0, ie. local install of this PR.Originally posted by @harripj in hyperspy/hyperspy#2781 (comment)
The text was updated successfully, but these errors were encountered: