snap.tl.macs3 error

[HaikuoLi]

Hello!

I am trying to follow your tutorial [https://kzhang.org/SnapATAC2/version/2.6/tutorials/diff.html] to analyze my dataset. But I got error "RuntimeError: Some worker process has died unexpectedly" (error message below) when I am running snap.tl.macs3(adata, groupby='leiden', n_jobs=1). Could you please help?

2024-09-05 16:39:33 - INFO - Exporting fragments...
2024-09-05 16:39:50 - INFO - Calling peaks...
  0%|          | 0/8 [00:00<?, ?it/s]Process SpawnPoolWorker-25:
Traceback (most recent call last):
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/multiprocess/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/multiprocess/queues.py", line 387, in get
    return _ForkingPickler.loads(res)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/dill/_dill.py", line 303, in loads
    return load(file, ignore, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/dill/_dill.py", line 289, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/dill/_dill.py", line 444, in load
    obj = StockUnpickler.load(self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/dill/_dill.py", line 434, in find_class
    return StockUnpickler.find_class(self, module, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/snapatac2/__init__.py", line 2, in <module>
    from . import preprocessing as pp
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/snapatac2/preprocessing/__init__.py", line 6, in <module>
    from ._scrublet import scrublet, filter_doublets
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/snapatac2/preprocessing/_scrublet.py", line 12, in <module>
    from snapatac2.tools._embedding import spectral
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/snapatac2/tools/__init__.py", line 6, in <module>
    from ._diff import marker_regions, diff_test
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/snapatac2/tools/_diff.py", line 5, in <module>
    from scipy.stats import chi2, norm, zscore
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/scipy/stats/__init__.py", line 605, in <module>
    from ._stats_py import *
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/scipy/stats/_stats_py.py", line 37, in <module>
    from scipy.spatial.distance import cdist
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/scipy/spatial/__init__.py", line 110, in <module>
    from ._kdtree import *
  File "/home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/scipy/spatial/_kdtree.py", line 4, in <module>
    from ._ckdtree import cKDTree, cKDTreeNode
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/hl737/.conda/envs/ds_notebook/lib/python3.12/site-packages/scipy/spatial/_ckdtree.cpython-312-x86_64-linux-gnu.so)
  0%|          | 0/8 [00:01<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File <timed eval>:1

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/snapatac2/tools/_call_peaks.py:155, in macs3(adata, groupby, qvalue, replicate, replicate_qvalue, max_frag_size, selections, nolambda, shift, extsize, min_len, blacklist, key_added, tempdir, inplace, n_jobs)
    152     return _snapatac2.find_reproducible_peaks(merged, others, blacklist)
    154 logging.info("Calling peaks...")
--> 155 peaks = _par_map(_call_peaks, [(x,) for x in fragments.values()], n_jobs)
    156 peaks = {k: v for k, v in zip(fragments.keys(), peaks)}
    157 if inplace:

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/snapatac2/tools/_call_peaks.py:221, in _par_map(mapper, args, nprocs)
    219 while len(jobs) > 0:
    220     if any(map(lambda p: not p.is_alive(), procs)):
--> 221         raise RuntimeError("Some worker process has died unexpectedly.")
    223     remaining = []
    224     for i, job in jobs:

RuntimeError: Some worker process has died unexpectedly.

Snap version 2.6.0

Thank you in advance!

[kaizhang]

From the error message, it looks like you may not have a working scipy installation. Can you try importing relevant scipy modules and see if it gives you the same error?

[HaikuoLi]

It was solved by reinstalling scipy and then reinstalling snapatac2. Thanks!

[HaikuoLi]

Sorry but can I have a follow up question? I am trying to identify enriched TF motifs by motifs = snap.tl.motif_enrichment( motifs=snap.datasets.cis_bp(unique=True), regions= marker_peaks, genome_fasta=snap.genome.mm10 ).
I got errors which seem to be related to polars. Do you have any ideas?

2024-09-05 21:23:37 - INFO - Fetching 271 sequences ...
2024-09-05 21:23:38 - INFO - Computing enrichment ...
100%|██████████| 1165/1165 [00:40<00:00, 28.62it/s]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/polars/_utils/construction/series.py:309, in _construct_series_with_fallbacks(constructor, name, values, dtype, strict)
    308 try:
--> 309     return constructor(name, values, strict)
    310 except TypeError:

TypeError: 'float' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
File <timed exec>:1

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/snapatac2/tools/_motif.py:127, in motif_enrichment(motifs, regions, genome_fasta, background, method)
    125 for key in result.keys():
    126     result[key]['adjusted p-value'] = _p_adjust_bh(result[key]['p-value'])
--> 127     result[key] = pl.DataFrame(result[key])
    128 return result

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/polars/dataframe/frame.py:360, in DataFrame.__init__(self, data, schema, schema_overrides, strict, orient, infer_schema_length, nan_to_null)
    355     self._df = dict_to_pydf(
    356         {}, schema=schema, schema_overrides=schema_overrides
    357     )
    359 elif isinstance(data, dict):
--> 360     self._df = dict_to_pydf(
    361         data,
    362         schema=schema,
    363         schema_overrides=schema_overrides,
    364         strict=strict,
    365         nan_to_null=nan_to_null,
    366     )
    368 elif isinstance(data, (list, tuple, Sequence)):
    369     self._df = sequence_to_pydf(
    370         data,
    371         schema=schema,
   (...)
    375         infer_schema_length=infer_schema_length,
    376     )

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/polars/_utils/construction/dataframe.py:159, in dict_to_pydf(data, schema, schema_overrides, strict, nan_to_null, allow_multithreaded)
    146     data_series = [
    147         pl.Series(
    148             name,
   (...)
    154         for name in column_names
    155     ]
    156 else:
    157     data_series = [
    158         s._s
--> 159         for s in _expand_dict_values(
    160             data,
    161             schema_overrides=schema_overrides,
    162             strict=strict,
    163             nan_to_null=nan_to_null,
    164         ).values()
    165     ]
    167 data_series = _handle_columns_arg(data_series, columns=column_names, from_dict=True)
    168 pydf = PyDataFrame(data_series)

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/polars/_utils/construction/dataframe.py:388, in _expand_dict_values(data, schema_overrides, strict, order, nan_to_null)
    385     updated_data[name] = s
    387 elif arrlen(val) is not None or _is_generator(val):
--> 388     updated_data[name] = pl.Series(
    389         name=name,
    390         values=val,
    391         dtype=dtype,
    392         strict=strict,
    393         nan_to_null=nan_to_null,
    394     )
    395 elif val is None or isinstance(  # type: ignore[redundant-expr]
    396     val, (int, float, str, bool, date, datetime, time, timedelta)
    397 ):
    398     updated_data[name] = F.repeat(
    399         val, array_len, dtype=dtype, eager=True
    400     ).alias(name)

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/polars/series/series.py:288, in Series.__init__(self, name, values, dtype, strict, nan_to_null)
    285         raise TypeError(msg)
    287 if isinstance(values, Sequence):
--> 288     self._s = sequence_to_pyseries(
    289         name,
    290         values,
    291         dtype=dtype,
    292         strict=strict,
    293         nan_to_null=nan_to_null,
    294     )
    296 elif values is None:
    297     self._s = sequence_to_pyseries(name, [], dtype=dtype)

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/polars/_utils/construction/series.py:294, in sequence_to_pyseries(name, values, dtype, strict, nan_to_null)
    291     except RuntimeError:
    292         return PySeries.new_from_any_values(name, values, strict=strict)
--> 294 return _construct_series_with_fallbacks(
    295     constructor, name, values, dtype, strict=strict
    296 )

File ~/.conda/envs/ds_notebook/lib/python3.12/site-packages/polars/_utils/construction/series.py:312, in _construct_series_with_fallbacks(constructor, name, values, dtype, strict)
    310 except TypeError:
    311     if dtype is None:
--> 312         return PySeries.new_from_any_values(name, values, strict=strict)
    313     else:
    314         return PySeries.new_from_any_values_and_dtype(
    315             name, values, dtype, strict=strict
    316         )

TypeError: unexpected value while building Series of type Int64; found value of type Float64: -inf

Hint: Try setting `strict=False` to allow passing data with mixed types.

I even got the same error when only one peak is specified in "regions", for example: motifs = snap.tl.motif_enrichment( motifs=snap.datasets.cis_bp(unique=True), regions= {'0':['chr1:182408480-182408981']}, genome_fasta=snap.genome.mm10 )

Appreciate your help!

[kaizhang]

which version are you using?

[HaikuoLi]

Thanks for continuing checking my issue, which has not been fixed still.

Version:
snapatac2 '2.7.0'
scanpy==1.9.8 anndata==0.10.9 umap==0.5.5 numpy==1.26.4 scipy==1.14.1 pandas==2.1.1 scikit-learn==1.5.1 statsmodels==0.14.1 igraph==0.11.6 pynndescent==0.5.11

Get error with one single code motifs = snap.tl.motif_enrichment( motifs=snap.datasets.cis_bp(unique=True), regions= {'0':['chr1:182408480-182408981']}, genome_fasta=snap.genome.mm10 )

I am attaching how it looks like in my workspace.
snap_motif.pdf