Decrosstalk resulting in NaN values for Dendrite Experiments

[Ahad-Allen]

Currently, the Decrosstalk module running through LIMS results in downstream output files for experiments within the session, such as the roi_traces.h5, consisting of only 10,000 values that are all NaN. This was discussed at the OpenScope meeting on Friday with @nataliaorlova / @jeromelecoq and it was mentioned that the issue should be brought up here with to the attention of @danielsf.

The output of a session level jobs on LIMS with this specific failure can be viewed here: http://lims2/job_logs?id=1211685068&utf8=%E2%9C%93. The specific error appearing to be "ERROR:ophys_etl.modules.decrosstalk.decrosstalk:ICA did not converge for any ROIs when applying decrosstalk to ophys_experiment_id: 1211453844 (1211453846)"

[danielsf]

ICA failing to converge for any ROIs is a known issue with our implementation of decrosstalk. Sometimes FastICA does not converge to the level of stringency we want. I don't think we (whoever "we" are) have every decided on what course of action to take in that case. The trace files full of NaNs is produced because modules downstream of decrosstalk expect there to be something (even if that something is a trace file full of invalid ROIs).

The way forward is to decide on an algorithmic strategy to address cases where FastICA fails to converge. This will likely impact all users of the ophys_etl_pipeline so, whatever happens, the learning mFISH team will need to sign off on it (because nothing around here is allowed to be simple....)

[nataliaorlova]

@Ahad-Allen, this experiment is a dendritic imaging plane - I don't think we ever expected LIMS processing pipeline to work on those planes starting all the way from segmentation. This plane has no cells, he ones that do occasionally show up there are probably leaked from the coupled plane. So, probably it's coupled plane had ROIs segmented, and there probably will be a lot of biological coupling in the signals coming from two planes. Because of this I'd anticipate decrosstalking to not work from the box.

[jeromelecoq]

Failure to converge probably mean the negentropy (or whatever metric you chose in fastICA to identify non-gaussianity) fails to go up, so perhaps the signal is too weak to be detected on the paired plane. If that is the case, I suspect the identity matrix would be the best answer and we could propagate the original ROI trace forward. Does the data store the convergence values (or similar) for each ROI? This would be one way to make these outcomes more visible.

I was not involved in pushing this pipeline to the finish line so I do not know whether you discussed that case and whether what I just said makes entire sense.