Run full Valero conversion for subjects fCamk1 and fCamk2 #66

h-mayorquin · 2023-06-22T16:44:56Z

This is the final debuggin before doing the full conversion. I got my hands on 70 per cent of the data. I should soon have access to all of it (still downloading it). This is a PR for preparing for that conversion it contains the following:

Final additions of metadata for the UP and Down states.
Warnings instead of assertions when certain files are not found.
Modification of run_conversion functions to add_to_nwbfile in line with the latest changes in neuroconv.
First draft of the convert_all_sessions.py script.
Addition of edge cases in certain interfaces (for example, the SleepIntervals interface sometimes receives only one interval and then requires special logic for that.

Todo:

Run conversion stubs for all sessions with subjects fCamk1 and fCamk1.
Debug for errors
run inspector
Run full conversion to see if there are memory problems.
Get a sense of conversion time and compression ratios.
Something else?

h-mayorquin · 2023-06-22T18:31:55Z

The inspector output for the sessions here:

Found 62 issues over 12 files:
      14 - BEST_PRACTICE_VIOLATION
      48 - BEST_PRACTICE_SUGGESTION
**************************************************


0  BEST_PRACTICE_VIOLATION
==========================

0.0  fCamk1_200827_sess9.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk1/fCamk1_200827_sess9/fCamk1_200827_110712/Basler_acA1280-60gc__21606137__20200827_110730202.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.1  fCamk1_200827_sess9.nwb and 1 other file: check_time_interval_time_columns - 'TimeIntervals' object with name 'SleepStates'
       Message: ['stop_time'] are time columns but the values are not in ascending order. All times should be in seconds with respect to the session start time.

0.2  fCamk1_200901_sess12.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk1/fCamk1_200901_sess12/fCamk1_200901_115118/Basler_acA1280-60gc__21606137__20200901_115140678.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.3  fCamk1_200902_sess13.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk1/fCamk1_200902_sess13/fCamk1_200902_115511/Basler_acA1280-60gc__21606137__20200902_115537938.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.4  fCamk1_200904_sess15.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk1/fCamk1_200904_sess15/fCamk1_200904_120435/Basler_acA1280-60gc__21606137__20200904_120507437.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.5  fCamk1_200908_sess16.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk1/fCamk1_200908_sess16/fCamk1_200908_124143/Basler_acA1280-60gc__21606137__20200908_124220793.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.6  fCamk1_200909_sess17.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk1/fCamk1_200909_sess17/fCamk1_200909_115825/Basler_acA1280-60gc__21606137__20200909_115853843.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.7  fCamk1_200910_sess18.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk1/fCamk1_200910_sess18/fCamk1_200910_131411/Basler_acA1280-60gc__21606137__20200910_131440006.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.8  fCamk1_200911_sess19.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk1/fCamk1_200911_sess19/fCamk1_200911_112340/Basler_acA1280-60gc__21606137__20200911_112406258.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.9  fCamk2_201012_sess1.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk2/fCamk2_201012_sess1/fCamk2_201012_130347/Basler_acA1280-60gc__21606137__20201012_130405941.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.10  fCamk2_201013_sess2.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
        Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk2/fCamk2_201013_sess2/fCamk2_201013_105930/Basler_acA1280-60gc__21606137__20201013_105946653.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.11  fCamk2_201014_sess3.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
        Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk2/fCamk2_201014_sess3/fCamk2_201014_112432/Basler_acA1280-60gc__21606137__20201014_112455106.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

0.12  fCamk2_201015_sess4.nwb: check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
        Message: The external file '/media/heberto/One Touch/Buzsaki/ValeroM/fCamk2/fCamk2_201015_sess4/fCamk2_201015_110238/Basler_acA1280-60gc__21606137__20201015_110255856.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

1  BEST_PRACTICE_SUGGESTION
===========================

1.13  fCamk1_200827_sess9.nwb and 11 other files: check_description - 'VectorData' object with name 'gain_to_uV'
        Message: Description ('no description') is a placeholder.

1.14  fCamk1_200827_sess9.nwb and 11 other files: check_description - 'VectorData' object with name 'offset_to_uV'
        Message: Description ('no description') is a placeholder.

1.15  fCamk1_200827_sess9.nwb and 11 other files: check_description - 'VectorData' object with name 'channel_name'
        Message: Description ('no description') is a placeholder.

1.16  fCamk1_200827_sess9.nwb and 11 other files: check_description - 'ProcessingModule' object with name 'behavior'
        Message: Description ('No description.') is a placeholder.

…xplorerInterface

buzsaki_lab_to_nwb/valero/behaviorinterface.py

CodyCBakerPhD · 2023-06-23T14:28:28Z

For the 'not ascending' check, are you using the dev branch of inspector or the latest release?

CodyCBakerPhD · 2023-06-23T14:31:28Z

Just to make sure the DANDI filenames look OK and such, and get rid of the relative path errors, can you try dandi organize and then rerun and share the inspection?

…blems Solve memory overflow for Valero Conversion

h-mayorquin · 2023-06-23T14:59:43Z

I am using dev for inspector, the error is related to this:

NeurodataWithoutBorders/nwbinspector#375

Now I am testing if Neuroscope will be enough if I wll need the new CellExplorerRecordingInterface coupled with SpikeInterface/spikeinterface#1741

CodyCBakerPhD · 2023-06-23T17:55:31Z

I am using dev for inspector, the error is related to this:
NeurodataWithoutBorders/nwbinspector#375

Ah, right - I remember seeing the issue, but I thought a fix was pushed through? Or no?

h-mayorquin · 2023-06-26T14:29:29Z

@CodyCBakerPhD
I pushed a solution for a similar issue, that one is still unadressed.

h-mayorquin · 2023-06-28T16:11:54Z

@CodyCBakerPhD

I ran dandi organize and then inspector before, those are the errors in the post-organize folder:

0  BEST_PRACTICE_VIOLATION
==========================

0.0  sub-fCamk1/sub-fCamk1_ses-fCamk1-200827-sess9_behavior+ecephys+image+ogen.nwb and 1 other file: check_time_interval_time_columns - 'TimeIntervals' object with name 'SleepStates'
       Message: ['stop_time'] are time columns but the values are not in ascending order. All times should be in seconds with respect to the session start time.

0.1  sub-fCamk1/sub-fCamk1_ses-fCamk1-200827-sess9_behavior+ecephys+image+ogen.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=8284.673333333332 and rate=0.03333333333466726 instead of timestamps.

0.2  sub-fCamk1/sub-fCamk1_ses-fCamk1-200901-sess12_behavior+ecephys+image+ogen.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=8093.441333333334 and rate=0.03333333333284827 instead of timestamps.

0.3  sub-fCamk1/sub-fCamk1_ses-fCamk1-200902-sess13_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=11471.841333333334 and rate=0.03333333333466726 instead of timestamps.

0.4  sub-fCamk1/sub-fCamk1_ses-fCamk1-200904-sess15_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=10659.585333333333 and rate=0.03333333333466726 instead of timestamps.

0.5  sub-fCamk1/sub-fCamk1_ses-fCamk1-200908-sess16_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=12492.321333333333 and rate=0.03333333333466726 instead of timestamps.

0.6  sub-fCamk1/sub-fCamk1_ses-fCamk1-200909-sess17_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=10546.049333333332 and rate=0.03333333333466726 instead of timestamps.

0.7  sub-fCamk1/sub-fCamk1_ses-fCamk1-200910-sess18_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=15388.193333333333 and rate=0.03333333333466726 instead of timestamps.

0.8  sub-fCamk1/sub-fCamk1_ses-fCamk1-200911-sess19_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=9474.369333333332 and rate=0.03333333333466726 instead of timestamps.

0.9  sub-fCamk2/sub-fCamk2_ses-fCamk2-201012-sess1_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
       Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=6988.801333333334 and rate=0.03333333333284827 instead of timestamps.

0.10  sub-fCamk2/sub-fCamk2_ses-fCamk2-201013-sess2_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
        Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=7750.337333333334 and rate=0.03333333333284827 instead of timestamps.

0.11  sub-fCamk2/sub-fCamk2_ses-fCamk2-201014-sess3_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
        Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=7674.337333333334 and rate=0.03333333333284827 instead of timestamps.

0.12  sub-fCamk2/sub-fCamk2_ses-fCamk2-201015-sess4_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
        Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=8183.617333333334 and rate=0.03333333333284827 instead of timestamps.

0.13  sub-fCamk3/sub-fCamk3_ses-fCamk3-201111-sess20_behavior+ecephys+image.nwb: check_regular_timestamps - 'ImageSeries' object at location '/acquisition/ImageSeriesTrackingVideo1'
        Message: TimeSeries appears to have a constant sampling rate. Consider specifying starting_time=5968.321333333333 and rate=0.03333333333284827 instead of timestamps.

1  BEST_PRACTICE_SUGGESTION
===========================

1.14  sub-fCamk1/sub-fCamk1_ses-fCamk1-200827-sess9_behavior+ecephys+image+ogen.nwb and 27 other files: check_description - 'ElectrodeGroup' object at location '/general/extracellular_ephys/Group 4'
        Message: Description ('no description') is a placeholder.

1.15  sub-fCamk1/sub-fCamk1_ses-fCamk1-200827-sess9_behavior+ecephys+image+ogen.nwb and 27 other files: check_description - 'ElectrodeGroup' object at location '/general/extracellular_ephys/Group 3'
        Message: Description ('no description') is a placeholder.

1.16  sub-fCamk1/sub-fCamk1_ses-fCamk1-200827-sess9_behavior+ecephys+image+ogen.nwb and 27 other files: check_description - 'ElectrodeGroup' object at location '/general/extracellular_ephys/Group 2'
        Message: Description ('no description') is a placeholder.

1.17  sub-fCamk1/sub-fCamk1_ses-fCamk1-200827-sess9_behavior+ecephys+image+ogen.nwb and 27 other files: check_description - 'ElectrodeGroup' object at location '/general/extracellular_ephys/Group 1'
        Message: Description ('no description') is a placeholder.

1.18  sub-fCamk1/sub-fCamk1_ses-fCamk1-200827-sess9_behavior+ecephys+image+ogen.nwb and 27 other files: check_description - 'VectorData' object with name 'gain_to_uV'
        Message: Description ('no description') is a placeholde

run full conversion for subjects 1 and 2

af7d0cd

h-mayorquin requested a review from CodyCBakerPhD June 22, 2023 16:44

modify stimuli interface to account for chopped stimuli at the end

2f822a2

move sleep interface to events, add verbose for file sizes, use CellE…

066a516

…xplorerInterface

h-mayorquin mentioned this pull request Jun 23, 2023

Avoid redundant timestamp creation in add_electrical_series for recording objects without time vector catalystneuro/neuroconv#495

Merged

add more verbosity

e885d41

CodyCBakerPhD reviewed Jun 23, 2023

View reviewed changes

buzsaki_lab_to_nwb/valero/behaviorinterface.py Outdated Show resolved Hide resolved

Merge pull request #67 from catalystneuro/change_interface_memory_pro…

f113c27

…blems Solve memory overflow for Valero Conversion

h-mayorquin changed the title ~~Run full Valero conversion for subjects fCamk1 and fCamk1~~ Run full Valero conversion for subjects fCamk1 and fCamk2 Jun 26, 2023

h-mayorquin added 4 commits June 28, 2023 13:37

add condition for adding ecephys metadata

b944b9d

correct video interface name

dab48ed

problem with sessions with no available videos solved

0f9919f

add description to behavioral module and refactor

530457a

Merge branch 'master' into final_conversion_valero

1fc72f3

h-mayorquin enabled auto-merge July 4, 2023 15:31

CodyCBakerPhD approved these changes Jul 4, 2023

View reviewed changes

h-mayorquin merged commit fa37396 into master Jul 4, 2023
1 check passed

h-mayorquin deleted the final_conversion_valero branch July 4, 2023 15:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run full Valero conversion for subjects fCamk1 and fCamk2 #66

Run full Valero conversion for subjects fCamk1 and fCamk2 #66

h-mayorquin commented Jun 22, 2023 •

edited

Loading

h-mayorquin commented Jun 22, 2023

CodyCBakerPhD commented Jun 23, 2023

CodyCBakerPhD commented Jun 23, 2023

h-mayorquin commented Jun 23, 2023

CodyCBakerPhD commented Jun 23, 2023

h-mayorquin commented Jun 26, 2023

h-mayorquin commented Jun 28, 2023

Run full Valero conversion for subjects fCamk1 and fCamk2 #66

Run full Valero conversion for subjects fCamk1 and fCamk2 #66

Conversation

h-mayorquin commented Jun 22, 2023 • edited Loading

h-mayorquin commented Jun 22, 2023

CodyCBakerPhD commented Jun 23, 2023

CodyCBakerPhD commented Jun 23, 2023

h-mayorquin commented Jun 23, 2023

CodyCBakerPhD commented Jun 23, 2023

h-mayorquin commented Jun 26, 2023

h-mayorquin commented Jun 28, 2023

h-mayorquin commented Jun 22, 2023 •

edited

Loading