[Bug]: "Maximum Recursion Depth Exceeded" displaying NWB when testing Jupyter notebook

[rcpeene]

What happened?

I have a github workflow that tests my Jupyter notebooks with pytest and nbmake. Anytime I have a cell that interactively displays parts of an NWB file (such as a cell that simply says nwb), it runs fine in the actual Jupyter program on vscode, but fails during the pytests on Github, yielding the error maximum recursion depth exceeded. See this failed test as an example.
When running the pytest locally on my windows machine, this is not a problem.

Steps to Reproduce

- Running Ubuntu
- Create a Jupyter notebook that opens an nwb file with PyNWB
- Include a cell that displays an nwb like follows: `nwb`
- Make sure pytest and nbmake are installed
- In the command prompt, run `pytest --nbmake ./test_nb.ipynb`

Traceback

_ /home/runner/work/openscope_databook/openscope_databook/docs/experiments/illusion.ipynb _
[gw3] linux -- Python 3.10.12 /usr/bin/python3
---------------------------------------------------------------------------
nwb
---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
File ~/.local/lib/python3.10/site-packages/IPython/core/formatters.py:344, in BaseFormatter.__call__(self, obj)
    342     method = get_real_method(obj, self.print_method)
    343     if method is not None:
--> 344         return method()
    345     return None
    346 else:

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:618, in Container._repr_html_(self)
    616 html_repr += "<div class='container-wrap'>"
    617 html_repr += f"<div class='container-header'><div class='xr-obj-type'><h3>{header_text}</h3></div></div>"
--> 618 html_repr += self._generate_html_repr(self.fields,is_field=True)
    619 html_repr += "</div>"
    620 return html_repr

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:629, in Container._generate_html_repr(self, fields, level, access_code, is_field)
    627     for key, value in fields.items():
    628         current_access_code = f"{access_code}.{key}" if is_field else f"{access_code}['{key}']"
--> 629         html_repr += self._generate_field_html(key,value,level,current_access_code)
    630 elif isinstance(fields, list):
    631     for index, item in enumerate(fields):

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:657, in Container._generate_field_html(self, key, value, level, access_code)
    655     html_content = self._generate_html_repr(value.fields, level + 1, access_code, is_field=True)
    656 elif isinstance(value, (list, dict, np.ndarray)):
--> 657     html_content = self._generate_html_repr(value,level+1,access_code,is_field=False)
    658 else:
    659     html_content = f'<span class="field-key">{value}</span>'

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:629, in Container._generate_html_repr(self, fields, level, access_code, is_field)
    627     for key, value in fields.items():
    628         current_access_code = f"{access_code}.{key}" if is_field else f"{access_code}['{key}']"
--> 629         html_repr += self._generate_field_html(key,value,level,current_access_code)
    630 elif isinstance(fields, list):
    631     for index, item in enumerate(fields):

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:655, in Container._generate_field_html(self, key, value, level, access_code)
    652     html_content = value.__repr_html__()
    654 elif hasattr(value, "fields"):
--> 655     html_content = self._generate_html_repr(value.fields,level+1,access_code,is_field=True)
    656 elif isinstance(value, (list, dict, np.ndarray)):
    657     html_content = self._generate_html_repr(value, level + 1, access_code, is_field=False)

    [... skipping similar frames: Container._generate_html_repr at line 629 (1 times)]

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:657, in Container._generate_field_html(self, key, value, level, access_code)
    655     html_content = self._generate_html_repr(value.fields, level + 1, access_code, is_field=True)
    656 elif isinstance(value, (list, dict, np.ndarray)):
--> 657     html_content = self._generate_html_repr(value,level+1,access_code,is_field=False)
    658 else:
    659     html_content = f'<span class="field-key">{value}</span>'

    [... skipping similar frames: Container._generate_html_repr at line 629 (1 times)]

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:655, in Container._generate_field_html(self, key, value, level, access_code)
    652     html_content = value.__repr_html__()
    654 elif hasattr(value, "fields"):
--> 655     html_content = self._generate_html_repr(value.fields,level+1,access_code,is_field=True)
    656 elif isinstance(value, (list, dict, np.ndarray)):
    657     html_content = self._generate_html_repr(value, level + 1, access_code, is_field=False)

    [... skipping similar frames: Container._generate_html_repr at line 629 (2 times), Container._generate_field_html at line 655 (1 times), Container._generate_field_html at line 657 (1 times)]

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:633, in Container._generate_html_repr(self, fields, level, access_code, is_field)
    631     for index, item in enumerate(fields):
    632         access_code += f'[{index}]'
--> 633         html_repr += self._generate_field_html(index,item,level,access_code)
    634 elif isinstance(fields, np.ndarray):
    635     html_repr += self._generate_array_html(fields, level)

    [... skipping similar frames: Container._generate_field_html at line 655 (2 times), Container._generate_html_repr at line 629 (2 times), Container._generate_field_html at line 657 (1 times)]

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:633, in Container._generate_html_repr(self, fields, level, access_code, is_field)
    631     for index, item in enumerate(fields):
    632         access_code += f'[{index}]'
--> 633         html_repr += self._generate_field_html(index,item,level,access_code)
    634 elif isinstance(fields, np.ndarray):
    635     html_repr += self._generate_array_html(fields, level)

    [... skipping similar frames: Container._generate_field_html at line 655 (980 times), Container._generate_html_repr at line 629 (980 times), Container._generate_field_html at line 657 (489 times), Container._generate_html_repr at line 633 (489 times)]

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:657, in Container._generate_field_html(self, key, value, level, access_code)
    655     html_content = self._generate_html_repr(value.fields, level + 1, access_code, is_field=True)
    656 elif isinstance(value, (list, dict, np.ndarray)):
--> 657     html_content = self._generate_html_repr(value,level+1,access_code,is_field=False)
    658 else:
    659     html_content = f'<span class="field-key">{value}</span>'

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:633, in Container._generate_html_repr(self, fields, level, access_code, is_field)
    631     for index, item in enumerate(fields):
    632         access_code += f'[{index}]'
--> 633         html_repr += self._generate_field_html(index,item,level,access_code)
    634 elif isinstance(fields, np.ndarray):
    635     html_repr += self._generate_array_html(fields, level)

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:655, in Container._generate_field_html(self, key, value, level, access_code)
    652     html_content = value.__repr_html__()
    654 elif hasattr(value, "fields"):
--> 655     html_content = self._generate_html_repr(value.fields,level+1,access_code,is_field=True)
    656 elif isinstance(value, (list, dict, np.ndarray)):
    657     html_content = self._generate_html_repr(value, level + 1, access_code, is_field=False)

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:629, in Container._generate_html_repr(self, fields, level, access_code, is_field)
    627     for key, value in fields.items():
    628         current_access_code = f"{access_code}.{key}" if is_field else f"{access_code}['{key}']"
--> 629         html_repr += self._generate_field_html(key,value,level,current_access_code)
    630 elif isinstance(fields, list):
    631     for index, item in enumerate(fields):

File ~/.local/lib/python3.10/site-packages/hdmf/container.py:644, in Container._generate_field_html(self, key, value, level, access_code)
    641 def _generate_field_html(self, key, value, level, access_code):
    642     """Generates HTML for a single field."""
--> 644     if isinstance(value,(int,float,str,bool)):
    645         return f'<div style="margin-left: {level * 20}px;" class="container-fields"><span class="field-key"' \
    646                f' title="{access_code}">{key}: </span><span class="field-value">{value}</span></div>'
    648     if hasattr(value, "generate_html_repr"):

RecursionError: maximum recursion depth exceeded in __instancecheck__

Operating System

Linux

Python Executable

Python

Python Version

3.9

Package Versions

argcomplete==3.2.1
attrs==21.2.0
Automat==20.2.0
Babel==2.8.0
bcrypt==3.2.0
blinker==1.4
certifi==2020.6.20
chardet==4.0.0
click==8.0.3
cloud-init==23.3.3
colorama==0.4.4
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
cryptography==3.4.8
dbus-python==1.2.18
distro==1.7.0
distro-info==1.1+ubuntu0.2
httplib2==0.20.2
hyperlink==21.0.0
idna==3.3
importlib-metadata==4.6.4
incremental==21.3.0
jeepney==0.7.1
Jinja2==3.0.3
jsonpatch==1.32
jsonpointer==2.0
jsonschema==3.2.0
keyring==23.5.0
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
MarkupSafe==2.0.1
mercurial==6.1.1
more-itertools==8.10.0
netifaces==0.11.0
oauthlib==3.2.0
packaging==23.2
pexpect==4.8.0
pipx==1.4.1
platformdirs==4.1.0
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.1
Pygments==2.11.2
PyGObject==3.42.1
PyHamcrest==2.0.2
PyJWT==2.3.0
pyOpenSSL==21.0.0
pyparsing==2.4.7
pyparted==3.11.7
pyrsistent==0.18.1
pyserial==3.5
python-apt==2.4.0+ubuntu2
python-debian==0.1.43+ubuntu1.1
python-magic==0.4.24
pytz==2022.1
PyYAML==5.4.1
requests==2.25.1
SecretStorage==3.3.1
service-identity==18.1.0
six==1.16.0
sos==4.5.6
ssh-import-id==5.11
systemd-python==234
tomli==2.0.1
Twisted==22.1.0
ubuntu-advantage-tools==8001
ufw==0.36.1
urllib3==1.26.5
userpath==1.9.1
wadllib==1.3.6
WALinuxAgent==2.2.46
zipp==1.0.0
zope.interface==5.4.0

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this bug was not already reported?

[weiglszonja]

I’ve seen a recursion error like this before, I investigated a little bit and it was about linking timestamps across multiple behavior objects like spatial series, not sure if you have the same:
hdmf-dev/hdmf#1010 (comment)

[rly]

I can reproduce this issue with the file from dandiset 000336 that you shared with me in a separate issue, and with a neuropixels file from dandiset 000053. Indeed it looks like the same issue as hdmf-dev/hdmf#1010 and @weiglszonja has a nice MWE to reproduce it. (@weiglszonja sorry that this issue hasn't been looked at recently!)

@stephprince Would you have some bandwidth to take a look at this issue? The issue is with the Container._repr_html_ in HDMF.

[stephprince]

yes, I can take a look at this.