Fix docs, add citation info, update repo infra (#1362)

.coveragerc

@@ -3,4 +3,10 @@ branch = True
 source = src/
 omit =
     src/pynwb/_version.py
+    src/pynwb/_due.py
     src/pynwb/testing/*
+
+[report]
+exclude_lines =
+    pragma: no cover
+    @abstract

.github/ISSUE_TEMPLATE/bug_report.md

@@ -20,14 +20,14 @@ assignees: ''
 <!--Please describe your environment according to the following bullet points.-->
 
     Python Executable: Conda or Python
-    Python Version: Python 3.5, 3.6, or 3.7
+    Python Version: Python 3.6, 3.7, or 3.8
     Operating System: Windows, macOS or Linux
-    HDMF Version: 
+    HDMF Version:
     PyNWB Version:
 
 ## Checklist
 
-- [ ] Have you ensured the feature or change was not already [reported](https://github.com/NeurodataWithoutBorders/pynwb/issues)?
+- [ ] Have you ensured the bug was not already [reported](https://github.com/NeurodataWithoutBorders/pynwb/issues)?
 - [ ] Have you included a brief and descriptive title?
 - [ ] Have you included a clear description of the problem you are trying to solve?
 - [ ] Have you included a minimal code snippet that reproduces the issue you are encountering?

.github/PULL_REQUEST_TEMPLATE.md

@@ -4,13 +4,14 @@ What was the reasoning behind this change? Please explain the changes briefly.
 
 ## How to test the behavior?
 ```
-Show here how to reproduce the new behavior (can be a bug fix or a new feature)
+Show how to reproduce the new behavior (can be a bug fix or a new feature)
 ```
 
 ## Checklist
 
+- [ ] Did you update CHANGELOG.md with your changes?
 - [ ] Have you checked our [Contributing](https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/docs/CONTRIBUTING.rst) document?
-- [ ] Have you ensured the PR description clearly describes problem and the solution?
+- [ ] Have you ensured the PR clearly describes the problem and the solution?
 - [ ] Is your contribution compliant with our coding style? This can be checked running `flake8` from the source directory.
 - [ ] Have you checked to ensure that there aren't other open [Pull Requests](https://github.com/NeurodataWithoutBorders/pynwb/pulls) for the same change?
-- [ ] Have you included the relevant issue number using `#XXX` notation where `XXX` is the issue number?
+- [ ] Have you included the relevant issue number using "Fix #XXX" notation where XXX is the issue number? By including "Fix #XXX" you allow GitHub to close issue #XXX when the PR is merged.

.github/PULL_REQUEST_TEMPLATE/release.md

@@ -0,0 +1,20 @@
+Prepare for release of PyNWB [version]
+
+### Before merging:
+- [ ] Minor releases: Update package versions in `requirements.txt`, `requirements-dev.txt`, `requirements-doc.txt`, `requirements-min.txt` as needed. See https://requires.io/github/NeurodataWithoutBorders/pynwb/requirements/?branch=dev
+- [ ] Check legal file dates and information in `Legal.txt`, `license.txt`, `README.rst`, `docs/source/conf.py`, and any other locations as needed
+- [ ] Update `setup.py` as needed
+- [ ] Update `README.rst` as needed
+- [ ] Update `src/pynwb/nwb-schema` submodule as needed. Check the version number manually.
+- [ ] Update changelog (set release date) in `CHANGELOG.md` and any other docs as needed
+- [ ] Run tests locally including gallery tests and validation tests, and inspect all warnings and outputs (`python test.py -v > out.txt`)
+- [ ] Test docs locally (`make apidoc`, `make html`)
+- [ ] Push changes to this PR and make sure all PRs to be included in this release have been merged
+- [ ] Check that the readthedocs build for this PR succeeds (build latest to pull the new branch, then activate and
+  build docs for new branch): https://readthedocs.org/projects/pynwb/builds/
+
+### After merging:
+1. Create release by following steps in `docs/source/make_a_release.rst` or use alias `git pypi-release [tag]` if set up
+2. After the CI bot creates the new release (wait ~10 min), update the release notes on the [GitHub releases page](https://github.com/NeurodataWithoutBorders/pynwb/releases) with the changelog
+3. Check that the readthedocs "latest" and "stable" builds run and succeed
+4. Update [conda-forge/pynwb-feedstock](https://github.com/conda-forge/pynwb-feedstock)

.gitignore

@@ -59,6 +59,9 @@ docs/tmpl.index.rst
 tests/coverage/htmlcov
 .coverage
 
+# duecredit output
+.duecredit.p
+
 # tox
 .tox
 

CHANGELOG.md

@@ -11,6 +11,11 @@
 - Update documentation, CI GitHub processes. @oruebel @yarikoptic, @bendichter, @TomDonoghue, @rly
   (#1311, #1336, #1351, #1352, #1345, #1340, #1327)
 - Set default `neurodata_type_inc` for `NWBGroupSpec`, `NWBDatasetSpec`. @rly (#1295)
+- Block usage of h5py 3+ for now. h5py>=2.9, <3 is supported. (#1355)
+- Fix incompatibility issue with downstream github-release tool used to deploy releases to GitHub. @rly (#1245)
+- Fix issue with Sphinx gallery. @rly
+- Add citation information to documentation and support for duecredit tool. @rly
+- Remove use of ColoredTestRunner for more readable verbose test output. @rly
 - Add support for nwb-schema 2.3.0. @rly (#1245, #1330)
   - Add optional `waveforms` column to the `Units` table.
   - Add optional `strain` field to `Subject`.
@@ -21,7 +26,25 @@
   - Clarify documentation for electrode impedance and filtering.
   - Set the `stimulus_description` for `IZeroCurrentClamp` to have the fixed value "N/A".
   - See https://nwb-schema.readthedocs.io/en/latest/format_release_notes.html for full schema release notes.
-- Add support for HDMF 2.5.3. @rly @ajtritt (#1325, #1355, #1360, #1245, #1287)
+- Add support for HDMF 2.5.5 and upgrade HDMF requirement from 2.1.0 to 2.5.5. @rly @ajtritt
+  (#1325, #1355, #1360, #1245, #1287). This includes several relevant features and bug fixes, including:
+  - Fix issue where dependencies of included types were not being loaded in namespaces / extensions.
+  - Add `HDF5IO.get_namespaces(path=path, file=file)` method which returns a dict of namespace name mapped to the
+    namespace version (the largest one if there are multiple) for each namespace cached in the given HDF5 file.
+  - Add methods for automatic creation of `MultiContainerInterface` classes.
+  - Add ability to specify a custom class for new columns to a `DynamicTable` that are not `VectorData`,
+    `DynamicTableRegion`, or `VocabData` using `DynamicTable.__columns__` or `DynamicTable.add_column(...)`.
+  - Add support for creating and specifying multi-index columns in a `DynamicTable` using `add_column(...)`.
+  - Add capability to add a row to a column after IO.
+  - Add method `AbstractContainer.get_fields_conf`.
+  - Add functionality for storing external resource references.
+  - Add method `hdmf.utils.get_docval_macro` to get a tuple of the current values for a docval_macro, e.g., 'array_data'  
+    and 'scalar_data'.
+  - `DynamicTable` can be automatically generated using `get_class`. Now the HDMF API can read files with extensions
+    that contain a DynamicTable without needing to import the extension first.
+  - Add `EnumData` type for storing data that comes from a fixed set of values.
+  - Add `AlignedDynamicTable` type which defines a DynamicTable that supports storing a collection of subtables.
+  - Allow `np.bool_` as a valid `bool` dtype when validating.
   - See https://github.com/hdmf-dev/hdmf/releases for full HDMF release notes.
 
 ## PyNWB 1.4.0 (August 12, 2020)

MANIFEST.in

@@ -1,4 +1,4 @@
-include license.txt Legal.txt versioneer.py src/pynwb/_version.py
+include license.txt Legal.txt versioneer.py src/pynwb/_version.py src/pynwb/_due.py
 include requirements.txt requirements-dev.txt requirements-doc.txt requirements-min.txt
 include test.py tox.ini
 

README.rst

@@ -37,17 +37,20 @@ Build Status
 Overall Health
 ==============
 
+.. image:: https://github.com/NeurodataWithoutBorders/pynwb/workflows/Run%20coverage/badge.svg
+    :target: https://github.com/NeurodataWithoutBorders/pynwb/actions?query=workflow%3A%22Run+coverage%22
+
 .. image:: https://codecov.io/gh/NeurodataWithoutBorders/pynwb/branch/dev/graph/badge.svg
     :target: https://codecov.io/gh/NeurodataWithoutBorders/pynwb
 
 .. image:: https://requires.io/github/NeurodataWithoutBorders/pynwb/requirements.svg?branch=dev
      :target: https://requires.io/github/NeurodataWithoutBorders/pynwb/requirements/?branch=dev
      :alt: Requirements Status
-     
+
 .. image:: https://readthedocs.org/projects/pynwb/badge/?version=latest
      :target: https://pynwb.readthedocs.io/en/latest/?badge=latest
      :alt: Documentation Status
-     
+
 .. image:: https://img.shields.io/pypi/l/pynwb.svg
      :target: https://github.com/neurodatawithoutborders/pynwb/blob/dev/license.txt
      :alt:    PyPI - License
@@ -82,6 +85,11 @@ Contributing
 
 For details on how to contribute to PyNWB see our `contribution guidelines <docs/CONTRIBUTING.rst>`_.
 
+Citing NWB
+==========
+
+Oliver Rübel, Andrew Tritt, Ryan Ly, Benjamin K. Dichter, Satrajit Ghosh, Lawrence Niu, Ivan Soltesz, Karel Svoboda, Loren Frank, Kristofer E. Bouchard, "The Neurodata Without Borders ecosystem for neurophysiological data science", bioRxiv 2021.03.13.435173, March 15, 2021, doi: https://doi.org/10.1101/2021.03.13.435173
+
 LICENSE
 =======
 

docs/gallery/domain/brain_observatory.py

@@ -142,9 +142,9 @@
 )
 
 ########################################
-# The Allen Institute does not include the raw imaging signal, as this data would make the file too large. Instead, these
-# data are  preprocessed, and a dF/F fluorescence signal extracted for each region-of-interest (ROI). To store the chain
-# of computations necessary to describe this data processing pipeline, pynwb provides a "processing module" with
+# The Allen Institute does not include the raw imaging signal, as this data would make the file too large. Instead,
+# these data are preprocessed, and a dF/F fluorescence signal extracted for each region-of-interest (ROI). To store the
+# chain of computations necessary to describe this data processing pipeline, pynwb provides a "processing module" with
 # interfaces that simplify and standardize the process of adding the steps in this provenance chain to the file:
 ophys_module = nwbfile.create_processing_module(
     name='ophys_module',

docs/source/conf.py

@@ -65,17 +65,16 @@
     'gallery_dirs': ['tutorials'],
     'subsection_order': ExplicitOrder(['../gallery/general', '../gallery/domain']),
     'backreferences_dir': 'gen_modules/backreferences',
-    'download_section_examples': False,
     'min_reported_time': 5
 }
 
 intersphinx_mapping = {
     'python': ('https://docs.python.org/3.8', None),
-    'numpy': ('https://docs.scipy.org/doc/numpy/', None),
+    'numpy': ('https://numpy.org/doc/stable/objects.inv', None),
     'matplotlib': ('https://matplotlib.org', None),
-    'h5py': ('http://docs.h5py.org/en/latest/', None),
+    'h5py': ('https://docs.h5py.org/en/latest/', None),
     'hdmf': ('https://hdmf.readthedocs.io/en/latest/', None),
-    'pandas': ('http://pandas.pydata.org/pandas-docs/stable/', None)
+    'pandas': ('https://pandas.pydata.org/pandas-docs/stable/', None)
 }
 
 # Add any paths that contain templates here, relative to this directory.
@@ -326,5 +325,5 @@ def skip(app, what, name, obj, skip, options):
 
 def setup(app):
     app.connect('builder-inited', run_apidoc)
-    app.add_stylesheet("theme_overrides.css")  # overrides for wide tables in RTD theme
+    app.add_css_file("theme_overrides.css")  # overrides for wide tables in RTD theme
     app.connect("autodoc-skip-member", skip)

docs/source/index.rst

@@ -33,6 +33,7 @@ breaking down the barriers to data sharing in neuroscience.
    overview_intro
    overview_software_architecture
    overview_nwbfile
+   overview_citing
 
 .. toctree::
    :maxdepth: 2

docs/source/overview_citing.rst

@@ -0,0 +1,54 @@
+.. _citing:
+
+Citing PyNWB
+============
+
+BibTeX entry
+------------
+
+If you use PyNWB in your research, please use the following citation:
+
+.. code-block:: bibtex
+
+    @article {R{\"u}bel2021.03.13.435173,
+     author = {R{\"u}bel, Oliver and Tritt, Andrew and Ly, Ryan and Dichter, Benjamin K. and Ghosh, Satrajit and Niu, Lawrence and Soltesz, Ivan and Svoboda, Karel and Frank, Loren and Bouchard, Kristofer E.},
+     title = {The Neurodata Without Borders ecosystem for neurophysiological data science},
+     elocation-id = {2021.03.13.435173},
+     year = {2021},
+     doi = {10.1101/2021.03.13.435173},
+     publisher = {Cold Spring Harbor Laboratory},
+     abstract = {The neurophysiology of cells and tissues are monitored electrophysiologically and optically in diverse experiments and species, ranging from flies to humans. Understanding the brain requires integration of data across this diversity, and thus these data must be findable, accessible, interoperable, and reusable (FAIR). This requires a standard language for data and metadata that can coevolve with neuroscience. We describe design and implementation principles for a language for neurophysiology data. Our software (Neurodata Without Borders, NWB) defines and modularizes the interdependent, yet separable, components of a data language. We demonstrate NWB{\textquoteright}s impact through unified description of neurophysiology data across diverse modalities and species. NWB exists in an ecosystem which includes data management, analysis, visualization, and archive tools. Thus, the NWB data language enables reproduction, interchange, and reuse of diverse neurophysiology data. More broadly, the design principles of NWB are generally applicable to enhance discovery across biology through data FAIRness.Competing Interest StatementThe authors have declared no competing interest.},
+     URL = {https://www.biorxiv.org/content/early/2021/03/15/2021.03.13.435173},
+     eprint = {https://www.biorxiv.org/content/early/2021/03/15/2021.03.13.435173.full.pdf},
+     journal = {bioRxiv}
+    }
+
+Using duecredit
+-----------------
+
+Citations can be generated using duecredit_. To install duecredit, run ``pip install duecredit``.
+
+You can obtain a list of citations for your Python script, e.g., ``yourscript.py``, using:
+
+.. code-block:: bash
+
+   cd /path/to/your/module
+   python -m duecredit yourscript.py
+
+Alternatively, you can set the environment variable ``DUECREDIT_ENABLE=yes``
+
+.. code-block:: bash
+
+   DUECREDIT-ENABLE=yes python yourscript.py
+
+Citations will be saved in a hidden file (``.duecredit.p``) in the current directory. You can then use the duecredit_
+command line tool to export the citations to different formats. For example, you can display your citations in
+BibTeX format using:
+
+.. code-block:: bash
+
+   duecredit summary --format=bibtex
+
+For more information on using duecredit, please consult its `homepage <https://github.com/duecredit/duecredit>`_.
+
+.. _duecredit: https://github.com/duecredit/duecredit

requirements-dev.txt

@@ -1,7 +1,9 @@
-codecov==2.1.8
-coverage==5.2
-flake8==3.8.3
-flake8-debugger==3.1.0
-flake8-print==3.1.4
-importlib-metadata<2
-tox==3.17.1
+# pinned dependencies to reproduce an entire development environment to use PyNWB, run PyNWB tests, check code style,
+# compute coverage, and create test environments
+codecov==2.1.11
+coverage==5.5
+flake8==3.9.1
+flake8-debugger==4.0.0
+flake8-print==4.0.0
+importlib-metadata==4.0.1
+tox==3.23.0

requirements-doc.txt

@@ -1,3 +1,4 @@
+# dependencies to generate the documentation for PyNWB
 sphinx
 matplotlib
 sphinx_rtd_theme

requirements.txt

@@ -1,6 +1,7 @@
+# pinned dependencies to reproduce an entire development environment to use PyNWB
 h5py==2.10.0
 hdmf==2.5.5
-numpy==1.18.5
-pandas==0.25.3
+numpy==1.19.3
+pandas==1.1.5
 python-dateutil==2.8.1
 setuptools==56.0.0

setup.cfg

@@ -15,18 +15,25 @@ exclude =
   .git,
   .tox,
   __pycache__,
-  docs/_build/,
+  build/,
   dist/,
   src/pynwb/nwb-schema/
+  docs/_build/,
   docs/source/conf.py
-  versioneer.py
+  docs/source/tutorials/
+  versioneer.py,
+  src/pynwb/_version.py
+  src/pynwb/_due.py
 per-file-ignores =
-  docs/gallery/*:E402,
-  docs/source/tutorials/*:E402
+  docs/gallery/*:E402,T001
   src/pynwb/io/__init__.py:F401
   src/pynwb/legacy/io/__init__.py:F401
   tests/integration/__init__.py:F401
   src/pynwb/testing/__init__.py:F401
+  src/pynwb/validate.py:T001
+  setup.py:T001
+  test.py:T001
+  scripts/*:T001
 
 [metadata]
 description-file = README.rst

setup.py

@@ -36,7 +36,6 @@
     'package_data': {'pynwb': ["%s/*.yaml" % schema_dir, "%s/*.json" % schema_dir]},
     'classifiers': [
         "Programming Language :: Python",
-        "Programming Language :: Python :: 3.5",
         "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",

src/pynwb/__init__.py

@@ -279,3 +279,21 @@ def export(self, **kwargs):
 from ._version import get_versions  # noqa: E402
 __version__ = get_versions()['version']
 del get_versions
+
+from ._due import due, BibTeX  # noqa: E402
+due.cite(BibTeX("""
+@article {R{\"u}bel2021.03.13.435173,
+    author = {R{\"u}bel, Oliver and Tritt, Andrew and Ly, Ryan and Dichter, Benjamin K. and Ghosh, Satrajit and Niu, Lawrence and Soltesz, Ivan and Svoboda, Karel and Frank, Loren and Bouchard, Kristofer E.},
+    title = {The Neurodata Without Borders ecosystem for neurophysiological data science},
+    elocation-id = {2021.03.13.435173},
+    year = {2021},
+    doi = {10.1101/2021.03.13.435173},
+    publisher = {Cold Spring Harbor Laboratory},
+    abstract = {The neurophysiology of cells and tissues are monitored electrophysiologically and optically in diverse experiments and species, ranging from flies to humans. Understanding the brain requires integration of data across this diversity, and thus these data must be findable, accessible, interoperable, and reusable (FAIR). This requires a standard language for data and metadata that can coevolve with neuroscience. We describe design and implementation principles for a language for neurophysiology data. Our software (Neurodata Without Borders, NWB) defines and modularizes the interdependent, yet separable, components of a data language. We demonstrate NWB{\textquoteright}s impact through unified description of neurophysiology data across diverse modalities and species. NWB exists in an ecosystem which includes data management, analysis, visualization, and archive tools. Thus, the NWB data language enables reproduction, interchange, and reuse of diverse neurophysiology data. More broadly, the design principles of NWB are generally applicable to enhance discovery across biology through data FAIRness.Competing Interest StatementThe authors have declared no competing interest.},
+    URL = {https://www.biorxiv.org/content/early/2021/03/15/2021.03.13.435173},
+    eprint = {https://www.biorxiv.org/content/early/2021/03/15/2021.03.13.435173.full.pdf},
+    journal = {bioRxiv}
+}
+"""), description="The Neurodata Without Borders ecosystem for neurophysiological data science",  # noqa: E501
+         path="pynwb/", version=__version__, cite_module=True)
+del due, BibTeX