Validation of network models

[pgleeson]

There are various packages/initiatives out there investigating ways to validate models in computational neuroscience, including NeuronUnit and the HBP Validation Framework.

While validation of a single neuron model against single cell recordings is a well understood problem, it's less clear how a network model should be validated...

Thoughts @apdavison @rgerkin @scrook @JohnGriffiths?

[apdavison]

My group together with Sonja Grün's lab have been developing tools for anatomical validation (e.g. laminar distribution of different synapse types in hippocampus CA1 - https://github.com/pedroernesto/HippoNetworkUnit) and for functional validation based on MEA recordings (e.g. https://github.com/INM-6/NetworkUnit). @mdenker and @pedroernesto can give more information.

I think one of the main challenges in validating network models at the functional level is that many of the physiological signals are spatiotemporally filtered aggregates across many neurons, e.g. LFP, two-photon calcium imaging, so you need good models of how these signals are generated, and you first need to validate these generative models...

[pedroernesto]

In addition, the network structure should be validated as well, an issue that strongly depends on the brain area, the anatomical feature considered and the available data.
For instance, we have functional layers in Hippocampus CA1, but not columns as in Neocortex. The probability distribution of anatomical features such as connections or neurite lengths across those units can impact the signals produced by the network dynamics.

[rgerkin]

Our focus in NeuronUnit has previously been on data-driven validation of physiology for single neuron models, but we are planning to support network models in 2019 after our upcoming release. This is an area where we could use a lot of feedback about the kinds of things that people care about reproducing in their network models (i.e. specific measures that have correlates in experimental analyses), as well as any public sources of data (if any). We would like to adopt tools and workflows for computing these measures to the extent that they exist now (as Python packages or as code that could be realized as such). @scrook @JustasB

[apdavison]

@rgerkin Does it make sense to put all neuroscience-related validation code into NeuronUnit, as you seem to be suggesting, or would it be better to maintain separate packages for different sub-domains?

I'm not sure what the answer is, but we should discuss it now rather than risk duplication of effort.

More concretely, does it make sense to merge NeuronUnit and NetworkUnit, or would it be better to develop them as separate packages?

[apdavison]

also see Gutzen et al. (2018)

[rgerkin]

@apdavison I am familiar with Gutzen et al, having reviewed it! I think it may be better to keep network stuff in NetworkUnit, although I think we will still have some SciUnit- and general neuroscience toolchain integration to offer there, as well as an expanded set of network measures.