# 15.3 Rate models

The gain function $F(h)$ of rate models can always be chosen such that, for constant input $h_{0}=R\,I_{0}$, the population activity $A_{0}=F(h_{0})$ in the stationary state of asynchronous firing is correctly described. The dynamic equations that describe the approach to the stationary state in a rate model are, however, to a certain degree ad hoc. This means that the analysis of transients as well as the stability analysis in recurrent networks will, in general, give different results in rate models than in spiking neuron models.

# 15.3.1 Rate models have slow transients

In Section 15.2 we have seen that spiking neuron models with a large amount of escape noise exhibit a population activity $A(t)$ that follows the input potential $h(t)$. In this case, it is therefore reasonable to define a rate model $A(t)=F(h(t))$ in which the momentary activity $A(t)$ reflects the momentary input potential $h(t)$. An example is Eq. (15.1), which corresponds to a ’quasi-stationary’ treatment of the population activity, because the transform $F$ is identical to the stationary gain function, except for a change in the units of the argument, as discussed in the text after Eq. (15.1). A similar argument can also be made for exponential integrate-and-fire neurons with diffusive noise, as we will see later in this section.

If we insert Eq. (15.2) into Eq. (15.1) we find

 $A(t)=F[h(t)]=F\left[{R\over\tau_{m}}\int_{0}^{\infty}\exp{(-{s\over\tau_{m}})}% I(t-s){\text{d}}s\right]\,.$ (15.13)

Eq. (15.13) makes explicit that the population activity in the rate model reflects a low-pass filtered version of the input current. The transient response to a step in the input current is therefore slow. Fig. 15.9: Slow (colored) diffusive noise versus white diffusive noise. A population of integrate-and-fire models with a time constant of τm=20\tau_{m}=20ms was simulated and responses to a step stimulus reported in time bins of 11ms. A. Colored noise with a filtering time constant τs\tau_{s}=10ms leads to an abrupt, instantaneous response. B. White noise leads to a smoothly increasing fairly slow response. Figures adapted from (77).

Since differential equations are more convenient than integrals, we rewrite the input potential in the form of Eq. (15.3) which we repeat here for convenience

 $\tau_{m}\,{{\text{d}}h(t)\over{\text{d}}t}=-h+R\,I(t)\,.$ (15.14)

The input potential $h(t)$ resulting from the integration of Eq. (15.14) is to be inserted into the function $F$ to arrive at the population activity $A(t)=F[h(t)]$. Note that the input current $I(t)$ in Eq. (15.14) can arise from external sources, from other populations or from recurrent activity in the network itself.

Example: Wilson-Cowan differential equation

Sometimes one finds in the literature rate models of the form

 $\tau_{A}{{\text{d}}\bar{A}(t)\over{\text{d}}t}=-\bar{A}(t)+F(h(t))\,,$ (15.15)

which have an additional time constant $\tau_{A}$. Therefore the transient response to a step in the input would be even slower than that of the rate model in Eq. (15.13).

In the derivation of Wilson and Cowan (553) the time constant $\tau_{A}$ arises from time-averaging over the ’raw’ activity variable with a sliding window of duration $\tau_{A}$. Thus, even if the ’raw’ activity has sharp transients, the time-averaged variable $\bar{A}(t)$ in Eq. (15.15) is smooth. In the theory of Wilson and Cowan, the differential equation for the population activity takes the form (552; 553)

 $\tau_{A}{{\text{d}}\bar{A}(t)\over{\text{d}}t}=-\bar{A}(t)+(1-\Delta^{\rm abs}% )\,F(h(t))$ (15.16)

so as to account for absolute refractoriness of duration $\Delta^{\rm abs}$; cf. the integral equation (14.10) in Chapter 14. $F$ has a sigmoidal shape. The input potential $h(t)$ comprises input from the same as well as from other populations.

Since there is no reason to introduce the additional low-pass filter with time constant $\tau_{A}$, we advise against the use of the model defined in Eqs. (15.15) or (15.16). However, we may set

 $\tau_{m}{{\text{d}}{A}(t)\over{\text{d}}t}=-{A}(t)+g_{\sigma}(I(t))\,,$ (15.17)

where $I(t)$ is the input current (as opposed to the input potential). The current can be attributed to input from other populations or from recurrent coupling within the population. The low-pass filter in Eq. (15.17) replaces the low-pass filter in Eq. (15.14) so that the two rate models are equivalent after an appropriate rescaling of the input, even for complex networks (345); see also Exercises.

# 15.3.2 Networks of rate models

Let us consider a network consisting of $K$ populations. Each population contains a homogeneous population of neurons. The input into population $k$ arising from other populations $n$ and from recurrent coupling within the population is described as

 $I_{k}(t)=\sum_{n}C_{kn}\,w_{kn}\int_{0}^{\infty}\alpha(s)A_{n}(t-s)\,{\text{d}% }s\,.$ (15.18)

Here $A_{n}(t)$ is the activity of population $n$ and $C_{kn}$ is the number of presynaptic neurons in population $n$ that are connected to a typical neuron in population $k$; the time course and strength of synaptic connections are described by $\alpha$ and $w_{kn}$, respectively; see Chapter 12.

We describe the dynamics of the input potential $h_{k}$ of population $k$ with the differential equation (15.14) and use for each population the quasi-stationary rate model $A_{n}(t)=F_{n}(h_{n})$ where $F_{n}$ is the gain function of the neurons in population $n$. The final result is

 $\tau_{m}\,{{\text{d}}h_{k}(t)\over{\text{d}}t}=-h_{k}+R\,\sum_{n}C_{kn}\,w_{kn% }\int_{0}^{\infty}\alpha(s)F_{n}(h_{n}(t-s))\,{\text{d}}s\,.$ (15.19)

Eq. (15.19) is the starting point for some of the models in Part IV of this book.

Example: Population with self-coupling

If we have a single population, we can drop the indices in Eq. (15.19) so as to arrive at

 $\tau_{m}\,{{\text{d}}h(t)\over{\text{d}}t}=-h+J_{0}\int_{0}^{\infty}\alpha(s)F% (h(t-s))\,{\text{d}}s\,$ (15.20)

where $J$ is the strength of the feedback. Thus, a single population with self-coupling is described by a single differential equation for the input potential $h$. The population activity is simply $A(t)=F(h(t))$.

Stationary states are found as discussed in Chapter 12. If there are three fixed points, the middle one is unstable while the other two (at low and high activity, respectively) are stable under the dynamics (15.20). The fixed points calculated from Eq. (15.20) are correct. However, stability under the dynamics (15.20) does not guarantee stability of the original network of spiking neurons. Indeed, as we have seen in Chapters 13 and 14, the fixed points of high and low activity may loose stability with respect to oscillations, even in a single homogeneous population. The rate dynamics of Eq. (15.20) cannot correctly account for these oscillations. In fact, for instantaneous synaptic current pulses $\alpha(s)=q\delta(s)$, where $\delta$ denotes the Dirac $\delta$-function, Eq. (15.20) reduces to a one-dimensional differential equation which can never give rise to oscillatory solutions.

# 15.3.3 Linear-Nonlinear-Poisson and improved transients

To improve the description of transients, we start from Eq. (15.13), but insert an arbitrary filter $\kappa$,

 $A(t)=F(h(t))=F\left[\int_{0}^{\infty}\kappa(s)I(t-s){\text{d}}s\right]\,.$ (15.21)

Eq. (15.21) is called the Linear-Nonlinear-Poisson (LNP) model (94; 478). It is also called a cascade model because it can be interpreted as a sequence of three processing steps. First, input is filtered with an arbitrary linear filter $\kappa$, which yields the input potential $h$. Second, the result is passed through a nonlinearity $F$. Third, in case of a single neurons, spikes are generated by an inhomogeneous Poisson process with rate $F(h(t))$. Since, in our model of a homogeneous population, we have many similar neurons, we drop the third step and interpret the rate $F(h(t))$ directly as the population activity.

For $\kappa(s)=(R/\tau_{m})\exp(-{s/\tau_{m}})$ we are back at Eq. (15.13). The question arises whether we can make a better choice of the filter $\kappa$ than a simple low-pass with the membrane time constant $\tau_{m}$. In Chapter 11 it is shown how an optimal filter $\kappa$ can be determined experimentally by reverse correlation techniques.

Here we are interested in deriving the optimal filter from the complete population dynamics. The LNP model in Eq. (15.21) is an approximation of the population dynamics that is more correctly described by the Fokker-Planck equations in Ch. 13 or by the integral equation of time-dependent renewal theory in Ch. 14. Let us recall that in both approaches, we can linearize the population equations around a stationary state of asynchronous firing $A_{0}$ which is obtained with a mean input $I_{0}$ at some noise level $\sigma$. The linearization of the population dynamics about $A_{0}$ yields a filter $G_{I}(s)$. We use this filter, and arrive at a variant of Eq. (15.21)

 $A(t)=\tilde{F}\left(\int_{0}^{\infty}G_{I}(s)I(t-s){\text{d}}s\right)\,,$ (15.22)

where $\tilde{F}(x)=g_{\sigma}(x/c)$ is a scaled version of the frequency-current curve $g_{\sigma}(I)$ and $c=\int_{0}^{\infty}G_{I}(s){\text{d}}s$ a constant which matches the slope of the gain function at the reference point for the linearization (373). Models with this, or similar choices of $\tilde{F}$, describe transient peaks in the population activity surprisingly well; see e.g., (214; 33; 373).

Rate models based on Eq. (15.22) can also be used to describe coupled populations. Stability of a stationary state $A_{0}$ is correctly described by Eq. (15.22), if the filter $G_{I}(s)$ in the argument on the right-hand-side reflects the linearization of the full population dynamics around $A_{0}$, but not if the filter is derived by linearization around some other value of the activity.

Example: Effective rate model for Exponential Integrate-and-Fire neurons

We denote the stationary gain function of exponential integrate-and-fire neurons by $g_{\sigma}(I)$ and the linear filter arising from linearization around a stationary activity $A_{0}$ by $G_{I}(s)$. For exponential integrate-and-fire neurons $G_{I}(s)$ has the high-frequency behavior of a low-pass filter $\propto A_{0}{R\over\Delta_{T}\tau_{m}}{1\over{\omega}}$; see Tab. 15.2.

We recall that the Fourier-transform of an exponential filter also yields a high-frequency behavior $\propto 1/\omega$ with the inverse filter time constant as cut-off frequency. This suggests that, for the exponential integrate-and-fire model, we can approximate the time course of $G_{I}(s)$ as an exponential filter with an effective time constant $\tau_{\rm eff}\propto\tau_{m}/A_{0}$. This leads back to Eq. (15.22), but with an exponential filter $G(s)$. Hence, we are now nearly back to Eq. (15.13), except that the time constant of the exponential is different.

We now switch from the frequency current curve $g_{\sigma}(I)$ to the equivalent description of the gain function $F(h)=g_{\sigma}(h/R)$ where the argument of $F$ has units of a potential. It is convenient to implement the exponential filter in the form of a differential equation for the the input potential (373)

 $\tau_{\rm eff}(t){{\text{d}}h\over{\text{d}}t}=-h+R\,I(t)\,,$ (15.23)

with an effective time constant

 $\tau_{\rm eff}(t)=\tau_{m}\,{\Delta_{T}\,F^{\prime}\over A_{0}(t)}\,,$ (15.24)

where $A_{0}(t)$ is the activity averaged over one or a few previous time steps and $F^{\prime}={\text{d}}F/{\text{d}}h$ is the derivative of the gain function at an appropriately chosen reference point $h_{0}$, so that $F(h_{0})\approx\langle A_{0}(t)\rangle$ is the long-term average of the activity.

In each time step, we update the input using Eq. (15.23) and calculate the activity as $A(t)=F(h(t))$ with $F(h)=g_{\sigma}(h/R)$. Such a rate model gives an excellent approximation of the activity in a population of exponential integrate-and-fire neurons (Fig. 15.10). Note that fast transients are well described, because the effective time constant is shortened as soon as the population activity increases. Fig. 15.10: Population of exponential integrate-and-fire neurons. The population activity A⁢(t)A(t) arising from a explicit simulation of NN model neurons (black solid line) can be approximated by a rate model with effective time constant τeff⁢(t)\tau_{\rm eff}(t) (gray). The stimulation is shown in the bottom panel. Figure adapted from (373).

So far we have focused on the initial transient after a step in the input current. After the initial transient, however, follows a second, much slower phase of adaptation during which the population response decreases, even if the stimulation is kept constant. For single neurons, adaptation has been discussed in Chapter 6.

In a population of neurons, adaptation can be described as an effective decrease in the input potential. If a population of non-adaptive neurons has an activity described by the gain function $A(t)=F(h(t))$, then the population rate model for adaptive neurons is

 $A(t)=F(h(t)-a(t))$ (15.25)

where $a(t)$ describes the amount of adaptation that neurons have accumulated

 $\tau_{a}(A){{\text{d}}a\over{\text{d}}t}=a_{\infty}(A)-a$ (15.26)

where $a_{\infty}(A)$ is the asymptotic level of adaptation that is attained if the population continuously fires at a constant rate $A$. The asymptotic level is approached with a time constant $\tau_{a}$. Eqs. (15.25) and (15.26) are a simplified version of the phenomenological model proposed in Benda and Herz (48).

Suppose that $a_{\infty}(A)=c\,A$ is linear in the population activity with a constant $c>0$ and $\tau_{a}(A)=\tau_{a}$ is independent of $A$. Then Eq. (15.26) can be integrated and yields $a(t)=\int_{0}^{\infty}\gamma(s)\,A(t-s){\text{d}}s$ with a filter $\gamma(s)=(c/\tau_{a})\,\exp(-s/\tau_{a})$. We insert the result in Eq. (15.25) and find
 $A(t)=F\left(h(t)-\int_{0}^{\infty}\gamma(s)\,A(t-s){\text{d}}s\right)\,.$ (15.27)