Persistence and Turing instability in a cross-diffusive predator–prey system with generalist predator

Abstract

In this paper, we propose and investigate persistence and Turing instability of a cross-diffusion predator–prey system with generalist predator. First, by virtue of the comparison principle, we obtain sufficient conditions of persistence for a corresponding reaction–diffusion system without self- and cross-diffusion. Second, by using the linear stability analysis, we prove that under some conditions the unique positive equilibrium solution is locally asymptotically stable for the corresponding ODE system and the corresponding reaction–diffusion system without cross-diffusion and self-diffusion. Hence it does not belong to the classical Turing instability. Third, under some appropriate sufficient conditions, we obtain that the uniform positive equilibrium solution is linearly unstable for the cross-reaction–diffusion and partial self-diffusion system. The results indicate that cross-diffusion and partial self-diffusion play an important role in the study of Turing instability about reaction–diffusion systems with generalist predator. Fourth, we elaborate on the relations between the theoretical results and the cross-diffusion predator–prey system by relying on some examples. In the end, we conclude our findings and give a brief discussion.

1 Introduction

It is well known that ecosystems are characterized by the interaction of species with a wide range of spatial and temporal scales natural environment. From the early beginning of this discipline, one of the most important interactions that affect the dynamics of all species is predation. Therefore, predator–prey models have become the key to population system investigation. Predator–prey dynamic has been one of the main research topics in mathematical ecology since the pioneering work of Lotka [1] and Volterra [2]. The main purpose of mathematical ecology is to investigate the relationship between different species and their living environment. As one of the most important interactions, the interaction between the predator and the prey has aroused great concern and predator–prey models with self-diffusion and cross-diffusion have been extensively investigated by many specialist and scholars [3–10]. More and more evidence indicates that in many cases where predators must search, share, and compete for food, ratio-dependent predator–prey models are considered reasonable. To put it briefly, the reaction functions have important effect on the dynamical behavior of predator–prey modes, the functional response is classified into four types: Lotka–Volterra type, Holling type II (Michaelis–Menten type), generalized Holling type III (Sigmoidal), and IV functional response. In [3], Zhang et al. performed Hopf and steady state bifurcation analysis in a ratio-dependent predator–prey model and derived explicit conditions for the existence of non-constant steady states that emerged through steady state bifurcation from related constant steady states. Li [4] studied stability and bifurcation of a ratio-dependent predator–prey model with cross-diffusion by virtue of Fourier decomposition, fixed index theory, bifurcation theory, energy estimates, and the differential method of implicit function and inverse function, and obtained the asymptotic stabilities of nonnegative constant solutions, the existence and nonexistence of non-constant positive steady states, the bifurcation and multiplicity of positive solutions. For more works on predator–prey systems with ratio-dependent functional response, we refer to [11–13].

The formation of spatial patterns is one of the important fields in biological research. The pioneering work of pattern formation was conducted and investigated by Turing [14], who derived the appropriate conditions under which the spatially homogeneous steady state is stable in the absence of diffusion but becomes unstable in the presence of diffusion, yielding a spatially heterogeneous steady state. This process is well known as diffusion-driven instability, and the associated spatial pattern is usually named the Turing pattern. Since the seminal work of Turing in 1952, the most widely investigated model for spatial pattern formation is the reaction–diffusion model [15–22]. It is worth noticing that Yan and Zhang investigated a diffusion two-species predator–prey system with the Beddington–DeAngelis functional response and subject to homogeneous Neumann boundary conditions by means of the upper and lower solutions method and the monotone iteration principle in [11]. Sun et al. [23] studied pattern formation in a predator–prey diffusion model with stage structure for the predator by using the bootstrap technique and higher-order energy estimates. Sebestyén et al. [18] analyzed stability of patterns and of constant steady states for a cross-diffusion system with two dependent variables by virtue of numerical simulations and numerical models. Lacitignola et al. [24] studied Turing pattern formation on the sphere for a morphochemical reaction–diffusion model for electrodeposition by means of the lumped surface finite element method and the systematic numerical simulations and numerical method. In [25], Song et al. investigated pattern dynamics in a Gierer–Meinhardt model with a saturating term by the linear stability analysis, the multiple scales methods, and the numerical simulations. Peng and Zhang [26] investigated Turing instability and pattern induced by cross-diffusion in a predator–prey system with Allee effect by the multiple time scales and the weakly nonlinear analysis. For more research on Turing instability in a cross-diffusive system, please refer to [27–30] and relevant references therein.

It is worth pointing out that the above results show that it has been known that stable equilibrium solutions of ordinary differential equations can be destabilized by the introduction of diffusion for different species. Mathematically speaking, Turing pattern arising from the reaction–diffusion ecological system admits at least one non-constant steady state. Therefore, we are particularly interested in a class of reaction–diffusion predator–prey models which account for predator–prey–mutualist and generalist predator [5, 31–33].

In this paper, motivated by the recent papers [9, 31–34], we will investigate a class of three-species predator–prey cross-diffusion systems with generalist predator. The aim of this paper is to construct a mathematical model to describe ecosystems with mutualist interaction and generalist predator, and further investigate Turing instability of the predator–prey ecosystem by virtue of mathematical analysis and numerical examples. The rest of the paper is organized as follows. The reaction–diffusion predator–prey model with generalist is introduced in Sect. 2. In Sect. 3, we demonstrate that the unique positive equilibrium is persistence for predator–prey reaction–diffusion system with no self- and cross-diffusion by virtue of the comparison principle. In Sect. 4, we give linear stability analysis of uniform positive state for ODES by means of the Routh–Hurwitz criterion. We prove that a uniform equilibrium solution is also linearly asymptotically stable under the same conditions for the predator–prey system with generalist predator without self- and cross-diffusion in Sect. 5. Section 6 is devoted to deriving the conditions on the parameter values such that the positive equilibrium solution is Turing unstable for a strongly coupled reaction–diffusion system. In Sect. 7, by some numerical examples we illustrate correctness of our results. Finally, we give some conclusions and biological discussions in Sect. 8.

2 The mathematical model

In this section, in order to establish the mathematical model, we will base on the following three assumptions on biological background: The prey growth is logistic and independent of the mutualist; The mutualist is independent of the predator; Predation in the absence of mutualism is Lotka–Volterra predation. To take into account the intra-specific and inter-specific population pressures between the generalist predator and the competitor–mutualist, we introduce self-diffusion and cross-diffusions. To account for this scenario, we obtain the following strongly coupled predator–prey reaction–diffusion system with generalist predator:

$$\begin{aligned} \textstyle\begin{cases} \frac{\partial u_{1}}{\partial t}-\Delta [(d_{1}+k_{11}u_{1}+k_{12}u _{2}+\frac{k_{13}}{\beta + u_{3}})u_{1}]=u_{1}(a-u_{1}-\frac{\alpha u _{2}}{1+\gamma u_{3}}), &x\in \Omega,t>0, \\ \frac{\partial u_{2}}{\partial t}-\Delta [(d_{2}+k_{21}u_{1}+k_{22}u _{2}+k_{23}u_{3})u_{2}]=u_{2}(b-u_{2}+\mu u_{1}+\eta u_{3}), &x \in \Omega,t>0, \\ \frac{\partial u_{3}}{\partial t}-\Delta [(d_{3}+\frac{k_{31}}{m+u _{1}}u_{2}+k_{33}u_{3})u_{3}]=u_{3}(c-\frac{u_{3}}{1+\delta u_{1}}), &x\in \Omega,t>0, \\ \partial_{n} u_{1}=\partial_{n} u_{2}=\partial_{n} u_{3}=0, &x \in \partial \Omega, t>0, \\ u_{i}(x,0)=u_{i0}(x)\ge (\not \equiv )0, &x\in \Omega, i=1,2,3,\hspace{-2pt} \end{cases}\displaystyle \end{aligned}$$

(1)

where Ω is a bounded and connected domain in $\mathbf{R}^{n}$ with smooth boundary ∂Ω. Here $u_{2}$, $u_{1}$, and $u_{3}$ represent the population densities of the predator, the prey, and the mutualist, respectively. Ω represents the domain in which three species inhabit. $\partial_{n}$ is the directional derivative normal to ∂Ω. Δ denotes the usual Laplacian operator in space Ω. $k_{ii}$ ($i=1,2,3$), $k_{12}, k_{13}, k_{21}, k_{23}$, and $k_{31}$ are nonnegative constants. $k_{ii}$ is the self-diffusion rate of ith species and $k_{ij}$ ($i\neq j$) is the cross-diffusion rate of ith species due to the pressure of the presence of jth species. a, b, c, α, γ, μ, η, δ, m, and β are all positive constants. In diffusion terms, the positive constant $d_{i}$ ($i=1,2,3$), which is usually termed self-diffusion coefficient, represents the natural dispersive force of movement of an individual. a, b, c are intrinsic growth rates of the three species, respectively, while α, μ, η, δ describe inter-species interactions. The homogeneous Neumann boundary condition indicates that the model is self-contained and there is zero population flux across the boundary. This system with cross-diffusion represents a model which involves interacting and migration in the same habitat Ω among generalist predator $u_{2}$, prey $u_{1}$, and $u_{3}$, while $u_{3}$ and $u_{1}$ are of symbiotic mutualist. For more biological meaning of the parameters, we refer to [17, 31–33].

In this model, predator $u_{2}$ diffuses with flux

$$\begin{aligned} \mathbf{J}&=-\nabla \bigl(d_{2} u_{2}+k_{21}u_{1}u_{2}+k_{22}u_{2}^{2}+k _{23}u_{3}u_{2} \bigr) \\ &=-k_{21}u_{2} \nabla u_{1}-(d_{2}+k_{21}u_{1}+k_{23}u _{3})\nabla u_{2}-k_{23}\nabla u_{3}. \end{aligned}$$

As $-k_{21}u_{2} <0$, the part $-k_{21}u_{2} \nabla u_{1}$ of the flux is directed toward the decreasing population density of prey $u_{1}$. When the predator is chasing the prey, the flux should be directed toward the increasing population density of the prey as in [35]. However, in a natural ecosystem, a great number of prey species congregate and form a huge group to protect themselves from the attack of predator.

The first prey $u_{1}$ diffuses with flux

$$\begin{aligned} \mathbf{J}_{1}&=-\nabla \biggl(d_{1}u_{1}+k_{11}u_{1}^{2}+k_{12}u_{2} u_{1}+\frac{k _{13}}{\beta + u_{3}}u_{1} \biggr) \\ &=- \biggl(d_{1}+2k_{11}u_{1}+k_{12}u_{2} +\frac{k _{13}}{\beta + u_{3}} \biggr)\nabla u_{1}-k_{12}u_{1} \nabla u_{2}+\frac{k _{13}u_{1}}{(\beta + u_{3})^{2}}\nabla u_{3}. \end{aligned}$$

The second prey $u_{3}$ diffuses with flux

$$\begin{aligned} \mathbf{J}_{2}&=-\nabla \biggl(d_{3}u_{3}+ \frac{k_{31}}{m+u_{1}}u_{2} u_{3}+k _{33}{u_{3}}^{2} \biggr) \\ &=\frac{k_{31}u_{2} u_{3}}{(m+u_{1})^{2}}\nabla u_{1}-\frac{k _{31}u_{3}}{m+u_{1}}\nabla u_{2}- \biggl(d_{3}+ \frac{k_{31}u_{2}}{m+u_{1}}+2k _{33}u_{3} \biggr)\nabla u_{3}. \end{aligned}$$

We observe that, as $-k_{12}u_{1}<0$ and $-\frac{k_{31}u_{3}}{m+u_{1}}<0$, the parts $-k_{12}u_{1}\nabla u _{2}$ and$-\frac{k_{31}u_{3}}{m+u_{1}}\nabla u_{2}$ of the flux are directed toward the decreasing population density of predator $u_{2}$. While $\frac{k_{13}u_{1}}{(\beta + u_{3})^{2}}>0$ (or $\frac{k_{31}u_{2} u_{3}}{(m+u_{1})^{2}}>0$) implies that the flux of $u_{1}$(or $u_{3}$) is directed toward the increasing population density of $u_{3}$(or $u_{1}$), that is, the two mutualist chase each other.

After adding these items, the cross-diffusion predator–prey system (1) means that in addition to the dispersive force, the diffusion also depends on species pressure from other species. Hence, the species in (1) are not homogeneously distributed based on the reason of self-diffusion and cross-diffusions.

In the above system (1), when $a=1, b=2, c=1, \eta =0$, and $k_{11}=k_{12}=k_{13}=k_{22}=k_{23}=k_{31}=k_{33}=0$, $d_{1}=d_{3}=0$, Tian et al. [31] investigated Turing pattern formation in a predator–prey–mutualist system by using the stability analysis and the Leray–Schauder degree theory. However, Tian et al. only studied the effects of diffusion and cross-diffusion between predator $u_{2}$ and prey $u_{1}$. Wen and Fu [33] investigated Turing instability for a competitor–competitor–mutualist model with nonlinear cross-diffusion effects by using a comparison method and a linearization technique when $k_{23}=0, \eta =0, \mu =-\mu$, but they did not research the effects of cross-diffusion between predator $u_{2}$ and prey $u_{3}$. Therefore, in this work we will study Turing instability of the more complex cross-diffusion predator–prey with generalist predator (1).

3 Persistence for the PDES without self-diffusion and cross-diffusion

By solving the equations $a-u_{1}- \frac{\alpha u_{2}}{1+\gamma u_{3}}=0$, $b-u_{2}+\mu u_{1}+\eta u _{3}=0$, $c-\frac{u_{3}}{1+\delta u_{1}}=0$, it is easy to know that model (1) has a unique constant positive equilibrium $\mathbf{u}^{*}=(u_{1}^{*},u_{2}^{*},u_{3}^{*})^{T}=(u_{1}^{*}, b+c \eta +(\mu +c\eta \delta )u_{1}^{*}, c(1+\delta u_{1}^{*}))^{T}$ provided that $a(1+\gamma )>(b+c\eta )\alpha$, where $u_{1}^{*}=\frac{-[c\eta \delta \alpha +\mu \alpha +\alpha +1-c a\delta \gamma ]+\sqrt{(c\eta \delta \alpha +\mu \alpha +\alpha +1-c a\delta \gamma )^{2}-4 c \delta \gamma [(b+c\eta )\alpha -a(1+\gamma )]}}{2c \delta \gamma }$.

From the viewpoint of ecology, the constant positive steady-state solution implies the coexistence of both the predator and the prey.

In this section, we always assume that $k_{ij}=0$. We will show that system (1) without self-diffusion and cross-diffusion is persistent.

Theorem 1

Assume that the inequalities

$$\begin{aligned}& a(1+\gamma )>(b+c\eta )\alpha\quad \textit{and}\quad c\delta > \gamma +( \mu +\eta \delta )\alpha \end{aligned}$$

(2)

hold, then system (1) is persistent.

Proof

By using the maximum principle of the parabolic type equation, we easily know that all the solutions of (1) are nonnegative since the initial value is nonnegative. By virtue of the first equation in (1), we can obtain

$$ \frac{\partial u_{1}}{\partial t}-d_{1} \Delta u=u_{1} \biggl(a-u_{1}- \frac{ \alpha u_{2}}{1+\gamma u_{3}} \biggr)\leq a u_{1} \biggl(1-\frac{u_{1}}{a} \biggr). $$

By use of the comparison principle for parabolic equation, we get $u_{1}\leq w$, where w is the solution of

$$ w_{t}-d_{1}\Delta w=aw \biggl(1-\frac{w}{a} \biggr), \qquad \partial_{n} w=0,\qquad w(x,0)=u _{10}(x)\geq (\not \equiv )0. $$

Since $\lim_{t\rightarrow \infty }{w(x,t)}=a$, we have $\lim_{t\rightarrow \infty }{u_{1}(x,t)}\leq a$, then for arbitrary $\varepsilon >0$, there exists $t_{1}>0$ such that when $t>t_{1}$, $u_{1}(x,t)\leq \bar{c}_{1}$, where $\bar{c}_{1}=a+\varepsilon$. This implies that

$$\begin{aligned} \limsup_{t\to \infty }\max_{\bar{\Omega }}u_{1}(x,t) \leq a. \end{aligned}$$

(3)

Similarly,

$$\begin{aligned} \frac{\partial u_{3}}{\partial t}-d_{3}\Delta u_{3}&=u_{3} \biggl(c-\frac{u _{3}}{1+\delta u_{1}} \biggr) \\ &\leq c u_{3} \biggl(1- \frac{u_{3}}{c[1+\delta (a+ \varepsilon )]} \biggr) \end{aligned}$$

for $t>t_{1}$. Thus, for the above ε, there exists $t_{3}>t_{1}$ such that when $t>t_{3}$, $u_{3}(x,t)\leq \bar{c}_{3}$, where $\bar{c}_{3}=c[1+\delta (a+\varepsilon )]+\varepsilon$. Therefore,

$$\begin{aligned} \limsup_{t\to \infty }\max_{\bar{\Omega }}u_{3}(x,t) \leq c(1+\delta a). \end{aligned}$$

(4)

Consequently, from the second equation in (1) we furthermore have

$$ \frac{\partial u_{2}}{\partial t}-d_{2}\Delta u_{2}=u_{2}(b-u_{2}+ \mu u_{1}+\eta u_{3})\leq u_{2}(b+a\mu +c\eta +a \eta \delta +\mu \varepsilon +\eta \delta \varepsilon +\eta \varepsilon -u_{2}). $$

So, for $\varepsilon >0$, there exists $t_{2}>0$ such that when $t>t_{2}$, $u_{2}(x,t)\leq \bar{c}_{2}$, where $\bar{c}_{2}=b+a\mu +c \eta +a\eta \delta +\mu \varepsilon +\eta \delta \varepsilon +\eta \varepsilon +\varepsilon$. Thus, we can obtain

$$\begin{aligned} \limsup_{t\to \infty }\max_{\bar{\Omega }}u_{2}(x,t) \leq b+a\mu +c \eta +a\eta \delta . \end{aligned}$$

(5)

On the other hand, according to the third equation in (1), we have

$$ \frac{\partial u_{3}}{\partial t}-d_{3}\Delta u_{3}=u_{3} \biggl(c-\frac{u _{3}}{1+\delta u_{1}} \biggr)\geq u_{3}(c-u_{3}). $$

Thus, for the above $\varepsilon >0$, there exists $t_{4}>0$ such that when $t>t_{4}$, $u_{3}(x,t)\geq \underline{c}_{3}$, where $\underline{c}_{3}=c-\varepsilon$. Further, we can get

$$\begin{aligned} \liminf_{t\to \infty }\min_{\bar{\Omega }}u_{3}(x,t) \geq c. \end{aligned}$$

(6)

Similar to the above argument, we can deduce that

$$\begin{aligned} \frac{\partial u_{2}}{\partial t}-d_{2}\Delta u_{2}&=u_{2}(b-u_{2}+ \mu u_{1}+\eta u_{3}) \\ &\geq u_{2}(b+ \underline{c}_{3}\eta -u_{2}),\quad t>t _{4}, \end{aligned}$$

and

$$\begin{aligned} \frac{\partial u_{1}}{\partial t}-d_{1} \Delta u&=u_{1} \biggl(a-u_{1}- \frac{ \alpha u_{2}}{1+\gamma u_{3}} \biggr) \\ &\geq u_{1} \biggl(a- \frac{\alpha \bar{c}_{2}}{1+ \delta \underline{c}_{3}} \biggr),\quad t>t_{2}, t_{4}. \end{aligned}$$

This implies that for $\varepsilon >0$, there exist $t_{5}>max\{t_{1},t _{4}\}$ and $t_{6}>max\{t_{2},t_{4}\}$ such that when $t>t_{5}$, $u_{2}\geq \underline{c}_{2}$ and when $t\geq t_{6}$, $u_{1}\geq \underline{c}_{1}$, where $\underline{c}_{2}=b+\eta (c-\varepsilon )- \varepsilon$, $\underline{c}_{1}=\frac{a+a\delta (c-\varepsilon )- \alpha (b+a\mu +c\eta +a\eta \delta +\mu \varepsilon +\eta \delta \varepsilon +\eta \varepsilon +\varepsilon )}{1+\delta (c-\varepsilon )}-\varepsilon$. Therefore, we can obtain

$$\begin{aligned}& \begin{aligned} &\liminf_{t\to \infty }\min_{\bar{\Omega }}u_{2}(x,t) \geq b+c\eta >0, \\ &\liminf_{t\to \infty }\min_{\bar{\Omega }}u_{1}(x,t) \geq \frac{a(1+c \delta )-\alpha (b+a\mu +c\eta +a\eta \delta )}{1+c\delta }>0. \end{aligned} \end{aligned}$$

(7)

It is worth noticing that since $c\delta >\gamma +(\mu +\eta \delta ) \alpha$, we have $1+c\delta >1+\gamma +(\mu +\eta \delta )\alpha$. Further, we can have $a(1+c\delta )>a(1+\gamma )+a(\mu +\eta \delta ) \alpha$. By virtue of the condition $a(1+\gamma )>(b+\eta c)\alpha$, we can obtain $a(1+c\delta )>(b+c\eta )\alpha +a(\mu +\eta \delta ) \alpha$, therefore the above inequality holds.

It is well known that a system of several species is said to be persistent if all species are persistent, while a species with density $u(x,t)~(x\in \Omega, t>0)$ is said to be persistent if $u(x,t)>0$ for all $x\in \Omega, t>0$ and

$$\begin{aligned} \liminf_{t\to \infty }\min_{\bar{\Omega }}u(x,t)>0. \end{aligned}$$

(8)

According to the definition of persistence for species and the above results of proof, we can conclude that system (1) is persistent. The proof is completed. □

4 Stability of the positive equilibrium solution of the ODE system

In this section, we mainly research the stability for the ODE system by virtue of the stability theory. It is easy to know that the ODE system of (1) is as follows:

$$\begin{aligned} \textstyle\begin{cases} \frac{d u_{1}}{ dt}=u_{1}(a-u_{1}-\frac{\alpha u_{2}}{1+\gamma u_{3}}), \\ \frac{d u_{2}}{d t} = u_{2}(b-u_{2}+\mu u_{1}+\eta u_{3}), \\ \frac{d u_{3}}{d t}=u_{3}(c-\frac{u_{3}}{1+\delta u_{1}}), \\ u_{i}(0)=u_{i0}\quad \mbox{for }i=1,2,3. \end{cases}\displaystyle \end{aligned}$$

(9)

Theorem 2

Suppose that the conditions of the parameters satisfy

$$\begin{aligned} a(1+\gamma )>(b+c\eta )\alpha,\qquad \mu \geq (b+c\eta )\delta, \qquad \alpha \delta \eta \leq 3\gamma,\quad \alpha \leq 2. \end{aligned}$$

(10)

Then the stationary uniform solution of system (9) is locally asymptotically stable.

Proof

For simplicity, throughout this paper, we denote

$$ \mathbf{F(u)}= \begin{pmatrix} f_{1} (u_{1},u_{2},u_{3}) \\ f_{2} (u_{1},u_{2},u_{3}) \\ f_{3} (u_{1},u_{2},u_{3}) \end{pmatrix} = \begin{pmatrix} u_{1}(a-u_{1}-\frac{\alpha u_{2}}{1+\gamma u_{3}}) \\ u_{2}(b-u_{2}+\mu u_{1}+\eta u_{3}) \\ u_{3}(c-\frac{u_{3}}{1+\delta u_{1}}) \end{pmatrix} . $$

Noticing $\mathbf{F(u^{*})}=0$, a direct calculation yields

$$ \mathbf{F_{u} \bigl(u^{*} \bigr)}= \begin{pmatrix} -u_{1}^{*} & -\frac{\alpha u_{1}^{*}}{1+\gamma u_{3}^{*}} & \frac{ \alpha \gamma u_{1}^{*} u_{2}^{*}}{(1+\gamma u_{3}^{*})^{2}} \\ \mu u_{2}^{*} & -u_{2}^{*} & \eta u_{2}^{*} \\ \frac{\delta {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} & 0 & -\frac{u _{3}^{*}}{1+\delta u_{1}^{*}} \end{pmatrix} . $$

The linearization of problem (9) at positive equilibrium $\mathbf{u}^{*}$ is

$$\begin{aligned} \frac{d \mathbf{u}}{ dt}=\mathbf{F_{u} \bigl(u^{*} \bigr)}\mathbf{u},\quad \mathbf{u}(0)=(u_{10},u_{20},u_{30})^{T}. \end{aligned}$$

(11)

The characteristic polynomial of $\mathbf{F_{u}(u^{*})}$ is given by $\phi (\lambda )=\lambda^{3}+A_{2} \lambda^{2}+A_{1} \lambda +A_{0}$, where

$$\begin{aligned}& A_{0}=\frac{u_{1}^{*} u_{2}^{*}u_{3}^{*}}{1+\delta u_{1}^{*}}+\frac{ \delta \alpha \eta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u _{1}^{*})^{2}(1+\gamma u_{3}^{*})} +\frac{\alpha \mu u_{1}^{*} u_{2} ^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+\gamma u_{3}^{*})}- \frac{\alpha \delta \gamma u_{1}^{*} {u_{2}^{*}}^{2} {u_{3}^{*}}^{2}}{(1+ \delta u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}}, \\& A_{1}=u_{1}^{*} u_{2}^{*} + \bigl(u_{1}^{*}+u_{2}^{*} \bigr) \frac{u_{3}^{*}}{1+ \delta u_{1}^{*}}+\frac{\alpha \mu u_{1}^{*} u_{2}^{*}}{1+\gamma u _{3}^{*}}-\frac{\alpha \gamma \delta u_{1}^{*} u_{2}^{*} {u_{3}^{*}} ^{2}}{(1+\delta u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}}, \\& A_{2}=u_{1}^{*}+u_{2}^{*}+ \frac{u_{3}^{*}}{1+\delta u_{1}^{*}}. \end{aligned}$$

It is evident that if conditions (10) hold, then it is not hard to verify that $A_{2}$, $A_{1}$, $A_{0}$ are positive, and $A_{2} A_{1}-A_{0}>0$. Hence, according to the Routh–Hurwitz criterion for stability, we can conclude that $\phi (\lambda )=0$ only has roots with negative real parts. Therefore $\mathbf{u^{*}}$ is locally asymptotically stable. □

5 Steady state solution of the PDE system without self-diffusion and cross-diffusion

In this section, our main objective is to search for the conditions on the parameter values such that the positive stationary uniform solution is linearly stable in the absence of self-diffusion and cross-diffusion. We now consider system (1) with no self-diffusion and cross-diffusion of the following form:

$$\begin{aligned} \textstyle\begin{cases} \frac{\partial u_{1}}{\partial t}-d_{1}\Delta u_{1}=u_{1}(a-u_{1}-\frac{ \alpha u_{2}}{1+\gamma u_{3}}), &x\in \Omega,t>0, \\ \frac{\partial u_{2}}{\partial t}-d_{2}\Delta u_{2}=u_{2}(b-u_{2}+ \mu u_{1}+\eta u_{3}), &x\in \Omega,t>0, \\ \frac{\partial u_{3}}{\partial t}-d_{3}\Delta u_{3}=u_{3}(c-\frac{u _{3}}{1+\delta u_{1}}), &x\in \Omega,t>0, \\ \partial_{n} u_{1}=\partial_{n} u_{2}=\partial_{n} u_{3}=0, &x \in \partial \Omega, t>0, \\ u_{i}(x,0)=u_{i0}(x)\ge (\not \equiv )0, &x\in \Omega, i=1,2,3. \end{cases}\displaystyle \end{aligned}$$

(12)

In order to research the local asymptotic stability of the parabolic system, we set up the following notation.

Notation

Let $0=\mu_{1}<\mu_{2}<\mu_{3}< \cdots \rightarrow \infty$ be the eigenvalues of −Δ on Ω with the homogeneous Neumann boundary condition and $E(\mu_{i})$. We define the following space decomposition:

• (i) $\mathbf{X_{ij}}:=\{\mathbf{c}\cdot \phi_{ij}: \mathbf{c}\in \mathbf{R}^{3}\}$, where $\{\phi_{ij}\}$ are orthonormal basis of $E(\mu_{i})$ for $j=1,\ldots,\operatorname{dim} E(\mu_{i})$;

• (ii) $\mathbf{X}:=\{\mathbf{u}\in [C^{1}(\overline{\Omega })]^{3} : \partial_{n} u_{1}=\partial_{n} u_{2}=\partial_{n} u_{3}=0\mbox{ on }\partial \Omega \}$ and so $\mathbf{X}=\bigoplus^{\infty }_{i=1}\mathbf{X_{i}}$, where $\mathbf{X_{i}}= \bigoplus^{\operatorname{dim} E(\mu_{i})}_{j=1}\mathbf{X}_{ij}$.

Theorem 3

Assume that the conditions of (10) hold, then the stationary uniform solution $\mathbf{u^{*}}$ of system (12) is locally asymptotically stable.

Proof

The linearization of (12) at $\mathbf{u^{*}}$ can be expressed by

$$ \frac{\partial \mathbf{u}}{dt}= \bigl(D \Delta +\mathbf{F_{u} \bigl(u^{*} \bigr)} \bigr) \mathbf{u}, $$

where

$$ D= \begin{pmatrix} d_{1}& 0 & 0 \\ 0 & d_{2} & 0 \\ 0 & 0 & d_{3} \end{pmatrix} . $$

It is well known that if, for each $i\ge 1$, Xi is invariant under the operator $D\Delta +\mathbf{F_{u}({u^{*}})}$, then problem (12) has a non-trivial solution of the form $\boldsymbol{\psi }=\mathbf{c}\phi \exp (\lambda t)$ if and only if $(\lambda,\mathbf{c})$ is an eigenpair for the matrix $-\mu_{i} D+ \mathbf{F_{u}(u^{*})}$, where c is a constant vector. Therefore the equilibrium $\mathbf{u^{*}}$ is locally asymptotically stable if all eigenvalues λ have negative real parts for all $\mu_{i}$.

The characteristic polynomial of $-\mu_{i} D+\mathbf{F_{u}({u^{*}})}$ is given by

$$ \rho_{i}(\lambda )=\lambda^{3}+B_{2} \lambda^{2}+B_{1}\lambda +B_{0}, $$

where

$$\begin{aligned}& B_{0}= \biggl(\mu_{i} d_{3}+\frac{u_{3}^{*}}{1+\delta u_{1}^{*}} \biggr) \bigl(\mu_{i} d _{1}+u_{1}^{*} \bigr) \bigl(\mu_{i} d_{2}+u_{2}^{*} \bigr) \\& \hphantom{B_{0}=}{}+\frac{d_{3} \mu_{i} \alpha \mu u_{1}^{*} u_{2}^{*}}{1+\gamma u_{3}^{*}}+\frac{\alpha \mu u_{1} ^{*} u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+\gamma u_{3}^{*})} \\& \hphantom{B_{0}=}{}+\frac{\delta \alpha \eta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u _{1}^{*})^{2}(1+\gamma u_{3}^{*})}-\frac{(\mu_{i} d_{2}+u_{2}^{*}) \alpha \delta \gamma u_{1}^{*} {u_{2}^{*}} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2} (1+\gamma u_{3}^{*})^{2}}, \\& B_{1}= \bigl(u_{1}^{*}+\mu_{i}d_{1} \bigr) \bigl(u_{2}^{*}+\mu_{i}d_{2} \bigr) \\& \hphantom{B_{0}=}{}+ \bigl(u_{1}^{*}+u _{2}^{*}+ \mu_{i} d_{1}+ \mu_{i} d_{2} \bigr) \biggl(\mu_{i} d_{3}+\frac{u_{3}^{*}}{1+ \delta u_{1}^{*}} \biggr) \\& \hphantom{B_{0}=}{}+\frac{\alpha \mu u_{1}^{*} u_{2}^{*}}{1+\gamma u _{3}^{*}}-\frac{\alpha \gamma \delta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}}, \\& B_{2}=\mu_{i}(d_{1}+d_{2}+d_{3})+u_{1}^{*}+u_{2}^{*}+ \frac{u_{3}^{*}}{1+ \delta u_{1}^{*}} . \end{aligned}$$

Let $\lambda_{1i}$, $\lambda_{2i}$, $\lambda_{3i}$ be the three roots of $\rho_{i}(\lambda )=0$. In order to obtain the linear stability of $\mathbf{u^{*}}$, we need to prove that there exists a positive constant ζ such that

$$ \operatorname{Re}\{\lambda_{1i}\}, \operatorname{Re}\{ \lambda_{2i}\}, \operatorname{Re}\{ \lambda_{3i}\}\leq - \zeta \quad \mbox{for all }i\geq 1. $$

(13)

In fact, assume that $\lambda_{1i}$, $\lambda_{2i}$, $\lambda_{3i}$ all have negative real parts, that is, $\operatorname{Re}\{\lambda_{1i}\}$, $\operatorname{Re}\{\lambda_{2i}\}$, $\operatorname{Re}\{\lambda_{3i}\}<0$. Let $\lambda =\mu_{i} \tau$, then

$$ \rho_{i}(\lambda )=\mu_{i}^{3} \tau^{3}+B_{2} \mu_{i}^{2} \tau^{2}+B _{1}\mu_{i} \tau +B_{0} \stackrel{\triangle }{=}\widetilde{\rho }_{i}( \tau ). $$

Notice that $\mu_{i}\to \infty$ as $i\to \infty$, we can calculate that

$$ \lim_{i\to \infty } {\frac{\widetilde{\rho }_{i}(\tau )}{\mu _{i}^{3}}}=\tau^{3}+(d_{1}+d_{2}+d_{3}) \tau^{2}+(d_{1} d_{2}+d_{1} d _{3}+d_{2} d_{3})\tau +d_{1} d_{2} d_{3}. $$

Since $d_{1} d_{2}+d_{1} d_{3}+d_{2} d_{3}>0 \mbox{ and } d_{1} d _{2} d_{3}>0$ hold, by the Routh–Hurwitz criterion, the three roots $\tau_{1i}$, $\tau_{2i}$, $\tau_{3i}$ of $\widetilde{\rho }_{i}(\tau )=0$ have negative real parts. Therefore we can conclude that there exists a positive constant ζ̃ such that $\operatorname{Re}\{\tau_{1i}\}, \operatorname{Re} \{\tau_{2i}\}, \operatorname{Re}\{\tau_{3i}\}\leq -\widetilde{\zeta }$. According to continuity, we see that there exists a positive constant $i_{0}$ such that the three roots of $\widetilde{\rho }_{i}(\tau )=0$ satisfy $\operatorname{Re}\{\tau_{1i}\}, \operatorname{Re}\{\tau_{2i}\}, \operatorname{Re}\{\tau_{3i}\}\leq - \widetilde{\zeta }/2$ for all $i\geq i_{0}$. Then $\operatorname{Re}\{\lambda_{1i}\}, \operatorname{Re}\{\lambda_{2i}\}, \operatorname{Re}\{\lambda_{3i}\}\leq -\frac{\mu_{i} \widetilde{\zeta }}{2}\leq -\frac{\widetilde{\zeta }}{2}$ for all $i\geq i_{0}$. Let $\zeta =\min \{\frac{\widetilde{\zeta }}{2}, \max_{1\leq i\leq i_{0}} \operatorname{Re}\{\lambda_{1i}\}, \operatorname{Re}\{\lambda_{2i}\}, \operatorname{Re}\{ \lambda_{3i}\}\}$, then we can obtain that (13) holds. Applying the Routh–Hurwitz criterion for stability, the three roots of $\rho_{i}(\lambda )$ all have negative real parts. Thus the proof is completed. □

6 Turing instability induced by self-diffusion and cross-diffusion

According to Theorem 3, we can easily find that if only free diffusion is introduced to the ODE system (9), the uniform positive stationary solution is also locally stable, which means that under conditions (10), only the PDE with no self-diffusion and cross-diffusion cannot induce Turing instability. In this section, we will mainly discuss the case which is a well-known phenomenon of cross-diffusion driven instability. Now let us investigate the effects of self-diffusion and cross-diffusion for system (1).

For simplicity, we denote that $\mathbf{K(u)}=(d_{1} u_{1} +k_{11}u _{1}^{2}+k_{12}u_{1} u_{2}+\frac{k_{13}u_{1}}{\beta + u_{3}}, d_{2} u _{2}+k_{21}u_{1} u_{2}+k_{22}u_{2}^{2}+k_{23}u_{2} u_{3}, d_{3} u_{3}+\frac{k _{31}}{m+u_{1}}u_{2} u_{3}+k_{33}u_{3}^{2} )^{T}$. Linearizing system (1) at $\mathbf{u^{*}}$, we have

$$\begin{aligned} \textstyle\begin{cases} \frac{\mathbf{\partial u}}{\mathbf{\partial t}}=(\mathbf{K_{u}}\Delta +\mathbf{F_{u}(u^{*})})\mathbf{u}, &x\in \Omega,t>0, \\ \mathbf{\partial _{n} u}=0, &x\in \partial \Omega, t>0, \\ \mathbf{u}(x,0)=(u_{10}(x),u_{20}(x),u_{30}(x))^{T}, &x\in \Omega, \end{cases}\displaystyle \end{aligned}$$

(14)

where

$$\begin{aligned} &\mathbf{K_{u} \bigl(u^{*} \bigr)} \\ &\quad\!\!= \begin{pmatrix} d_{1}+2k_{11} u_{1}^{*}+k_{12}u_{2}^{*}+\frac{k_{13}}{\beta +u_{3} ^{*}} & k_{12}u_{1}^{*} & - \frac{k_{13} u_{1}^{*}}{(\beta + u_{3}^{*})^{2}} \\ k_{21} u_{2}^{*} & d_{2}+k_{21}u_{1}^{*}+2k_{22}u_{2}^{*}+k_{23}u _{3}^{*} & k_{23} u_{2}^{*} \\ -\frac{k_{31}u_{2}^{*} u_{3}^{*}}{(m+ u_{1}^{*})^{2}} & \frac{k_{31}u _{3}^{*}}{m+u_{1}^{*}} & d_{3}+\frac{k_{31}u_{2}^{*}}{m+u_{1}^{*}}+2k _{33}u_{3}^{*}\! \end{pmatrix} . \end{aligned}$$

It is well known that if for each $i\ge 1$, X is invariant under the operator $\mathbf{K_{u}(u^{*})}\Delta + \mathbf{F_{u}(u^{*})}$, then problem (1) has a non-trivial solution of the form $\boldsymbol{\psi }=\mathbf{c}\phi \exp (\lambda t)$ if and only if $(\lambda,\mathbf{c})$ is an eigenpair for the matrix $-\mu_{i} \mathbf{K_{u}(u^{*})}+\mathbf{F_{u}(u^{*})}$, where c is a constant vector. Therefore the equilibrium $\mathbf{u^{*}}$ is unstable if at least one eigenvalue λ has a positive real part for some $\mu_{i}$.

By some computations, the characteristic polynomial of $-\mu_{i} \mathbf{K_{u}(u^{*})}+\mathbf{F_{u}(u^{*})}$ is given by

$$ \psi_{i}(\lambda )=\lambda^{3}+C_{2} \lambda^{2}+C_{1} \lambda +C_{0}, $$

(15)

where

$$\begin{aligned}& C_{2}=\mu_{i} \biggl[d_{1}+d_{2}+d_{3}+2(k_{11}+k_{21})u_{1}^{*}+ \biggl(k_{12}+2k _{22}+\frac{k_{31}}{m+u_{1}^{*}} \biggr)u_{2}^{*} \\& \hphantom{ C_{2}=}{}+(k_{23}+2k_{33})u_{3}^{*}+ \frac{k _{13}}{\beta +u_{3}^{*}} \biggr]+u_{1}^{*}+u_{2}^{*}+ \frac{u_{3}^{*}}{1+ \delta u_{1}^{*}}, \\& C_{1}= \bigl(\mu_{i} L_{1}+u_{1}^{*} \bigr) \bigl(\mu_{i} L_{2}+u_{2}^{*} \bigr)+ \bigl(\mu_{i} L_{1}+u_{1}^{*} \bigr) \biggl(\mu_{i} L_{3}+\frac{u_{3}^{*}}{1+\delta u_{1}^{*}} \biggr) \\& \hphantom{ C_{1}=}{}+ \bigl(\mu_{i} L_{2}+u_{2}^{*} \bigr) \biggl(\mu_{i} L_{3}+\frac{u_{3}^{*}}{1+\delta u _{1}^{*}} \biggr) \\& \hphantom{ C_{1}=}{}- \biggl(\frac{k_{13}\mu_{i} u_{1}^{*}}{(\beta +u_{3}^{*})^{2}}+\frac{\alpha \gamma u_{1}^{*} u_{2}^{*}}{(1+\gamma u_{3}^{*})^{2}} \biggr) \biggl( \frac{\mu_{i} k_{31}u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}}+\frac{\delta {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \biggr) \\& \hphantom{ C_{1}=}{}- \biggl(\mu_{i} k_{12} u_{1}^{*}+ \frac{ \alpha u_{1}^{*}}{1+\gamma u_{3}^{*}} \biggr) \bigl( \mu_{i} k_{21} u_{2}^{*}- \mu u_{2}^{*} \bigr)- \bigl( \mu_{i} k_{23} u_{2}^{*}-\eta u_{2}^{*} \bigr) \frac{\mu_{i} k_{31 u_{3}^{*}}}{m+u_{1}^{*}}, \\& C_{0}= \biggl(\mu_{i} k_{12} u_{1}^{*}+ \frac{\alpha u_{1}^{*}}{1+\gamma u _{3}^{*}} \biggr) \bigl(\mu u_{2}^{*}- \mu_{i} k_{21} u_{2}^{*} \bigr) \biggl( \mu_{i} L_{3}+ \frac{u _{3}^{*}}{1+\delta u_{1}^{*}} \biggr) \\& \hphantom{C_{0}=}{}+ \bigl(\mu_{i} L_{1}+u_{1}^{*} \bigr) \bigl(\eta u_{2} ^{*}-\mu_{i} k_{23}u_{2}^{*} \bigr)\frac{\mu_{i} k_{31}u_{3}^{*}}{m+u_{1} ^{*}} \\& \hphantom{C_{0}=}{} + \bigl( \mu_{i} L_{1}+u_{1}^{*} \bigr) \bigl( \mu_{i} L_{2}+u_{2}^{*} \bigr) \biggl( \mu_{i} L _{3}+\frac{u_{3}^{*}}{1+\delta u_{1}^{*}} \biggr) \\& \hphantom{C_{0}=}{}+ \biggl(\mu_{i} k_{12}u_{1}^{*}+ \frac{ \alpha u_{1}^{*}}{1+\gamma u_{3}^{*}} \biggr) \bigl(\eta u_{2}^{*}- \mu_{i} k_{23}u _{2}^{*} \bigr) \biggl( \frac{\mu_{i} k_{31}u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}}+\frac{ \delta {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \biggr) \\& \hphantom{C_{0}=}{}+\frac{\mu_{i} k_{31} u_{3}^{*}}{m+u_{1}^{*}} \bigl(\mu u_{2}^{*}- \mu_{i} k_{21} u_{2}^{*} \bigr) \biggl( \frac{ \mu_{i} k_{13} u_{1}^{*}}{(\beta +u_{3}^{*})^{2}}+\frac{\alpha \gamma u_{1}^{*} u_{2}^{*}}{(1+\gamma u_{3}^{*})^{2}} \biggr) \\& \hphantom{ C_{0}=}{}- \bigl(\mu_{i} L_{2}+u_{2} ^{*} \bigr) \biggl(\frac{\mu_{i} k_{13} u_{1}^{*}}{(\beta +u_{3}^{*})^{2}}+\frac{ \alpha \gamma u_{1}^{*} u_{2}^{*}}{(1+\gamma u_{3}^{*})^{2}} \biggr) \biggl( \frac{ \mu_{i} k_{31}u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}}+\frac{\delta {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \biggr) . \end{aligned}$$

While

$$\begin{aligned}& L_{1}=d_{1}+2k_{11}u_{1}^{*}+k_{12}u_{2}^{*}+ \frac{k_{13}}{\beta +u _{3}^{*}}, \\& L_{2}=d_{2}+k_{21}u_{1}^{*}+2k_{22}u_{2}^{*}+k_{23}u_{3}^{*} , \\& L_{3}=d_{3}+2k_{33}u_{3}^{*}+ \frac{k_{31}u_{2}^{*}}{m+u_{1}^{*}} . \end{aligned}$$

Let $\lambda_{1}(\mu_{i})$, $\lambda_{2} (\mu_{i})$, $\lambda_{3} (\mu _{i})$ be the three roots of $\psi_{i} (\lambda )=0$, then $\lambda _{1}(\mu_{i})\lambda_{2}(\mu_{i})\lambda_{3}(\mu_{i})=-C_{0}$. In order to have at least one $\operatorname{Re}\lambda_{j}(\mu_{i})>0$, it is sufficient that $C_{0}<0$.

In the following we shall find out the conditions such that $C_{0}<0$. In fact $C_{0}=\operatorname{det}(\mu_{i}\mathbf{K_{u}(u^{*})}- \mathbf{F_{u}(u^{*})})$. By simple computation, it follows that

$$ C_{0}=P_{3} \mu_{i}^{3}+P_{2} \mu_{i}^{2}+P_{1}\mu_{i}- \operatorname{det} \bigl( \mathbf{F_{u} \bigl(u^{*} \bigr)} \bigr), $$

(16)

where

$$\begin{aligned}& P_{3}=L_{1} L_{2} L_{3}- \frac{k_{12} k_{23} k_{31}u_{1}^{*} {u_{2} ^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}} \\& \hphantom{P_{3}=}{}-\frac{ k_{13} k_{21}k_{31} u _{1}^{*} u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})(\beta +u_{3}^{*})^{2}}-\frac{ k_{13}k_{31} L _{2} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}(\beta +u_{3} ^{*})^{2}}-k_{12} k_{21}L_{3} u_{1}^{*} u_{2}^{*}-\frac{k_{23} k_{31} L_{1} u_{2}^{*} u_{3}^{*}}{m+u_{1}^{*}}, \\& P_{2}=L_{1} L_{3} u_{2}^{*}+L_{2} L_{3} u_{1}^{*} +\frac{L_{1} L_{2} u_{3}^{*}}{1+\delta u_{1}^{*}} + \frac{\eta k_{12} k_{31}u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}} -\frac{\delta k_{12} k_{23} u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \\& \hphantom{P_{2}=}{}-\frac{k_{23}k_{31} \alpha u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(1+\gamma u _{3}^{*})(m+u_{1}^{*})^{2}} -\frac{k_{21} k_{31}\alpha \gamma u_{1} ^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})(1+\gamma u_{3}^{*})} +\frac{ \mu k_{13} k_{31} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})(\beta +u_{3}^{*})^{2}} \\& \hphantom{P_{2}=}{} -\frac{k_{31} \alpha \gamma L_{2} u_{1}^{*} {u_{2} ^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2} (1+\gamma u_{3}^{*})^{2}} -\frac{k _{13} k_{31} u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2} ( \beta +u_{3}^{*})^{2}} -\frac{\delta k_{13} L_{2} u_{1}^{*} {u_{3}^{*}} ^{2}}{(1+\delta u_{1}^{*})^{2} (\beta +u_{3}^{*})^{2}} \\& \hphantom{P_{2}=}{}-\frac{k_{12} k _{21} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}-\frac{\alpha k_{21} L_{3} u_{1}^{*} u_{2}^{*}}{1+\gamma u_{3}^{*}}+\mu k_{12} L _{3} u_{1}^{*} u_{2}^{*}+ \frac{\eta k_{31}L_{1} u_{2}^{*} u_{3}^{*}}{m+u _{1}^{*}}-\frac{k_{23} k_{31} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{m+u_{1} ^{*}}, \\& P_{1}=\frac{L_{1} u_{2}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}} +\frac{L _{2} u_{1}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}+L_{3} u_{1}^{*} u_{2} ^{*} +\frac{k_{12}\eta \delta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+ \delta u_{1}^{*})^{2}} -\frac{k_{23}\alpha \delta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\gamma u_{3}^{*})(1+\delta u_{1}^{*})^{2}} \\& \hphantom{P_{1}=}{}+\frac{k _{31}\alpha \eta u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(1+\gamma u_{3} ^{*})(m+u_{1}^{*})^{2}}+\frac{\alpha \gamma \mu k_{31}u_{1}^{*} {u _{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})(1+\gamma u_{3}^{*})^{2}} -\frac{k _{31}\alpha \gamma u_{1}^{*} {u_{2}^{*}}^{3} u_{3}^{*}}{(m+u_{1}^{*})^{2} (1+\gamma u_{3}^{*})^{2}} \\& \hphantom{P_{3}=}{}-\frac{\alpha \gamma \delta L_{2} u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u _{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}} -\frac{\delta k_{13} u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} -\frac{ \alpha k_{21}u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+\gamma u_{3}^{*})(1+ \delta u_{1}^{*})} \\& \hphantom{P_{3}=}{}+\frac{k_{12} \mu u_{1}^{*} u_{2}^{*} u_{3}^{*}}{1+ \delta u_{1}^{*}} +\frac{\mu \alpha L_{3} u_{1}^{*} u_{2}^{*}}{1+ \gamma u_{3}^{*}} +\frac{k_{31} \eta u_{1}^{*} u_{2}^{*} u_{3}^{*}}{m+u _{1}^{*}}, \\& -\operatorname{det} \bigl(\mathbf{F_{u} \bigl(u^{*} \bigr)} \bigr) =\frac{u_{1}^{*} u_{2}^{*}u_{3}^{*}}{1+ \delta u_{1}^{*}} +\frac{\delta \alpha \eta u_{1}^{*} u_{2}^{*} {u_{3} ^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(1+\gamma u_{3}^{*})} \\& \hphantom{-\operatorname{det}(\mathbf{F_{u}(u^{*})})=}{}+\frac{\alpha \mu u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+\gamma u _{3}^{*})}-\frac{\alpha \delta \gamma u_{1}^{*} {u_{2}^{*}}^{2} {u _{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}} . \end{aligned}$$

Let $\tilde{P}(\nu )=P_{3} \nu^{3}+P_{2} \nu^{2}+P_{1}\nu -\operatorname{det}(\mathbf{F_{u}(u^{*})})$, and let $\tilde{\mu _{1}}$, $\tilde{\mu _{2}}$, and $\tilde{\mu _{3}}$ be the three roots of $\tilde{P}(\nu )=0$ with $\operatorname{Re}(\tilde{\mu }_{1})\leq \operatorname{Re}(\tilde{\mu }_{2})\leq \operatorname{Re}(\tilde{\mu } _{3})$. Then $\tilde{\mu }_{1} \tilde{\mu }_{2}\tilde{\mu }_{3}=\operatorname{det}( \mathbf{F_{u}(u^{*})})=-A_{0}<0$. Notice that direct calculations show that $P_{3}>0$, thus one of $\tilde{\mu }_{1}$, $\tilde{\mu }_{2}$, $\tilde{\mu }_{3}$ is real and negative and the product of the other two is positive.

Our following discussion will cover five cases.

Case 1

$k_{21}$ is a variation parameter and other parameters are fixed.

Consider the following limits:

$$\begin{aligned}& \lim_{k_{21}\to \infty } \frac{P_{3}}{k_{21}}=d_{1} L_{3} u _{1}^{*}+2k_{11}L_{3} {u_{1}^{*} }^{2}+\frac{ k_{13} d_{3} u_{1}^{*}}{ \beta +u_{3}^{*}}+ \frac{2 k_{13} k_{33}u_{1}^{*} u_{3}^{*}}{\beta +u _{3}^{*}}+\frac{k_{13} k_{31}\beta u_{1}^{*} u_{2}^{*}}{(\beta +u_{3} ^{*})^{2}(m+u_{1}^{*})} \\& \hphantom{\lim_{k_{21}\to \infty } \frac{P_{3}}{k_{21}}=}{}-\frac{k_{13} k_{31}{u_{1}^{*}}^{2} u_{2}^{*} u _{3}^{*}}{(m+u_{1}^{*})^{2} (\beta +u_{3}^{*})^{2}}:=h_{3}>0, \\& \lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}=L_{3} {u_{1} ^{*}}^{2}+\frac{(d_{1}+2k_{11}u_{1}^{*}+\frac{k_{13}}{\beta +u_{3} ^{*}})u_{1}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}} \\& \hphantom{ \lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}=}{}-\frac{k_{31}\alpha \gamma u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})(1+\gamma u_{3}^{*})} - \frac{k_{31}\alpha \gamma {u_{1}^{*}}^{2} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}} \\& \hphantom{ \lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}=}{}-\frac{k_{13}\delta {u_{1}^{*}}^{2} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} -\frac{\alpha L_{3} u_{1}^{*} u_{2}^{*}}{1+ \gamma u_{3}^{*}}:=h_{2}, \\& \lim_{k_{21}\to \infty }{\frac{P_{1}}{k_{21}}= \biggl(u_{1}^{*}- \frac{ \alpha \gamma \delta u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+\gamma u_{3}^{*})^{2}}-\frac{\alpha u_{2}^{*}}{1+\gamma u_{3} ^{*}} \biggr)\frac{u_{1}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}}:=h_{1}. \end{aligned}$$

Note that $\lim_{k_{21}\to \infty }{\tilde{P}(\nu )=\nu (h_{3} \nu^{2}+h_{2} \nu +h_{1})}$. Thus, when $u_{1}^{*}<\frac{\alpha \gamma \delta u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+ \gamma u_{3}^{*})^{2}}+\frac{\alpha u_{2}^{*}}{1+\gamma u_{3}^{*}}$, we show that $h_{1}<0<h_{3}$. A continuity argument shows that there exists a positive constant $k_{21}^{*}$ such that, for $k_{21}\geq k _{21}^{*}$, the three roots $\tilde{\mu }_{1}$, $\tilde{\mu }_{2}$, and $\tilde{\mu }_{3}$ of $\tilde{P}(\nu )=0$ are all real and satisfy

$$\begin{aligned}& \lim_{k_{21}\to \infty } {\tilde{\mu }_{1}= \frac{-h_{2}-\sqrt{h _{2}^{2}-4h_{1} h_{3}}}{2h_{3}}}, \\& \lim_{k_{21}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{21}\to \infty } {\tilde{\mu } _{3}=\frac{-h_{2}+\sqrt{h_{2}^{2}-4h_{1} h_{3}}}{2h_{3}}}, \end{aligned}$$

and we can conclude that $-\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}$. Furthermore, we can obtain

$$ \tilde{P}(\nu )< 0,\quad \mbox{when }\nu \in (-\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$

It is well known that since at least one eigenvalue $\mu_{i}$ of −Δ satisfies $\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})$ for some i, we have $C_{0}<0$ and the number of sign changes for the characteristic polynomial $\psi (\lambda )$ is either one or three. By using Descartes’ rule, $\psi (\lambda )$ has at least one positive root. Therefore, we can conclude the following theorem.

Theorem 4

Suppose that

$$ u_{1}^{*}< \frac{\alpha \gamma \delta u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+ \delta u_{1}^{*})(1+\gamma u_{3}^{*})^{2}}+ \frac{\alpha u_{2}^{*}}{1+ \gamma u_{3}^{*}} $$

(17)

and (10) hold. Then there exists a positive constant $k_{21}^{*}$ such that, for $k_{21}\geq k_{21}^{*}$, the uniform stationary solution $\mathbf{u^{*}}$ of system (1) is unstable.

Case 2

$k_{23}$ is a variation parameter and other parameters are fixed.

Consider the following limits:

$$\begin{aligned}& \lim_{k_{23}\to \infty } {\frac{P_{3}}{k_{23}}=L_{1} u_{3} \bigl( d _{3}+2k_{33} u_{3}^{*} \bigr)-\frac{ k_{12}k_{31} u_{1}^{*}{u_{2}^{*}}^{2} u _{3}^{*}}{(m+u_{1}^{*})^{2}}-\frac{k_{13} k_{31}u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(m+u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}}}:=e_{3}>0, \\& \lim_{k_{23}\to \infty }\frac{P_{2}}{K_{23}}= \bigl(d_{3}+2k_{33}u _{3}^{*} \bigr)u_{1}^{*} u_{3}^{*}+\frac{L_{1} {u_{3}^{*}}^{2}}{1+\delta u _{1}^{*}}-\frac{k_{12}\delta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+ \delta u_{1}^{*})^{2}} \\& \hphantom{\lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}=}{}-\frac{(1+2\gamma u_{3}^{*})k_{31}\alpha u_{1} ^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(1+\gamma u_{3}^{*})^{2}(m+u_{1}^{*})^{2}}-\frac{k _{13}\delta u_{1}^{*} {u_{3}^{*}}^{3}}{(1+\delta u_{1}^{*})^{2}( \beta +u_{3}^{*})^{2}} \\& \hphantom{\lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}}:=e_{2}, \\& \lim_{k_{23}\to \infty }{\frac{P_{1}}{k_{23}}= \biggl(1-\frac{\alpha \delta u_{2}^{*}}{(1+\gamma u_{3}^{*})(1+\delta u_{1}^{*})}- \frac{ \alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+ \gamma u_{3}^{*})^{2}} \biggr)\frac{u_{1}^{*} {u_{3}^{*}}^{2}}{1+\delta u _{1}^{*}}}:=e_{1} . \end{aligned}$$

Noticing $\lim_{k_{23}\to \infty }{\tilde{P}(\nu )=\nu (e_{3} \nu^{2}+e_{2} \nu +e_{1})}$. If the parameters satisfy $\frac{\alpha \delta u_{2}^{*}}{(1+\gamma u_{3}^{*})(1+\delta u_{1}^{*})}+\frac{ \alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+ \gamma u_{3}^{*})^{2}}>1$, then $e_{1}<0<e_{3}$. A continuity argument shows that there exists a positive constant $k_{23}^{*}$ such that, for $k_{23}\geq k_{23}^{*}$, the three roots $\tilde{\mu }_{1}$, $\tilde{\mu }_{2}$, and $\tilde{\mu }_{3}$ of $\tilde{P}(\nu )=0$ are all real and satisfy

$$\begin{aligned}& \lim_{k_{23}\to \infty } {\tilde{\mu }_{1}= \frac{-e_{2}-\sqrt{e _{2}^{2}-4e_{1} e_{3}}}{2e_{3}}}, \\& \lim_{k_{23}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{23}\to \infty } {\tilde{\mu } _{3}=\frac{-e_{2}+\sqrt{e_{2}^{2}-4e_{1} e_{3}}}{2e_{3}}}, \end{aligned}$$

and we can conclude that $-\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}$. Furthermore, we can obtain

$$ \tilde{P}(\nu )< 0, \quad \mbox{when }\nu \in (-\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$

It is worth noting that since at least one eigenvalue $\mu_{i}$ of −Δ satisfies $\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})$ for some i, therefore $C_{0}<0$ and the number of sign changes for the characteristic polynomial $\psi (\lambda )$ is either one or three. By using Descartes’ rule, $\psi (\lambda )$ has at least one positive root. Hence, we have the following theorem.

Theorem 5

If the condition

$$ \frac{\alpha \delta u_{2}^{*}}{(1+\gamma u_{3}^{*})(1+\delta u_{1} ^{*})}+\frac{\alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{(1+\delta u _{1}^{*})(1+\gamma u_{3}^{*})^{2}}>1 $$

(18)

and (10) are fulfilled, then there exists a positive constant $k_{23}^{*}$ such that the uniform stationary solution $\mathbf{u^{*}}$ of system (1) is unstable provided $k_{23}\geq k_{23}^{*}$.

Case 3

$k_{31}$ is a variation parameter and other parameters are fixed.

Consider the following limits:

$$\begin{aligned}& \lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}=\frac{L_{1} L _{2} u_{2}^{*}}{m+u_{1}^{*}}- \frac{ k_{12} k_{23} u_{1}^{*} {u_{2} ^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}}-\frac{k_{13} k_{21}u_{1}^{*} u _{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})(\beta +u_{3}^{*})^{2}}-\frac{k_{13} L _{2} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(\beta +u_{3}^{*})^{2}(m+u_{1} ^{*})^{2}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}=}{}-\frac{k_{12} k_{21}u_{1}^{*} {u_{2}^{*}}^{2}}{m+u_{1}^{*}}-\frac{k _{23}L_{1} u_{2}^{*} u_{3}^{*}}{m+u_{1}^{*}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}}:=f_{3}>0, \\& \lim_{k_{31}\to \infty }\frac{P_{2}}{K_{31}}=L_{3} {u_{1} ^{*}}^{2}+\frac{(d_{1}+2k_{11}u_{1}^{*}+\frac{k_{13}}{\beta +u_{3} ^{*}})u_{1}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}-\frac{k_{31}\alpha \gamma u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})(1+\gamma u _{3}^{*})} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}=}{}-\frac{k_{31}\alpha \gamma {u_{1}^{*}}^{2} {u_{2}^{*}}^{2} u _{3}^{*}}{(m+u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}} -\frac{k_{13} \delta {u_{1}^{*}}^{2} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}( \beta +u_{3}^{*})^{2}} - \frac{\alpha L_{3} u_{1}^{*} u_{2}^{*}}{1+ \gamma u_{3}^{*}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}}:=f_{2}, \\& \lim_{k_{31}\to \infty }\frac{P_{1}}{k_{31}}= \biggl(u_{2}^{*}+ \eta u_{3}^{*}+\frac{\alpha \eta u_{2}^{*} u_{3}^{*}}{(1+\gamma u_{3} ^{*})(m+u_{1}^{*})}+\frac{\mu \alpha u_{2}^{*}+2\alpha \gamma \mu u _{2}^{*} u_{3}^{*}}{(1+\gamma u_{3}^{*})^{2}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}=}{} -\frac{\alpha \gamma {u_{2}^{*}}^{2} u_{3}^{*}}{(m+ u_{1}^{*})(1+\gamma u_{3}^{*})^{2}} \biggr)\frac{u_{1}^{*} u_{2}^{*}}{m+u_{1}^{*}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}}:=f_{1}. \end{aligned}$$

It is worth mentioning that $\lim_{k_{31}\to \infty }{\tilde{P}( \nu )=\nu (f_{3}\nu^{2}+f_{2} \nu +f_{1})}$. Thus, when $u_{2}^{*}+ \eta u_{3}^{*}+\frac{\alpha \eta u_{2}^{*} u_{3}^{*}}{(1+\gamma u_{3} ^{*})(m+u_{1}^{*})}+\frac{\mu \alpha u_{2}^{*}+2\alpha \gamma \mu u _{2}^{*} u_{3}^{*}}{(1+\gamma u_{3}^{*})^{2}}<\frac{\alpha \gamma {u_{2}^{*}}^{2} u_{3}^{*}}{(m+ u_{1}^{*})(1+\gamma u_{3}^{*})^{2}}$, we show that $f_{1}<0<f_{3}$. A continuity argument shows that there exists a positive constant $k_{31}^{*}$ such that, for $k_{31}\geq k _{31}^{*}$, the three roots $\tilde{\mu }_{1}$, $\tilde{\mu }_{2}$, and $\tilde{\mu }_{3}$ of $\tilde{P}(\nu )=0$ are all real and satisfy

$$\begin{aligned}& \lim_{k_{31}\to \infty } {\tilde{\mu }_{1}= \frac{-f_{2}-\sqrt{f _{2}^{2}-4f_{1} f_{3}}}{2f_{3}}}, \\& \lim_{k_{31}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{31}\to \infty } {\tilde{\mu } _{3}=\frac{-f_{2}+\sqrt{f_{2}^{2}-4f_{1} f_{3}}}{2f_{3}}}, \end{aligned}$$

and we can conclude that $-\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}$. Furthermore, we can obtain

$$ \tilde{P}(\nu )< 0, \quad \mbox{when }\nu \in (-\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$

It is well known that since at least one eigenvalue $\mu_{i}$ of −Δ satisfies $\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})$ for some i, thus $C_{0}<0$ and the number of sign changes for the characteristic polynomial $\psi (\lambda )$ is either one or three. By using Descartes’ rule, $\psi (\lambda )$ has at least one positive root. Therefore, we can conclude the following theorem.

Theorem 6

Suppose that

$$ u_{2}^{*}+\eta u_{3}^{*}+ \frac{\alpha \eta u_{2}^{*} u_{3}^{*}}{(1+ \gamma u_{3}^{*})(m+u_{1}^{*})}+\frac{\mu \alpha u_{2}^{*}+2\alpha \gamma \mu u_{2}^{*} u_{3}^{*}}{(1+\gamma u_{3}^{*})^{2}}< \frac{ \alpha \gamma {u_{2}^{*}}^{2} u_{3}^{*}}{(m+ u_{1}^{*})(1+\gamma u _{3}^{*})^{2}} $$

(19)

and (10) are satisfied. Then there exists a positive constant $k_{31}^{*}$ such that, for $k_{31}\geq k_{31}^{*}$, the uniform stationary solution $\mathbf{u^{*}}$ of cross-diffusion predator–prey system (1) is unstable.

Case 4

$k_{13}$ is a variation parameter and other parameters are fixed.

Perform the following limits:

$$\begin{aligned}& \lim_{k_{13}\to \infty } \frac{P_{3}}{k_{13}}=\frac{1}{ \beta +u_{3}^{*}} \biggl(L_{2} L_{3}-\frac{k_{21}k_{31}u_{1}^{*} u_{2}^{*} u _{3}^{*}}{(m+u_{1}^{*})(\beta +u_{3}^{*})}-\frac{k_{31} L_{2} u_{1} ^{*} u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}(\beta +u_{3}^{*})}- \frac{k _{23} k_{31}u_{2}^{*} u_{3}^{*}}{m+u_{1}^{*}} \biggr) \\& \hphantom{ \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}}{}:=g_{3}>0, \\& \lim_{k_{13}\to \infty }\frac{P_{2}}{k_{13}}=\frac{L_{3} u_{2}^{*}}{\beta +u_{3}^{*}}+ \frac{L_{2} u_{3}^{*}}{(\beta +u_{3}^{*})(1+ \delta u_{1}^{*})}+\frac{k_{31}\eta u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})( \beta +u_{3}^{*})}+\frac{k_{31}\mu u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(m+u _{1}^{*})(\beta +u_{3}^{*})^{2}} \\& \hphantom{\lim_{k_{13}\to \infty }\frac{P_{2}}{k_{13}}=}{}-\frac{k_{31} u_{1}^{*} {u_{2}^{*}} ^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}}- \frac{\delta L_{2} u_{1}^{*} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \\& \hphantom{ \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}}{} :=g_{2}, \\& \lim_{k_{13}\to \infty }{\frac{P_{1}}{k_{13}}=\frac{u_{2}^{*} u_{3}^{*}}{\beta +u_{3}^{*}} \biggl( \frac{1}{1+\delta u_{1}^{*}}+\frac{k_{31} \eta }{m+u_{1}^{*}}-\frac{\delta u_{1}^{*} u_{3}^{*} }{1+\delta u_{1} ^{*}} \biggr)}:=g_{1}. \end{aligned}$$

Also notice that $\lim_{k_{13}\to \infty }{\tilde{P}(\nu )= \nu (g_{3}\nu^{2}+g_{2} \nu +g_{1})}$. Thus, when $\frac{k_{31}\eta (1+ \delta u_{1}^{*})}{m+ u_{1}^{*}}+1<\delta u_{1}^{*}u_{3}^{*}$, we show that $g_{1}<0<g_{3}$. A continuity argument shows that there exists a positive constant $k_{13}^{*}$ such that, for $k_{13}\geq k_{13}^{*}$, the three roots $\tilde{\mu }_{1}$, $\tilde{\mu }_{2}$, and $\tilde{\mu }_{3}$ of $\tilde{P}(\nu )=0$ are all real and satisfy

$$\begin{aligned}& \lim_{k_{13}\to \infty } {\tilde{\mu }_{1}= \frac{-g_{2}-\sqrt{g _{2}^{2}-4g_{1} g_{3}}}{2g_{3}}}, \\& \lim_{k_{13}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{13}\to \infty } {\tilde{\mu } _{3}=\frac{-g_{2}+\sqrt{g_{2}^{2}-4g_{1} g_{3}}}{2g_{3}}}, \end{aligned}$$

and we can conclude that $-\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}$. Furthermore, we can obtain

$$ \tilde{P}(\nu )< 0,\quad \mbox{ when }\nu \in (-\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$

It is remarkable that since at least one eigenvalue $\mu_{i}$ of −Δ satisfies $\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})$ for some i, hence $C_{0}<0$ and the number of sign changes for the characteristic polynomial $\psi (\lambda )$ is either one or three. By using Descartes’ rule, $\psi (\lambda )$ has at least one positive root. Therefore, we can obtain the following instability theorem.

Theorem 7

Assume that the parameters satisfy

$$ \frac{k_{31}\eta (1+\delta u_{1}^{*})}{m+ u_{1}^{*}}+1< \delta u_{1} ^{*}u_{3}^{*} $$

(20)

and (10). Then there exists a positive constant $k_{13}^{*}$ such that, for $k_{13}\geq k_{13}^{*}$, the uniform stationary solution $\mathbf{u^{*}}$ of cross-diffusion system with generalist (1) is unstable.

Case 5

$k_{22}$ is a variation parameter and other parameters are fixed.

Perform the following limits:

$$\begin{aligned}& \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}=2u_{2}^{*} \bigl(d _{1}+2k_{11}u_{1}^{*}+k_{12}u_{2}^{*} \bigr) \bigl(d_{3}+2k_{33}u_{3}^{*}\bigr) +\frac{2k _{31}{u_{2}^{*}}^{2}(d_{1}+2k_{11}u_{1}^{*}+k_{12}u_{2}^{*})}{m+u_{1} ^{*}} \\& \hphantom{ \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}=}{}+\frac{2k_{13}u_{2}^{*}(d_{3}+2k_{33}u_{3}^{*})}{\beta +u_{3} ^{*}}+\frac{2k_{13}k_{31}{u_{2}^{*}}^{2}(m\beta +mu_{3}^{*}+\beta u_{1}^{*})}{(m+u _{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} \\& \hphantom{ \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}}{}:=s_{3}>0, \\& \lim_{k_{22}\to \infty }{\frac{P_{2}}{k_{22}}}=2L_{3} u_{1} ^{*} u_{2}^{*}+\frac{2L_{1} u_{2}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}- \frac{k _{31}\alpha \gamma u_{1}^{*}{u_{2}^{*}}^{3} u_{3}^{*}}{(m+u_{1}^{*})^{2}(1+ \gamma u_{3}^{*})^{2}}-\frac{\delta k_{13}u_{1}^{*} u_{2}^{*} {u_{3} ^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}}:=s_{2}, \\& \lim_{k_{22}\to \infty }{\frac{P_{1}}{k_{22}}=\frac{2 u_{1} ^{*} u_{2}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}\biggl(1- \frac{\alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+\gamma u_{3}^{*})^{2}}\biggr)}:=s _{1} . \end{aligned}$$

It is worth pointing out that $\lim_{k_{22}\to \infty }{\tilde{P}( \nu )=\nu (s_{3}\nu^{2}+s_{2} \nu +s_{1})}$. Thus, when $\frac{\alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{ (1+\delta u_{1}^{*})(1+\gamma u _{3}^{*})^{2}}>1$, we show that $s_{1}<0<s_{3}$. A continuity argument shows that there exists a positive constant $k_{22}^{*}$ such that, for $k_{22}\geq k_{22}^{*}$, the three roots $\tilde{\mu }_{1}$, $\tilde{\mu }_{2}$, and $\tilde{\mu }_{3}$ of $\tilde{P}(\nu )=0$ are all real and satisfy

$$\begin{aligned}& \lim_{k_{22}\to \infty } {\tilde{\mu }_{1}= \frac{-s_{2}-\sqrt{s _{2}^{2}-4s_{1} s_{3}}}{2s_{3}}}, \\& \lim_{k_{22}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{22}\to \infty } {\tilde{\mu } _{3}=\frac{-s_{2}+\sqrt{s_{2}^{2}-4s_{1} s_{3}}}{2s_{3}}}, \end{aligned}$$

and we can conclude that $-\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}$. Furthermore, we can obtain

$$ \tilde{P}(\nu )< 0, \quad \mbox{when }\nu \in (-\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$

It is clear that since at least one eigenvalue $\mu_{i}$ of −Δ satisfies $\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})$ for some i, therefore $C_{0}<0$ and the number of sign changes for the characteristic polynomial $\psi (\lambda )$ is either one or three. By using Descartes’ rule, $\psi (\lambda )$ has at least one positive root. So, we can get the following instability theorem.

Theorem 8

Suppose that

$$ \frac{\alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{ (1+\delta u_{1}^{*})(1+ \gamma u_{3}^{*})^{2}}>1 $$

(21)

and (10) are fulfilled. Then there exists a positive constant $k_{22}^{*}$ such that, for $k_{22}\geq k_{22}^{*}$, the uniform stationary solution $\mathbf{u^{*}}$ of (1) is unstable.

Remark

• (A) By using the arguments similar to above, we can easily know that if the other constants are fixed, whereas self-diffusion coefficient $k_{11}$ or $k_{33}$ is sufficiently large, then the self-diffusion predator–prey system cannot induce Turing instability.

• (B) $k_{13}$, $k_{21}$, $k_{22}$, $k_{23}$, $k_{31}$ can be chosen as variation parameters since the number of sign changes for the polynomial (16) could be bigger than one for large values of $k_{13}$, $k _{21}$, $k_{22}$, $k_{23}$, $k_{31}$. By using Descartes’ rule, polynomial (16) could have at least a positive root, which leads to linear instability.

7 Examples

In this section, we will illustrate the correctness of our results by five examples. In order to facilitate the reading for cross-disciplinary readers, we need to point out that persistence means that all populations are persistent; symbiotic mutualism is a form of symbiosis, in which both sides in symbiosis are able to derive some kind of living benefit from each other; self-diffusion means that the per capita diffusion of each species is influenced only by its own density; cross-diffusion refers to the per capita diffusion rate of each species which is influenced by the other ones; and a generalist predator indicates a population that can feed on multiple species. The parameter values a, b, c are intrinsic growth rates of the three species, respectively, while α, μ, η, γ, δ describe inter-species interactions.

Example 1

This example is for Theorems 2, 3, and 4 for Case 1. Let $a=3$, $b=2$, $c=1$, $\alpha =1$, $\gamma =1$, $\delta =\frac{1}{2}$, $\eta =2$, $\mu =3$. Then $u_{1}^{*}=\frac{-9+ \sqrt{97}}{2}, u_{2}^{*}=-14+2\sqrt{97}, u_{3}^{*}=\frac{-5+ \sqrt{97}}{4}$. Using direct calculation, it is easy to see that conditions (10) and (17) all hold. Therefore the positive equilibrium $\mathbf{u^{*}}$ is linearly stable for ODEs (9) and reaction–diffusion system (12), whereas unstable for cross-reaction–diffusion system (1).

Example 2

This example is for Theorems 2, 3, and 5 for Case 2. Let $a=4$, $b=1$, $c=1$, $\alpha =2$, $\gamma =1$, $\delta =\frac{1}{2}$, $\eta =1$, $\mu =2$. Then $u_{1}^{*}=-6+2 \sqrt{11}, u_{2}^{*}=-13+5\sqrt{11}, u_{3}^{*}=-2+\sqrt{11}$. By using direct calculation, it is easy to find that conditions (10) and (18) all hold. Therefore the positive equilibrium $\mathbf{u^{*}}$ is linearly stable for ODEs (9) and reaction–diffusion system (12), whereas unstable for cross-reaction–diffusion system (1).

Example 3

This example is for Theorems 2, 3, and 6 for Case 3. Let $a=\frac{2}{5}$, $b=\frac{1}{10}$, $c=1$, $\alpha =2$, $\gamma =1$, $\delta =\frac{1}{2}$, $\eta =\frac{1}{10}$, $\mu = \frac{1}{10}$, $m=\frac{1}{10\text{,}000}$. Then $u_{1}^{*}=\frac{-31+ \sqrt{1041}}{10}$, $u_{2}^{*}=\frac{-51+3\sqrt{1041}}{200}$, $u_{3}^{*}=\frac{-11+ \sqrt{1041}}{20}$. By virtue of direct computation, it is easy to verify that conditions (10) and (19) all hold. Therefore the positive equilibrium $\mathbf{u^{*}}$ is linearly stable for ODEs (9) and reaction–diffusion system (12), whereas unstable for cross reaction–diffusion system (1).

Example 4

This example is for Theorems 2, 3, and 7 for Case 4. Let $a=3$, $b=1$, $c=1$, $\alpha =2$, $\gamma =1$, $\delta =1$, $\eta =1$, $\mu =2$, $k_{31}=8$, $m=1$. Then $u_{1}^{*}=-3+ \sqrt{11}, u_{2}^{*}=-7+3\sqrt{11}, u_{3}^{*}=-2+\sqrt{11}$. By direct calculation, it is easy to prove that conditions (10) and (20) all hold. Therefore the positive equilibrium $\mathbf{u^{*}}$ is linearly stable for ODEs (9) and reaction–diffusion system (12), whereas unstable for cross-reaction–diffusion system (1).

Example 5

This example is for Theorems 2, 3, and 8 for Case 5. Let $a=4$, $b=1$, $c=2$, $\alpha =2$, $\gamma =1$, $\delta =1$, $\eta =1$, $\mu =3$. Then $u_{1}^{*}=\frac{-5+\sqrt{41}}{4}, u_{2}^{*}=\frac{-13+5\sqrt{41}}{4}, u_{3}^{*}= \frac{-1+\sqrt{41}}{2}$. By use of direct calculation, it is easy to know that conditions (10) and (21) all hold. Therefore the positive equilibrium $\mathbf{u^{*}}$ is linearly stable for ODEs (9) and reaction–diffusion system (12), whereas unstable for cross-reaction–diffusion system (1).

8 Conclusions and discussions

In this paper, we investigate a strongly coupled cross-diffusion predator–prey system with generalist predator (1). By virtue of the comparison principle and mathematical analysis, we derive sufficient conditions of persistence of system (1) without self- and cross-diffusion. By linear stability analysis we obtain that a unique positive equilibrium is locally asymptotically stable for ODE and PDE systems without self-and cross-diffusion under certain conditions. Furthermore, we also get Turing instability emergence conditions for cross-diffusion and self-diffusion (only as $k_{22}$ is sufficiently large) predator–prey system. We illustrate the roles of diffusion, self-reaction diffusion, and nonlinear cross-diffusions in formation of stationary patterns. The above five instability theorems further show that self-diffusion and cross-diffusion can drive the development of spatially homogeneous density distribution and play an important role in driving Turing patterns in our predator–prey cross-diffusion system with generalist predator. We explain the correctness of the important results by virtue of numerical examples.

From a biological point of view, under parameter condition (2), we first prove that the predator–prey reaction–diffusion system (1) with no self- and cross-diffusion is persistent, and verify that its uniform positive steady state $\mathbf{u^{*}}$ is asymptotically stable for ODEs and PDEs of system (1) under condition (10). Hence, the system does not belong to the classical Turing instability scheme. Next, by a linear stability method, the results show that self-diffusions $k_{11}$, $k_{33}$ and cross-diffusion $k_{12}$ cannot produce instability for the predator–prey system with generalist and create Turing instability. Finally, we investigate the roles of cross-diffusions and self-diffusion in the stability of positive equilibrium $\mathbf{u^{*}}$ and find that most of them may be helpful to create Turing instability under certain parameter conditions. Actually, the corresponding uniform steady state of the system with cross-diffusions $k_{21}$, $k_{23}$, $k_{31}$, $k_{13}$ or only self-diffusion $k_{22}$ can be made unstable by increasing these cross-diffusion coefficients or self-diffusion to sufficiently large values, namely Turing instability can occur.

So self-diffusion and cross-diffusion can destabilize a uniform positive equilibrium which is stable for system (9) and for weakly coupled reaction–diffusion system (12), whereas Turing instability and the stationary patterns arise from the effect of cross-diffusion and partial self-diffusion. Our results obtained show not only that system (1) is capable of producing much more complex dynamics than the corresponding weakly coupled system (12) or single cross-diffusion system (1) with $k_{12}>0$, $k_{13}= k_{23}= k_{21}= k_{31}=0$, but also some interesting combining effects of intra-species competitions, cross-diffusion, and inter-species interactions.