Modeling the Allee effect and fear effect in predator–prey system incorporating a prey refuge

Abstract

In this paper, we consider a predator–prey model with Allee effect, fear effect and prey refuge. By considering the prey refuge as a parameter, we give the threshold condition for the stability of the system, and prove that the system undergoes a supercritical Hopf bifurcation. We show that increasing the prey refuge or Allee effect can make the dynamical behavior of the system more complicated; the fear effect or Allee effect has no influence on the prey density, but can lead to a decrease of the predator population at positive equilibrium.

1 Introduction

The predator–prey model is one of the basic models in the theoretical studies of ecology, and it has been studied extensively (see [1–14] and the references cited therein). On the other hand, prey species usually make use of refuges to decrease predation risk. Chen et al. [15] investigated the stable property of a predator–prey system with a constant number of prey refuges. The authors [16] showed that prey refuge has no influence on the stability of the system. Khajanchi and Banerjee [17] studied the uniform persistence and global asymptotic stability of a stage structured predator–prey model with prey refuge. Xiao et al. [18] considered the global stability of a stage structure predator–prey system with prey refuge, and pointed out that the model undergoes a Hopf bifurcation when the delay crosses some critical values. Xie et al. [19] studied the persistence and stability of a modified Leslie–Gower predator–prey model with prey refuge, and showed that the prey refuge has a positive effect on the persistence property. For more details in this direction, see [20–23].

Most studies of predator systems only consider direct killing by predators, because such predation can be easily observed in nature. However, prey respond to predation risk and exhibit different types of anti-predator responses, including habitat change, foraging, alertness, and different physiological changes. Apart from the direct killing, based on many experiments, Zanette et al. [24] showed that the song sparrows reduce by 40% the offspring by predation fears. Motivated by this, Wang et al. [25] considered the cost of fear into prey reproduction and investigated the following predator–prey system

$$ \begin{aligned} &u'(t) = a u F(f,v)-du-bu^{2}-muv, \\ &v'(t) = nmuv-ev, \end{aligned} $$

(1)

where a is the birth rate of prey; d is the natural death rate of prey; b is the density dependent coefficient; m is the capture rate; n is the conversion efficiency; e is the death rate of predator; f is the level of fear; $F (f, v)$ is the cost of anti-predator defence due to fear. From a biological point of view, $F(f,v)$ can be reasonably assumed to obey

$$ \begin{aligned} &F(0,v)=1,\qquad F(f,0)=1,\qquad \lim_{f\to +\infty }F(f,v)=0, \\ &\lim_{v\to +\infty }F(f,v)=0,\qquad \frac{\partial F(f,v)}{\partial f}< 0,\qquad \frac{\partial F(f,v)}{\partial v}< 0. \end{aligned} $$

(2)

They showed the cost of fear has no influence on the stability of the system (1). Considering the Holling type II functional response for system (1), [25] studied the stability of equilibria, and showed that a high amount of fear can stabilize the predator–prey system. Zhang et al. [26] investigated a Holling-II predator–prey model incorporating the fear effect and a prey refuge. They found that the fear effect cannot only reduce the population density of the predator at the positive equilibrium, but also stabilize the system. Kumar and Dubey [27] studied the stability of a delay prey-predator model with prey refuge and fear effect, and showed that the refuge below the threshold level is conducive to the system. Xiao and Li [28] considered a mutual interference predator–prey model with fear effect. Comparing with the corresponding predator–prey model without mutual interference, they concluded that the mutual interference can stabilize the predator–prey system. Sasmal and Takeuchi [29] studied a predator–prey system with fear effect, and discussed the multi-stability and Hopf bifurcation of the system. For more details in this direction, see [30–33].

On the other hand, the Allee effect can lead to the decrease of the intrinsic growth at low population densities, and make the system become unstable. Lin [34] studied the stability of a single species logistic model with Allee effect and feedback control, and showed the Allee effect makes the system become unstable. Ibarra and Flores [35] discussed the bifurcation of a Holling–Tanner predator–prey model with Allee effect. Wu et al. [36] concluded that the unique positive equilibrium is globally stable, and the Allee effect has no impact on the final density of the species. Guan and Chen [37] studied the bifurcation and stability of an amensalism with Allee effect, considering the growth reduction due to predator fear consider, Sasmal [38] studied a predator model with Allee effect in prey as follows:

$$ \textstyle\begin{cases} u'(t)=ru(1- \frac{u}{k})(u-\theta ) \frac{1}{1+fv}-auv, \\ v'(t)=a\alpha uv-mv, \end{cases} $$

(3)

where all the coefficients are positive constants; r is the intrinsic growth rate; k is the carrying capacity of the environment; a represents predation rate; α is the conversion efficiency of predator by consuming prey; m is the predator’s natural mortality rate; $0<\theta <k$ is expressed by a strong Allee effect; $F(f,v)=\frac{1}{1+fv}$ is the same meaning as (1). In the presence of fear, they showed that the fear effect can lead to the decrease of the per-capita growth rate, pointed out that fear does not affect the equilibrium stability, and the system has bi-stability among different equilibria.

Motivated by the above papers, the main purpose of this paper is to study the influence of the prey refuge on the stability of the system (3) as follows:

$$ \textstyle\begin{cases} u'(t)=ru(1- \frac{u}{k})(u-\theta _{0}) \frac{1}{1+f_{0}v}-a(1-\eta )uv, \\ v'(t)=a\alpha (1-\eta )uv-m_{0}v, \end{cases} $$

(4)

where $0<\eta <1$ is the prey refuge constant; $\eta u(t)$ is the capacity of a refuge at time t; the rest of the parameters have the same meaning as system (3).

For simplicity, let

$$ \begin{aligned}& N= \frac{u}{k},\qquad P= \frac{av}{rk},\qquad \tau =a \alpha kt, \\ &\epsilon = \frac{a\alpha }{r},\qquad \theta = \frac{\theta _{0}}{k},\qquad f= \frac{rkf_{0}}{a},\qquad m= \frac{m_{0}}{a\alpha k}. \end{aligned} $$

Rewriting τ as t, system (4) is reduced to

$$ \textstyle\begin{cases} N'(t)= \frac{N}{\epsilon } ( \frac{(1-N)(N-\theta )}{1+fP}-(1-\eta )P ), \\ P'(t)=P((1-\eta )N-m), \end{cases} $$

(5)

where $0<\theta <1$ and $0<\eta <1$, the remaining parameters are positive constants.

The organization of this paper is as follows. In Sect. 2, the stability of the equilibria of the system (5) is investigated. In Sect. 3, the impacts of the fear effect, Allee effect and prey refuge on the species are discussed. Finally, a brief conclusion is drawn.

2 Main results

In this section, we study the existence and stability of the equilibria point of the system (5). Obviously, system (5) always has a trivial equilibrium $E_{0}(0,0)$ and two boundary equilibria $E_{1}(1,0)$ and $E_{2}(\theta,0)$. The positive equilibrium point of the system (5) is determined by the following equation:

$$ \textstyle\begin{cases} \frac{(1-N)(N-\theta )}{1+fP}-(1-\eta )P=0, \\ (1-\eta )N-m=0. \end{cases} $$

(6)

Substituting $N=\frac{m}{1-\eta }$ into the first equation of (6), we have

$$ fP^{2}+P-H=0, $$

(7)

where $H=\frac{(1-\eta -m)(m-\theta (1-\eta ))}{(1-\eta )^{3}}$. If $m\geq 1$, that is $H<0$, then there does not exist a positive equilibrium. If $1-\frac{m}{\theta }<\eta <1-m$, then $H>0$. Hence, Eq. (7) has a positive solution,

$$ P^{*}= \frac{-1+\sqrt{1+4fH}}{2f}. $$

Let $N^{*}=\frac{m}{1-\eta }$, then system (5) has a positive equilibrium $E^{*} (N^{*}, P^{*} )$.

Now we discuss the stability of equilibria, and we obtain the following lemmas.

Lemma 1

The trivial equilibrium$E_{0}(0,0)$of the system (5) is always a stable node.

Proof

The Jacobian matrix of the system (5) at $E_{0}$ is calculated as

$$ J_{E_{0}}= \begin{bmatrix} - \frac{\theta }{\epsilon }&0 \\ 0&-m \end{bmatrix}. $$

We obtain $\operatorname{Tr} J_{E_{0}}=-\frac{\theta }{\epsilon }-m<0$ and $\operatorname{Det} J_{E_{0}}= \frac{m \theta }{\epsilon } >0$, thus the equilibrium $E_{0}$ is a stable node. This completes the proof of Lemma 1. □

Lemma 2

• (1) If$0<\eta <1-m$, $E_{1}(1,0)$of the system (5) is a saddle point.

• (2) If$\eta =1-m$, $E_{1}(1,0)$of the system (5) is an attracting saddle node, which includes a stable parabolic sector.

• (3) If$1-m<\eta <1$, $E_{1}(1,0)$of the system (5) is locally asymptotically stable.

Proof

The Jacobian matrix of the system (5) at $E_{1}$ is calculated as

$$ J_{E_{1}}= \begin{bmatrix} \frac{\theta -1}{\epsilon }& \frac{\eta -1}{\epsilon } \\ 0&1-\eta -m \end{bmatrix}. $$

Then the eigenvalues $\lambda _{1}=\frac{\theta -1}{\epsilon }<0$ and $\lambda _{2}=1-\eta -m$. If $0<\eta <1-m$, that is $\lambda _{2}>0$, the boundary equilibrium $E_{1}$ is a saddle point. If $1-m<\eta <1$, that is $\lambda _{2}<0$, the boundary equilibrium $E_{1}$ is stable.

When $\eta =1-m$, we have $\lambda _{2}=0$. Then system (5) can be rewritten as

$$ \textstyle\begin{cases} N'(t)= \frac{N}{\epsilon } ( \frac{(1-N)(N-\theta )}{1+fP}-mP ), \\ P'(t)=mP(N-1). \end{cases} $$

(8)

We transform the equilibrium $E_{1}$ to the origin by making a transformation that $X=N-1, Y=P$. Then we have a Taylor expansion at the origin as follows:

$$ \textstyle\begin{cases} X'(t)= \frac{\theta -1}{\epsilon }X- \frac{m}{\epsilon }Y+ \frac{\theta -2}{\epsilon }X^{2}- \frac{(\theta -1)f+m}{\epsilon }XY+Q_{1}(X,Y), \\ Y'(t)=mXY, \end{cases} $$

(9)

where $Q_{1}(X,Y)$ is $C^{\infty }$ functions of at least the third order in terms of $(X,Y)$.

Let $X_{1}=\frac{\theta -1}{\epsilon }X-\frac{m}{\epsilon }Y, Y_{1}=Y$. Introducing a new time variable τ by $\tau =-\frac{1-\theta }{\epsilon }t$, rewriting τ as t, we have

$$ \textstyle\begin{cases} X_{1}'(t)=X_{1}+ \frac{(-2+\theta )\epsilon }{(\theta -1)^{2}}X_{1}^{2}- \frac{\epsilon m^{2}+f\theta ^{2}-2f\theta -m\theta +f+3m}{(\theta -1)^{2}}X_{1}Y_{1} \\ \phantom{X_{1}'(t)=}{}- \frac{(\epsilon m^{2}+f\theta ^{2}-2f\theta +f+m)m}{\epsilon (\theta -1)^{2}}Y_{1}^{2}+Q_{2}(X_{1},Y_{1}), \\ Y_{1}'(t)= \frac{\epsilon ^{2} m}{(\theta -1)^{2}}X_{1}Y_{1}+ \frac{\epsilon m^{2}}{(\theta -1)^{2}}Y_{1}^{2}, \end{cases} $$

(10)

where $Q_{2}(X_{1},Y_{1})$ is a $C^{\infty }$ function of at least the third order in terms of $(X_{1},Y_{1})$.

Therefore, the coefficient of $Y_{1}^{2}$ is $\frac{\epsilon m^{2}}{(\theta -1)^{2}}>0$. According to Theorem 7.1 in Zhang et al. [39], we can conclude that $E_{1}(1,0)$ is an attracting saddle node, which includes a stable parabolic sector. This completes the proof of Lemma 2. □

Lemma 3

• (1) If$0<\eta <1-\frac{m}{\theta }$, $E_{2}(\theta,0)$of the system (5) is unstable.

• (2) If$\eta =1-\frac{m}{\theta }$, $E_{2}(\theta,0)$of the system (5) is a repelling saddle node, which includes a unstable parabolic sector.

• (3) If$1-\frac{m}{\theta }<\eta <1$, $E_{2}(\theta,0)$of the system (5) is a saddle point.

Proof

The Jacobian matrix of the system (5) at $E_{2}$ is calculated as

$$ J_{E_{2}}= \begin{bmatrix} \frac{\theta (1-\theta )}{\epsilon }& \frac{\theta (\eta -1)}{\epsilon } \\ 0&\theta (1-\eta )-m \end{bmatrix}. $$

Then the eigenvalues $\lambda _{1}=\frac{\theta (1-\theta )}{\epsilon }>0$ and $\lambda _{2}=\theta (1-\eta )-m$. If $0<\eta <1-\frac{m}{\theta }$, that is, $\lambda _{2}>0$, the boundary equilibrium $E_{2}$ is unstable. If $1-\frac{m}{\theta }<\eta <1$, that is, $\lambda _{2}<0$, the boundary equilibrium $E_{2}$ is a saddle point.

When $\eta =1-\frac{m}{\theta }$, we obtain $\lambda _{2}=0$. Then system (5) can be rewritten as

$$ \textstyle\begin{cases} N'(t)= \frac{N}{\epsilon } ( \frac{(1-N)(N-\theta )}{1+fP}- \frac{m}{\theta }P ), \\ P'(t)=mP ( \frac{N}{\theta }-1 ). \end{cases} $$

(11)

We transform the equilibrium $E_{2}$ to the origin by making a transformation that $X=N-\theta, Y=P$. Then we have a Taylor expansion at the origin as follows:

$$ \textstyle\begin{cases} X'(t)= \frac{\theta (1-\theta )}{\epsilon }X- \frac{m}{\epsilon }Y+ \frac{1-2\theta }{\epsilon }X^{2}- \frac{\theta ^{2}(\theta -1)f+m}{\epsilon \theta }XY+Q_{3}(X,Y), \\ Y'(t)= \frac{m}{\theta }XY, \end{cases} $$

(12)

where $Q_{3}(X,Y)$ is a $C^{\infty }$ function of at least the third order in terms of $(X,Y)$.

Let $X_{1}=\frac{\theta (1-\theta )}{\epsilon }X- \frac{m}{\epsilon }Y, Y_{1}=Y$. Introducing a new time variable τ by $\tau =\frac{\theta (1-\theta )}{\epsilon }t$, rewriting τ as t, we have

$$ \textstyle\begin{cases} X_{1}'(t)=X_{1}- \frac{f\theta ^{5}-2f\theta ^{4}+f\theta ^{3}+\epsilon m^{2}+3m\theta ^{2}-m\theta }{(\theta -1)^{2}\theta ^{3}}Y_{1}- \frac{(2\theta -1)\epsilon }{\theta ^{2}(\theta -1)^{2}}X_{1}^{2} \\ \phantom{X_{1}'(t)=}{}- \frac{f\theta ^{5}-2f\theta ^{4}+f\theta ^{3}+\epsilon m^{2}+3m\theta ^{2}-m\theta }{\theta ^{3}(\theta -1)^{2}}X_{1}Y_{1} \\ \phantom{X_{1}'(t)=}{}- \frac{(f\theta ^{5}-2f\theta ^{4}+f\theta ^{3}+\epsilon m^{2}+m\theta ^{2})m}{\epsilon (\theta -1)^{2}\theta ^{3}}Y_{1}^{2}+Q_{4}(X_{1},Y_{1}), \\ Y_{1}'(t)= \frac{\epsilon ^{2} m}{\theta ^{3}(\theta -1)^{2}}X_{1}Y_{1}+ \frac{\epsilon m^{2}}{\theta ^{3}(\theta -1)^{2}}Y_{1}^{2}, \end{cases} $$

(13)

where $Q_{4}(X_{1},Y_{1})$ is a $C^{\infty }$ function of at least the third order in terms of $(X_{1},Y_{1})$.

Therefore, the coefficient of $Y_{1}^{2}$ is $\frac{\epsilon m^{2}}{\theta ^{3}(\theta -1)^{2}}>0$. According to Theorem 7.1 in Zhang et al. [39], we can conclude that $E_{2}(\theta,0)$ is a repelling saddle node, which includes an unstable parabolic sector. This completes the proof of Lemma 3. □

Lemma 4

• (1) If$1-\frac{m}{\theta }<\eta <1-\frac{2m}{\theta +1}$, $E^{*} (N^{*}, P^{*} )$of the system (5) is stable.

• (2) If$1-\frac{2m}{\theta +1}<\eta <1-m$, $E^{*} (N^{*}, P^{*} )$of the system (5) is unstable.

Proof

The Jacobian matrix of the system (5) at $E^{*}$ is calculated as

$$ J_{E^{*}}= \begin{bmatrix} \frac{m((1+\theta )(1-\eta )-2m)}{\epsilon (1-\eta )^{2}(1+fP^{*})} & - \frac{m}{\epsilon (1-\eta )} ( \frac{(1-m-\eta )(m-\theta (1-\eta ))f}{(1-\eta )^{2}(1+fP^{*})^{2}}+(1- \eta ) ) \\ P^{*}(1-\eta ) &0 \end{bmatrix}. $$

By calculation, we obtain $\operatorname{Det} J_{E^{*}}=\frac{mP^{*}}{\epsilon } (\frac{(1-m-\eta )(m-\theta (1-\eta ))f}{(1-\eta )^{2}(1+fP^{*})^{2}}+(1- \eta ) )> 0$ and $\operatorname{Tr} J_{E^{*}}=\frac{m((1+\theta )(1-\eta )-2m)}{\epsilon (1-\eta )^{2}(1+fP^{*})}$.

Therefore, if $1-\frac{m}{\theta }<\eta <1-\frac{2m}{\theta +1}$, that is $\operatorname{Tr} J_{E^{*}}>0$, then the boundary equilibrium $E^{*}$ of the system (5) is unstable. If $1-\frac{2m}{\theta +1}<\eta <1-m$, that is $\operatorname{Tr} J_{E^{*}}<0$, then the boundary equilibrium $E^{*}$ of the system (5) is stable. This completes the proof of Lemma 4. □

Lemma 5

If$\eta =1-\frac{2m}{\theta +1}$, system (5) undergoes a supercritical Hopf bifurcation at positive equilibrium$E^{*}$and there exists a stable limit cycle around$E^{*}$.

Proof

Form the proof of Lemma 4, when $\eta =1-\frac{2m}{\theta +1}$, we have $\operatorname{Det} J_{E^{*}}>0$ and $\operatorname{Tr} J_{E^{*}}=0$. The positive equilibrium $E^{*}$ becomes a non-hyperbolic equilibrium, and the Jacobian matrix at $E^{*}$ has a pair of imaginary roots. In order to ensure the occurrence of the Hopf bifurcation, the transverse conditions of Hopf bifurcation should be checked. By a straightforward computation, we have

$$ {\frac{\mbox{d}}{\mbox{d}\eta }\operatorname{Tr}[J_{E^{*}}]}\bigg|_{ \eta =1- \frac{2m}{\theta +1}}= - \frac{(\theta +1)^{3}}{\epsilon (\sqrt{2m(f(1+\theta )(1-\theta )^{2}+2m)}+2m )} \neq 0. $$

Hence, the positive equilibrium $E^{*}$ loses its stability through a non-degenerate Hopf bifurcation.

Now we will calculate the Lyapunov number $l_{1}$ at $E^{*}$ to determine the stability of the limit cycle. We first translate the equilibrium $E^{*}$ of the system (5) to the origin by employing the transformations $N=\hat{x}-N^{*}$ and $P=\hat{y}-P^{*}$. Note that $N^{*}=\frac{1+\theta }{2}$ and $\eta =1-{ \frac{ (1-\theta )^{2} }{4 P^{*} ( f{ P^{*}}+1 ) }}$. Thus, system (5) in a neighborhood of the origin can be written as

$$ \textstyle\begin{cases} \dot{x} = a_{10}x+a_{01}\hat{y}+a_{20} \hat{x}^{2}+a_{11}\hat{x}\hat{y}+a_{02}\hat{y}^{2}+a_{30}\hat{x}^{3} \\ \phantom{\dot{x} =}{} +a_{21}\hat{x}^{2} \hat{y}+a_{12}\hat{x}\hat{y}^{2}+a_{03}\hat{y}^{3}, \\ \dot{\hat{y}} = b_{10}\hat{x}+b_{01}\hat{y}+b_{20} \hat{x}^{2}+b_{11}\hat{x}\hat{y}+b_{02}\hat{y}^{2}+b_{30}\hat{x}^{3} \\ \phantom{\dot{\hat{y}} =}{} +b_{21}\hat{x}^{2} \hat{y}+b_{12}\hat{x}\hat{y}^{2}+b_{03}\hat{y}^{3}, \end{cases} $$

(14)

where $a_{10}=0$, $a_{01}=-\frac{(1+\theta )(1-\theta )^{2} (2fP^{*}+1 )}{8\epsilon P^{*}(fP^{*}+1)^{2}}$, $a_{20}=-\frac{1+\theta }{2\epsilon (fP^{*}+1)}$, $a_{11}=-\frac{(1-\theta )^{2}(2fP^{*}+1)}{4\epsilon P^{*}(fP^{*}+1)^{2}}$, $a_{02}=\frac{(1+\theta )(1-\theta )^{2}f^{2}}{8\epsilon (fP^{*}+1)^{3}}$, $a_{30}=-\frac{1}{\epsilon (fP^{*}+1)}$, $a_{12}=\frac{(1-\theta )^{2}f^{2}}{4\epsilon (fP^{*}+1)^{3}}$, $a_{21}=\frac{(1+\theta )(fP^{*}+1)^{2}f}{2\epsilon (fP^{*}+1)^{4}}$, $a_{03}=-\frac{(1+\theta )(1-\theta )^{2}f^{3}}{8\epsilon (fP^{*}+1)^{4}}$, $b_{10}=\frac{(1-\theta )^{2}}{4(fP^{*}+1)}$, $b_{01}=\frac{(1-\theta )^{2}(1+\theta )}{8P^{*}(fP^{*}+1)}-m$, $b_{11}=\frac{(1-\theta )^{2}}{4P^{*}(fP^{*}+1)}$, $b_{20}=b_{02}=b_{30}=b_{21}=b_{12}=b_{03}=0$.

The first Lyapunov number $l_{1}$ [40] to determine the stability of limit cycle for a planar system (14) is given by the following formula:

$$\begin{aligned} l_{1} ={} & \frac{-3\pi }{2a_{01}\varPhi ^{3/2}}\bigl\{ \bigl[a_{10}b_{10} \bigl(a_{11}^{2}+a_{11}b_{02}+a_{02}b_{11} \bigr)+a_{10}a_{01}\bigl(b_{11}^{2}+a_{20}b_{11}+a_{11}b_{02} \bigr) \\ &{}+b_{10}^{2}(a_{11}a_{02}+2a_{02}b_{02})-2a_{10}b_{10} \bigl(b_{02}^{2}-a_{20}a_{02} \bigr)-2a_{10}a_{01}\bigl(a_{20}^{2}-b_{20}b_{02} \bigr) \\ &{}-a_{01}^{2}(2a_{20}b_{20}+b_{11}b_{20})+ \bigl(a_{01}b_{10}-2a_{10}^{2}\bigr) (b_{11}b_{02}-a_{11}a_{20})\bigr] \\ &{}-\bigl(a_{10}^{2}+a_{01}b_{10}\bigr) \bigl[3(b_{10}b_{03}-a_{01}a_{30})+2a_{10}(a_{21}+b_{12})+(b_{10}a_{12}-a_{01}b_{21}) \bigr] \bigr\} \\ ={}&- \frac{3\pi (1+\theta )(1-\theta )^{4}(2fP^{*}+1)}{32\epsilon ^{2} \varPhi ^{3/2} (fP^{*}+1)^{4} P^{*} }, \end{aligned}$$

where $\varPhi =\sqrt{\frac{(1+\theta )(1-\theta )^{4}(2fP^{*}+1)}{32\epsilon (fP^{*}+1)^{3}P^{*}}}$.

Obviously $l_{1}<0$, the system (5) undergoes a supercritical Hopf bifurcation at positive equilibrium $E^{*}$ and there exists a stable limit cycle around $E^{*}$. This completes the proof of Lemma 5.

It follows from Lemma 6 that the trivial equilibrium point $E_{0}$ is always a stable node. Hence, in the rest of this section, we only consider the stability of the boundary equilibria $E_{1}, E_{2}$ and the positive equilibrium $E^{*}$ of the system with the prey refuge and Allee effect varies. When $m\ge 1$, there does not exist a positive equilibrium point of the system (5). According to Lemmas 2 and 3, we have the following theorem. □

Theorem 1

When$m\ge 1$and$0<\eta <1$, the boundary equilibrium$E_{1}$is stable and the boundary equilibrium$E_{2}$is a saddle point.

From Theorem 1, if the death rate of the predator is large enough, then the predator will be extinct, and the prey will tend to extinction or the carrying capacity of the environment depending on the choices of the initial density of prey.

In the following, we only discuss $0< m<1$. According to Lemmas 2–4 and considering the prey refuge as a parameter, we have the following theorems.

Theorem 2

When$\theta \le 2m-1$and$m<1$, we have:

• (1) If$0<\eta <1-m$, $E_{1}$and$E_{2}$are saddle points, $E^{*}$is stable.

• (2) If$\eta =1-m$, $E_{1}$is an attracting saddle node, including a stable parabolic sector, and$E_{2}$is a saddle point.

• (3) If$1-m<\eta <1$, $E_{1}$is stable and$E_{2}$is a saddle point.

Theorem 3

When$2m-1<\theta \le m<1$, we have:

• (1) If$0<\eta <1-\frac{2m}{1+\theta }$, $E_{1}$and$E_{2}$are saddle points, $E^{*}$is unstable.

• (2) If$\eta =1-\frac{2m}{1+\theta }$, $E_{1}$and$E_{2}$are saddle points, and a Hopf bifurcation exists at$E^{*}$.

• (3) If$1-\frac{2m}{1+\theta }<\eta <1-m$, $E_{1}$and$E_{2}$are saddle points and$E^{*}$is stable.

• (4) If$\eta =1-m$, $E_{1}$is an attracting saddle node, including a stable parabolic sector and$E_{2}$is a saddle point.

• (5) If$1-m<\eta <1$, $E_{1}$is stable and$E_{2}$is a saddle point.

Theorem 4

When$m<\theta <1$, we get:

• (1) If$0<\eta <1-\frac{m}{\theta }$, $E_{1}$is a saddle point, $E_{2}$is unstable.

• (2) If$\eta =1-\frac{m}{\theta }$, $E_{1}$is a saddle point, and$E_{2}$is a repelling saddle node, which includes a unstable parabolic sector.

• (3) If$1-\frac{m}{\theta }<\eta <1-\frac{2m}{1+\theta }$, $E_{1}$and$E_{2}$are saddle points and$E^{*}$is unstable.

• (4) If$\eta =1-\frac{2m}{1+\theta }$, $E_{1}$and$E_{2}$are saddle points, and a Hopf bifurcation exists at$E^{*}$.

• (5) If$1-\frac{2m}{1+\theta }<\eta <1-m$, $E_{1}$and$E_{2}$are saddle points and$E^{*}$is stable.

• (6) If$\eta =1-m$, $E_{1}$is an attracting saddle node, which includes a stable parabolic sector, and$E_{2}$is a saddle point.

• (7) If$1-m<\eta <1$, $E_{1}$is stable and$E_{2}$is a saddle point.

Note that the trivial equilibrium point $E_{0}$ is always a stable node. If $m<\theta <1$ and $0<\eta \leq 1-\frac{m}{\theta }$, it follows from Theorem 4 that $E_{1}$ is a saddle point and $E_{2}$ is unstable. Obviously, the solutions of the system (5) are positive and bounded. Also, for system (1.5) there do not exist limit cycles. Hence, $E_{0}$ is globally asymptotically stable. Then we have the following remark.

Remark 1

If $m<\theta <1$ and $0<\eta \leq 1-\frac{m}{\theta }$, the trivial equilibrium $E_{0}$ is globally asymptotically stable.

If $m\leq \theta$, without the prey refuge, the author [38] pointed out that both prey and predator will be extinct. However, it follows from Theorems 3 and 4, the dynamics behavior of the system (5) becomes more complicated. With increase of the prey refuge, the behavior of the positive equilibrium $E^{*}$ changes from instability to stability. System (5) undergoes a supercritical Hopf bifurcation and there exists a stable limit cycle around $E^{*}$. If the prey refuge is large enough, the positive equilibrium $E^{*}$ disappears, that is, the predator will be extinct.

3 Discussion and numerical simulations

In this section, we show the influence of the Allee effect, fear effect and prey refuge on the dynamics behavior of the system (5).

Firstly, we study the impact of the fear effect on the dynamics behavior of the system (5). Note that $N^{*}=\frac{m}{1-\eta }$, then the fear effect has no influence on the prey density. By calculating, we have

$$ \frac{\partial P^{*}}{\partial f}=- \frac{2H^{2}}{\sqrt{(1+4fH)}(\sqrt{1+4fH}+1+2fH)}< 0, $$

that is, the predator density is a strictly decreasing function with respect to fear effect. Then we show that increasing the fear effect can decrease the predator density, that is, fear effect can lead to the decrease of the predator population. It follows from Theorems 2–4 that the cost of the fear effect has no influence on the stability of the system (5), which is in accord with that for the predator–prey model with the linear functional response [25].

Secondly, we study the impact of the Allee effect on the dynamics behavior of the system (5). Obviously, the Allee effect has no impact on the prey density. The existence of the positive equilibrium point $E^{*}(N^{*},P^{*})$ implies that $1-\frac{m}{\theta }<\eta <1-m$. By simple computation, one has

$$ \frac{\partial P^{*}}{\partial \theta }=- \frac{1-\eta -m}{(1-\eta )^{2}\sqrt{1+4fH}}< 0, $$

that is the predator density is a strictly decreasing function with respect to the Allee effect. With the increase of Allee effect, the density of the predator population decreases, while the density of the prey population does not change. From Theorem 2, if the Allee effect is small enough, then the dynamics behavior of the system (5) is relatively simple, that is the positive equilibrium $E^{*}$ is stable when it exists (see Fig. 1(a)). However, if the Allee effect is large enough, the dynamics behavior of the system (5) becomes more complicated. From Theorems 3 and 4, the behavior of the positive equilibrium $E^{*}$ changes from stability to instability, and there exists a stable limit cycle around $E^{*}$ (see Fig. 1(b)). If the prey refuge is small enough, the positive equilibrium $E^{*}$ can even disappear (see Fig. 1(c)). Hence, increasing the amount of the Allee effect has a negative effect on the stability of the system (5).

Figure 1

Dynamic behavior of the system (5) with $\eta =0.1, \varepsilon =1, f=0.2, m=0.6$

Thirdly, we investigate the impact of prey refuge on the dynamics behavior of the system (5). By computing the derivative along the $N^{*}$ and $P^{*}$ with respect to η, respectively, we obtain

$$ \frac{\partial N^{*}}{\partial \eta }= \frac{m}{(1-\eta )^{2}}>0 $$

and

$$ \frac{\partial P^{*}}{\partial \eta }= \frac{-\theta (1-\eta )^{2}+2m(1+\theta )(1-\eta )-3m^{2}}{(1-\eta )^{4}\sqrt{1+4fH}}. $$

The above inequality shows that $N^{*}$ is a strictly increasing function with respect to prey refuge. That is, increasing the value of prey refuge can increase prey density.

Let $F(z)=-\theta z^{2}+2m(1+\theta )z-3m^{2}$, where $z=1-\eta$. Then the sign of $\frac{\partial P^{*}}{\partial \eta }$ is determined by the sign of $F(z)$. Note that the existence of the positive equilibrium point $E^{*}(N^{*},P^{*})$ implies that $m<1$ and $1-\frac{m}{\theta }<\eta <1-m$. Hence, we only consider $m< z<\frac{m}{\theta }$.

By calculation, $F(m)=-(1-\theta )m^{2}<0$ and $F(\frac{m}{\theta })=\frac{(1-\theta )m^{2}}{\theta }>0$. Solving the equation $F(z)=0$, we have $z_{1}=\frac{m(1+\theta )-m\sqrt{\theta ^{2}-\theta +1}}{\theta }$, where $m< z_{1}<\frac{m}{\theta }$. Then $F(z)<0$ for $m< z< z_{1}$ and $F(z)>0$ for $z_{1}< z<\frac{m}{\theta }$. Hence, we have

$$ \frac{\partial P^{*}}{\partial \eta }< 0 \quad\text{for all } 1-z_{1}< \eta < 1-m $$

and

$$ \frac{\partial P^{*}}{\partial \eta }>0 \quad\text{for all } 1-\frac{m}{\theta }< \eta < 1-z_{1}. $$

Therefore, when $1-z_{1}<\eta <1-m$ or $1-\frac{m}{\theta }<\eta <1-z_{1}$, the predator density is a strictly decreasing or increasing function with respect to prey refuge, respectively (see Fig. 2). With the increase of prey refuge, the density of the prey population increases. Then the prey refuge is benefit to the density of the prey population. However, when $1-z_{1}<\eta <1-m$, increasing the amount of prey refuge can protect more prey from predation. Due to lack of food resources, it leads to the decrease of the density of the predator population. When $1-\frac{m}{\theta }<\eta <1-z_{1}$, the predator density is an increasing function with respect to prey refuge. Then with the increase of the prey population, more and more preys have to leave the shelter. It leads to more food for the predator. Hence, the predator density is increasing.

Figure 2

The relation between positive equilibrium $E^{*}$ with prey refuge η

In the case without the prey refuge, that is $\eta =0$, system (5) is reduced to the system in [38]. The author [38] pointed out that both prey and predator converge to O for every initial conditions if $m\leq \theta$. However, with influence of prey refuge, it follows from Theorems 3 and 4 that the dynamics behavior of the system (5) becomes even more complicated than that of the corresponding system in[38]. Figure 3 shows the dynamics behavior of positive equilibrium $E^{*}$ in $(\eta,\theta )$ plane when $m<\theta <1$.

Figure 3

The regions of the existence and stability of positive equilibrium $E^{*}$ in $(\eta,\theta )$ plane when $m<\theta <1$

Now, we consider the dynamics behavior of the system (5) when $m<\theta$, that is the Allee effect is large enough. More precisely, if prey refuge η is larger than $1-m$, there does not exist the positive equilibrium $E^{*}$ (see Fig. 4(a)). That is, if the shelter is large enough, then it leads to less food resources for the predator. Hence, the predator is extinct. With the decrease of prey refuge, a stable positive equilibrium $E^{*}$ appears (see Fig. 4(b)). Then both prey and predator can survive. When prey refuge passes through $1-\frac{2m}{1+\theta }$, system (5) undergoes a supercritical Hopf bifurcation at positive equilibrium $E^{*}$ and there exists a stable limit cycle around $E^{*}$ (see Fig. 4(c)). When prey refuge is small enough, the positive equilibrium $E^{*}$ becomes unstable. Then both prey and predator are extinct (see Fig. 4(d)). Finally, if the prey refuge is less than $1-\frac{m}{\theta }$, the positive equilibrium $E^{*}$ disappears. Then system (5) is still extinct (see Fig. 4(e)). In this situation, the shelter is not enough for sustaining the prey and predator to survival.

Figure 4

Dynamic behavior of the system (5) with $\theta =0.6, \varepsilon =1, f=0.2, m=0.4$

4 Conclusion

In this paper, we consider a fear effect predator–prey model with the Allee effect and a prey refuge. By analyzing the characteristic equation of the corresponding linearized system and considering prey refuge as parameter, we obtain the threshold condition for the stability of the system (5). We show that prey refuge plays an important role in the dynamics of the system (5). Comparing with the corresponding predator–prey model without prey refuge [38], we find that the prey refuge can make the dynamics behavior of the system (5) more complicated.