Predator-prey dynamics with Allee effect in prey refuge

Abstract

In this paper we establish a predator-prey model with a refuge and an open habitat for prey. The Allee effect in a prey refuge and the environment carrying capacity of prey are considered. According to biology of prey and predator, fast and slow time scales are considered in some parameters. Based on two different time scales, the system is divided into a fast system and a slow system. Applying the singular perturbation techniques, we analyze the dynamics on the slow system. The stability analyses are performed, and the Hopf bifurcation occurs when the environment carrying capacity of prey is greater than a critical value. This value is an increasing function of the Allee effect. By calculating the first Lyapunov coefficient, the stable periodical oscillation is shown. It is shown that the carrying capacity of prey and the Allee effect of prey in the refuge can influence biological environment.

1 Introduction

Allee effect is a phenomenon in biology characterized by a positive correlation between population size or density and the mean individual fitness of a population or species. In general, these facilitative behaviors for Allee effect mechanisms include mate limitation, cooperative defense and environmental conditioning [1, 2]. Allee effects have been shown to be present in all major taxonomic groups of animals [3]. Examples include social spiders [4], meerkats [5], African wild dogs [6], white-winged choughs [7] and red-backed voles [8].

In [9], based on habitat-selection theory, Morris illustrated how qualitative and quantitative differences of habitats affect population growth. The result was that the growth rate of prey in an open habitat should be different from that in refuges. In 1996, Knight and Morris [10] observed an open wetland and a covered wetland (like a refuge) along the coasts of Hudson and James Bays in northern Canada to assess the habitat choices of red-backed voles. They found that there was higher density of voles in the open wetland than in the covered wetland. In predator-prey ecological systems, it is the same that there is more population in open habitats than in refuges for prey. Furthermore, given dispersal populations can be sustained in habitats with conditions outside the sink habitats (refuges). Without immigration, sink populations face extinction because deaths exceed births (e.g., because of unfavorable abiotic conditions, scant resources) [11]. It is natural that there is migration of prey between open habitat and refuges, and the movement rate is a constant [12].

In 2002, Morris performed some tests on red-backed voles and deer mice in Canada’s Rocky Mountains [8]. These tests revealed an Allee effect and suggested that the Allee effect occurs only at small population sizes for small mammals. Furthermore, Morris [9] pointed out that ecologists are likely to debate whether Allee effects are common or rare, but there can be little doubt that an Allee effect occurs at low population sizes. In [11] the authors also pointed out that populations at low density experience Allee effects in many species. Hence, we consider an Allee effect in prey refuges in a predator-prey system. In this paper we study the Allee effect and the environment carrying capacity of prey to gain insight on the impact of them on the dynamics of a predator-prey system.

The Allee effect has numerous impacts on population dynamics, distribution and conservation [4–8] and attracts much attention in biomathematics. In predator-prey systems, many authors have considered the Allee effect in prey [13–18]. In [13] the Allee effect is considered in both the richer habitat and the poorer habitat (refuge). Two models with refuge and without refuge are separated. The authors considered linear functional response. It is shown that the impact of evolution is enhanced by the availability of refuges and the Allee effect. In [14] the Allee effect and type III functional response are considered in a predator-prey system. It is shown that the Allee effect can promote system collapse. In [15], the authors extend the work of Auger [19] considering the Allee effect in prey population. The functional response is linear. At the slow time scale, saddle-node, supercritical Hopf and Bogdanov-Takens bifurcations caused by the Allee effect are found. In [18] the Allee effect is incorporated into a predator-prey model with Holling II type functional response. The authors found that the Allee effect of prey species increases the extinction risk of both predators and prey and can lead to unstable periodical oscillation.

According to [20], Holling II functional response may be more appropriate for homogeneous systems. Based on the observation of Knight [10], the tests of Morris [8, 9] and the results of [11], the Allee effect is more likely to occur in prey refuges. Previous predator-prey models did not consider either the Allee effect in prey refuges or Holling II functional response. Hence, we consider the Allee effect in prey refuges and Holling II functional response in this paper. The goal of this work is to study the impact of Allee effect in prey refuges and the environment carrying capacity of prey on the dynamics of the predator-prey system. In this paper we consider fast and slow time scales and apply the singular perturbation techniques to reduce the complete system to an aggregated model that describes the dynamics of the total number of prey and the number of predator at the slow time scale.

The paper is organized as follows. In Section 2 we establish a predator-prey model with Allee effect in prey refuge and Holling II type functional response. We separate fast and slow equations and carry out equilibria. In Section 3 we discuss the stability of equilibrium points in the slow dynamics. In Section 4, Hopf bifurcations are studied. In Section 5 we carry out numerical simulations. We end the paper with a brief discussion.

2 Modeling

We use the well-known Rosenzweig-MacArthur predator-prey model with logistic growth function of the prey and Holling II functional response. For predator, the reproductive rate responds only to the rate of prey killed per predator. Then we can set up the compartmental model as follows:

$$ \textstyle\begin{cases} \frac{dx_{1}}{dt} = C_{21} x_{2}-C_{12} x_{1}+r_{1}x_{1} (1-\frac{x_{1}}{K} )-\frac{b x_{1}y}{a+x_{1}}, \\ \frac{dx_{2}}{dt} =C_{12} x_{1}-C_{21}x_{2}+r_{2}x_{2} (1-\frac{x_{2}}{K} ) (\frac{x_{2}}{A}-1 ), \\ \frac{dy}{dt} = \frac{cb x_{1}y}{a+x_{1}}-dy. \end{cases} $$

(2.1)

Here $x_{1}$ represents the number of prey in an open habitat, $x_{2}$ represents the number of prey in a refuge, y represents the number of predators in the open habitat. In this model, we consider the Allee effect only in the equation of $x_{2}$.

All the parameters are listed in Table 1. If $bc\leq d$, predators can never grow, and then we assume $bc>d$.

Table 1 Parameters

It is a fact that the movement of prey is on a faster time scale than the growth and the death. According to the references [21–23], it is reasonable to consider two time scales for these parameters in the model. Let $r_{i}=\epsilon\tilde{r_{i}}$, $b=\epsilon\tilde{b}$, $d=\epsilon \tilde{d}$. Here $\epsilon\ll1$ is a small positive parameter, which means that movements have a larger speed than that associated to growth and death processes. We obtain the fast system

$$ \textstyle\begin{cases} \frac{dx_{1}}{dt} = C_{21} x_{2}-C_{12} x_{1}+\epsilon [ \tilde{r_{1}}x_{1} (1-\frac{x_{1}}{K} )- \frac{ \tilde{b} x_{1}y}{a+x_{1}} ], \\ \frac{dx_{2}}{dt} =C_{12} x_{1}-C_{21}x_{2}+ \epsilon [ \tilde{r_{2}}x_{2} (1- \frac{x_{2}}{K} ) (\frac {x_{2}}{A}-1 ) ], \\ \frac{dy}{dt} = \epsilon [\frac{c \tilde{b} x_{1}y}{a+x_{1}}- \tilde{d}y ]. \end{cases} $$

(2.2)

Let $x=x_{1}+x_{2}$ be the total number of prey, $u_{i}=\frac{x_{i}}{x}$ ($i=1,2$) be the proportions of prey in patch i. We rewrite the fast system as follows:

$$ \textstyle\begin{cases} \frac{du_{1}}{dt} = C_{21} u_{2}-C_{12} u_{1}+\epsilon [ \tilde{r_{1}}u_{1}(1-u_{1}) (1- \frac{u_{1}x}{K} )-(1-u_{1})\frac{\tilde{b} u_{1}y}{a+u_{1}x} \\ \hphantom{\frac{du_{1}}{dt} =}{} -\tilde{r_{2}}u_{1}u_{2} (1- \frac{u_{2}x}{K} ) (\frac{u_{2}x}{A}-1 ) ], \\ \frac{dx}{dt} =\epsilon x [\tilde{r_{1}}u_{1} (1-\frac{u_{1}x}{K} )+\tilde{r_{2}}u_{2} (1- \frac {u_{2}x}{K} ) (\frac{u_{2}x}{A}-1 )- \frac{\tilde{b} u_{1}y}{a+u_{1}x} ], \\ \frac{dy}{dt} = \epsilon y [\frac{c\tilde{b} u_{1}x}{a+u_{1}x}-\tilde{d} ]. \end{cases} $$

(2.3)

According to the method of [21], we put $\epsilon=0$ in (2.3) and obtain the equilibrium $(u^{*}_{1},u^{*}_{2})=(\frac {C_{21}}{C_{12}+C_{21}},\frac{C_{12}}{C_{12}+C_{21}})$ for the fast system. By replacing $u_{i}$ by $u^{*}_{i}$ ($i=1,2$) in the 2nd and 3rd equations of (2.3), we obtain the following two-dimensional system:

$$ \textstyle\begin{cases} \frac{dx}{dt} =x [ \tilde{r_{1}}u^{*}_{1} (1-\frac{u^{*}_{1}x}{K} )+\tilde {r_{2}}u^{*}_{2} (1- \frac{u^{*}_{2}x}{K} ) (\frac{u^{*}_{2}x}{A}-1 )-\frac{\tilde{b} u^{*}_{1}y}{a+u^{*}_{1}x} ], \\ \frac{dy}{dt} = y [\frac{\tilde{b}c u^{*}_{1}x}{a+u^{*}_{1}x}-\tilde{d} ]. \end{cases} $$

(2.4)

In this model, the purpose is to study the impact of Allee effect and the carrying capacity of prey. Here we only consider the particular case $C_{12}=C_{21}$. Then $u^{*}_{1}=u^{*}_{2}=\frac{1}{2}$ and system (2.4) is

$$ \textstyle\begin{cases} \frac{dx}{dt} =x [ \frac{\tilde{r_{1}}}{2} (1-\frac{x}{2K} )+\frac{\tilde{r_{2}}}{2} (1- \frac{x}{2K} ) (\frac{x}{2A}-1 )- \frac{\tilde{b} y}{2a+x} ], \\ \frac{dy}{dt} = y [\frac{\tilde{b}c x}{2a+x}-\tilde{d} ]. \end{cases} $$

(2.5)

Let the right-hand side of (2.5) equal to zero, we obtain

$$\textstyle\begin{cases} x =0,\qquad y= \frac{2a+x}{2\tilde{b}} (1- \frac{x}{2K} ) (\frac{\tilde {r_{2}}x}{2A}+\tilde{r_{1}}- \tilde{r_{2}} ); \\ y =0,\qquad x= \frac{2a\tilde{d}}{\tilde{b}c-\tilde{d}}. \end{cases} $$

Thus, the equilibrium $E_{0}=(0,0)$ always exists. For the equilibria without predator, it should satisfy the equation

$$\biggl(\frac{x}{2K}-1 \biggr) \biggl( \frac{\tilde{r_{2}}x}{2A}+\tilde{r_{1}}-\tilde{r_{2}} \biggr)=0. $$

If $\tilde{r_{1}}<\tilde{r_{2}}$, there are two boundary equilibria $E_{1}=(\frac{2A(\tilde{r_{2}}-\tilde{r_{1}})}{\tilde{r_{2}}},0)$ and $E_{2}=(2K,0)$.

If $\tilde{r_{1}}\geq\tilde{r_{2}}$, there is only one boundary equilibrium $E_{2}=(2K,0)$.

Denote $K_{1}=\frac{a\tilde{d}}{\tilde{b}c-\tilde{d}}$ and $A_{1}=\frac {\tilde{r_{2}}x^{*}}{2(\tilde{r_{2}}-\tilde{r_{1}})}$. Then the coexistence equilibrium $E^{*}=(x^{*},y^{*})$ exists under the condition $K>K_{1}$ and $A< A_{1}$ if $\tilde{r_{2}}>\tilde{r_{1}}$. Here,

$$\begin{aligned}& x^{*} =\frac{2a\tilde{d}}{\tilde{b}c-\tilde{d}}, \\& y^{*} =\frac{ac}{\tilde{b}c-\tilde{d}} \biggl(1-\frac{x^{*}}{2K} \biggr) \biggl( \frac{\tilde {r_{2}}x^{*}}{2A}+\tilde{r_{1}}-\tilde{r_{2}} \biggr). \end{aligned}$$

Next we will study the dynamics at the slow time scale.

3 Dynamics of model (2.5) for the slow system

We first calculate the Jacobian matrix for system (2.5)

$$J=\left ( \textstyle\begin{array}{@{}cc@{}} \frac{\tilde{r_{1}}-\tilde{r_{2}}}{2}+(\frac{\tilde{r_{2}}x}{2A}+\frac {\tilde{r_{2}}-\tilde{r_{1}}}{2K})x -\frac{3\tilde{r_{2}}}{8AK}x^{2}-\frac{2a\tilde{b}y}{(2a+x)^{2}} & -\frac{\tilde{b}x}{2a+x} \\ \frac{2a\tilde{b}cy}{(2a+ x)^{2}} & \frac{\tilde{b}cx}{2a+x}-\tilde{d} \end{array}\displaystyle \right ). $$

It is easy to calculate and obtain that

$$J_{E_{0}}=\left ( \textstyle\begin{array}{@{}cc@{}} \frac{\tilde{r_{1}}-\tilde{r_{2}}}{2} & 0\\ 0 & -\tilde{d} \end{array}\displaystyle \right ). $$

Then $E_{0}$ is a node point if $\tilde{r_{1}}<\tilde{r_{2}}$ and a saddle point if $\tilde{r_{1}}>\tilde{r_{2}}$, and a saddle-node bifurcation occurs at $\tilde{r_{1}}=\tilde{r_{2}}$.

For $E_{1}=(\frac{2A(\tilde{r_{2}}-\tilde{r_{1}})}{\tilde{r_{2}}},0)$ when $\tilde{r_{1}}<\tilde{r_{2}}$.

$$J_{E_{1}}=\left ( \textstyle\begin{array}{@{}cc@{}} a_{11} & a_{12} \\ 0 & a_{22} \end{array}\displaystyle \right ), $$

where

$$\begin{aligned}& a_{11} =\frac{\tilde{r_{2}}-\tilde{r_{1}}}{2} \biggl[1- \frac{A(\tilde{r_{2}}-\tilde {r_{1}})}{K\tilde{r_{2}}} \biggr], \\& a_{12} =-\frac{2A\tilde{b}(\tilde{r_{2}}-\tilde{r_{1}})}{2a\tilde {r_{2}}+2A(\tilde{r_{2}}-\tilde{r_{1}})}< 0, \\& a_{22} =\frac{A(\tilde{b}c-\tilde{d})(\tilde{r_{2}}-\tilde{r_{1}})-a\tilde {d}\tilde{r_{2}}}{a\tilde{r_{2}}+A(\tilde{r_{2}}-\tilde{r_{1}})}. \end{aligned}$$

Note that $A\leq K$ and $\tilde{r_{2}}-\tilde{r_{1}}<\tilde{r_{2}}$, then $a_{11}>0$. Hence, $E_{1}$ is an unstable source if $A>A_{1}$ and is a saddle if $A< A_{1}$, and a saddle-node bifurcation occurs at $A=A_{1}$.

For $E_{2}=(2K,0)$ when $\tilde{r_{1}}<\tilde{r_{2}}$.

$$J_{E_{2}}=\left ( \textstyle\begin{array}{@{}cc@{}} a_{11} & a_{12} \\ 0 & a_{22} \end{array}\displaystyle \right ), $$

where

$$\begin{aligned}& a_{11} =\frac{\tilde{r_{2}}-\tilde{r_{1}}}{2} \biggl[1- \frac{K\tilde{r_{2}}}{A(\tilde {r_{2}}-\tilde{r_{1}})} \biggr], \\& a_{22} =\frac{K(\tilde{b}c-\tilde{d})-a\tilde{d}}{a+K}. \end{aligned}$$

Note that $\frac{A(\tilde{r_{2}}-\tilde{r_{1}})}{K\tilde{r_{2}}}<1$, then $a_{11}<0$. Hence, $E_{2}$ is an unstable source if $K>K_{1}$ and is a saddle if $K< K_{1}$, and a saddle-node bifurcation occurs at $K=K_{1}$.

For $E_{2}=(2K,0)$ when $\tilde{r_{1}}\geq\tilde{r_{2}}$. In this case $a_{11}<0$. Hence, $E_{2}$ is an unstable source if $K>K_{1}$ and is a saddle if $K< K_{1}$, and a saddle-node bifurcation occurs at $K=K_{1}$.

For $E^{*}=(x^{*},y^{*})$,

$$J_{E^{*}}=\left ( \textstyle\begin{array}{@{}cc@{}} a_{11} & a_{12} \\ a_{21} & 0 \end{array}\displaystyle \right ), $$

where

$$\begin{aligned} a_{12} &=\frac{\tilde{d}}{c}< 0, \\ a_{21} &=\frac{(\tilde{b}c-\tilde{d})^{2}}{2a\tilde{b}c}y^{*}>0. \end{aligned} $$

The characteristic equation of $J_{E^{*}}$ is

$$\lambda^{2}-a_{11}\lambda+\frac{\tilde{d}}{c}a_{21}=0. $$

Here,

$$\begin{aligned} a_{11} =&\frac{\tilde{r_{1}}-\tilde{r_{2}}}{2}+ \biggl(\frac{\tilde {r_{2}}x}{2A}+ \frac{\tilde{r_{2}}-\tilde{r_{1}}}{2K} \biggr)x^{*} -\frac{3\tilde{r_{2}}}{8AK} \bigl(x^{*} \bigr)^{2} \\ &{}-\frac{\tilde{b}c-\tilde{d}}{\tilde{b}c} \biggl[\frac{\tilde{r_{1}}-\tilde{r_{2}}}{2}+ \biggl(\frac{\tilde{r_{2}}x}{4A}+ \frac {\tilde{r_{2}}-\tilde{r_{1}}}{4K} \biggr)x^{*} -\frac{\tilde{r_{2}}}{8AK} \bigl(x^{*} \bigr)^{2} \biggr] \\ =&-\frac{x^{*}}{4K} \biggl[\frac{\tilde{b}c+\tilde{d}}{\tilde{b}c} \biggl(\frac {\tilde{r_{2}}x^{*}}{2A}+ \tilde{r_{1}}-\tilde{r_{2}} \biggr) +\frac{\tilde{r_{2}}x^{*}}{2A} \biggr] \\ &{}+\frac{1}{2} \biggl[\frac{\tilde{d}}{\tilde{b}c} \biggl(\frac{\tilde {r_{2}}x^{*}}{2A}+ \tilde{r_{1}}-\tilde{r_{2}} \biggr) +\frac{\tilde{r_{2}}x^{*}}{2A} \biggr]. \end{aligned}$$

Note the condition of existence of $E^{*}$ is $A<\frac{\tilde {r_{2}}x^{*}}{2(\tilde{r_{2}}-\tilde{r_{1}})}$. Then $\frac{\tilde{r_{2}}x^{*}}{2A}+\tilde{r_{1}}-\tilde{r_{2}}>0$. Denote

$$K_{2}=\frac{a\tilde{d}}{\tilde{b}c-\tilde{d}} \biggl[1 +\frac{\frac{\tilde{r_{2}}x^{*}}{2A}+\tilde{r_{1}}-\tilde{r_{2}}}{ \frac{\tilde{d}}{\tilde{b}c}(\frac{\tilde{r_{2}}x^{*}}{2A}+\tilde {r_{1}}-\tilde{r_{2}})+\frac{\tilde{r_{2}}x^{*}}{2A}} \biggr]>K_{1}>0. $$

Then

$$K>K_{2} \quad \Leftrightarrow\quad a_{11}>0. $$

Hence, $E^{*}$ is a source if $K>K_{2}$ and $E^{*}$ is a sink if $K< K_{2}$.

The existence and stability of equilibria can be summered in the following Theorem 3.1.

Theorem 3.1

• (1) $E_{0}=(0,0)$ always exists and is a node point if $\tilde {r_{1}}<\tilde{r_{2}}$ and a saddle point if $\tilde{r_{1}}>\tilde{r_{2}}$, and a saddle-node bifurcation occurs at $\tilde{r_{1}}=\tilde{r_{2}}$.

• (2) $E_{1}=(\frac{2A(\tilde{r_{2}}-\tilde{r_{1}})}{\tilde{r_{2}}},0)$ exists when $\tilde{r_{1}}<\tilde{r_{2}}$ and is an unstable source if $A>A_{1}$ and is a saddle if $A< A_{1}$, and a saddle-node bifurcation occurs at $A=A_{1}$.

• (3) $E_{2}=(2K,0)$ always exists and is an unstable source if $K>K_{1}$ and is a saddle if $K< K_{1}$, and a saddle-node bifurcation occurs at $K=K_{1}$.

• (4) $E^{*}=(x^{*},y^{*})$ exists under the condition $K>K_{1}$ and $A< A_{1}$ if $\tilde{r_{2}}>\tilde{r_{1}}$, and is a source if $K>K_{2}$ and is a sink if $K_{1}< K< K_{2}$.

4 Hopf bifurcation analysis

In this section we study the dynamical behavior of $E^{*}$ when $K=K_{2}$. The characteristic equation of $J_{E^{*}}$ is

$$\lambda^{2}-a_{11}\lambda+\frac{\tilde{d}}{c}a_{21}=0. $$

The roots are

$$\lambda_{1,2}=\frac{1}{2}a_{11}\pm \frac{1}{2}\sqrt{a^{2}_{11}- \frac {4\tilde{d}}{c}a_{21}}:=\mu(K)\pm i\omega(K). $$

If $K=K_{2}$, $a_{11}=0$, and then the eigenvalues are $\lambda_{1,2}=\pm i\omega$ and the Hopf bifurcation will occur. In this case,

$$\omega^{2}=\frac{\tilde{d}}{c}a_{21}= \frac{\tilde{d}(\tilde{b}c-\tilde {d})^{2}}{2a\tilde{b}c^{2}}y^{*}. $$

Denote $P=\frac{\tilde{r_{2}}x^{*}}{2A}+\tilde{r_{1}}-\tilde{r_{2}}$. The transversality condition is satisfied:

$$\mu{'}(K)=\frac{x^{*}}{8K^{2}} \biggl[\frac{\tilde{b}c+\tilde{d}}{\tilde{b}c} P+ \frac{\tilde{r_{2}}x^{*}}{2A} \biggr]>0. $$

Now we calculate the first Lyapunov coefficient. Rewrite the coordinate of $E_{2}$ when $K=K_{2}$:

$$\bigl(x^{0},y^{0} \bigr) = \biggl( \frac{2a\tilde{d}}{\tilde{b}c-\tilde{d}}, \frac{acP}{(\tilde{b}c-\tilde{d})(\frac{\tilde{d}}{\tilde{b}c}+\frac {P-r}{P}+1)} \biggr). $$

Note that the slow system (2.5) is

$$ \textstyle\begin{cases} \dot{x} = \frac{x}{2} (1- \frac{x}{2K_{2}} ) (\frac{\tilde {r_{2}}x}{2A}+ \tilde{r_{1}}- \tilde{r_{2}} ) (2a+ x)-\tilde{b} x y, \\ \dot{y} = -\tilde{d}(2a+x) y +\tilde{b}cx y. \end{cases} $$

(4.1)

Translate the origin of the coordinates to this equilibrium by the change of variables

$$\textstyle\begin{cases} x = x^{0}+\xi_{1}, \\ y = y^{0}+\xi_{2}. \end{cases} $$

Since $a_{11}=0$, $\frac{\tilde{r_{2}}x^{0}}{4A}(1-\frac{x^{0}}{2K_{2}})= \frac{1}{2\tilde{b}c}(\frac{\tilde{b}c+\tilde{d}}{2K_{2}}x^{0}-\tilde{d})P$. From $\tilde{b}y^{0}=\frac{1}{2}(2a+x^{0})(1-\frac{x^{0}}{2K_{2}})P$, we obtain the new system

$$ \textstyle\begin{cases} \dot{\xi_{1}} = - \tilde{b}x^{0}\xi_{2}+a_{20}\xi_{1}^{2}- \tilde{b}\xi_{1}\xi _{2}+a_{30} \xi^{3}_{1}+o (\vert \xi \vert ^{4} ), \\ \dot{\xi_{2}} =(\tilde{b}c-\tilde{d})y^{0} \xi_{1}+(\tilde{b}c-\tilde{d})\xi_{1}\xi_{2}. \end{cases} $$

(4.2)

Denote $r=\tilde{r_{1}}-\tilde{r_{2}}$ and then $P>r$. Here,

$$\begin{aligned}& a_{20} = -\frac{1}{2\tilde{d}[(2\tilde{b}c+\tilde{d})P-\tilde{b}c r]} \bigl[ \bigl( \tilde{b}^{2}c^{2}+ \tilde{b}c\tilde{d}+\tilde{d}^{2} \bigr)P^{2}+\tilde {b}c(2 \tilde{b}c+\tilde{d})Pr+\tilde{b}^{2}c^{2}r^{2} \bigr]< 0, \\& a_{30} = -\frac{\tilde{b}c-\tilde{d}}{4a\tilde{d}^{2}[(2\tilde {b}c+\tilde{d})P-\tilde{b}c r]} \bigl[ \bigl(\tilde{b}^{2}c^{2}+2 \tilde{b}c\tilde{d}+2\tilde{d}^{2} \bigr)P^{2} \\& \hphantom{a_{30} =}{}-(\tilde {b}c+ \tilde{d}) (2\tilde{b}c+\tilde{d})Pr +\tilde{b}c( \tilde{b}c+\tilde{d})r^{2} \bigr] \\& \hphantom{a_{30} } = -\frac{\tilde{b}c-\tilde{d}}{4a\tilde{d}^{2}[(2\tilde{b}c+\tilde {d})P-\tilde{b}c r]} \bigl\{ \bigl[(\tilde{b}c+\tilde{d})P- \tilde{b}cr \bigr]( \tilde{b}c+\tilde {d}) (P-r)+\tilde{d}^{2}P^{2} \bigr\} < 0. \end{aligned}$$

This system can be represented as

$$\dot{\xi}=B\xi+\frac{1}{2}C(\xi,\xi)+\frac{1}{6}D(\xi, \xi,\xi)+o \bigl(\vert \xi \vert ^{4} \bigr), $$

where $B = B(K)$, and the multilinear functions C and D take on the planar vectors $\xi=(\xi_{1},\xi_{2})$, $\eta=(\eta_{1},\eta_{2})$ and $\zeta=(\zeta_{1},\zeta_{2})$ the values

$$C(\xi,\eta)=\left ( \textstyle\begin{array}{@{}c@{}} 2a_{20}\xi_{1}\eta_{1}-\tilde{b}(\xi_{1}\eta_{2}+\xi_{2}\eta_{1}) \\ (\tilde{b}c-\tilde{d})(\xi_{1}\eta_{2}+\xi_{2}\eta_{1}) \end{array}\displaystyle \right ), $$

and

$$D(\xi,\eta,\zeta)=\left ( \textstyle\begin{array}{@{}c@{}} -6a_{30}\xi_{1}\eta_{1}\zeta_{1} \\ 0 \end{array}\displaystyle \right ). $$

Write the matrix $B(K)$ in the form

$$B=\left ( \textstyle\begin{array}{@{}cc@{}} 0& -\tilde{b}x^{0}\\ (\tilde{b}c-\tilde{d})y^{0} & 0 \end{array}\displaystyle \right ). $$

The eigenvalues of B are

$$\lambda_{1}=\frac{2a\tilde{b}c}{\tilde{b}c-\tilde{d}}\omega i,\qquad \lambda_{2}=- \frac{2a\tilde{b}c}{\tilde{b}c-\tilde{d}}\omega i. $$

From

$$Bq=\lambda_{2}q,\qquad B^{T}p=\lambda_{1}p, $$

we can calculate that the eigenvectors are

$$q=\left ( \textstyle\begin{array}{@{}c@{}} \tilde{d} \\ i \omega c \end{array}\displaystyle \right ),\qquad p= \left ( \textstyle\begin{array}{@{}c@{}} \omega c\\ i\tilde{d} \end{array}\displaystyle \right ). $$

To achieve the necessary normalization $\langle p,q\rangle=1$, we can take, for example,

$$q=\left ( \textstyle\begin{array}{@{}c@{}} \tilde{d} \\ i \omega c \end{array}\displaystyle \right ),\qquad p= \frac{1}{2\omega c \tilde{d}}\left ( \textstyle\begin{array}{@{}c@{}} \omega c\\ i\tilde{d} \end{array}\displaystyle \right )=\left ( \textstyle\begin{array}{@{}c@{}} \frac{1}{2\tilde{d}}\\ i\frac{1}{2\omega c } \end{array}\displaystyle \right ). $$

And we calculate

$$C(q,q)=\left ( \textstyle\begin{array}{@{}c@{}} 2a_{20}\tilde{d}^{2}-\tilde{b}(2\omega c \tilde{d}i) \\ (\tilde{b}c-\tilde{d})(2\omega c \tilde{d}i) \end{array}\displaystyle \right ), $$

$$C(q,\bar{q})=\left ( \textstyle\begin{array}{@{}c@{}} 2a_{20}\tilde{d}^{2} \\ 0 \end{array}\displaystyle \right ), $$

and

$$D(q,q,\bar{q})=\left ( \textstyle\begin{array}{@{}c@{}} -6a_{30}\tilde{d}^{3} \\ 0 \end{array}\displaystyle \right ). $$

Then we obtain

$$\begin{aligned}& g_{20} =\bigl\langle p,C(q,q)\bigr\rangle =a_{20} \tilde{d}+(\tilde{b}c-\tilde{d})\tilde{d}-\tilde {d}c\omega i, \\& g_{11} =\bigl\langle p,C(q,\bar{q})\bigr\rangle =a_{20} \tilde{d}, \\& g_{21} =\bigl\langle p,C(q,q,\bar{q})\bigr\rangle =3a_{30} \tilde{d}^{2}. \end{aligned}$$

The first Lyapunov coefficient is

$$\begin{aligned} l(K)&=\frac{1}{2\omega^{2}}Re(ig_{20}g_{11}+ \omega g_{21}) \\ &=\frac{\omega\tilde{d}^{2}}{2\omega^{2}}(ca_{20}+a_{30}) \\ &< 0. \end{aligned} $$

Therefore, a unique and stable limit cycle bifurcates from the equilibrium $E^{*}$ via the Hopf bifurcation for $K>K_{2}$. This result is different from the result in [18].

Theorem 4.1

If $K>K_{2}$, a unique and stable limit cycle bifurcates from the equilibrium $E^{*}$ via the Hopf bifurcation.

5 Numerical simulation

Now we perform some simulations for the dynamics of the slow system (Figures 1-4) and the original system (Figure 4). According to some literature works, the values of parameters are listed in Table 2. We choose $a=10$, $\tilde{b}=0.5$, $c=0.5$, $\tilde{d}=0.07$. Then $K_{1}=3.89$. Some simulations are performed by Maple software (Figures 1-4).

Figure 1

Hopf bifurcation occurs. $\epsilon=0.1$ and $\tilde{r}_{1}=0.2$, $\tilde{r}_{2}=0.1$, $K=7.505$, $A=1$. Then $\tilde{r}_{1}>\tilde{r}_{2}$ and $K=K_{2}>K_{1}$.

Table 2 The value of parameters

If we increase the environment carrying capacity of prey, Figure 2 shows that the period of predator oscillation is very long. In additional, a different Allee effect of prey in a refuge can lead to stable oscillation or stable endemic equilibrium state (Figure 3).

Figure 2

Comparison of oscillation with different  K .

Figure 3

Comparison of oscillation with different  A .

At last we compare the dynamics of the original system with the slow system when ϵ is small (Figure 4). From Figure 4, we can see that the Hopf bifurcation occurs in both the original system and the slow system. The dynamics are similar, which also shows that the full dynamical system can be characterized by the dynamics on the slow manifold.

Figure 4

Hopf bifurcation occurs in both the original system and the slow system.

6 Discussion

In this article we conducted mathematical analyses of a predator-prey model with Allee effect in prey refuge and Holling II type functional response. According to biology of prey and predator, fast and slow time scales are considered. Applying the singular perturbation techniques, we separate the dynamics of the model into two time scales. Based on theoretical analyses and numerical simulations (Figure 4), we show that the full dynamical system can be characterized by the dynamics on the slow manifold in the long run. Then we analyze the stability of the system on the slow time scale. Our results show that when the carrying capacity for the prey population K is greater than some value $K_{2}$, the Hopf bifurcation will occur. According to Figure 2, if the carrying capacity for the prey is ensured abundant, the period will be longer and the amount of predators fluctuates greatly. In additional, reducing the Allee effect of prey will lead to stable periodical oscillation (Figure 3), which is different from the result of [18].

In nature, the growth rate of prey in a refuge should be less than that in an open habitat because of limitation of mating. According to Theorem 3.1, this difference will lead to the equilibria without predator, and the equilibria without predator is unstable. This means that biological environment without predator is unstable in nature, which is in good agreement with some practical phenomena. For example [27], Kaiba forest located in the northern margin of the Colorado Grand Canyon, Arizona. In 1925, to protect the black tailed deer, people killed all the natural enemies of the forest. The number of deer in the forest began to increase sharply to 100,000, but by 1942 the number reduced to 8,000. The reason is that the original ecosystem had been destroyed, more than its stability threshold, and thus became unstable.

Furthermore, when the carrying capacity of prey (K) is bigger than a critical value ($K_{1}$), the coexistence equilibrium with prey and predator is stable, i.e., animals and their natural enemies can coexist. When the carrying capacity of prey is less than the critical value, prey and predator can not coexist. This can also explain why a lot of animals and natural enemies coexist in nature [27, 28]. In 1930s, Hawaii’s snail disaster is a very good example. In the Hawaii islands, the big snails were bred in a small region and eaten by human beings. Some old snails were thrown in the wild, which meant the carrying capacity of snail increased. A few years later, a large number of breeding snails were everywhere. This meant that snails and humans coexisted. People sprayed chemicals and cultivated land for 15 consecutive years, but also could not remove the snails [27]. According to this conclusion, human beings can control the amount of prey and predator by adjusting the carrying capacity of prey. Based on these analyses, it is easy to see that the carrying capacity of prey and the Allee effect of prey in the refuge can influence biological environment. In addition, in this model the migration rate of prey can affect the number of equilibria and the stability. However, the purpose of this paper was to study the impact of Allee effect and the carrying capacity of prey. Hence, we have only considered the particular case in this paper, and the general case will be considered in the future.