The dynamics of a Leslie type predator–prey model with fear and Allee effect

Abstract

In this article, we discuss the dynamics of a Leslie–Gower ratio-dependent predator–prey model incorporating fear in the prey population. Moreover, the Allee effect in the predator growth is added into account from both biological and mathematical points of view. We explore the influence of the Allee and fear effect on the existence of all positive equilibria. Furthermore, the local stability properties and possible bifurcation behaviors of the proposed system about positive equilibria are discussed with the help of trace and determinant values of the Jacobian matrix. With the help of Sotomayor’s theorem, the conditions for existence of saddle-node bifurcation are derived. Also, we show that the proposed system admits limit cycle dynamics, and its stability is discussed with the value of first Lyapunov coefficient. Moreover, the numerical simulations including phase portrait, one- and two-parameter bifurcation diagrams are performed to validate our important findings.

1 Introduction

One of the prominent topics for ecologists, biologists, and even mathematicians is the study of various mechanisms related to the interaction between prey and predators. The typical model of such interactions between two species was first proposed in the form of a simplest mathematical model [1, 2]. The key factor in a predator–prey relationship is that the growth of predator depends on its functional response to the prey population, which describes the amount of prey each predator eats per unit of time. Many researchers have developed and implemented various forms of interaction terms to understand the more realistic situation. Thus the predation function depends on prey only such as Holling type classification [3, 4], whereas ratio-dependent [5], Beddington [6], and Crowley Martin [7] are the functions of both prey and predator. In real-world situations, [8, 9] clearly confirm that the ratio-dependent interaction has a greater description of predation rates than the prey-dependent interaction. The authors in [10] assumed that the predator growth function is different from the predator’s predation function, and its reduction has a reciprocal relationship with per capita availability of its preferred food, the so-called Leslie type predator–prey model. Despite its rich and varying complexity, there has been greater focus paid by several researchers to modify and build the Leslie type model in several ways [7, 11, 12]. In some real situations, fractional-order systems are found to be more accurate than integer-order ones. For this reason, a fractional-order predator–prey system has been investigated by several researchers, and numerous outstanding results have been derived, for example, see [13, 14].

The Allee effect is one of the most interesting phenomena in ecology, which is characterized by a positive correlation between the population size and the per capita growth rate for a population [15]. The Allee effect can be caused by various environmental factors including difficulties in finding mating partners at low density, genetic inbreeding, social felicitation of reproduction, low probability of successful mating, depletion in inbreeding rate, etc. Recently, there has been greater interest from many researchers to study the effect of incorporating the Allee effect in the predator–prey dynamics. In the ratio-dependent predator–prey model, the authors in [16] compared the dynamical properties with and without the Allee effect in logistic prey growth term and showed that the inclusion of Allee effect can remove oscillation behavior of species densities. Further, the conditions for occurrence of various bifurcation behaviors were derived and verified numerically with the help of parametric diagrams and phase portraits. The same model with discrete time was investigated in [17], and the authors showed that varying integral step size can arise chaotic orbits via period-doubling and invariant circle. A Leslie type model with ratio-dependent type interaction term with state impulsive feedback control was considered, and the conditions for existence, uniqueness, and orbital asymptotical stability of periodic solution of order-1 were obtained by applying the geometry theory of differential equations [18]. The authors [19] considered a strong Allee effect in the prey growth function in the Holling type II model with density-dependent death rate for predators and examined various types of bifurcation behavior in the considered model. Since there have been fewer studies related to the double Allee effect in the predator–prey system, in [20] the authors attended the presence of double Allee effect in the ratio-dependent predator–prey system in the prey growth function, where they studied various types of bifurcation behavior and showed bi-stability behavior in the phase plane implying that the considered system dynamics is sensitive in choosing the initial conditions.

Predators will actively affect the ecosystem and the prey reproduction process by consuming them and by affecting the behavior and physiology of the survivor indirectly [21]. Compared to direct predation, these forms of behavioral and physiological modifications related to fear of predation are more efficient and longer-lasting evolutionary effects [21, 22]. For starters, birds respond with anti-predator defences to the sounds of predators and flee from their nests at the first sign of danger [22]. Fear of song sparrows during the breeding season by eliminating direct predation, recognizing that the understanding of predation risk alone reduces the amount of offspring produced by 40 percent per year [23]. Recently, many mathematical models have been studied with effect of fear in the prey population due to anti-predator defense, see for details [24–26].

The authors in [27] showed that the predator–prey model with Beddington–DeAngelis type interaction has both stabilizing and destabilizing effect on incorporating the fear term in the prey. In [28] the effect of fear of predator in the prey–predator model was investigated and the existence, stability of equilibria, and occurrence of limit cycle in the considered system were studied. The authors in [29] studied the dynamics of cost of fear and the Allee effect in the predator–prey model, also with disease in the prey population, while the dynamics of a prey-predator system with fear and group defense was considered in [30]. The predator populations in the natural world are more vulnerable to Allee effects than their prey, and because they are bigger, populations are more likely to undergo an Allee effect (predator populations are usually much smaller than prey populations). The model with Allee effect in the predator population was considered in [31, 32]. Since there have been various dynamics and bifurcation behavior arising in the nonlinear population model, in [33] the authors attempted to study the predator–prey system with nonlinear harvesting and showed that the considered system has unstable periodic orbit by finding the value of first Lyapunov coefficient. They also showed the existence of saddle-node and transcritical bifurcation by Sotomayor’s theorem, and Bogdanov–Takens bifurcation by finding universal unfolding near cusp. To understand the dynamics of fear and Allee effect, more works are required, and this fact has motivated our current study.

Encouraged by the above discussion, we aim to investigate the combined effect of incorporating both fear and Allee effect in a Leslie type predator–prey model with ratio-dependent type interaction. To the best of our knowledge, this assumption has not been studied. We are interested in a way the system exhibits various complex dynamical behaviors. The main highlights of this article are summarized as follows:

• The predator–prey model with fear effect in prey growth term and Allee effect in predator growth term is considered and analyzed.

• The proposed models utilize the fact that predators have to go for alternative food when there is low density of their favorite food.

• To confirm the long term coexistence of species of the proposed model, permanence analysis is performed.

• The impact of introduced parameters on system dynamics is investigated through different types of bifurcations.

• By taking fear and Allee effect constant as a bifurcation parameter, the proposed theoretical results are verified through numerical simulations.

This article is organized as follows: In Sect. 2, we describe the formation of the system along with its assumptions, and some preliminaries of the system are given in Sect. 3. In Sect. 4, we address the existence of the system’s equilibria and local stability with numerical examples. In Sect. 5, we explore the complex dynamical behavior of our system by varying the Allee parameter with a numerical example. We give numerical simulations in Sect. 6, and the conclusion is given in the last section.

2 The mathematical model

The mathematical model of the modified Leslie–Gower predator–prey model as in [11, 12] is given by

$$\begin{aligned} &\frac{dx}{dt}=xf_{1}(x,K_{1})-g(x,y)y, \end{aligned}$$

(2.1a)

$$\begin{aligned} &\frac{dy}{dt}=s y \biggl(1-\frac{y}{n x+c} \biggr), \end{aligned}$$

(2.1b)

where $x(t)$, $y(t)$ are the prey and predator biomass at time t, $f_{1}(x,K_{1})$ is a function representing the gross rate of increase in the prey population with carrying capacity $K_{1}$ in the lack of predators and satisfying $f_{1}(0,K_{1})>0$, $f_{1}(K_{1},K_{1})=0$, $\frac{\partial f_{1}}{\partial x}\leq 0$, $\frac{\partial f_{1}}{\partial t}|_{x=K_{1}}<0$, and $\frac{\partial f_{1}}{\partial K_{1}}>0$. And nx in (2.1b) is the prey-dependent carrying capacity of the predator, and it is assumed that c is a constant taken for the alternative food source, which in turns means that the predator proportion is small compared to the alternative food source. We consider the logistic growth of the form $f_{1}(x,K_{1})=r_{0}-e-b x$, with $K_{1}=\frac{r_{0}-e}{b}$. Also, as in [5], the ratio-dependent (ratio between prey and predator) in Michaelis–Menten type interaction is implemented, which is of the form $g (\frac{x}{y} )=\frac{p x}{x+ a y}$, where $p,a$ are predators searching level and handling time spent for each prey. Then (2.1a) takes the form

$$\begin{aligned} \frac{dx}{dt}&=r_{0} x f_{2}(k,y)-e x-b x^{2}-\frac{p x y}{x+ a y}, \end{aligned}$$

(2.2)

where $r_{0},e$, and b are positive constants, representing the birth rate, death rate, and death rate caused by intra-species competition. The dynamical analysis of a Leslie–Gower model with ratio-dependent interaction is performed in [34]. Since the prey production is indirectly influenced by the fear effect [35], we also attempted to include the fear effect in the prey growth term in (2.2). Hence, we multiply the function $f_{2}(k,y)=\frac{1}{1+k y}$ with the birth rate $r_{0}$, which is the cost of anti-predator defense due to fear [26], and for other fruitful results see [27–29]. Here, the quantity k measures the fear level on prey, and by the biological meanings of k, y, and $f_{2}(x,y)$, it is appropriate to take that

$$\begin{aligned} &f_{2}(0,y)=1,\qquad f_{2}(k,0)=1,\qquad \lim_{k\to \infty }f_{2}(k,y)=0, \\ &\lim_{y\to \infty }f_{2}(k,y)=0,\qquad \frac{\partial f_{2} (k,y)}{\partial k}< 0,\qquad \frac{\partial f_{2}(k,y)}{\partial y}< 0. \end{aligned}$$

Then (2.2) with fear term is of the following form:

$$\begin{aligned} \frac{dx}{dt}=\frac{r_{0} x}{1+k y}-e x-b x^{2}-\frac{p x y}{x+ a y}. \end{aligned}$$

(2.3)

Since there have been fewer studies related to the presence of Allee effect on the predator population, in [12] the authors attempted a Leslie–Gower model with Holling type interaction model and studied the dynamics with effect to the Allee effect in both prey and predator, respectively. Hence, it is reasonable and more realistic to take the Allee effect into account for the predator population. We arrive at the following model with fear in the prey population and the Allee effect in the predator population:

$$\begin{aligned} &\frac{dx}{dt}=\frac{r_{0} x}{1+k y}-e x-b x^{2}- \frac{p x y}{x+a y}:=g_{1}(x,y), \end{aligned}$$

(2.4a)

$$\begin{aligned} &\frac{dy}{dt}=s y \biggl(\frac{y}{y+m}-\frac{ y}{n x+c} \biggr):=g_{2}(x,y), \\ &\quad x(0)>0, y(0)>0, \end{aligned}$$

(2.4b)

where $\frac{y}{y+m}$ is the term for the Allee effect and m is the Allee effect constant. The per capita growth rate of the predator is reduced from s to $\frac{sy}{y+m}$. The bigger the m, the stronger the Allee effect, the slower the per capita growth rate of the predator population, and we always assume H1 $r_{0}>e$, i.e., the reproduction rate of prey is always greater than its death rate.

3 Preliminaries

3.1 Positivity

Biologically, the survival of the population is positivity. We follow Theorem A.4 in [36] to prove the positivity of system (2.4a)–(2.4b). We define the state space of (2.4a)–(2.4b) as $X= \{ (x,y)\in \mathcal{R}^{2}_{+} \}$ due to the biological meaning. Now the functions $g_{1}(\cdot,\cdot ),g_{2}(\cdot,\cdot )$ are continuously differentiable and locally Lipschitz in X. Then, by Theorem A.4 in [36], we conclude that, for a sufficiently large number S, the solution of the initial value problem (2.4a)–(2.4b) exists and is unique in the interval $[0,S)$.

3.2 Boundedness

Boundedness, whether there is a well-specified beginning or end of a case. As a consequence of limited resources, boundaries can be viewed as a natural constraint on growth. To prove the boundedness of system (2.4a)–(2.4b), we use the lemma of Chen [37].

Lemma 3.1

(Chen [37])

Assume that $\epsilon,\eta >0$ with $u(0)>0$. Then, for $\frac{du}{dt}\leq u(t)(\epsilon -\eta u(t))$, we have

$$\begin{aligned} \lim_{t\to \infty }\sup u(t)\leq \frac{\epsilon }{\eta }, \end{aligned}$$

and for $\frac{du}{dt}\geq u(t)(\epsilon -\eta u(t))$, we have

$$\begin{aligned} \lim_{t\to \infty } u(t)\geq \frac{\epsilon }{\eta }. \end{aligned}$$

Proposition 3.1

Assume that $r_{0}>e$ and $\frac{r_{0}-e}{b}+n+\epsilon _{1}>q m$, where $\epsilon _{1}>0$ is an arbitrary small number. Then all the solutions $(x(t),y(t))$ of system (2.4a)–(2.4b) with the initial conditions $x(0)>0$, $y(0)>0$ are bounded.

Proof

From (2.4a), we have

$$\begin{aligned} \frac{dx}{dt}&=\frac{r_{0} x}{1+k y}-e x-b x^{2}-\frac{p x y}{x+a y} \\ &\leq x(r_{0}-e-b x). \end{aligned}$$

Since $r_{0}>e$ by H1, applying Lemma 3.1, we obtain

$$\begin{aligned} \lim_{t\to \infty }\sup x(t)\leq \frac{r_{0}-e}{b}. \end{aligned}$$

(3.1)

Thus, there exists $\tilde{T_{1}}\in \mathbb{R}$, $\tilde{T_{1}}>0$ for some arbitrary constant $\epsilon _{1}>0$ such that

$$\begin{aligned} x(t)\leq \frac{r_{0}-e}{b}+\epsilon _{1}, \quad\forall t> \tilde{T_{1}}. \end{aligned}$$

Also, (2.4b) can be written as

$$\begin{aligned} \frac{dy}{dt}=\frac{n x+c-m-y}{(y+m)(n x+c)}. \end{aligned}$$

Thus, for all $t>\tilde{T_{1}}$, we have

$$\begin{aligned} \frac{dy}{dt}\leq \frac{s y^{2}}{m c} \biggl(\frac{n(r_{0}-e)}{b}+c- m + \epsilon _{1}- y \biggr). \end{aligned}$$

Assume that $\frac{n(r_{0}-e)}{b}+c+\epsilon _{1}> m$. Then, again by Lemma 3.1, we have

$$\begin{aligned} \lim_{t\to \infty }\sup y(t)\leq \frac{n(r_{0}-e)}{b }+c+ \epsilon _{1}-m. \end{aligned}$$

(3.2)

Thus, $\tilde{T_{2}}>\tilde{T_{1}}$, $\tilde{T_{2}}\in \mathbb{R}$ for some arbitrary constant $\epsilon _{2}>0$ such that

$$\begin{aligned} y(t)\leq \frac{n(r_{0}-e)}{b }+c+\epsilon _{1}-m+\epsilon _{2},\quad \forall t>\tilde{T_{2}}. \end{aligned}$$

Therefore, the solutions of system (2.4a)–(2.4b) are bounded. □

3.3 Permanence

The system’s permanence is important as it ensures that the species present now will not be extinct in the future. System (2.4a)–(2.4b) is said to be permanent if there exist positive constants $M_{3}$ and $M_{4}$ such that each positive solution $(x(t,x_{0},y_{0}),y(t,x_{0},y_{0}))$ of system (2.4a)–(2.4b) with the initial condition $(x_{0},y_{0})\in int (\mathbb{R}_{+}^{2})$ satisfies

$$\begin{aligned} &\min \Bigl\{ \lim_{t\to \infty } \inf x(t,x_{0},y_{0}), \lim_{t\to \infty }\inf y(t,x_{0},y_{0}) \Bigr\} \geq M_{3}, \\ &\max \Bigl\{ \lim_{t\to \infty } \inf x(t,x_{0},y_{0}), \lim_{t\to \infty }\inf y(t,x_{0},y_{0}) \Bigr\} \leq M_{4}. \end{aligned}$$

Proposition 3.2

System (2.4a)–(2.4b) with the initial conditions $x(0)>0$, $y(0)>0$ is permanent if $\frac{r_{0}}{1+k M_{2}}-e-p>0$ and $n-m q>0$.

Proof

From (2.4a), we have

$$\begin{aligned} \frac{dx}{dt}&\geq x \biggl[\frac{r_{0}}{1+k M_{2}}-e-b x-p \biggr] \end{aligned}$$

(3.3)

$$\begin{aligned} &\geq x(w_{0}- b x), \end{aligned}$$

(3.4)

where $M_{2}$ is the upper bound of y population and $w_{0}=\frac{r_{0}}{1+k M_{2}}-e-p$.

Assume that $w_{0}>0$. Then, by applying Lemma 3.1, we obtain

$$\begin{aligned} \lim_{t\to \infty }\inf x(t)\geq \frac{\omega _{0}}{b}. \end{aligned}$$

Also, from (2.4b), we have

$$\begin{aligned} \frac{dy}{dt}&\geq s y^{2} \biggl(\frac{1}{y+m}- \frac{1}{c} \biggr) \\ &\geq \frac{s y^{2}}{m c}(c-m - y). \end{aligned}$$

Assume that $c-m >0$, again by using Lemma 3.1, we obtain

$$\begin{aligned} \lim_{t\to \infty }\inf y(t)\geq c-m. \end{aligned}$$

Choose $M_{3}=\min \{ \frac{\omega _{0}}{b},c-m \}$, and from (3.1) and (3.2), choose $M_{4}=\max \{ \frac{r_{0}-e}{b}, \frac{n(r_{0}-e)}{b }+c+ \epsilon _{1}-m \}$. Then the condition of permanence of system (2.4a)–(2.4b) follows. □

4 Equilibria and local stability analysis

4.1 Existence of equilibria

In order to obtain the equilibrium points of system (2.4a)–(2.4b), we consider the following equations:

$$\begin{aligned} \frac{r_{0} }{1+k y}-e -b x-\frac{p y}{x+a y}=0 \quad\text{and}\quad \frac{1}{y+m}-\frac{1 }{n x+c}=0. \end{aligned}$$

(4.1)

Simple computation yields the following equilibrium points of system (2.4a)–(2.4b):

• Trivial equilibrium $E_{0}(0,0)$ exists by [38].

• Predator free equilibrium $\bar{E}(\bar{x},0)=(\frac{r_{0}-e}{b},0)$ always exists by H1.

• Prey free equilibrium $\hat{E}(0,\hat{y})=(0,c-m)$ exists if and only if $c-m>0$.

• Interior equilibrium $E^{*}(x^{*},y^{*})$, where $x^{*}=\frac{y^{*}+m-c}{n}$, and $y^{*}$ is the positive root of the cubic equation

  $$\begin{aligned} {y^{*}}^{3}+3 A_{1} {y^{*}}^{2}+3 A_{2} y^{*}+ A_{3}=0, \end{aligned}$$

  (4.2)

  where

  $$\begin{aligned} &A_{1}= \frac{b k (a n+2) (m-c)+(a n+1) (b+e k n)+k n^{2} p}{3 b k (a n+1)}, \\ &A_{2}= \frac{p n^{2} + k b (m - c)^{2} + (2 b + e k n + a b n) (m - c) + n (1 + a n) (e - r)}{3 k b (1 + a n)}, \\ &A_{3}=\frac{b (m - c)^{2} + n (e - r) (m - c)}{k b (1 + a n)}. \end{aligned}$$

By introducing a transformation $z=y^{*}+A_{1}$, (4.2) can be reduced to

$$\begin{aligned} l(z)=z^{3}+3 p_{1} z+p_{2}=0, \end{aligned}$$

(4.3)

where $p_{1}=A_{2}-A_{1}^{2}$ and $p_{2}=A_{3}-3 A_{1} A_{2}+2 A_{1}^{3}$. By [39], the existence of the positive roots of (4.3) can be stated in the following lemma.

Lemma 4.1

Existence of positive roots of (4.3):

• (a) If $p_{2}<0$, (4.3) has a single positive root;

• (b) Suppose $p_{2}>0$ and $p_{1}<0$, then:

  • (1) If $p_{2}^{2}+4 p_{1}^{3}>0$, (4.3) has no positive roots;

  • (2) If $p_{2}^{2}+4 p_{1}^{3}=0$, (4.3) has a positive root of multiplicity two;

  • (3) If $p_{2}^{2}+4 p_{1}^{3}<0$, (4.3) has two positive roots;

• (c) If $p_{2}=0$ and $p_{1}<0$, (4.3) has a unique positive root.

Moreover, algebraic computations show that (4.3) has two positive roots $z_{1}= \frac{\sqrt[3]{(-4 p_{2}+4 \sqrt{4 p_{1}^{3}+p_{2}^{2}})^{2}}-4 p_{1}}{2 \sqrt[3]{-4 p_{2}+4 \sqrt{4 p_{1}^{3}+p_{2}^{2}}}}$ and $z_{2}=-\frac{z_{1}}{2}+ \frac{\sqrt{z_{1}^{3}+4 p_{2}}}{2 \sqrt{z_{1}}}$. It is noted that if (4.3) has one positive root, it must be the same as $z_{1}$. Suppose $p_{2}>0,p_{1}<0$, and using the Descartes rule of sign change, we discuss the existence of possible interior equilibria of system (2.4a)–(2.4b) in the following cases.

Case: (i) If $m< c$, i.e., $A_{1}>0$, (4.2) has almost two positive roots, then by Lemma 4.1, model (2.4a)–(2.4b) has

• (1) No positive interior equilibrium by Lemma 4.1.

• (2) A unique positive interior equilibrium point of multiplicity two, i.e., $E_{1}=(x_{1},y_{1})=(\frac{z_{1,2}-A_{1}+m-c}{n}, z_{1,2}-A_{1})$ with $z_{1,2}>A_{1}$ and $y_{1}>c-m$. In this case, $z_{1}=z_{2}$.

• (3) Two positive interior equilibrium points, namely $E_{2,3}=(x_{2,3},y_{2,3})=(\frac{z_{1,2}-A_{1}+m-c}{n}, z_{1,2}-A_{1})$ with $z_{1,2}>A_{1}$.

Case: (ii) If $m=c$ shows $A_{1}>0,A_{3}=0$, then one root becomes zero and (4.2) has at most one positive root, then the interior equilibrium is $E_{4}=(x_{4},y_{4})=(\frac{z_{1}-A_{1}+m-c}{n},z_{1}-A_{1})$ with $z_{1}>A_{1}$ and $m -c=0$.

Case: (iii) If $m>c$ implies $A_{1}>0$ and (4.2) has a unique positive root, then the interior equilibrium is $E_{5}=(x_{5},y_{5})=(\frac{z_{1}-A_{1}+m-c}{n},z_{1}-A_{1})$.

Note that the Allee effect term m plays an important role for the existence of interior equilibrium points. Moreover, if $m< c$, the two equilibria $E_{2}$ and $E_{3}$ collide to get $E_{1}$ and disappear if m crosses some particular threshold value, it states possible saddle-node bifurcation around $E_{1}$. Furthermore, if $m=c$, then one of the equilibria $E_{2},E_{3}$ will disappear and one will collide with $E_{0}$ another with $E_{4}$, then there may be possible existence of transcritical bifurcation around $E_{0}$.

Hence, the expression for the critical threshold value m for the above scenario of the existence of positive equilibria is difficult to find, so we discuss it with the help of numerical example and nullcline analysis.

Example 4.1

For system (2.4a)–(2.4b), consider the fixed parameter values $e=0.05$, $b=1$, $p=1$, $s=0.5$, $a=1$ $c=0.2$, $n=1$, $r_{0}=1$, $k=1$ and varying m. If $m=0.03$, system (2.4a)–(2.4b) has $\hat{E}(0,0.17)$ and no positive interior equilibrium point (given in Fig. 1(a)) and, if $m=0.08586597105258$, we get the unique positive interior equilibrium $E_{1}(0.07208,0.18621)$ with $\hat{E}(0,0.114134)$, and for $m=0.12$, we get two positive interior equilibria $E_{2}(0.156899, 0.236899)$, $E_{3}(0.018512,0.098512)$ with $\hat{E}(0,0.08)$. If $m=c=0.2$, we have a unique positive interior equilibrium $E_{4}(0.25,0.25)$, both Ê and $E_{3}$ collide with $E_{0}$, and for $m=0.3>c$, $E_{3}$ disappears and we have $E_{5}(0.341366,0.241366)$. Finally, in all cases we have $E_{0}(0,0)$ and $\bar{E}(0.95,0)$. This is shown in Fig. 1(b). Since the value of a fear function decreases as the level of fear increases as in [40], the parameter k also has the same capability for the possible existence of saddle-node bifurcation, see Fig. 1(c). The saddle-node bifurcation plot for varying m is given in Fig. 2(a) and for k in Fig. 2(b). The region for existence of interior equilibria in $k-m$ plane is given in Fig. 2(c).

Figure 1

The blue line represents prey nullcline and the black line represents predator nullcline, (a) when $k=1$, $m=0.03$, there is no interior equilibrium point. (b) Fixing $k=1$ and varying m shows that system (2.4a)–(2.4b) has a unique equilibrium $E_{1}$ in blue dot, two interior equilibria $E_{2}$, $E_{3}$ in black dot, one equilibrium $E_{3}$ coincides with $E_{0}$ and has a unique interior equilibrium $E_{4}$ in red dot for $m=c$, $E_{3}$ disappears and $E_{5}$ is given in red dot for $m>c$. (c) Taking $m=0.08586$, the scenario for the existence of no equilibria, unique equilibrium, and two equilibria by varying fear parameter k

Figure 2

(a) Saddle-node bifurcation diagram by fixing $k=1$ and m varying in (0, 0.2), the blue dotted line represents stable $E_{2}$ and the red line represents unstable $E_{3}$. (b) Saddle-node bifurcation diagram by fixing $m=0.1$ and k varying in (0, 1.3). (c) The black dark curve represents $p_{2}^{2}+4 p_{1}^{3}=0$ which separates Region I (no interior equilibria) and II (two interior equilibria), thed blue dashed line represents $m=c$ which separates Region II and Region III (unique interior equilibria)

Thus, the trace(Tr) and determinant (Det) values of the Jacobian matrix play a prominent role in studying the local stability of system (2.4a)–(2.4b), we discuss the local stability in the following subsection.

4.2 Local stability analysis

The Jacobian matrix of the system at any equilibrium point $(x,y)$ is given by

$$\begin{aligned} &J= \begin{pmatrix} \frac{r_{0}}{1+k y}-e-2 b x-\frac{a p y^{2}}{(x+ a y)^{2}}&\frac{-r_{0} k x}{(1+k y)^{2}}- \frac{p x^{2}}{(x+a y)^{2}} \\ \frac{s n y^{2}}{(n x+c)^{2}}& \frac{s y^{2}+2 m s y}{(y+m)^{2}}- \frac{2 s y}{n x+c} \end{pmatrix}. \end{aligned}$$

(4.4)

Note that the Jacobian matrix cannot be evaluated at $E_{0}(0,0)$. To study the stability of system (2.4a)–(2.4b) at $E_{0}$, we transform the ratio-dependent predator–prey system to an equivalent system by making use of transformation $x=x$, $y=\frac{y}{x}$, then the resultant system takes the form:

$$\begin{aligned} &\frac{dx}{dt}=\frac{r_{0} x}{1+ k y x}-e x-b x^{2}- \frac{p y x}{1+a y}, \end{aligned}$$

(4.5)

$$\begin{aligned} &\frac{dy}{dt}=s y \biggl(\frac{ y x}{y x+m}-\frac{ y x}{n x+c} \biggr)-y \biggl(\frac{r_{0}}{1+k y x}-e-b x-\frac{p y}{1+a y} \biggr). \end{aligned}$$

(4.6)

Thus, the above system has the same equilibrium $E_{0}=(0,0)$ and axial equilibrium $\bar{E}(\bar{x},0)=(\frac{r_{0}-e}{b},0)$ after re-transfer to its original variables. The Jacobian matrix of system (2.4a)–(2.4b) at $E_{0}$ from (4.5) is given as

$$\begin{aligned} J_{E_{0}}= \begin{pmatrix} r_{0}-e & 0 \\ 0&-r_{0}+e \end{pmatrix}. \end{aligned}$$

The Jacobian matrix of system (2.4a)–(2.4b) at the axial equilibria points Ē and Ê is given by

$$\begin{aligned} &J_{\bar{E}}= \begin{pmatrix} -r_{0}+e&-\frac{k(r_{0}-e)}{b}-p \\ 0&0 \end{pmatrix}, \qquad J_{\hat{E}}= \begin{pmatrix} 0&0 \\ \frac{s n (c-m)^{2}}{c^{2}}&-\frac{s(c-m)^{2}}{c^{2}} \end{pmatrix}. \end{aligned}$$

Since the sign of the eigenvalues of $J_{E_{0}},J_{\bar{E}},J_{\hat{E}}$ determines the local stability of system (2.4a)–(2.4b) around the equilibrium points $E_{0}$, Ē, Ê, thus the conditions for local stability around the $E_{0}$, Ē, Ê are summarized in the following proposition.

Proposition 4.1

Assume that H1 holds, then $E_{0}$ is saddle, since the eigenvalues of $J_{E_{0}}$ are $\lambda _{1}=r_{0}-e>0$ and $\lambda _{2}=-r_{0}+e<0$. Also the eigenvalues of $J_{\bar{E}}$ are $\lambda _{1}=-r_{0}+e<0$ and $\lambda _{2}=0$, thus Ē always has stable manifold along the x-axis. Further $c-m>0$ implies the existence of Ê, so if Ê exists then one of the eigenvalues of $J_{\hat{E}}$ is $\lambda _{2}=-\frac{s(c-m)^{2}}{c^{2}}<0$ and another one $\lambda _{1}=0$, then Ê always has stable manifold along y-axis. And if $m=c$, then Ê collides with $E_{0}$.

Let $E^{*}(x^{*},y^{*})$ be an arbitrary interior equilibrium whose Jacobian matrix is given as follows:

$$\begin{aligned} J_{E^{*}}= \begin{pmatrix} -b x^{*}+\frac{p x^{*} y^{*}}{(x^{*}+a y^{*})^{2}}&- \frac{r_{0} k x^{*}}{(1+k y^{*})^{2}}- \frac{p {x^{*}}^{2}}{(x^{*}+a y^{*})^{2}} \\ \frac{s n {y^{*}}^{2}}{(n x^{*}+c)^{2}}&- \frac{s {y^{*}}^{2}}{(y^{*}+m)^{2}} \end{pmatrix}. \end{aligned}$$

(4.7)

Thus, the characteristic equation of the above matrix is

$$\begin{aligned} \lambda ^{2}+\operatorname{Tr}(J_{E^{*}})\lambda +\operatorname{Det}(J_{E^{*}})=0, \end{aligned}$$

(4.8)

where

$$\begin{aligned} &\operatorname{Tr}(J_{E^{*}})=x^{*} \biggl(-b + \frac{p y^{*}}{(x^{*}+a y^{*})^{2}} \biggr)-\frac{s {y^{*}}^{2}}{(y^{*}+m)^{2}}, \\ &\operatorname{Det}(J_{E^{*}})=s x^{*} {y^{*}}^{2} \biggl[ \biggl(b - \frac{p y^{*}}{(x^{*}+a y^{*})^{2}} \biggr) \biggl( \frac{1}{(y^{*}+m)^{2}} \biggr) \\ &\phantom{\operatorname{Det}(J_{E^{*}})=}{}+ \biggl(\frac{r_{0} k}{(1+k y^{*})^{2}}+ \frac{p x^{*}}{(x^{*}+a y^{*})^{2}} \biggr) \biggl( \frac{n}{(n x^{*}+c)^{2}} \biggr) \biggr]. \end{aligned}$$

Thus, $E^{*}$ is locally asymptotically stable if it satisfies Routh–Hurwitz criteria that are $\operatorname{Tr}(J_{E^{*}})<0$ and $\operatorname{Det}(J_{E^{*}})>0$. Thus, we have the following proposition.

Proposition 4.2

The arbitrary interior equilibrium point $E^{*}$ of system (2.4a)–(2.4b) is locally asymptotically stable if $x^{*} (-b +\frac{p y^{*}}{(x^{*}+a y^{*})^{2}} )< \frac{s {y^{*}}^{2}}{(y^{*}+m)^{2}}$ and $-b +\frac{p y^{*}}{(x^{*}+a y^{*})^{2}}>0$.

Here, we give an example for the local stability of system (2.4a)–(2.4b).

Example 4.2

We chose the parameter values as given in Example 4.1 and vary the parameters m and k. If we choose $k=2$ and $m=0.17$, we have two interior equilibrium points $E_{2}(0.144371,0.174371)$ and $E_{3}(0.00471853,0.0347185)$, and their corresponding trace and determinant values are $\operatorname{Tr}(J_{E_{2}})=-0.0247796<0$, $\operatorname{Det}(J_{E_{2}})=0.0333904>0$ and $\operatorname{Tr}(J_{E_{3}})=0.0862325>0$, $\operatorname{Det}(J_{E_{3}})=0.00112236<0$. Then the equilibrium $E_{2}$ is locally asymptotically stable, while $E_{3}$ is unstable. Moreover, $E_{2}$ satisfies Proposition 4.1, which is $0.103414=x_{2} (-b +\frac{p y_{2}}{(x_{2}+a y_{2})^{2}} )< \frac{s {y_{2}}^{2}}{(y_{2}+m)^{2}}=0.128194$ and $-b +\frac{p y_{2}}{(x_{2}+ a y_{2})^{2}}=0.716306>0$, i.e., $E_{2}$ is locally asymptotically stable (see Fig. 6(a)).

5 Bifurcation analysis

In this section, we focus on the possible existence of various bifurcation behaviors of system (2.4a)–(2.4b). In the nonlinear population interaction model, the most common phenomenon is the existence of periodic solutions around the equilibrium point, that is, the local birth or death of a periodic solution, the so-called Hopf bifurcation. Moreover, the two branches of stable and unstable equilibria created collide and disappear and are defined as saddle-node bifurcation. Finally, the exchange of stability properties of two equilibria is known as transcritical bifurcation. Hence, the bifurcation behavior of system (2.4a)–(2.4b) is discussed in what follows.

5.1 Saddle-node bifurcation

Theorem 5.1

Suppose $\operatorname{Det}(J_{E^{*}})|_{E_{1}}=0$ for some critical threshold $m=m_{s}$, then system (2.4a)–(2.4b) admits a saddle-node bifurcation at $E_{1}$.

Proof

By taking m as the bifurcation parameter, we use Sotomayor’s theorem as in [41] to prove that system (2.4a)–(2.4b) admits a saddle-node bifurcation. According to [41], since $\operatorname{Det}(J_{E_{1}})=\lambda _{1} \lambda _{2}=0$, either of the eigenvalues must be zero and another is negative, also $\operatorname{Tr}(J_{E_{1}})<0$. Let $g=(g_{1},g_{2})^{T}$, where $g_{1}$, $g_{2}$ are given in Sect. 2. Then the matrix $J_{E^{*}}$ at $E_{1}(x_{1},y_{1})$ is written by

$$\begin{aligned} J_{E_{1}}=Dg(x_{1},y_{1})= \begin{pmatrix} (-b x_{1}+\frac{p x_{1} y_{1}}{(x_{1}+a y_{1})^{2}} )& (-\frac{r_{0} k x_{1}}{(1+k y_{1})^{2}}- \frac{p {x_{1}}^{2}}{(x_{1}+a y_{1})^{2}} ) \\ \frac{s n {y_{1}}^{2}}{(n x_{1}+c)^{2}}&- \frac{s {y_{1}}^{2}}{(y_{1}+m_{s})^{2}} \end{pmatrix}. \end{aligned}$$

Let $m=m_{s}$ be the threshold value for which $J_{E_{1}}$ has the eigenvalue zero, demands $\operatorname{Det}(J_{E_{1}})=0$ at $m=m_{s}$. Also, let the matrices $J_{E_{1}}$ and ${J_{E_{1}}}^{T}$ have the eigenvectors $v=(v_{1},v_{2})^{T}$ and $w=(w_{1},w_{2})^{T}$ for the zero eigenvalue, which yields

$$\begin{aligned} v=& \biggl(\frac{1}{(y_{1}+m_{s})^{2}},\frac{n}{(n x_{1}+c)^{2}} \biggr)^{T}, \qquad w= \biggl(\frac{-s n {y_{1}}^{2}}{(n x_{1}+c)^{2}},-b x_{1}+ \frac{p x_{1} y_{1}}{(x_{1}+a y_{1})^{2}} \biggr)^{T}. \end{aligned}$$

Furthermore, we can get $g_m(E_1;m_s)=\left(\begin{array}{c}0\\ \frac{-s{y_1}^2}{{(y_1+m_s)}^2}\end{array}\right)$.

Now

$$\begin{aligned} \Omega _{1}=& W^{T} g_{m}(E_{1};m_{s}) = \biggl(-b x_{1}+ \frac{p x_{1} y_{1}}{(x_{1}+a y_{1})^{2}} \biggr) \biggl( \frac{-s {y_{1}}^{2}}{(y_{1}+m_{s})^{2}} \biggr). \end{aligned}$$

(5.1)

Therefore, $\Omega _{1}\neq 0$ at $m=m_{s}$, and

$$\begin{aligned} D^{2} g(E_{1};m_{s}) (v,v)={}& \begin{pmatrix} \frac{\partial ^{2} g_{1}}{\partial x^{2}}v_{1}^{2}+2 \frac{\partial ^{2} g_{1}}{\partial x\, \partial y}v_{1} v_{2}+ \frac{\partial ^{2} g_{1}}{\partial y^{2}}v_{2}^{2} \\ \frac{\partial ^{2} g_{2}}{\partial x^{2}}v_{1}^{2}+2 \frac{\partial ^{2} g_{2}}{\partial\, x \partial y}v_{1} v_{2}+ \frac{\partial ^{2} g_{2}}{\partial y^{2}}v_{2}^{2} \end{pmatrix} \\ ={}& \begin{pmatrix} \alpha _{1} v_{1}^{2} + \alpha _{2} v_{1} v_{2}+ \alpha _{3} v_{2}^{2} \\ \alpha _{4} v_{1}^{2} + \alpha _{5} v_{1} v_{2}+ \alpha _{6} v_{2}^{2} \end{pmatrix}, \end{aligned}$$

where

$$\begin{aligned} &\alpha _{1}=-2 b+\frac{a p y_{1}^{2}}{(x_{1}+a y_{1})^{3}},\qquad \alpha _{2}=- \frac{r_{0} k}{(1+k y_{1})^{2}}- \frac{2 p a x_{1} y_{1}}{(x_{1}+a y_{1})^{3}}, \qquad \alpha _{4}=- \frac{2 s n^{2} y_{1}^{2}}{(n x_{1}+c)^{3}}, \\ &\alpha _{3}=\frac{2 r_{0} k^{2} x_{1}}{(1+k y_{1})^{3}}+ \frac{2 p a x_{1}^{2}}{(x_{1}+a y_{1})^{3}},\qquad \alpha _{5}=0,\qquad \alpha _{6}=- \frac{2 s m_{s} y_{1}}{(y_{1}+m_{s})^{3}}, \end{aligned}$$

also

$$\begin{aligned} \Omega _{2}=W^{T} D^{2} g(E_{1};m_{s}) (v,v)= \begin{pmatrix} w_{1} & w_{2} \end{pmatrix} \begin{pmatrix} \alpha _{1} v_{1}^{2} + \alpha _{2} v_{1} v_{2}+ \alpha _{3} v_{2}^{2} \\ \alpha _{4} v_{1}^{2} + \alpha _{5} v_{1} v_{2}+ \alpha _{6} v_{2}^{2} \end{pmatrix}\neq 0. \end{aligned}$$

(5.2)

Thus, system (2.4a)–(2.4b) admits a saddle-node bifurcation around $E_{1}(x_{1},y_{1})$, if $\Omega _{1}\neq 0$ and $\Omega _{2}\neq 0$ by Sotomayor’s theorem [41]. To confirm $\Omega _{1},\Omega _{2}\neq 0$, we calculated it numerically. Therefore, we can say that the number of interior equilibria of system (2.4a)–(2.4b) changes from zero to two, when the value of m varies from one side of threshold value $m=m_{s}$ to the other side. □

Example 5.1

Let us take the parameter values as in Example 4.1, when $k=1$ the two equilibria $E_{2}, E_{3}$ collide at some critical magnitude of m, i.e., $m=m_{s}=0.08586597105258$ and a unique equilibrium $E_{1}$ emerges and disappears for $m<0.08586597105258=m_{s}$, which ensures the possible existence of saddle-node bifurcation near $E_{1}$ (see Fig. 2(a)). In addition, by [41] the values $\Omega _{1}=-0.0302379\neq 0$, $\Omega _{2}=7.70607\neq 0$ confirm that system (2.4a)–(2.4b) undergoes saddle-node bifurcation near $E_{1}$.

5.2 Transcritical bifurcation

System (2.4a)–(2.4b) admits transcritical bifurcation, if there occurs a possible exchange of stability properties between equilibria. From Sect. 4, system (2.4a)–(2.4b) has the boundary equilibria $\hat{E}(0,\hat{y})$ and $\bar{E}(\bar{x},0)$. Also, when $m=c$, $E_{2}$ and Ê coincide with $E_{0}$. Note that if Ê and Ē exist, then they are always stable and also $E_{0}$ is always a saddle by Proposition 4.1. Then, there is no chance of exchange of stability properties between equilibria.

Remark 5.1

System (2.4a)–(2.4b) does not admit transcritical bifurcation behavior.

5.3 Hopf bifurcation

In this subsection, the possible occurrence of limit cycle dynamics (Hopf bifurcation) near the arbitrary interior equilibrium $E^{*}(x^{*},y^{*})$ is to be analyzed by varying m and fixing all other parameters as constant.

Theorem 5.2

The necessary and sufficient conditions for system (2.4a)–(2.4b) undergoing Hopf bifurcation at $m=m_{h}$ around the interior equilibrium point $E^{*}$ are that $\operatorname{Tr}(J_{E^{*}})|_{m=m_{h}}=0$, $\operatorname{Det}(J_{E^{*}})|_{m=m_{h}}>0$, and $\frac{d(\operatorname{Tr}(J_{E^{*}}))}{dm}|_{m=m_{h}} \neq 0$.

Proof

To prove that system (2.4a)–(2.4b) exhibits limit cycle dynamics, we need to consider that at some threshold value $m=m_{h}$ the trace value of $J_{E^{*}}$ should be zero. The threshold value $m_{h}$ is calculated by

$$\begin{aligned} x^{*}(m) \biggl(-b +\frac{p y^{*}(m)}{(x^{*}(m)+a y^{*}(m))^{2}} \biggr)-\frac{s {y^{*}(m)}^{2}}{ (y^{*}(m)+m)^{2}}\bigg|_{m=m_{h}}=0. \end{aligned}$$

Thus, the characteristic equation (4.8) in Sect. 4 becomes

$$\begin{aligned} \lambda ^{2}+\operatorname{Det}(J_{E^{*}})|_{m=m_{h}}=0. \end{aligned}$$

(5.3)

Now, equation (5.3) must have the eigenvalues $\lambda _{1,2}=\pm i\omega _{0}$, where $\omega _{0}=\sqrt{\operatorname{Det}(J_{E^{*}}) |_{m=m_{h}}}$, with $\operatorname{Det}(J_{E^{*}})>0$ at $m=m_{h}$. Next, we shall verify the transversality condition at some value of $m=m_{h}$, $\lambda _{1,2}=\alpha (m)\pm \beta (m)$, where $\alpha (m)=\frac{1}{2}(x^{*}(m),y^{*}(m))$ and

$$\begin{aligned} \beta (m)=\sqrt{\operatorname{Det}\bigl(x^{*}(m),y^{*}(m)\bigr)-\frac{1}{4}\operatorname{Tr}^{2} \bigl(x^{*}(m),y^{*}(m)\bigr)}. \end{aligned}$$

Now, $\frac{d}{dm}\alpha (m)|_{m=m_{h}}=\frac{1}{2} [\frac{d}{d m}\operatorname{Tr}(x^{*}(m),y^{*}(m)) ]_{m=m_{h}}$.

Thus, the transversality condition is satisfied if $\frac{d}{dm}\operatorname{Tr}(x^{*}(m),y^{*}(m))\neq 0$ at $m=m_{h}$. Therefore, the system undergoes Hopf bifurcation at $m=m_{h}$. □

5.3.1 Direction and stability

To study the direction of the possible occurrence of limit cycle oscillation and its stability nature originating around the arbitrary equilibrium $E^{*}$, we derive the first Lyapunov coefficient ($l_{1}$) and use the results in [42].

Let the translation $u=x-x^{*}$ and $v=y-y^{*}$, and transform $E^{*}$ of system (2.4a)–(2.4b) into origin, we get

$$\begin{aligned} \textstyle\begin{cases} \frac{du}{dt}= \frac{r_{0} (u+x^{*})}{1+k (v+y^{*})}-e (u+x^{*})-b (u+x^{*})^{2}- \frac{p (u+x^{*}) (v+y^{*})}{(u+x^{*})+a (v+y^{*})}, \\ \frac{dv}{dt}=s (v+y^{*}) ( \frac{(v+y^{*})}{(v+y^{*})+m}- \frac{ (v+y^{*})}{n(u+x^{*})+c} ). \end{cases}\displaystyle \end{aligned}$$

(5.4)

On expanding (5.4) using Taylor’s series expansion at $(u,v)=(0,0)$, we obtain

$$\begin{aligned} \textstyle\begin{cases} \dot{u}=a_{10} u+a_{01} v+a_{20} u^{2}+a_{11} u v+a_{02} v^{2}+a_{30} u^{3} \\ \phantom{\dot{u}=}{}+a_{21}u^{2} v+a_{12} u v^{2}+a_{03} v^{3}+O( \vert u,v \vert ^{4}), \\ \dot{v}=b_{10} u+b_{01} v+b_{20} u^{2}+b_{11} u v+b_{02} v^{2}+b_{30} u^{3} \\ \phantom{\dot{v}=}{}+b_{21}u^{2} v+b_{12} u v^{2}+b_{03} v^{3}+O( \vert u,v \vert ^{4}), \end{cases}\displaystyle \end{aligned}$$

(5.5)

where

$$\begin{aligned} &a_{10}=-b x^{*}+\frac{p x^{*} y^{*}}{(x^{*}+a y^{*})^{2}},\qquad a_{01}=- \frac{r_{0} k x^{*}}{(1+k y^{*})^{2}}- \frac{p {x^{*}}^{2}}{(x^{*}+a y^{*})^{2}}, \\ &a_{20}=-b+\frac{a p {y^{*}}^{2}}{2(x^{*}+a y^{*})^{3}},\qquad a_{11}=- \frac{r_{0} k}{(1+k y^{*})^{2}}- \frac{2 a p x^{*} y^{*}}{(x^{*}+a y^{*})^{3}}, \\ &a_{02}=\frac{ r_{0} k^{2} x^{*}}{(1+k y^{*})^{3}}+ \frac{ a p {x^{*}}^{2}}{(x^{*}+a y^{*})^{3}},\qquad a_{30}=- \frac{ a p {y^{*}}^{2}}{2(x^{*}+a y^{*})^{4}}, \\ &a_{21}= \frac{2 a p x^{*} y^{*}- p a^{2} {y^{*}}^{2}}{(x^{*}+a y^{*})^{4}},\qquad a_{12}=\frac{ r_{0} k^{2}}{(1+k y^{*})^{3}}+ \frac{2 a^{2} p x^{*} y^{*}-a p {x^{*}}^{2}}{(x^{*}+a y^{*})^{4}}, \\ &a_{03}=-\frac{ r_{0} k^{3} x^{*}}{(1+k y^{*})^{4}}- \frac{a p {x^{*}}^{2}}{(x^{*}+a y^{*})^{4}},\qquad b_{10}= \frac{s n {y^{*}}^{2}}{(n x^{*}+c)^{2}}, \qquad b_{01}=-\frac{s {y^{*}}^{2}}{(y^{*}+m)^{2}}, \\ &b_{20}=-\frac{2 s n^{2} {y^{*}}^{2}}{(n x^{*}+c)^{3}}, \qquad b_{11}=0,\qquad b_{02}=- \frac{2 s m y^{*}}{(y^{*}+m)^{3}}, \\ &b_{30}=\frac{s n^{3} {y^{*}}^{2}}{(n x^{*}+c)^{2}} b_{21}=0, \qquad b_{12}=0,\qquad b_{03}=\frac{2 s m y^{*}- s m^{2}}{3(y^{*}+m)^{4}}. \end{aligned}$$

We rewrite system (5.5) after removing the terms of degree above 3, which takes the form

$$\begin{aligned} \dot{U}=J_{E^{*}} U+B(U), \end{aligned}$$

(5.6)

where

$$\begin{aligned} U= \begin{pmatrix} u \\ v \end{pmatrix}\quad \text{and}\quad B= \begin{pmatrix} B_{1} \\ B_{2}\end{pmatrix}= \begin{pmatrix} a_{20}u^{2}+a_{11}u v+a_{02}v^{2}+a_{30}u^{2} \\ +a_{21}u^{2} v+a_{12}u v^{2}+a_{03} v^{3} \\ b_{20}u^{2}+b_{11}u v+b_{02}v^{2}+b_{30}u^{2} \\ +b_{21}u^{2} v+b_{12}u v^{2}+b_{03} v^{3}\end{pmatrix}. \end{aligned}$$

Assume $i\theta _{0}$ to be the eigenvalue of $J_{E^{*}}$ and its eigenvector is given by $\hat{v}=(a_{01,i\theta _{0}-a_{10}})^{T}$. Now define

$$\begin{aligned} Q=\bigl(\operatorname{Re}\bigl(\hat{v},-\operatorname{Im}(\hat{v})\bigr)\bigr)= \begin{pmatrix} a_{01}&0 \\ -a_{10}&-\theta _{0}\end{pmatrix}. \end{aligned}$$

Let us use the transformation $U=Q V$ or $V=Q^{-1}U$, where $V=(v_{1},v_{2})^{T}$ in (2.4a)–(2.4b), we obtain

$$\begin{aligned} \dot{V}=\bigl(Q^{-1} J_{E^{*}} Q\bigr)V+Q^{-1}B(QV). \end{aligned}$$

Thus, the above system takes the form

$$\begin{aligned} \begin{pmatrix} \dot{v_{1}} \\ \dot{v_{2}}\end{pmatrix}= \begin{pmatrix} 0 &-\theta _{0} \\ \theta _{0}&0\end{pmatrix} \begin{pmatrix} v_{1} \\ v_{2}\end{pmatrix}+ \begin{pmatrix} S^{1}(v_{1},v_{2};m=m_{h}) \\ S^{2}(v_{1},v_{2};m=m_{h})\end{pmatrix}, \end{aligned}$$

(5.7)

where $S^{1}$ and $S^{2}$ are nonlinear in u and v, given by

$$\begin{aligned} S^{1}(v_{1},v_{2};m=m_{h})= \frac{1}{a_{01}}B_{1}, \qquad S^{2}(v_{1},v_{2};m=m_{h})=- \frac{1}{\theta _{0} a_{01}}(a_{01}B_{1}+a_{01}B_{2}), \end{aligned}$$

with

$$\begin{aligned} B_{1}={}& \bigl(a_{20} a_{01}^{2}-a_{11} a_{01} a_{10}+a_{02} a_{10}^{2} \bigr)v_{1}^{2}+ \theta _{0}(2 a_{02} a_{10}-a_{11} a_{01})v_{1} v_{2}+\theta _{0}^{2} a_{02} v_{2}^{2} \\ &{}+\bigl(a_{12} a_{01} a_{10}^{2}-a_{03} a_{10}^{3}+a_{30} a_{01}^{3}-a_{21} a_{01}^{2} a_{10}\bigr)v_{1}^{3}+ \theta _{0}\bigl(2 a_{12} a_{10} a_{01} \\ &{}-a_{21} a_{01}^{2}-3 a_{03} a_{10}^{2}\bigr)v_{1}^{2} v_{2} +\theta _{0}^{2}(a_{12} a_{01}-3 a_{03} a_{10})v_{1} v_{2}^{2}- \theta _{0}^{3} a_{03}v_{2}^{3}, \\ B_{2}={}& \bigl(b_{20} a_{01}^{2}-b_{11} a_{01} a_{10}+b_{02} a_{10}^{2} \bigr)v_{1}^{2}+ \theta _{0}(2 b_{02} a_{10}-b_{11} a_{01})v_{1} v_{2}+\theta _{0}^{2} b_{02} v_{2}^{2} \\ &{}+\bigl(b_{30} a_{01}^{3}+b_{21}a_{01}a_{10}^{2}-b_{21}a_{01}^{2} a_{10}-b_{03} a_{10}^{3} \bigr)v_{1}^{3}+\theta _{0}\bigl(2 b_{12} a_{01} a_{10} \\ &{}-b_{21} a_{01}^{2}-3 b_{03} a_{10}^{2}\bigr)v_{1}^{2} v_{2} +\theta _{0}^{2}(b_{12}a_{01}-3 b_{03} a_{10})v_{1} v_{2}^{2}- \theta _{0}^{3} b_{03}v^{3}. \end{aligned}$$

By using the normal form theory in [42], we need the following quantity $l_{1}$ to study the properties of the limit cycle, which is calculated as follows:

$$\begin{aligned} l_{1}={}&\frac{1}{16} \bigl[S^{1}_{v_{1} v_{1} v_{1}}+S^{1}_{v_{1} v_{2} v_{2}}+S^{2}_{v_{1} v_{1} v_{2}}+S^{2}_{v_{2} v_{2} v_{2}} \bigr]+ \frac{1}{16 \theta _{0}} \bigl[S^{1}_{v_{1} v_{2}} \bigl(S^{1}_{v_{1} v_{1} }+S^{1}_{v_{2} v_{2}} \bigr) \\ &{}-S^{2}_{v_{1} v_{2}} \bigl(S^{2}_{v_{1} v_{1}}+S^{2}_{v_{2} v_{2}} \bigr)-S^{1}_{v_{1} v_{1}}S^{2}_{v_{1} v_{1}}+S^{1}_{v_{2} v_{2}}S^{2}_{v_{2} v_{2}} \bigr], \end{aligned}$$

where the partial derivative values are calculated at the critical point, i.e., $(v_{1},v_{2};m)=(0,0;m_{h})$. By applying the result given in [42], the existing limit cycle (Hopf bifurcation) is supercritical and subcritical if $l_{1}<0$ and $l_{1}>0$ correspondingly.

Example 5.2

Here, we choose the fixed parameter values $e=0.05$, $b=1$, $p=1$, $s=0.5$, $a=1$ $c=0.2$, $n=1$, $r_{0}=1$, $k=2.7055$ and varying m. We observe at some critical threshold $m=m_{h}=0.2=c$, system (2.4a)–(2.4b) has the positive equilibrium $E_{4}(x_{4}=0.154807,y_{4}=0.154807)$, also the trace of $J_{E_{4}}$ is zero $(\operatorname{Tr}(m_{h})=0)$ and the determinant of $J_{E_{4}}$ is positive $(\operatorname{Det}(m_{h})=0.0345387>0)$ and $\frac{d}{d m}\alpha (m)|_{m=m_{h}}=\frac{1}{2} [\frac{d}{d m}\operatorname{Tr}(x_{4}(m),y_{4}(m)) ]_{m=m_{h}}=0.268271\neq 0$ ensures that system (2.4a)–(2.4b) has limit cycle oscillation near $E_{4}$ at $m_{h}$. Also, system (2.4a)–(2.4b) has $\theta _{0}= 0.185849$ and the first Lyapunov coefficient $l_{1}=-1.53282<0$. Therefore, the existing Hopf bifurcation around $E_{4}$ is supercritical (unstable and surrounded by a stable limit cycle), see Figs. 3, 4, 5, and 6.

Figure 3

The two-parameter bifurcation diagram for system (2.4a)–(2.4b), k varying in $(0,4)$ and m varying in $(0,0.4)$. From Fig. 2(c) the black dark curve ($E_{1}$ exists) represents $\operatorname{Det}(E_{1})=0$ which separates Region I (no interior equilibria) and Region II ($E_{2}$ and $E_{3}$ exist), the black dashed line ($E_{4}$ exists) represents $m=c$ which separates Region II and Region III ($E_{5}$ exists), and the blue line (Hopf bifurcation curve) represents $\operatorname{Tr}(E_{2,4,5})=0$ which separates the stable and unstable region of equilibria $E_{2,4,5}$. Red dot represents the Bogdanov–Takens point, which is the intersecting point of $\operatorname{Det}(E_{1})=0$ and $\operatorname{Tr}(E_{2,4,5})=0$

Figure 4

The bifurcation diagram by fixing $k=2.5$ and varying m in $(0.16,0.2)$. In (a), (b) the x and y components of equilibria Ê, $E_{2}$, $E_{3}$ are plotted respectively, the blue line represents Ê and it is always stable, the red line represents $E_{3}$ and it is unstable, the red dotted line represents an unstable branch of $E_{2}$, and the black line represents a stable branch of $E_{2}$. The complex behavior of system (2.4a)–(2.4b) by separating critical values of m by black dotted lines into the regions namely A, B, C, D, and E. In A, Ê only exists and in B, C, D the Ê, $E_{2}$, $E_{3}$ exist. In C, the unstable $E_{2}$ is surrounded by stable limit cycle (green curve), and when $m=c=0.2$ the Ê, $E_{3}$ coincide with the saddle equilibrium $E_{0}$

Figure 5

The bifurcation diagram by fixing $m=0.18$ (red curve), $m=0.2$ (blue curve), $m=0.22$ (green curve) and varying k in $(2,4)$. The existence of limit cycle (Hopf bifurcation) for various values of m

Figure 6

Phase portrait for system (2.4a)–(2.4b) for case (i) $E_{2}$ for $m< c$, we have (a), (b), and (c); for case (ii) $E_{4}$ for $m=c$, we have (d), (e), and (f); for case (iii) $E_{5}$ for $m>c$, we have (g), (h), and (i). Here (a), (d), and (g) are locally asymptotically stable; (b), (e), and (h) are periodic solutions; (c), (f), and (i) are unstable phase portraits. The red line represents prey nullcline, the black line represents predator nullcline, the red and green arrow lines represent stable trajectories along x and y axes respectively, which approach Ê and Ē. The black dashed arrow line represents the separatrix curve separating the trajectories which approach Ē and interior equilibria $E_{2,4,5}$ in the x-y plane

5.4 Bogdanov–Takens bifurcation

In the previous subsection, we discussed the existence of one-parameter bifurcation for system (2.4a)–(2.4b). In this subsection the possible existence of a codimension-2 bifurcation, namely Bogdanov–Takens (BT) bifurcation for (2.4a)–(2.4b), is analyzed. For this, we have to take two parameter values m and k as the varying parameters and fix all other parameter constants. Here, we only discuss it numerically.

Remark 5.2

System (2.4a)–(2.4b) has a Bogdanov–Takens bifurcation for the varying parameters m and k if it satisfies $\operatorname{Det}(J_{E^{*}})=0$ and $\operatorname{Tr}(J_{E^{*}})=0$ at the critical threshold value $(m,k)=(m_{b}, k_{b})$.

Example 5.3

Take other parameter as $e=0.05$, $b=1$, $p=1$, $s=0.5$, $a=1$ $c=0.2$, $n=1$, $r_{0}=1$ and choose $m_{b}=0.117545$, $k_{b}=1.6083$, we get the unique interior equilibrium $E_{1}(0.058598,0.14105)$ and $\operatorname{Det}(J_{E_{1}})=\operatorname{Tr}(J_{E_{1}})=0$. Hence, system (2.4a)–(2.4b) has Bogdanov–Takens bifurcation around $E_{1}$ (see Figs. 3 and 8).

Remark 5.3

In most of the existing literature, authors have incorporated fear and Allee effect separately on predator–prey models, see [27, 28, 32, 40, 43]. For example, the issue on the impact of fear effect in the predator–prey model with Holling type II predator–prey interaction was reported in [40], where they stated that fear constant is able to stabilize the proposed system at an interior steady state, that is, both the species can exist together, or it can create the oscillatory coexistence of species populations. Weak Allee effect on the stability of a discrete-time predator–prey model was given in [43], and they observed that the system becomes stable from chaotic dynamics as the Allee parameter increases. Authors in [44] discussed a food chain model that experienced strange dynamics by varying Allee effect parameters, that is, the system state changes from stable to chaotic via period doubling as Allee effect increases. The effects of both Allee and fear parameters in predator–prey interaction model with prey-dependent functional response were considered in [29]. It is worth mentioning that our derived ratio-dependent modified Leslie type model (2.4a)–(2.4b) incorporates fear effect in the prey population and Allee effect in the predators growth. Moreover, system (2.4a)–(2.4b) exhibits complex dynamics with respect to both fear and Allee parameters, which is explained theoretically through bifurcation analysis and is verified in terms of numerical simulations.

6 Numerical simulations

In this section, in order to obtain a better visualization of how different parameter selection values influence the system (2.4a)–(2.4b) dynamics, numerical simulations are shown. Since we aim to investigate the effect of fear(k) and the Allee effect(m), all other parameters of system (2.4a)–(2.4b) are fixed as $e=0.05$, $b=1$, $p=1$, $h=0.5$, $n=0.2$, $q=1$, $r_{0}=1$. The scenario of no equilibrium, unique interior equilibrium, and two interior equilibria is given in Example 4.1 and showed in Fig. 1 with respect to the parameters m and k. The saddle-node bifurcation behavior for system (2.4a)–(2.4b) for parameter m is shown in Fig. 2(a) and for k in Fig. 2(b), where the blue dashed line represents the stable equilibrium branch and the red line represents the unstable equilibrium branch. The regions for existence of interior equilibria are shown in Fig. 2(c), where Region I is the space below the black darker line, here interior equilibrium point is not possible. Region II is the space between the black dashed line and the black thick line, here two interior equilibrium points arise. On and above the black dashed line representing Region III, the unique interior equilibrium point appears. In Fig. 3, the two-parameter bifurcation diagram is plotted, since the trace value is zero on the blue line, which separates the stable and unstable region, and the red dot denotes the Bogdanov–Takens bifurcation point, i.e., the intersection point of trace and determinant equals zero.

In order to show the birth of periodic solution of system (2.4a)–(2.4b), fixing $k=2.5$ and varying m in $(0,0.2)$, the one-parameter bifurcation diagram is plotted in Fig. 4. When $0.16< m<0.174$, the only equilibrium Ē exists and is stable. For $0.174< m<0.185$, the periodic solution around the interior equilibrium $E_{2}$ exists and it is stable in $0.185< m<0.2$. Hence, it states that the Allee parameter has a stabilizing effect on system (2.4a)–(2.4b). Also, in Fig. 5, the one-parameter bifurcation diagram is plotted with respect to the parameter k considering the cases in Lemma 4.1: For case (i) $m< c=0.18$ in red, for case (ii) $m=c=0.2$ in blue, and for case (iii) $m>c=0.22$ in green, which shows the birth of periodic solution on increasing k. Therefore, the fear parameter k has a destabilizing role for system (2.4a)–(2.4b). The existence of stable, periodic, and unstable phase portraits around the equilibrium $E_{2}$ for different choice of m and k is shown in Fig. 6.

The time trajectories may tend to either Ē and Ê or $E_{2}$, it depends on the choice of the initial condition. As in Fig. 6, there is a separatrix curve separating the trajectories that approach $\bar{E}=(0,0.3)$ and $E_{2}=(0.137202,0.167202)$, thus system (2.4a)–(2.4b) has three stable equilibria at $m=0.17$, $k=2.14$ and the remaining parameters are the same as in Example 4.1. Since the system is sensitive to the initial condition, we attempt to evaluate the $133\times 133$ initial conditions in prey vs predator plane and mark in different colors in 7(c), red color approaches Ê, green color approaches Ē, and blue color approaches $E_{2}$. In Figs. 7(a) and 7(b), it is shown that the different initial conditions approach different equilibria in time trajectories for both prey and predator, respectively.

Figure 7

(a) and (b) Time trajectories for prey and predator populations with various initial conditions, where green curves are the trajectories approaching Ē, blue curves are the trajectories approaching $E_{2}$, and red curves are the trajectories approaching Ê. (c) Bi-stability region in prey vs predator plane, i.e., initial conditions green, blue, and red points approach Ē, $E_{2}$, and Ê respectively

The local amplification of two-parameter spaces k and m in Fig. 3 around the BT point (Bogdanov–Takens bifurcation point) is shown in Fig. 8(a) for clear view, which is $(k,m)=(1.6083,0.1175466)$, and the corresponding phase portrait around $E_{1}=(0.0585234, 0.140977)$ is shown Fig. 8(b), which is the cusp of co-dimension 2.

Figure 8

(a) Local amplification of Fig. 3 around BT point. (b) Phase portrait for system (2.4a)–(2.4b) around BT point

The values of trace and determinant are shown in the colored density graph for $E_{2}$ in Figs. 9(a), 9(b) and for $E_{3}$ in Figs. 9(c), 9(d). In Fig. 9(a) the black dashed line $(\operatorname{Tr}(E_{2})=0)$ states that the trace value changes from negative to positive, also in Fig. 9(b) the determinant value is always positive. Hence system (2.4a)–(2.4b) shows limit cycle oscillations. However, in Fig. 9(c), even the trace values change from negative to positive, the determinant values are always negative in Fig. 9(d). Therefore, system (2.4a)–(2.4b) is stable near $E_{2}$ and unstable near $E_{3}$. Finally, in Fig. 10, the two-parameter bifurcation diagram is plotted for the equilibria $E_{2}$, $E_{4}$, and $E_{5}$, which includes Hopf bifurcation curve, stable and unstable regions with different choices of system parameters. Note that if $E_{3}$ exists, it is always unstable.

Figure 9

For system (2.4a)–(2.4b), black dashed lines represent trace value is zero, (a) Trace values for $E_{2}$, which changes from negative to positive. (b) Determinant values for $E_{2}$, which is always positive. (c) Trace values for $E_{3}$, which changes from negative to positive. (d) Determinant values for $E_{3}$, which is always negative. Also, confirms the results in Fig. 3

Figure 10

Two-parameter bifurcation diagram for system (2.4a)–(2.4b) with different choice of parameter values, the blue curve represents Hopf bifurcation curve, the black curve represents determinant zero (below in (b), (c) and above in (d), system (2.4a)–(2.4b) has no interior equilibria), and the black dashed curve represents $m=c$

7 Conclusion

In the present study, the fear effect in the prey population and the Allee effect in the predator population were considered in the modified Leslie type predator–prey model with ratio-dependent interaction term. First, we discuss the positivity, boundedness, and permanence analysis for system (2.4a)–(2.4b). Then the condition for existence of possible equilibria of system (2.4a)–(2.4b) is derived which showed that two equilibria arise, annihilate, and disappear with influence of both fear and Allee effects. Furthermore, we studied the local stability of all possible equilibria. For certain threshold values of Allee effect term, system (2.4a)–(2.4b) has two interior equilibria. Hence, we discussed the analytical conditions for local stability of arbitrary interior equilibria and plotted the stable and unstable regions using two-parameter bifurcation diagram. The colored map of trace and determinant values supports the study of stability of two possible interior equilibria. In addition, the considered system experiences the various bifurcation behaviors by varying a suitable parameter, and we observed that the increase of fear effect can change the system dynamics from stable to unstable via Hopf bifurcation. Also, we found that the existing Hopf bifurcation is supercritical by calculating the first Lyapunov coefficient. We showed that the Allee effect in predator helps to stabilize the unstable behavior caused by fear effect. Additionally, the existence of saddle-node bifurcation is discussed with the help of Sotomayor’s theorem. It is important to analyze the effect of fear and Allee effects on other system parameters, for this we have plotted various two-parameter bifurcation diagrams with the choice of different parameter values. Hence, the fear and the Allee effects play a vital role in the system dynamical behavior, it is important to analyze the long term survival for both species. Thus, it could be interesting and meaningful to study the dynamics of a predator–prey model with the Allee effects in both prey and predator. However, these terms will increase the complexity of the system, and we will leave this as future research.