On three-term conjugate gradient method for optimization problems with applications on COVID-19 model and robotic motion control

The three-term conjugate gradient (CG) algorithms are among the efficient variants of CG algorithms for solving optimization models. This is due to their simplicity and low memory requirements. On the other hand, the regression model is one of the statistical relationship models whose solution is obtained using one of the least square methods including the CG-like method. In this paper, we present a modification of a three-term conjugate gradient method for unconstrained optimization models and further establish the global convergence under inexact line search. The proposed method was extended to formulate a regression model for the novel coronavirus (COVID-19). The study considers the globally infected cases from January to October 2020 in parameterizing the model. Preliminary results have shown that the proposed method is promising and produces efficient regression model for COVID-19 pandemic. Also, the method was extended to solve a motion control problem involving a two-joint planar robot.


Introduction
Consider the following optimization model: The nonlinear CG methods play an important role in solving large-scale optimization models due to the modesty of their memory requirements and nice convergence properties. Generally, the iterates of the CG methods are usually determined through the following recursive computational scheme: where t k is the step-size computed along the search direction d k . For the first iteration, d 0 is always the steepest descent direction, that is, d 0 = -g 0 [3]. However, subsequent directions are recursively determined by where the scalar β k is known as the CG coefficient whose different form determines a different CG methods.
The following line search procedures have been used in the convergence analysis and implementations of the already existing CG methods [4]. The convergence analysis often requires the line search to satisfy the exact line search, the Wolfe or strong Wolfe (SWP) line search. The exact line search requires the step-size t k to satisfy (1.4) The standard line search requires computing t k such that the cost function is minimized along d k satisfying f (x k + t k d k ) ≤ f (x k ) + δt k g T k d k , (1.5) g(x k + t k d k ) T d k ≥ σ g T k d k . (1.6) The SWP is to compute t k satisfying (1.5) and where 0 < δ < σ < 1. Presently, there are several known formulas for different CG parameters (see [4][5][6][7][8][9][10]). One of the most efficient algorithms among the well-known formulas is the PRP [7,8] defined by g T k y k-1 g k-1 2 , (1.8) where y k-1 = g kg k- 1 . From the computational point of view, the PRP algorithm performs better than most CG algorithms, and the convergence result has been established under some line search procedures. However, for a general function, the PRP method fails with regard to the global convergence under the Wolfe line search procedure. This is because the direction of search d k is not descent for a general objective function [4]. This problem inspired numerous researchers to study the global convergence of PRP method under inexact line search. Interestingly, considering the general function, Yuan et al. [11] proved the global convergence of PRP method using a modified Wolfe line search procedure. More practical approaches of the line search have been employed to identify a step-size capable of achieving adequate reduction in the objective function f (x) at minimal cost.
Recently, Rivaie et al. [12] proposed a variant of PRP method by replacing the term g k-1 2 in the denominator of PRP with d k-1 2 as follows: 9) and showed that the method converges globally under the exact line search. However, Dai [13] pointed out a wrong inequality used in the convergence result of RMIL method and suggested some necessary corrections as follows: 0, otherwise, (1.10) and further established the global convergence under the exact line search. Preliminary results have been presented using the same benchmark test problems with different initial guess to illustrate the efficiency of the modified method. More recently, Yousif [14] modified the work of Dai [13] and showed that RMIL+ converges globally under the strong Wolfe line search. For more reference on the convergence analysis of the CG method, please refer to the following references [15][16][17][18][19].
It is worthy to note that the sufficient descent property plays a crucial role in the convergence analysis of the CG methods including the RMIL method. In this regard, several variants of the CG methods have been defined to satisfy (1.11) independent of the line search technique used. One of the efficient variants of the CG methods is the three-term CG method where the search direction d k contains three terms. One of the classical three-term methods was proposed by Beale [20], using the coefficient β HS k [5]. The author constructed a new direction of search as follows: where d t is the restart direction and The performance of this method was improved using an efficient restart strategy developed by McGuire [21]. The first three-term PRP algorithm (TTPRP) was defined by Zhang et al. [22] with the formula given as where β k is the PRP method defined in (1.8) and θ k = - . An attractive feature of this method is that the descent condition holds independent of any line search, and the global convergence was established under a modified Armijo line search. Based on the structure of TTPRP, Liu et al. [23] extended the coefficient of RMIL (1.9) to defined a three-term CG method known as TTRMIL with formula as follows: where β k is defined by (1.9) and θ k = - The global convergence of this method was proved under the standard Wolfe line search. However, the proposed TTRMIL method in (1.13) employed the RMIL method; Dai [13] pointed out some errors in the convergence result and suggested some correction given in [14]. Motivated by this, we propose a modification of TTRMIL in the next section. For more references about the three-term CG method, interested readers may refer to [24][25][26][27].
The rest of the paper would be structured as follows. In the next section, a modified TTRMIL method is given with its algorithm. The sufficient descent property and the global convergence of the new modification are studied in Sect. 3. Preliminary results based on some unconstrained optimization problems are presented to illustrate the performance of the method in Sect. 4. The proposed modification was extended to formulate a parameterized model for cases of COVID-19 in Sect. 5. In Sect. 6, the application in motion control is presented. Finally, the concluding remark and some recommendations of the study are presented in Sect. 7.

TTRMIL+ method and its algorithm
Motivated by the comments made by Dai [13] on the convergence of RMIL method, as discussed in the preceding section, we propose a modified TTRMIL, named TTRMIL+, by replacing β k in (1.13) with the β k given in (1.10) as follows: From (1.13) and (2.2), it is obvious that the difference between these two methods is the CG parameter β k employed by each method in defining their search directions d k . This is a little change that has a great impact in the convergence analysis of RMIL+. It is interesting to note that the TTRMIL+ reduces to the classical RMIL+ method under the exact minimization condition. The following algorithm describes the proposed TTRMIL+. The following assumptions are very important and usually required in the convergence analysis of most CG algorithms.
From Assumption 2.1 and [16,28], it implies that there exist positive constants γ and b such that But the function f (x) decreases as k → +∞, hence, from Assumption 2.1, the sequence {x k } generated by Algorithm 1 is said to be contained in a bounded region. This implies that the sequence {x k } is bounded. The convergence analysis of the new method would be studied in the next section.

Convergence analysis
In this section, we establish the sufficient descent condition and global convergence properties of the proposed TTRMIL+ method.
The following theorem indicates that the search direction of TTRMIL+ method satisfies the sufficient descent condition. Proof We will prove by induction. For k = 0 and from (2.1), we have g T 0 d 0 =g 0 2 , so that the sufficient descent condition (1.12) is satisfied. Suppose that (1.12) is true for k -1, that is, g T k-1 d k-1 =g k-1 2 . According to the value of β RMIL+ k (1.10), we have two cases.
Remark 3.2 Since the proposed method satisfies the sufficient descent condition (1.12), then, for all k ≥ 0, we have Now, we will establish the global convergence of the TTRMIL+ method by first providing the following lemma to show that the standard Wolfe line search gives a lower bound for the step-size t k as follows.
Proof From the standard Wolfe condition (1.6) and by subtracting g T k d k in the both sides, and using Lipschitz continuity (2.3), we get Since d k is a descent direction and also σ < 1, that implies (3.2) is true.
The following lemma is the Zoutendijk condition [29], which plays an important role in the analysis of the global convergence properties for CG method.
Proof From the standard Wolfe condition (1.5) and (3.2), we have Hence, from Assumption (2.1), we get the Zoutendijk condition (3.3) and hence the proof.
We present a global convergence results of the proposed TTRMIL+ CG method using the standard Wolfe line search.
Proof Suppose by contradiction that (3.4) is not true. Then ∀k ≥ 0, we can find a positive constant c so that Here, we have two cases.

Numerical experiments
In this part, we report the numerical experiments to demonstrate the efficiency of the TTRMIL+ method in comparison with the RMIL [12], RMIL+ [13], PRP [7,8], and TTR-MIL [23] methods. For comparing the computational performance, we consider some test problems from Andrei [30], and Jamil and Yang [31]. Most of initial points are also considered by Andrei [30] and implemented using dimensions starting from 2 to 50,000. The test problems and their initial points are presented in Table 1 All numerical results of the RMIL, RMIL+, and PRP methods are listed in Table 2 and those of the TTRMIL and TTRMIL+ methods in Table 3. A method is said to have failed if the NOI is more than 10,000 and the terminating criteria stated above have not been satisfied. The failure is symbolized with 'F' . We also use the performance profile tool of Dolan Let P be the set of test problems with n p being the number of test problem. S is the set of methods and n s is the number of methods. For each method s ∈ S and problem p ∈ P, let j p,s denote either NOI, NOF, or CPU time required to solve problem p by method s. Then the performance profile is defined as follows:  where τ > 0, and r p,s is the performance ratio that can be obtained by Generally, the method with the high performance profile value ρ s (τ ) is considered the best method for a given τ value. In other words, the method where the curve dominates the very top is the most efficient method compared to the others.
According to Table 2, the RMIL method was able to solve 66% of the problems, RMIL+ 75%, and PRP 71%. Meanwhile, based on Table 3, the TTRMIL method solved 93% of the problems and the proposed TTRMIL+ 94%. In this regard, the TTRMIL+ method is considered a better method when compared to the RMIL, RMIL+, and PRP methods, but competes with the TTRMIL method in terms of NOI, CPU time, and NOF. From the performance profile in Figs. 1-3, we can see that the TTRMIL+ method is efficient and promising with regard to solving unconstrained optimization problems compared to the RMIL, RMIL+, PRP, and TTRMIL methods.

Application of TTRMIL+ to parameterized COVID-19 model
Coronavirus disease often called COVID-19 is an acute vector-borne disease that surfaced in 2019. This disease is caused by the newly discovered coronavirus (SARS-CoV-2) and can be transmitted through droplets produced when an infected person exhales, sneezes,  Recently, numerous studies modeled various aspects of the coronavirus outbreak, and application of numerical methods on some COVID-19 models was also studied. In this paper, we consider the global COVID-19 outbreak from January to September, 2020, model the confirmed cases into an unconstrained optimization problem, and finally apply TTRMIL+ to obtain the solution of the parameterized model.
where x i , i = 1, 2, . . . , p, p > 0 is the predictor, y is the response variable, and ε is the error. This type of problem often arises in the fields of management, finance, economics, accounting, physics, and many more. The regression analysis is a statistical modeling tool used to estimate the relationships between a dependent variable and one or more independent variables. To derive the linear regression function, we compute y such that y = a 0 + a 1 x 1 + a 2 x 2 + · · · + a p x p + ε, (5.2)  where the parameters of the regression are defined by a 0 , . . . , a p . The regression analysis estimates the regression parameters a 0 , a 1 , . . . , a p such that the value of the error ε is minimized. An instance where the linear regression method is the relationship between y and x is approximated with a straight line. However, such a case infrequently occurs, and thus, the nonlinear regression process is often used. In this study, we consider the nonlinear regression approach.
To formulate the approximate function, we consider the data from the global confirmed cases of COVID-19 from January to September, 2020. The detailed description of the process follows from the statistics presented in Table 4 which are taken from the data obtained by the World Health Organization [WHO] [33]. We have data for nine months (Jan-Sept), the data for the months would be denoted by x-variable and the confirmed cases corresponding to these months would be denoted by the y-variable. For fitting the data, we only consider the data for eight months (Jan-Aug), and reserve the data for September for error analysis.
From the above data, we obtain the following approximate function for the nonlinear least square method: f (x) = -26,029.59 + 14,557.39x + 3290.077x 2 .
(5.3) Denoting the number of months by x j and the corresponding confirmed cases by y j , then, we can transform the least squares problem (5.3) into the following unconstrained minimization model: The nonlinear quadratic function for the least squares problem is derived using the data utilized from Jan-Aug, 2020, which is further used to formulate the corresponding unconstrained optimization model. Obviously, it can be observed that data x j and the value of y j possess some parabolic relations with the regression parameters u 0 , u 1 , and u 2 and the regression function (5.4).
Using the data from The data considered to generate the unconstrained optimization model are data from Jan-August, and the data for Sept is reserved for computing the relative errors of the predicted data. Applying the proposed TTRMIL+ method on model (5.6) under the strong Wolfe line search, we obtain the following results presented in Table 5.
One of the major challenges is computing the values of u 0 , u 1 , u 2 using matrix inverse [34]. To overcome this difficulty, we implement the proposed TTRMIL+ using different initial points. The computation would be terminated if the following conditions hold.
1. The algorithm fails to solve the model. 2. The number of iterations exceeds 1000. This point is denoted as 'Fail' .

Trend line method
A trend line is a line drawn under pivot lows or over pivot highs to show the prevailing direction of price. In this section, we estimate the data for COVID-19 for a period of nine (9) months using the proposed TTRMIL+ and least squares methods. The trend line is plotted using the Microsoft Excel software based on data from Table 4. The trend line   Fig. 4 using the actual data from Table 4. Further, to illustrate the efficiency of the proposed method, we compare the approximation functions of TTRMIL+ method with the functions of trend line and least square methods as follows.
The ideal purpose of regression analysis is estimating the parameters a 0 , a 1 , . . . , a p such that the error ε is minimized. From the results presented in Table 6, it is obvious that the proposed TTRMIL+ CG method has the least relative error compared to the least square and trend line methods which implied that the method is applicable to real-life situations. For other references regarding modeling, analysis, and prediction of COVID-19 cases, one can see [35].

Application TTRMIL+ in motion control
This section demonstrates the performance of the proposed TTRMIL+ CG method on motion control of a two-joint planar robotic manipulator. As presented in [36], the following model describes a discrete-time kinematics equation of two-joint planar robot manipulator at the position level (μ k ) = η k , (6.1) where μ k ∈ R 2 and η k ∈ R 2 denote the joint angle vector and the end effector vector position, respectively. The vector-valued function (·) represents the kinematics function which has the following structure: (μ k ) = τ 1 cos(μ 1 ) + τ 2 cos(μ 1 + μ 2 ) τ 1 sin(μ 1 ) + τ 2 sin(μ 1 + μ 2 ) , (6.2) with τ 1 and τ 2 denoting the length of the first and second rod, respectively. In the case of motion control, at each instantaneous computational time interval [t k , t k+1 ) ⊆ [0, t f ] with t f being the end of task duration, the following nonlinear least squares model is to be minimized: where k denotes the end effector controlled track. Similar to the approach presented in [37][38][39], the end effector, used in this experiment, is controlled to track a Lissajous curve given as  Fig. 5(a), where the end effector trajectory and the desired path are plotted in Fig. 5(b). Finally, the errors recorded on Perusing through these figures, it can be seen from Figs. 5(a) and 5(b) that the TTRMIL+ successfully accomplished the task at hand. The error recorded in the course of the task is relatively low as can be seen from Figs. 5(c) and 5(d), which confirms the efficiency of the proposed TTRMIL+.

Conclusion
This paper presented a modified conjugate gradient method for unconstrained optimization models. The proposed TTRMIL+ method replaced RMIL in TTRMIL with a new modification known as RMIL+. The sufficient descent condition and the convergence proof of TTRMIL+ are studied under the standard Wolfe line search. Some unconstrained benchmark test problems are considered to illustrate the performance of the proposed method. The result obtained showed that the TTRMIL+ method is efficient and promising. The method was further applied to a parameterized COVID-19 model, and the result obtained showed that TTRMIL+ produced a good regression model and thus can be used in regression analysis. Finally, we applied the method to solve a practical problem of motion control. Future work includes studying the new algorithm on nonlinear least squares problems as discussed in [40]. Furthermore, we shall consider other problems in our future research as presented in the following references [41][42][43][44].