Skip to main content

The LS-SVM algorithms for boundary value problems of high-order ordinary differential equations

Abstract

This paper introduces the improved LS-SVM algorithms for solving two-point and multi-point boundary value problems of high-order linear and nonlinear ordinary differential equations. To demonstrate the reliability and powerfulness of the improved LS-SVM algorithms, some numerical experiments for third-order, fourth-order linear and nonlinear ordinary differential equations with two-point and multi-point boundary conditions are performed. The idea can be extended to other complicated ordinary differential equations.

Introduction

High-order boundary value problems for ordinary differential equations are used to model different problems in some fields such as biology, economics, and engineering. Due to the importance of high-order ordinary differential equations, a considerable size of research work has been carried out about this problem. Among others, finite difference method [1] was proposed to solve two-point boundary value problems for high-order linear and nonlinear ordinary differential equations. Homotopy perturbation method [2, 3] was used for the solution of fourth-order and sixth-order boundary value problems. Ali [4] proposed the optimal homotopy asymptotic method to solve multi-point boundary value problems. Adomian decomposition method [5,6,7,8,9,10] was presented for solving two-point and multi-point boundary value problems of high-order ordinary differential equations. Haar wavelets method [11] and Shannon wavelet method [12] were proposed to solve boundary value problems of high-order ordinary differential equations. Doha [13] proposed spectral Galerkin algorithms based on Jacobi polynomials for solving two-point boundary value problems of third-order and fifth-order ordinary differential equations. Doha [14] proposed spectral Galerkin algorithms by using Chebyshev polynomials of the third and fourth kinds for solving high even-order differential equations. Shifted Jacobi collocation method [15] was proposed for solving nonlinear high-order multi-point boundary value problems. Saadatmandi and Dehghan [16] discussed sinc-collocation method for solving multi-point boundary value problems. Variational iteration method [17,18,19] was applied to solving two-point boundary value problems of high-order linear and nonlinear ordinary differential equations. Although these numerical methods provide good approximations to the solution, the approximate solution derivatives are discontinuous and can seriously affect the stability of the solution.

Neural network, which is one of machine intelligence techniques, has universal function approximation capabilities [20,21,22], and the solution obtained from the neural network is differentiable and in closed analytic form. Neural network has been widely used for solving ordinary differential equations [23, 24], partial differential equations [25,26,27], fractional differential equations [28,29,30], and integro-differential equations [31, 32]. Chakraverty and Mall [33] analyzed a regression-based neural network algorithm to solve two-point boundary value problems of fourth-order linear ordinary differential equations. Malek [34] proposed a novel hybrid method based on optimization techniques and feed forward artificial neural networks methods for two-point boundary value problems of fourth-order ordinary differential equations. Mai-Duy [35] discussed radial basis function networks for boundary value problems of high-order ordinary differential equations directly. However, artificial neural network has several drawbacks, such as the need for a large number of controlling parameters and the difficult choice of the number of hidden units. Furthermore, its training procedure is time-consuming and can be trapped in local minima.

SVM algorithms [36] were introduced by Vapnik in the framework of statistical learning theory. SVM algorithms map the input data into a high-dimensional feature space using a feature map. SVM algorithms can achieve a global optimum by solving a convex quadratic programming problem. Meanwhile, SVM algorithms adopt the structural risk minimization principle, which has a better generalization performance. LS-SVM algorithms [37] are a modification of SVM algorithms. LS-SVM algorithms change inequality constraints to equality constraints and regard the sum of squared errors loss function as experience loss of the training set. LS-SVM algorithms will deal with a set of linear equations instead of a quadratic optimization problem, which reduces the computation time of model learning significantly and improves higher solution accuracy. Therefore, LS-SVM algorithms have various applications in the area of pattern recognition [38], fault diagnosis [39], and time-series prediction [40, 41]. In addition, LS-SVM algorithms have been successfully applied for solving differential equations [42, 43], differential algebraic equations [44, 45], and integral equations [46].

LS-SVM algorithms are only used to solve two-point boundary value problems of second-order linear ordinary differential equations [42]. To the best of our knowledge, there are not too many results on LS-SVM algorithms for solving two-point and multi-point boundary value problems of high-order linear and nonlinear ordinary differential equations. The main goal of the present thesis is to develop improved LS-SVM algorithms to solve two-point and multi-point boundary value problems of high-order linear and nonlinear ordinary differential equations.

The remainder of this paper is organized as follows. First, Sect. 2 introduces least squares support vector machines. A brief overview of LS-SVM algorithms for solving ordinary differential equations is provided, and some definitions are given in Sect. 3. Following, in Sect. 4, the proposed LS-SVM algorithms for solving two-point boundary value problems of high-order linear and nonlinear ordinary differential equations and multi-point boundary value problems of high-order linear and nonlinear ordinary differential equations are discussed. In Sect. 5, we present five numerical examples to exhibit the accuracy and the efficiency of our proposed LS-SVM algorithms. Finally, concluding remarks are presented in Sect. 6.

Least squares support vector machines

Consider a given training data set \(\{(x_{i}, y_{i})|x_{i}\in R^{n}, y _{i}\in R\}_{i=1}^{N}\) (in this paper \(n=1\)), where \(\{x_{i}\}_{i=1} ^{N}\) are input data points and \(\{y_{i}\}_{i=1}^{N}\) are the corresponding output data points. One assumes that the underlying function describing the relation between input points and output points has the following form:

$$ y(x)=\boldsymbol{\omega }^{T}\boldsymbol{\phi }({x)}+b, $$
(1)

where ω and b are parameters of the model that have to be determined and \(\boldsymbol{\phi }({x)}\) is the nonlinear feature map which maps an input space into a higher dimensional feature space. Then, the optimal solution is sought in that space by minimizing the residual between the model outputs and the measurements [47]. To this end, the LS-SVM model in the primal is formulated as the following optimization problem [37, 48]:

$$ \min _{\boldsymbol{\omega },b, e _{i}} J(\boldsymbol{\omega },\boldsymbol{e})= \frac{1}{2} \boldsymbol{\omega }^{T}\boldsymbol{\omega }+ \frac{1}{2}\gamma \boldsymbol{e}^{T}\boldsymbol{e} $$
(2)

subject to

$$ y_{i}={\boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{i})+b}+e_{i},\quad i=1,2,\ldots,N, $$

where γ is a positive regularization parameter and \(e_{i}\) is the error of the ith input data. The first term is a regularization term, while the second one minimizes the training errors.

The optimization problem with equality constraints (2) can be solved by using the Lagrange multipliers method

$$ L(\boldsymbol{\omega }, b, \alpha _{i},e_{i} )= \frac{1}{2} \boldsymbol{\omega }^{T}\boldsymbol{\omega }+ \frac{1}{2}\gamma \boldsymbol{e}^{T}\boldsymbol{e}-\sum _{i=1}^{N}\alpha _{i}\bigl[{\boldsymbol{ \omega } ^{T} \boldsymbol{\phi }(x_{i})+b}+e_{i}-y_{i} \bigr], $$
(3)

where \(\alpha _{i}\) (\(i=1,2,\ldots,N\)) are Lagrange multipliers that can be positive or negative in the LS-SVM formulation.

According to the KKT conditions, we will obtain

$$ \textstyle\begin{cases} \frac{\partial L}{\partial \boldsymbol{\omega }}= \boldsymbol{\omega }-\sum_{i=1}^{N}\alpha _{i} \boldsymbol{\phi }(x _{i})=0;\\ \frac{\partial L}{\partial b}=\sum_{i=1}^{N}\alpha _{i}=0; \\ \frac{\partial L}{\partial e_{i}}=\alpha _{i} -\gamma e_{i}=0;\\ \frac{\partial L}{ \partial \alpha _{i}}={\boldsymbol{\omega }^{T} \boldsymbol{\phi }(x _{i})+b}-y_{i}+e_{i}=0. \end{cases} $$
(4)

When ω and \(e_{i}\) are eliminated from a system of Eq. (4), we obtain the following linear system:

(5)

where \(\varTheta _{ij}=K(x_{i},x_{j})=\boldsymbol{\phi }(x_{i})^{T} \boldsymbol{\phi }(x_{j})\) (\(i,j=1,2,\ldots,N\)) is the ijth entry of the positive definite kernel matrix; \(\boldsymbol{y}=[y_{1},y_{1},\ldots,y _{N}]^{T};\boldsymbol{\alpha }=[\alpha _{1},\alpha _{1},\ldots,\alpha _{N}]^{T}\) and \(I_{N-1}=[1,1,\ldots,1]\).

Finally, the LS-SVM model in the dual form can be described as

$$ y(x)=\sum_{i=1}^{N}\alpha _{i}K(x_{i},x)+b. $$
(6)

Brief overview of LS-SVM model for solving ODEs and some definitions

In this section, a brief overview of LS-SVM algorithms for solving ordinary differential equations is provided and some definitions are given.

With regard to the initial value problem of the first-order linear ordinary differential equation in the following form [42]:

$$ \textstyle\begin{cases} \frac{dy}{dx}=a(x)y(x)+r(x),& x\in [a,c],\\ y(a)=A, \end{cases} $$
(7)

the authors in [42] assume that a general approximate solution is \(y=\boldsymbol{\omega }^{T}\boldsymbol{\phi }({x)}+b\), where ω and b are the parameters to be solved. Then the interval \([a, c]\) is discretized into a series of collocation points by using collocation methods [49], and the optimal values of the parameters ω and b are obtained by solving the optimization problem with constraints, see [42]. According to the Lagrange multipliers method [50], the optimization problem with constraints is transformed into the Lagrangian function which is composed of the LS-SVM cost function and constraints that the approximate solution \(y=\boldsymbol{\omega }^{T}\boldsymbol{\phi }({x)}+b\) satisfies the given first-order linear ordinary differential equation and the initial condition at the collocation points. The described methodology is applicable for solving other types of differential equations including second-order boundary value problems, partial differential equations, and descriptor systems [42,43,44].

The feature map ϕ is not explicitly known in general, so the kernel function will be introduced. By utilizing Mercer’s theorem [36], the derivative of the kernel function is defined as [42, 44]

$$ \nabla _{n,m}\bigl(K(x_{i},x_{j})\bigr)= \frac{\partial ^{n+m}(K(u,v))}{\partial u ^{n}\partial v^{m}}\bigg|_{u=x_{i},v=x_{j}}=\boldsymbol{\phi }^{(n)}(x_{i})^{T} \boldsymbol{\phi }^{(m)}(x_{j})=[\varTheta _{n,m}]_{i,j}. $$
(8)

In this paper, the RBF kernel \(K(u,v)=\exp (-(u-v)^{2}/\sigma ^{2})\) is considered as a kernel function, then we can obtain

$$\begin{aligned}& \begin{aligned} \nabla _{1,3} \bigl(K(x_{i},x_{j})\bigr) &=\frac{\partial ^{4}(K(u,v))}{\partial u \partial v^{3}}\bigg|_{u=x_{i},v=x_{j}}= \boldsymbol{\phi }^{(1)}(x_{i})^{T} \boldsymbol{\phi }^{(3)}(x_{j})=[\varTheta _{1,3}]_{i,j} \\ &=- \biggl[\frac{12}{ \sigma ^{4}}-\frac{12}{\sigma ^{2}} \biggl[\frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{2}+\biggl[\frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{4} \biggr]K(x _{i},x_{j}); \end{aligned} \\& \begin{aligned} \nabla _{2,3}\bigl(K(x_{i},x_{j})\bigr)&= \frac{\partial ^{5}(K(u,v))}{ \partial u^{2}\partial v^{3}}\bigg|_{u=x_{i},v=x_{j}}=\boldsymbol{\phi } ^{(2)}(x_{i})^{T} \boldsymbol{\phi }^{(3)}(x_{j})=[\varTheta _{2,3}]_{i,j} \\ &= \biggl[\frac{60}{\sigma ^{4}}-\frac{20}{\sigma ^{2}} \biggl[\frac{2(x _{i}-x_{j})}{\sigma ^{2}} \biggr]^{2}+\biggl[\frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{4} \biggr] \frac{2(x_{i}-x_{j})}{\sigma ^{2}}K(x_{i},x_{j}); \end{aligned} \\& \begin{aligned} \nabla _{3,3}\bigl(K(x_{i},x_{j})\bigr)={}& \frac{\partial ^{6}(K(u,v))}{\partial u ^{3}\partial v^{3}}\bigg|_{u=x_{i},v=x_{j}}=\boldsymbol{\phi }^{(3)}(x_{i})^{T} \boldsymbol{\phi }^{(3)}(x_{j})=[\varTheta _{3,3}]_{i,j}\\ ={}& \biggl[\frac{120}{ \sigma ^{6}} -\frac{180}{\sigma ^{4}} \biggl[\frac{2(x_{i}-x_{j})}{ \sigma ^{2}} \biggr]^{2}+ \frac{30}{\sigma ^{2}} \biggl[\frac{2(x_{i}-x_{j})}{ \sigma ^{2}} \biggr]^{4}-\biggl[ \frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{6} \biggr]\\ &{}\times K(x_{i},x_{j}); \end{aligned} \\& \begin{aligned} \nabla _{3,4}\bigl(K(x_{i},x_{j})\bigr)={}& \frac{\partial ^{7}(K(u,v))}{\partial u^{3}\partial v^{4}}\bigg|_{u=x_{i},v=x_{j}}= \boldsymbol{\phi }^{(3)}(x_{i})^{T} \boldsymbol{\phi }^{(4)}(x_{j})=[ \varTheta _{3,4}]_{i,j} \\ ={}& \biggl[\frac{840}{\sigma ^{6}}-\frac{420}{\sigma ^{4}} \biggl[\frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{2}+\frac{42}{\sigma ^{2}} \biggl[\frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{4}- \biggl[\frac{2(x _{i}-x_{j})}{\sigma ^{2}} \biggr]^{6} \biggr] \\ &{}\times\frac{2(x_{i}-x_{j})}{\sigma ^{2}}K(x_{i},x_{j}); \end{aligned} \\& \begin{aligned} \nabla _{4,4}\bigl(K(x_{i},x_{j})\bigr)={}& \frac{\partial ^{8}(K(u,v))}{\partial u^{4}\partial v^{4}}\bigg|_{u=x_{i},v=x_{j}}= \boldsymbol{\phi }^{(4)}(x_{i})^{T} \boldsymbol{\phi }^{(4)}(x_{j})=[ \varTheta _{4,4}]_{i,j}\\ ={} &\biggl[\frac{1680}{\sigma ^{8}} -\frac{3360}{ \sigma ^{6}} \biggl[\frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{2}+ \frac{840}{ \sigma ^{4}} \biggl[\frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{4}- \frac{56}{ \sigma ^{2}} \biggl[\frac{2(x_{i}-x_{j})}{\sigma ^{2}} \biggr]^{6} \\ &{}+\biggl[\frac{2(x _{i}-x_{j})}{\sigma ^{2}} \biggr]^{8} \biggr]K(x_{i},x_{j}). \end{aligned} \end{aligned}$$

Boundary value problems of high-order ordinary differential equations

In this section, we formulate the improved LS-SVM algorithms to the solution of two-point and multi-point boundary value problems of high-order linear and nonlinear ordinary differential equations.

Two-point boundary value problems of high-order ordinary differential equations

The improved LS-SVM algorithms to the solution of two-point boundary value problems of high-order linear and nonlinear ordinary differential equations are described.

Nonlinear ordinary differential equations for two-point boundary value problems

Two-point boundary value problems of Mth-order nonlinear ordinary differential equations to be solved can be stated as follows:

$$ \frac{d^{M}y}{dx^{M}}+a_{M-1}(x)\,\frac{d^{M-1}y}{dx^{M-1}}+\cdots+a _{1}(x)\,\frac{dy}{dx}=f(x,y), \quad x \in [a,c], $$
(9)

subject to boundary conditions \(y^{(s)}(a)=p_{s}\), \(y^{(r)}(c)=q_{r}\), \(0\leq s\leq S\), \(0\leq r\leq R\), \(R=M-2-S\).

The interval \([a, c]\) is discretized into a series of collocation points \(\varOmega = \{a = x_{1} < x_{2} < \cdots < x_{N} = c\}\). Assume that a general approximate solution to (9) is \(y=\boldsymbol{\omega }^{T} \boldsymbol{\phi }({x)}+b\). The optimal values of the parameters ω and b are obtained by the following optimization problem:

$$ \min _{\boldsymbol{\omega }, b, \boldsymbol{e}, \boldsymbol{\xi }, y _{i}} J(\boldsymbol{\omega },\boldsymbol{e}, \boldsymbol{\xi })= \frac{1}{2}\boldsymbol{\omega }^{T} \boldsymbol{\omega }+\frac{1}{2} \gamma \boldsymbol{e}^{T} \boldsymbol{e}+\frac{1}{2}\gamma \boldsymbol{\xi }^{T}\boldsymbol{ \xi } $$
(10)

subject to

$$\begin{aligned} &\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=1} ^{M-1} \boldsymbol{\omega }^{T}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x _{i}) =f(x_{i},y_{i})+e_{i}, \quad i=2,\ldots,N-1; \\ &y_{i}={\boldsymbol{\omega } ^{T}\boldsymbol{\phi }(x_{i})+b}+\xi _{i},\quad i=2,\ldots,N-1; \\ & \boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{1})+b=p_{0}; \\ & \boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{N})+b=q_{0}; \\ & \boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(s)}(x_{1})=p_{s}, \quad s=1,2,\ldots,S; \\ &\boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(r)}(x_{N})=q_{t}, \quad r=1,2,\ldots,R. \end{aligned}$$

Theorem 1

Given a positive definite kernel function \(K: R\times R\rightarrow R\) and a regularization parameter \(\gamma \in R^{+}\), the solution to (10) is given by the following dual problem:

(11)

where \([\widehat{\varTheta }_{l,l'}]_{N-2}=[ \widetilde{\varTheta }_{M,M}]_{N-2}+ \overline{D}_{a_{l}}[ \overline{\varTheta }_{l,M}]_{N-2}+[\overline{ \varTheta }_{M,l'}]_{N-2}\overline{D}_{a_{l'}}^{T} + \overline{D}_{a_{l}}[\overline{ \varTheta }_{l,l'}]_{N-2} \overline{D}^{T}_{a_{l'}} +\gamma ^{-1}E\); \([\overline{ \varTheta }_{M,l'}]_{N-2}= [[\widetilde{\varTheta }_{M,1}]_{N-2},[ \widetilde{ \varTheta }_{M,2}]_{N-2},\ldots,[\widetilde{\varTheta }_{M,M-1}]_{N-2} ]\); \([\overline{\varTheta }_{l,M}]_{N-2}= [[\widetilde{ \varTheta } _{1,M}]_{N-2};[\widetilde{\varTheta }_{2,M}]_{N-2}; \ldots;[ \widetilde{\varTheta }_{M-1,M}]_{N-2} ]\); \(\overline{D}_{a_{l}}=[D_{a _{1}}, D_{a_{2}}, \ldots,D_{a_{M-1}}]\); \([\overline{\varTheta }_{l,l'}]_{N-2}=[ \widetilde{\varTheta }_{1:M-1,1:M-1}]_{N-2};\overline{D}_{a_{l'}}=[D_{a _{1}}, D_{a_{2}}, \ldots,D_{a_{M-1}}]\); \([\widetilde{\varTheta }_{0,0}]_{N-2}=[ \varTheta _{0,0}]_{2:N-1,2:N-1}+\gamma ^{-1}E\); \(l,l'=1,2,\ldots,M-1\); \(\boldsymbol{\alpha }=[\alpha _{2},\alpha _{3}, \ldots, \alpha _{N-1}]^{T}\); \(D _{a_{l'}}=\operatorname{diag}(a_{l'}(t_{2}),a_{l'}(t_{3}), \ldots,a_{l'}(t_{N}))\); \(D_{a _{l}}= \operatorname{diag}(a_{l}(t_{2}),a_{l}(t_{3}), \ldots,a_{l}(t_{N}))\); \([ \widehat{\varTheta }_{0,l'}]_{N-2}=[\widetilde{\varTheta }_{0,M}]_{N-2}+[\overline{ \varTheta }_{0,l'}]_{N-2} \overline{D}_{a_{l'}}^{T}\); \(\boldsymbol{I}_{1,N-2}=[1,1, \ldots,1]\); \(\boldsymbol{\beta }=[\beta _{0},\beta _{1}, \ldots,\beta _{S}]^{T}\); \([\overline{ \varTheta }_{0,l'}]_{N-2}= [[\widetilde{ \varTheta }_{0,1}]_{N-2},[ \widetilde{\varTheta }_{0,2}]_{N-2},\ldots,[\widetilde{\varTheta }_{0,M-1}]_{N-2} ]\); \(\boldsymbol{\eta }=[\eta _{2},\eta _{3},\ldots,\eta _{N-1}]^{T}\); \([\widehat{\varTheta }_{s,l'}^{1} ]_{N-2}=[\widetilde{\varTheta }_{0:S,M} ^{1} ]_{N-2}+[\overline{\varTheta }_{0:S,l'}^{1} ]_{N-2}\overline{D}_{a _{l'}}^{T}\); \([\widehat{ \varTheta }_{r,l'}^{N}]_{N-2}=[\widetilde{ \varTheta } _{0:R,M}^{N}]_{N-2}+[\overline{ \varTheta }_{0:R,l'}^{N}]_{N-2} \overline{D}_{a_{l'}}^{T}\); \([\overline{\varTheta }_{0:S,l'}^{1}]_{N-2}= [[\widetilde{\varTheta }_{0:S,1}^{1} ]_{N-2}, [\widetilde{\varTheta }_{0:S,2} ^{1} ]_{N-2},\ldots,[\widetilde{\varTheta }_{0:S,M-1}^{1} ]_{N-2} ]\); \(\boldsymbol{q}=[q_{0},q_{1},q_{2}, \ldots,q_{R}]^{T}\); \([\overline{ \varTheta }_{0:R,l'}^{N}]_{N-2}= [[\widetilde{\varTheta }_{0:R,1}^{N} ]_{N-2},[ \widetilde{\varTheta }_{0:R,2}^{N} ]_{N-2}, \ldots,[\widetilde{\varTheta }_{0:R,M-1} ^{N}]_{N-2} ]\); \(\boldsymbol{\lambda }=[\lambda _{0},\lambda _{1}, \lambda _{2},\ldots,\lambda _{R}]^{T}\); \([\overline{\varTheta }_{s,0}^{1}]_{N-2}= [ \widetilde{\varTheta }_{0:S,0}^{1}]_{N-2}\); \([\overline{\varTheta }_{r,0}^{N}]_{N-2}= [ \widetilde{\varTheta }_{0:R,0}^{N}]_{N-2}\); \(\boldsymbol{y}=[y_{2},y_{3},\ldots,y _{N-1}]\); \(0_{s,N-2}=0_{S+1,N-2}\); \([\widetilde{\varTheta }_{s,s'}]_{1,1}=[ \varTheta _{0:S,0:S}]_{1,1}\); \([\widetilde{\varTheta }_{r,s'}]_{N,1}=[ \varTheta _{0:R,0:S}]_{N,1}\); \([\widetilde{\varTheta }_{r,r'}]_{N,N}=[ \varTheta _{0:R,0:R}]_{N,N}\); \(f_{N-2}(\boldsymbol{x},\boldsymbol{y})=[f(x _{2},y_{2}),f(x_{3},y_{3}),\ldots,f(x_{N-1},y_{N-1}) ]^{T}\); \(\boldsymbol{p}=[p _{0},p_{1},p_{2}, \ldots,p_{S}]^{T}\); \(\frac{\partial f(\boldsymbol{x}, \boldsymbol{y})}{\partial y}= [\frac{\partial f(x,y)}{\partial y} |_{x=x_{2},y=y_{2}}, \frac{\partial f(x,y)}{\partial y}|_{x=x _{3},y=y_{3}},\ldots,\frac{\partial f(x,y))}{\partial y}|_{x=x_{N-1},y=y _{N-1}} ]\); \(D_{N-2}(\boldsymbol{y})= \operatorname{diag}(\frac{\partial f( \boldsymbol{x},\boldsymbol{y})}{\partial y} )\); \(\boldsymbol{1}_{r}=[1;0;\ldots;0]_{R+1,1}\); \(\boldsymbol{1}_{s}=[1;0;\ldots;0]_{S+1,1}\); \(0_{r,N-2}=0_{R+1,N-2}\); where \([\widetilde{\varTheta }_{m,n}]_{N-2}=[\varTheta _{m,n}]_{2:N-1,2:N-1}\), \([\widetilde{\varTheta }_{0:S,n}^{1}]_{N-2}= [[\varTheta _{0:S,n}]_{1,2},[ \varTheta _{0:S,n}]_{1,3},\ldots,[\varTheta _{0:S,n}]_{1,N-1} ]\) and \([\widetilde{\varTheta }_{0:R,n}^{N}]_{N-2}= [[\varTheta _{0:R,n}]_{N,2},[ \varTheta _{0:R,n}]_{N,3},\ldots,[\varTheta _{0:R,n}]_{N,N-1} ]\), \(m,n=0,1,\ldots,M \).

Proof

Consider the Lagrangian function of the optimization problem (10):

$$\begin{aligned} &L(\boldsymbol{\omega }, y_{i},\alpha _{i},\eta _{i},\beta _{0},\beta _{s}, \lambda _{0},\lambda _{r},b,e_{i},\xi _{i}) \\ &\quad =\frac{1}{2} \boldsymbol{\omega }^{T}\boldsymbol{\omega }+\frac{1}{2}\gamma \boldsymbol{e}^{T}\boldsymbol{e}+\frac{1}{2}\gamma \boldsymbol{\xi } ^{T}\boldsymbol{\xi } \\ &\qquad {}-\sum_{i=2}^{N-1} \alpha _{i} \Biggl[ \boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=1} ^{M-1}\boldsymbol{\omega }^{T}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x _{i}) -f(x_{i},y_{i})-e_{i} \Biggr] \\ &\qquad {}-{\sum_{i=2}^{N-1}\eta _{i}\bigl( \boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{i})+b}+\xi _{i} -y _{i}\bigr)-\beta _{0}\bigl(\boldsymbol{\omega ^{T}}\boldsymbol{\phi }(x_{1})+b-p _{0}\bigr)- \sum_{s=1}^{S}\beta _{s}\bigl( \boldsymbol{\omega ^{T}} \boldsymbol{\phi }^{(s)}(x_{1})-p_{s} \bigr) \\ &\qquad {}-\lambda _{0}\bigl( \boldsymbol{\omega ^{T}} \boldsymbol{\phi }(x_{N})+b -q_{0}\bigr)-\sum_{r=1}^{M-2-S} \lambda _{r}\bigl(\boldsymbol{\omega ^{T}}\boldsymbol{\phi } ^{(r)}(x_{N})-q_{r}\bigr). \end{aligned}$$
(12)

Then the KKT optimality conditions are given by

$$\begin{aligned}& \begin{gathered} \begin{aligned}\frac{\partial L}{\partial \boldsymbol{\omega }}={}& \boldsymbol{\omega }-\sum_{i=2}^{N-1}\alpha _{i} \Biggl[ \boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=1}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i}) \Biggr]-\sum _{i=2}^{N-1}\eta _{i} \boldsymbol{\phi }(x_{i})-\beta _{0}\boldsymbol{\phi }(x_{1}) \\ & {}-\sum_{s=1}^{S}\beta _{s}\boldsymbol{\phi }^{(s)}(x_{1})-\lambda _{0} \boldsymbol{\phi }(x_{N})-\sum _{r=1}^{M-2-S}\lambda _{r} \boldsymbol{\phi }^{(r)}(x_{N})=0; \end{aligned} \\ \frac{\partial L}{\partial \alpha _{i}}=\boldsymbol{\omega }^{T} \Biggl[\boldsymbol{\phi }^{(M)}(x _{i})+\sum_{l=1}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i}) \Biggr]-f(x_{i},y_{i})-e_{i}=0,\quad i=2,3, \ldots,N-1; \\ \frac{\partial L}{ \partial \eta _{i}}=\boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{i})+b+ \xi _{i}-y_{i}=0,\quad i=2,3,\ldots,N-1; \\ \frac{\partial L}{\partial \beta _{0}}=\boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{1})+b-p_{0}=0; \\ \frac{ \partial L}{\partial \beta _{s}}=\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(s)}(x_{1})-p_{s}=0,\quad s=1,2,\ldots,S; \\ \frac{\partial L}{\partial \lambda _{0}}=\boldsymbol{\omega }^{T}\boldsymbol{\phi }(x _{N})+b-q_{0}=0; \\ \frac{\partial L}{\partial \lambda _{r}}=\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(r)}(x_{N})-q_{r}=0,\quad r=1,2, \ldots,M-2-S; \\ \frac{ \partial L}{\partial b}=-\sum_{i=2}^{N-1}\eta _{i}-\beta _{0}-\lambda _{0}=0; \\ \frac{\partial L}{\partial y_{i}}=\eta _{i}+\alpha _{i} \frac{ \partial f(x_{i},y_{i})}{\partial y_{i}}=0,\quad i=2,3,\ldots,N-1; \\ \frac{ \partial L}{\partial e_{i}}=\alpha _{i}+\gamma e_{i}=0,\quad i=2,3, \ldots,N-1; \\ \frac{\partial L}{\partial \xi _{i}}=-\eta _{i}+\gamma \xi _{i}=0,\quad i=2,3, \ldots,N-1. \end{gathered} \end{aligned}$$
(13)

Finally, rewriting the above system in matrix form will result in (11). □

System (11) is solved by Newton’s method. Therefore, the LS-SVM model in the dual form becomes

$$ \begin{aligned}[b] \hat{y}(x) ={}&b+\sum _{i=2}^{N-1}\alpha _{i} \Biggl[\nabla _{M,0}\bigl(K(x_{i},x)\bigr)+ \sum _{l=1}^{M-1}a_{l}(x_{i})\nabla _{l,0}\bigl(K(x_{i},x)\bigr) \Biggr]\\ &{}+\sum _{i=2} ^{N-1}\eta _{i} \nabla _{0,0}\bigl(K(x_{i},x)\bigr) +\beta _{0} \nabla _{0,0}\bigl(K(x _{1},x)\bigr)+\sum _{s=1}^{S}\beta _{s}\nabla _{s,0}\bigl(K(x_{1},x)\bigr)\\ &{}+\lambda _{0} \nabla _{0,0}\bigl(K(x_{N},x)\bigr)+\sum _{r=1}^{M-2-S}\lambda _{r}\nabla _{r,0}\bigl(K(x _{N},x)\bigr). \end{aligned} $$
(14)

Linear ordinary differential equations for two-point boundary value problems

Two-point boundary value problems of Mth-order linear ordinary differential equations to be solved can be stated as follows:

$$ \frac{d^{M}y}{dx^{M}}+a_{M-1}(x)\,\frac{d^{M-1}y}{dx^{M-1}}+\cdots+a _{1}(x)\,\frac{dy}{dx}+a_{0}(x)y=r(x), \quad x \in [a,c], $$
(15)

subject to boundary conditions \(y^{(s)}(a)=p_{s}\), \(y^{(r)}(c)=q_{r}\), \(0\leq s\leq S\), \(0\leq r\leq R\), \(R=M-2-S\).

Assume that a general approximate solution to (15) is \(y= \boldsymbol{\omega }^{T}\boldsymbol{\phi }({x)}+b\). To obtain the optimal values of the parameters ω and b, collocation methods which discretize the interval \([a, c]\) into a series of collocation points \(\varOmega = \{a = x_{1} < x_{2} < \cdots < x_{N} = c \}\) can be used. Therefore, these parameters are obtained by solving the following optimization problem:

$$ \min _{\boldsymbol{\omega }, b, e_{i}} J( \boldsymbol{\omega },\boldsymbol{e})= \frac{1}{2}\boldsymbol{\omega } ^{T}\boldsymbol{\omega }+ \frac{1}{2}\gamma \boldsymbol{e}^{T} \boldsymbol{e} $$
(16)

subject to

$$ \begin{aligned} &\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=0} ^{M-1} \boldsymbol{\omega }^{T}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x _{i})+a_{0}(x_{i})b=r(x_{i})+e_{i},\quad i=2,\ldots,N-1; \\ & \boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{1})+b=p_{0}; \\ & \boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{N})+b=q_{0}; \\ & \boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(s)}(x_{1})=p_{s},\quad s=1,2, \ldots,S; \\ &\boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(r)}(x_{N})=q_{t},\quad r=1,2, \ldots,R. \end{aligned} $$

Theorem 2

Given a positive definite kernel function \(K: R\times R\rightarrow R\) and a regularization parameter \(\gamma \in R^{+}\), the solution to (16) is obtained by the following dual problem:

(17)

where \([\widehat{\varTheta }_{l,l'}]_{N-2}=[ \widetilde{\varTheta }_{M,M}]_{N-2}+ \overline{D}_{a_{l}}[ \overline{\varTheta }_{l,M}]_{N-2}+[\overline{ \varTheta }_{M,l'}]_{N-2}\overline{D}_{a_{l'}}^{T} + \overline{D}_{a_{l}}[\overline{ \varTheta }_{l,l'}]_{N-2}\overline{D}^{T}_{a_{l'}}+\gamma ^{-1}E\); \([\overline{ \varTheta }_{M,l'}]_{N-2}= [[\widetilde{\varTheta }_{M,0}]_{N-2},[ \widetilde{\varTheta }_{M,1}]_{N-2}, \ldots,[\widetilde{\varTheta }_{M,M-1}]_{N-2} ]\); \(\overline{D}_{a_{l'}}=[D_{a_{0}},D_{a_{1}}, \ldots,D_{a_{M-1}}]\); \([\overline{ \varTheta }_{l,M}]_{N-2}= [[\widetilde{\varTheta }_{0,M}]_{N-2}; [ \widetilde{ \varTheta }_{1,M}]_{N-2};\ldots;[\widetilde{\varTheta }_{M-1,M}]_{N-2} ]\); \(\overline{D}_{a_{l}}=[D_{a_{0}},D_{a_{1}}, \ldots,D_{a_{M-1}}]\); \([\overline{ \varTheta }_{l,l'}]_{N-2}=[ \widetilde{\varTheta }_{0:M-1,0:M-1}]_{N-2}\); \(D _{a_{l'}}=\operatorname{diag}(a_{l'}(t_{2}),a_{l'}(t_{3}), \ldots,a_{l'}(t_{N-1}))\); \(l,l'=0,1,\ldots, M-1\); \(D_{a_{l}}=\operatorname{diag}(a_{l}(t_{2}),a_{l}(t_{3}), \ldots,a_{l}(t_{N-1}))\); \(\boldsymbol{p}=[p_{0},p_{1},p_{2}, \ldots,p_{S}]^{T}\); \(\boldsymbol{q}=[q _{0},q_{1},q_{2}, \ldots,q_{R}]^{T}\); \([\widehat{\varTheta }_{s,l'}^{1}]_{N-2}= [ \widetilde{\varTheta }_{0:S,M}^{1}]_{N-2}+ [\overline{\varTheta }_{0:S,l'} ^{1}]_{N-2} \overline{D}_{a_{l'}}^{T}\); \([\widehat{\varTheta }_{r,l'}^{N}]_{N-2}=[ \widetilde{\varTheta }_{0:R,M}^{N}]_{N-2}+[\overline{\varTheta }_{0:R,l'} ^{N}]_{N-2}\overline{D}_{a_{l'}}^{T}\); \([\overline{\varTheta }_{0:S,l'} ^{1} ]_{N-2}= [[\widetilde{\varTheta }_{0:S,0}^{1} ]_{N-2},[ \widetilde{\varTheta }_{0:S,1}^{1} ]_{N-2},\ldots,[\widetilde{\varTheta }_{0:S,M-1} ^{1}]_{N-2} ]\); \(\boldsymbol{1}_{s}=[1;0; \ldots;0]_{S+1,1}\); \([\overline{ \varTheta }_{0:R,l'}^{N}]_{N-2}= [[\widetilde{\varTheta }_{0:R,0}^{N} ]_{N-2},[ \widetilde{\varTheta }_{0:R,1}^{N} ]_{N-2},\ldots,[\widetilde{\varTheta }_{0:R,M-1} ^{N}]_{N-2} ]\); \(\boldsymbol{\alpha }=[\alpha _{2},\alpha _{3},\ldots, \alpha _{N-1}]^{T}\); \([\widetilde{\varTheta }_{s,s'}]_{1,1}=[\varTheta _{0:S,0:S}]_{1,1}\); \([ \widetilde{\varTheta }_{r,s'}]_{N,1}=[ \varTheta _{0:R,0:S}]_{N,1}\); \([ \widetilde{\varTheta }_{r,r'}]_{N,N}=[\varTheta _{0:R,0:R}]_{N,N}\); \(\boldsymbol{\beta }=[\beta _{0},\beta _{1},\beta _{2},\ldots,\beta _{S}]^{T}\); \(\boldsymbol{\lambda }=[\lambda _{0},\lambda _{1},\lambda _{2}, \ldots, \lambda _{R}]^{T}\); \(\boldsymbol{1}_{r}=[1;0; \ldots;0]_{R+1,1}\); \(A=[a_{0}(x_{2}),a _{0}(x_{3}), \ldots,a_{0}(x_{N-1})]\); \(r(\boldsymbol{x})=[r(x_{2}),r(x_{3}),\ldots, r(x_{N-1})]^{T}\).

Proof

We construct the Lagrangian function of the optimization problem (16):

$$ \begin{aligned}[b] &L(\boldsymbol{\omega }, \alpha _{i},\beta _{0},\beta _{s},\lambda _{0}, \lambda _{r},b,e_{i})\\ &\quad =\frac{1}{2} \boldsymbol{\omega }^{T} \boldsymbol{\omega }+\frac{1}{2}\gamma \boldsymbol{e}^{T} \boldsymbol{e}\\ &\qquad {}-\sum_{i=2}^{N-1} \alpha _{i} \Biggl[\boldsymbol{\omega } ^{T}\boldsymbol{\phi }^{(M)}(x_{i})+\sum_{l=0}^{M-1} \boldsymbol{\omega }^{T} a_{l}(x_{i})\boldsymbol{\phi }^{(l)}(x _{i})+a_{0}(x_{i})b -r(x_{i})-e_{i} \Biggr]\\ &\qquad {}-\beta _{0}\bigl( \boldsymbol{\omega ^{T}} \boldsymbol{\phi }(x_{1})+b-p_{0}\bigr)-\sum _{s=1} ^{S}\beta _{s}\bigl(\boldsymbol{ \omega ^{T}} \boldsymbol{\phi }^{(s)}(x _{1})-p_{s}\bigr)- \lambda _{0}\bigl(\boldsymbol{\omega ^{T}}\boldsymbol{\phi }(x _{N})+b-q_{0}\bigr)\\ &\qquad {}-\sum_{r=1}^{M-2-S} \lambda _{r}\bigl(\boldsymbol{\omega ^{T}} \boldsymbol{\phi }^{(r)}(x_{N})-q_{r}\bigr). \end{aligned} $$
(18)

The conditions for optimality are as follows:

$$ \begin{aligned} &\begin{aligned}\frac{\partial L}{\partial \boldsymbol{\omega }}={}& \boldsymbol{\omega }-\sum_{i=2}^{N-1}\alpha _{i} \Biggl[ \boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=0}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i}) \Biggr]-\beta _{0} \boldsymbol{\phi }(x _{1})-\sum_{s=1}^{S} \beta _{s}\boldsymbol{\phi }^{(s)}(x_{1}) \\ & {} -\lambda _{0}\boldsymbol{\phi }(x_{N})-\sum _{r=1}^{M-2-S}\lambda _{r} \boldsymbol{\phi }^{(r)}(x_{N})=0; \end{aligned} \\ &\frac{\partial L}{\partial \alpha _{i}}=\boldsymbol{\omega }^{T} \Biggl[\boldsymbol{\phi }^{(M)}(x _{i})+\sum_{l=0}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i}) \Biggr]+a_{0}(x_{i})b -r(x_{i})-e_{i}=0,\quad i=2,3,\ldots,N-1; \\ &\frac{\partial L}{\partial \beta _{0}}=\boldsymbol{\omega }^{T}\boldsymbol{\phi }(x _{1})+b-p_{0}=0; \\ &\frac{\partial L}{\partial \beta _{s}}= \boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(s)}(x_{1})-p_{s}=0,\quad s=1,2,\ldots,S; \\ &\frac{\partial L}{\partial \lambda _{0}}=\boldsymbol{\omega }^{T} \boldsymbol{\phi }(x_{N})+b-q_{0}=0; \\ &\frac{\partial L}{\partial \lambda _{r}}=\boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(r)}(x_{N})-q _{r}=0,\quad r=1,2,\ldots,M-2-S; \\ &\frac{\partial L}{\partial b}=-\sum_{i=2} ^{N-1}a_{0}(x_{i}) \alpha _{i}-\beta _{0}-\lambda _{0}=0; \\ &\frac{\partial L}{\partial e_{i}}=\alpha _{i}+\gamma e_{i}=0,\quad i=2,3,\ldots,N-1. \end{aligned} $$
(19)

Finally, rewriting the above system in matrix form will result in (17). □

The linear system (17), which consists of unknowns \((\alpha , \beta , \lambda , b)\), is solved. The LS-SVM model in the dual form becomes

$$ \begin{aligned}[b] \hat{y}(x) ={}&\sum _{i=2}^{N-1}\alpha _{i} \Biggl[\nabla _{M,0}\bigl(K(x_{i},x)\bigr)+ \sum _{l=0}^{M-1}a_{l}(x_{i})\nabla _{l,0}\bigl(K(x_{i},x)\bigr) \Biggr]\\ &{}+\beta _{0} \nabla _{0,0}\bigl(K(x_{1},x)\bigr) +\sum_{s=1}^{S}\beta _{s} \nabla _{s,0}\bigl(K(x _{1},x)\bigr)\\ &{}+\lambda _{0} \nabla _{0,0}\bigl(K(x_{N},x)\bigr)+\sum _{r=1}^{R}\lambda _{r}\nabla _{r,0}\bigl(K(x_{N},x)\bigr)+b. \end{aligned} $$
(20)

Multi-point boundary value problems of high-order ordinary differential equations

The improved LS-SVM algorithms to the solution of multi-point boundary value problems of high-order linear and nonlinear ordinary differential equations are described.

Nonlinear ordinary differential equations for multi-point boundary value problems

Consider the following Mth-order nonlinear ordinary differential equations for multi-point boundary value problems [15]:

$$ \frac{d^{M}y}{dx^{M}}+a_{M-1}(x)\,\frac{d^{M-1}y}{dx^{M-1}}+\cdots+a _{1}(x)\,\frac{dy}{dx}=f(x,y), \quad x \in [a,c], $$
(21)

subject to \(y^{(q_{0})}(a)=s_{0}\), \(y^{(q_{j})}(x_{p_{j}})=s_{j}\), \(y^{(q _{M-1})}(c)=s_{M-1}\), \(x_{p_{j}}\in [a,c]\), \(p_{j}\in Z\), \(j=1,2,\ldots, M-2\), \(0 \leq q_{0},q_{1},\ldots, q_{M-1}\leq M-1\).

The interval \([a, c]\) is discretized into a series of collocation points \(\varOmega = \{a = x_{p_{0}}=x_{1} < x_{2} < \cdots<x_{p_{1}}<\cdots<x_{p_{2}}<\cdots<x _{p_{M-2}}<\cdots< x_{p_{M-1}}= x_{N} =c\}\). Assume that the approximate solution to (21) is \(y=\boldsymbol{\omega }^{T}\boldsymbol{\phi }( {x)}+b\), the primal optimization problem is described as follows:

$$ \min _{\boldsymbol{\omega }, b, e_{i}, \boldsymbol{\xi }, y_{i}} J( \boldsymbol{\omega },\boldsymbol{e}, \boldsymbol{\xi })=\frac{1}{2} \boldsymbol{\omega }^{T} \boldsymbol{\omega }+\frac{1}{2}\gamma \boldsymbol{e}^{T} \boldsymbol{e}+\frac{1}{2}\gamma \boldsymbol{\xi } ^{T} \boldsymbol{\xi } $$
(22)

subject to

$$ \begin{aligned} &\boldsymbol{\omega }^{T} \Biggl[\boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=1}^{M-1} a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i}) \Biggr]=f(x_{i},y _{i})+e_{i},\quad i=1,2,\ldots,N-M; \\ &y_{i}={\boldsymbol{\omega }^{T} \boldsymbol{\phi }(x_{i})+b}+\xi _{i},\quad i=1,2,\ldots,N-M; \\ & \boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(q_{0})}(x_{1})+b^{(q_{0})}=s _{0}; \\ &\boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(q_{j})}(x_{p _{j}})+b^{(q_{j})}=s_{j},\quad j=1,2, \ldots, M-2; \\ &\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(q_{M-1})}(x_{N})+b^{(q_{M-1})}=s_{M-1}. \end{aligned} $$

Theorem 3

Given a positive definite kernel function \(K: R\times R\rightarrow R\) and a regularization parameter \(\gamma \in R^{+}\), the solution to (22) is obtained by the following dual problem:

(23)

where \([\widehat{\varTheta }_{l,l'}]_{N-M}=[ \widetilde{\varTheta }_{M,M}]_{N-M}+ \overline{D}_{a_{l}}[ \overline{\varTheta }_{l,M}]_{N-M}+[\overline{ \varTheta }_{M,l'}]_{N-M}\overline{D}_{a_{l'}}^{T} + \overline{D}_{a_{l}} [\overline{\varTheta }_{l,l'}]_{N-M}\overline{D}^{T}_{a_{l'}}+ \gamma ^{-1}E\); \([\overline{\varTheta }_{M,l'}]_{N-M}= [[\widetilde{\varTheta } _{M,1}]_{N-M}, [\widetilde{ \varTheta }_{M,2}]_{N-M},\ldots,[ \widetilde{\varTheta }_{M,M-1}]_{N-M} ]\); \([\overline{\varTheta }_{l,M}]_{N-M}= [[\widetilde{ \varTheta }_{1,M}]_{N-M}; [\widetilde{\varTheta }_{2,M}]_{N-M};\ldots;[ \widetilde{\varTheta }_{M-1,M}]_{N-M} ]\); \(\overline{D}_{a_{l'}}=[D _{a_{1}},D_{a_{2}}, \ldots,D_{a_{M-1}}]\); \(\overline{D}_{a_{l}}=[D_{a_{1}},D _{a_{2}},\ldots,D_{a_{M-1}}]\); \([\overline{\varTheta }_{l,l'}]_{N-M}=[ \widetilde{\varTheta }_{1:M-1,1:M-1}]_{N-M}\); \(\boldsymbol{\alpha }=[\alpha _{2}, \ldots,\alpha _{p_{1}-1},\alpha _{p_{1}+1},\ldots,\alpha _{N-1}]^{T}\); \([ \widetilde{\varTheta }_{0,0}]_{N-M}= [\varTheta _{0,0}]_{1:N-M,1:N-M}+\gamma ^{-1}E\); \(l,l'=1,2, \ldots,M-1\); \(D_{a_{l}}=\operatorname{diag}(a_{l}(x_{2}), \ldots,a_{l}(x _{p_{1}-1}), a_{l}(x_{p_{1}+1}), \ldots, a_{l}(x_{N-1}))\); \(B=[\chi _{b_{0}}, \chi _{b_{1}},\ldots,\chi _{b_{M-1}}]\); \(D_{a_{l'}}=\operatorname{diag}(a_{l'}(x_{2}), \ldots,a _{l'}(x_{p_{1}-1}), a_{l'}(x_{p_{1}+1}), \ldots,a_{l'}(x_{N-1}))\); \([\widehat{\varTheta }_{0,l'}]_{N-M}=[ \widetilde{\varTheta }_{0,M}]_{N-M}+[\overline{ \varTheta }_{0,l'}]_{N-M}\overline{D}_{a_{l'}}^{T}\); \(\boldsymbol{s}=[s_{0},s _{1},s_{2}, \ldots,s_{M-1}]^{T}\); \([\overline{\varTheta }_{0,l'}]_{N-M}= [[\widetilde{ \varTheta }_{0,1}]_{N-M}, [\widetilde{\varTheta }_{0,2}]_{N-M},\ldots,[ \widetilde{\varTheta }_{0,M-1}]_{N-M} ]\); \(\boldsymbol{y}=[y_{2}, \ldots,y _{P_{1}-1},y_{P_{1}+1},\ldots,y_{N-1}]^{T}\); \([\widehat{\varTheta }_{q_{j},l'}]_{M,N-M}= [ \widetilde{\varTheta }_{q_{0}:q_{M-1},M}]_{M,N-M}+[\overline{\varTheta } _{q_{0}:q_{M-1},l'}]_{M,N-M} \overline{D}_{a_{l'}}^{T}\); \(\boldsymbol{\beta }=[\beta _{0},\beta _{1},\beta _{2},\ldots,\beta_{M-1}]^{T}\); \([\overline{\varTheta }_{q_{0}:q_{M-1},l'}]_{M,N-M}= [[ \widetilde{ \varTheta }_{q_{0}:q_{M-1},1}]_{M,N-M},[\widetilde{\varTheta } _{q_{0}:q_{M-1},2}]_{M,N-M},\ldots,[\widetilde{\varTheta }_{q_{0}:q_{M-1},M-1}]_{M,N-M} ]\); \([\widetilde{\varTheta }_{q_{j},0}]_{M,N-M}=[ \varTheta _{q_{0}:q_{M-1},0}]_{M,N-M}\); \(\boldsymbol{\eta }=[\eta _{2}, \ldots, \eta _{p_{1}-1},\eta _{p_{1}+1},\ldots,\eta _{N-1}]^{T}\); \([ \widetilde{\varTheta }_{q_{j},q_{j'}}]_{p_{j},p_{j'}}= [ \varTheta _{q_{0}:q_{M-1},q_{0}:q_{M-1}}]_{p_{0}:p_{M-1},p_{0}:p_{M-1}}\); \(D _{N-M}(\boldsymbol{y})= \operatorname{diag}(\frac{\partial f(\boldsymbol{x}, \boldsymbol{y})}{\partial y})\); \(f_{N-M}(\boldsymbol{x}, \boldsymbol{y})=[f(x_{2},y_{2}), \ldots, f(x_{p_{1}-1},y_{p_{1}-1}),f(x _{p_{1}+1},y_{p_{1}+1}),\ldots,f(x_{N-1},y_{N-1}) ]^{T}\); \(\frac{\partial f(\boldsymbol{x},\boldsymbol{y})}{\partial y}= [\frac{\partial f(x,y)}{ \partial y}|_{\substack{x=x_{2}\\y=y_{2}}},\ldots, \frac{\partial f(x,y)}{ \partial y}|_{\substack{x=x_{p_{1}-1}\\y=y_{p_{1}-1}}}, \frac{ \partial f(x,y)}{\partial y}|_{\substack{x=x_{p_{1}+1}\\y=y_{p _{1}+1}}},\ldots, \frac{\partial f(x,y))}{\partial y}|_{\substack{x=x _{N-1}\\y=y_{N-1}}} ]\), where \([\widetilde{\varTheta }_{m,n}]_{N-M}=[\varTheta _{m,n}]_{1:N-M,1:N-M};[ \widetilde{\varTheta }_{q_{0}:q_{M-1},m}]_{M,N-M}=[ \varTheta _{q_{0}:q_{M-1},m}]_{p _{0}:p_{M-1},1:N-M}\); \(m,n=0,1,\ldots,M-1\).

Proof

The Lagrangian function of the constrained optimization problem (22) is introduced as follows:

$$ \begin{aligned}[b] &L(\boldsymbol{\omega }, y_{i},\alpha _{i},\eta _{i},\beta _{j},b,e_{i}, \xi _{i})\\ &\quad =\frac{1}{2} \boldsymbol{\omega }^{T}\boldsymbol{\omega }+ \frac{1}{2}\gamma \boldsymbol{e}^{T}\boldsymbol{e}+\frac{1}{2}\gamma \boldsymbol{\xi }^{T}\boldsymbol{\xi }\\ &\qquad {}-\sum_{i=1}^{N-M} \alpha _{i} \Biggl[\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(M)}(x_{i})+ \sum _{l=1}^{M-1}\boldsymbol{\omega }^{T}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i})-f(x_{i},y_{i})-e_{i} \Biggr]\\ &\qquad {}-\sum_{i=1} ^{N-M}\eta _{i}\bigl(\boldsymbol{\omega }^{T}\boldsymbol{\phi }(x_{i})+b +\xi _{i}-y_{i}\bigr)-\sum_{j=0}^{M-1} \beta _{j} \bigl( \boldsymbol{\omega ^{T}}\boldsymbol{\phi }^{(q_{j})}(x_{p_{j}})+b^{(q _{j})}-s_{j} \bigr). \end{aligned} $$
(24)

The conditions for optimality

$$\begin{aligned}& \begin{gathered} \frac{\partial L}{\partial \boldsymbol{\omega }}= \boldsymbol{\omega }-\sum_{i=1}^{N-M}\alpha _{i} \Biggl[ \boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=1}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i})) \Biggr]-\sum _{i=1}^{N-M}\eta _{i} \boldsymbol{\phi }(x_{i})-\sum_{j=0}^{M-1}\beta _{j} \boldsymbol{\phi }^{(q_{j})}(x_{p_{j}})=0; \\ \frac{\partial L}{ \partial \alpha _{i}}=\boldsymbol{\omega }^{T} \Biggl(\boldsymbol{\phi } ^{(M)}(x_{i})+\sum_{l=1}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x _{i}) \Biggr) -f(x_{i},y_{i})-e_{i}=0,\quad i=1,2,\ldots,N-M; \\ \frac{\partial L}{\partial \eta _{i}}=\boldsymbol{\omega }^{T}\boldsymbol{\phi }(x _{i})+b+\xi _{i}-y_{i}=0,\quad i=1,2,\ldots,N-M; \\ \frac{\partial L}{\partial \xi _{i}}=-\eta _{i}+\gamma \xi _{i}=0,\quad i=1,2, \ldots,N-M; \\ \frac{\partial L}{\partial e_{i}}=\alpha _{i}+\gamma e_{i}=0,\quad i=1,2, \ldots,N-M; \\ \frac{ \partial L}{\partial \beta _{j}}=\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(q_{j})}(x_{p_{j}})+b^{(q_{j})}-s_{j}=0,\quad j=0,1, \ldots, M-1; \\ \frac{\partial L}{\partial b}=-\sum_{i=1}^{N-M}\eta _{i}-\sum_{j=0}^{M-1}\beta _{j}\chi _{b_{j}}=0,\quad \chi _{b_{j}}= \textstyle\begin{cases} 1,& q_{j}=0;\\ 0,& q_{j}=1,2,\ldots,M-1; \end{cases}\displaystyle \\ \frac{\partial L}{\partial y_{i}}=\eta _{i}+\alpha _{i} \frac{ \partial f(x_{i},y_{i})}{\partial y_{i}}=0 \end{gathered} \end{aligned}$$
(25)

can be written as a system in matrix form (23), after eliminating parameters ω and \(e_{i}\). □

System (23), which consists of \(3N-2M+1\) equations with unknowns \((\alpha ,\eta ,\beta ,b, y)\), is solved by Newton’s method. The LS-SVM model in the dual form becomes

$$ \begin{aligned}[b] \hat{y}(x) ={}&\sum _{i=1}^{N-M}\alpha _{i} \Biggl[\nabla _{M,0}\bigl(K(x_{i},x)\bigr)+ \sum _{l=1}^{M-1}a_{l}(x_{i})\nabla _{l,0}\bigl(K(x_{i},x)\bigr) \Biggr]\\ &{}+\sum _{i=1} ^{N-M}\eta _{i} \nabla _{0,0}\bigl(K(x_{i},x)\bigr) +\sum _{j=0}^{M-1}\beta _{j}\nabla _{q_{j},0}\bigl(K(x_{p_{j}},x)\bigr)+b. \end{aligned} $$
(26)

Linear ordinary differential equations for multi-point boundary value problems

Consider the following Mth-order linear ordinary differential equations for multi-point boundary value problems:

$$ \frac{d^{M}y}{dx^{M}}+a_{M-1}(x)\frac{d^{M-1}y}{dx^{M-1}}+\cdots+a _{1}(x)\frac{dy}{dx}+a_{0}(x)y=r(x), \quad x \in [a,c], $$
(27)

subject to \(y^{(q_{0})}(a)=s_{0}\), \(y^{(q_{j})}(x_{p_{j}})=s_{j}\), \(y^{(q _{M-1})}(c)=s_{M-1}\), \(x_{p_{j}}\in [a,c]\), \(p_{j}\in Z\), \(j=1,2,\ldots, M-2\), \(0 \leq q_{0},q_{1},\ldots, q_{M-1}\leq M-1\).

The interval \([a, c]\) is discretized into a series of collocation points \(\varOmega = \{a = x_{p_{0}}=x_{1} < x_{2} < \cdots<x_{p_{1}}<\cdots<x_{p_{2}}<\cdots<x _{p_{M-2}}<\cdots< x_{p_{M-1}}=x_{N} = c\}\). Suppose that the approximate solution to (27) is \(y=\boldsymbol{\omega }^{T}\boldsymbol{\phi }( {x)}+b\), the original optimal problem is described as follows:

$$ \operatorname*{min}\limits _{\boldsymbol{\omega }, b, e_{i}} J( \boldsymbol{\omega },\boldsymbol{e})= \frac{1}{2}\boldsymbol{\omega } ^{T}\boldsymbol{\omega }+ \frac{1}{2}\gamma \boldsymbol{e}^{T} \boldsymbol{e} $$
(28)

subject to

$$ \begin{aligned} &\boldsymbol{\omega }^{T} \Biggl[\boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=0}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i})+a_{0}(x_{i})b \Biggr]=r(x_{i})+e_{i}, \\ & \quad i=2,3,\ldots,p_{1}-1, p_{1}+1,\ldots, N-1; \\ &\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(q_{0})}(x_{1})+b^{(q_{0})}=s_{0}; \\ & \boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(q_{j})}(x_{p_{j}})+b^{(q _{j})}=s_{j},\quad j=1,2, \ldots, M-2; \\ &\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(q_{M-1})}(x_{N})+b^{(q_{M-1})}=s_{M-1}. \end{aligned} $$

Theorem 4

Given a positive definite kernel function \(K: R\times R\rightarrow R\) and a regularization parameter \(\gamma \in R^{+}\), the solution to (28) is obtained by the following dual problem:

(29)

where \([\widehat{\varTheta }_{l,l'}]_{N-M}=[ \widetilde{\varTheta }_{M,M}]_{N-M}+ \overline{D}_{a_{l}}[ \overline{\varTheta }_{l,M}]_{N-M}+[\overline{ \varTheta }_{M,l'}]_{N-M}\overline{D}_{a_{l'}}^{T} + \overline{D}_{a_{l}} [\overline{\varTheta }_{l,l'}]_{N-M}\overline{D}^{T}_{a_{l'}}+ \gamma ^{-1}E\); \([\overline{\varTheta }_{M,l'}]_{N-M}= [[\widetilde{\varTheta } _{M,0}]_{N-M},[\widetilde{ \varTheta }_{M,1}]_{N-M},\ldots,[ \widetilde{\varTheta }_{M,M-1}]_{N-M} ]\); \(\overline{D}_{a_{l'}}=[D _{a_{0}},D_{a_{1}}, \ldots,D_{a_{M-1}}]\); \(\overline{D}_{a_{l}}=[D_{a_{0}},D _{a_{1}},\ldots,D_{a_{M-1}}]\); \([\overline{\varTheta }_{l,l'}]_{N-M}=[ \widetilde{\varTheta }_{0:M-1,0:M-1}]_{N-M}\); \([\overline{\varTheta }_{l,M}]_{N-M}= [[\widetilde{ \varTheta }_{0,M}]_{N-M}; [\widetilde{\varTheta }_{1,M}]_{N-M};\ldots;[ \widetilde{\varTheta }_{M-1,M}]_{N-M} ]\); \(D_{a_{l'}}=\operatorname{diag}(a_{l'}(x _{2}), \ldots,a_{l'}(x_{p_{1}-1}),a_{l'}(x_{p_{1}+1}), \ldots,a_{l'}(x_{N-1}))\); \(\boldsymbol{s}=[s_{0},s_{1},s_{2}, \ldots,s_{M-1}]^{T}\); \(D_{a_{l}}=\operatorname{diag}(a _{l}(x_{2}), \ldots,a_{l}(x_{p_{1}-1}),a_{l}(x_{p_{1}+1}), \ldots,a_{l}(x_{N-1}))\); \(l,l'=0,1, \ldots,M-1\); \([\widehat{\varTheta }_{q_{j},l'}]_{M,N-M}=[\widetilde{\varTheta}_{q _{j},M}]_{M,N-M}+[\overline{\varTheta }_{q_{j},l'}]_{M,N-M} \overline{D} _{a_{l'}}^{T} \); \(\boldsymbol{\beta }=[\beta _{0},\beta _{1},\beta _{2},\ldots, \beta _{M-1}]^{T}\); \([\overline{\varTheta }_{q_{j},l'}]_{M,N-M}= [[ \widetilde{ \varTheta }_{q_{j},0}]_{M,N-M},[\widetilde{\varTheta }_{q_{j},1}]_{M,N-M},\ldots,[ \widetilde{\varTheta }_{q_{j},M-1}]_{M,N-M} ]\); \([ \widetilde{\varTheta }_{q_{j},l'}]_{M,N-M}= [\varTheta _{q_{j},l'}]_{p_{0}:p_{M-1},1:N-M}\); \([\widetilde{\varTheta }_{q_{j},q_{j'}}]_{p_{j},p_{j'}}=[ \varTheta _{q_{0}:q_{M-1},q_{0}:q_{M-1}}]_{p_{0}:p_{M-1},p_{0}:p_{M-1}}\); \(A=[a_{l}(x_{2}),\ldots,a_{l}(x_{p_{1}-1}), a_{l}(x_{p_{1}+1}), \ldots,a _{l}(x_{N-1})]\); \(\boldsymbol{\alpha }=[\alpha _{2},\ldots,\alpha _{x_{p_{1}-1}}, \alpha _{x_{p_{1}+1}},\ldots, \alpha _{N-1}]^{T}\); \(B=[\chi _{b_{0}},\chi _{b_{1}},\ldots,\chi _{b_{M-1}}]\); \(r( \boldsymbol{x})=[r(x_{2}),\ldots,r(x_{p _{1}-1}),r(x_{p_{1}+1}), \ldots,r(x_{N-1}) ]^{T}\).

Proof

The Lagrangian function of the optimization problem (28) becomes

$$ \begin{aligned}[b] &L(\boldsymbol{\omega }, \alpha _{i},\beta _{j},b,e_{i})\\ &\quad =\frac{1}{2} \boldsymbol{\omega }^{T}\boldsymbol{\omega }+\frac{1}{2}\gamma \boldsymbol{e}^{T}\boldsymbol{e}\\ &\qquad {}-\sum_{i=1}^{N-M} \alpha _{i} \Biggl[ \boldsymbol{\omega }^{T}\boldsymbol{\phi }^{(M)}(x_{i})+\sum_{l=0} ^{M-1}\boldsymbol{\omega }^{T}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x _{i})+a_{0}(x_{i})b -r(x_{i})-e_{i} \Biggr]\\ &\qquad {}-\sum_{j=0}^{M-1} \beta _{j} \bigl(\boldsymbol{\omega ^{T}}\boldsymbol{\phi }^{(q_{j})}(x_{p_{j}})+b ^{(q_{j})}-s_{j} \bigr). \end{aligned} $$
(30)

Setting the partial derivatives of the Lagrangian function to zero, we will obtain

$$\begin{aligned} &\frac{\partial L}{\partial \boldsymbol{\omega }}= \boldsymbol{\omega }-\sum_{i=1}^{N-M}\alpha _{i} \Biggl( \boldsymbol{\phi }^{(M)}(x_{i})+\sum _{l=0}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x_{i}) \Biggr)-\sum _{j=0}^{M-1}\beta _{j} \boldsymbol{\phi }^{(q_{j})}(x_{p_{j}})=0; \\ &\frac{\partial L}{ \partial \alpha _{i}}=\boldsymbol{\omega }^{T} \Biggl(\boldsymbol{\phi } ^{(M)}(x_{i})+\sum_{l=0}^{M-1}a_{l}(x_{i}) \boldsymbol{\phi }^{(l)}(x _{i}) \Biggr)+a_{0}(x_{i})b -r(x_{i})-e_{i}=0,\quad i=1,2,\ldots,N-M; \\ &\frac{ \partial L}{\partial e_{i}}=\alpha _{i}+\gamma e_{i}=0,\quad i=1,2, \ldots,N-M; \\ &\frac{\partial L}{\partial \beta _{j}}=\boldsymbol{\omega }^{T} \boldsymbol{\phi }^{(q_{j})}(x_{p_{j}})+b^{(q_{j})}-s_{j}=0,\quad j=0,1, \ldots, M-1; \\ &\frac{\partial L}{\partial b}=-\sum_{i=1}^{N-M}a_{0}(x_{i}) \alpha _{i}-\sum_{j=0}^{M-1}\beta _{j}\chi _{b_{j}}=0,\quad \chi _{b_{j}}= \textstyle\begin{cases} 1,& q_{j}=0;\\ 0, &q_{j}=1,2,\ldots,M-1. \end{cases}\displaystyle \end{aligned}$$
(31)

Finally, rewriting system (31) in matrix form will result in (29). □

System (29) with unknowns \((\alpha ,\beta , b)\) is solved. The LS-SVM model in the dual form becomes

$$ \begin{aligned}[b] \hat{y}(x) ={}&\sum _{i=1}^{N-M}\alpha _{i} \Biggl[\nabla _{M,0}\bigl(K(x_{i},x)\bigr)+ \sum _{l=0}^{M-1}a_{l}(x_{i})\nabla _{l,0}\bigl(K(x_{i},x)\bigr) \Biggr]\\ &{}+\sum _{j=0} ^{M-1}\beta _{j}\nabla _{q_{j},0}\bigl(K(x_{p_{j}},x)\bigr)+b. \end{aligned} $$
(32)

Numerical experiments

In this section, some numerical experiments are performed in order to demonstrate the reliability and powerfulness of the improved LS-SVM algorithms. The algorithms are applied to third-order, fourth-order linear and nonlinear ordinary differential equations with two-point boundary conditions and to third-order, fourth-order linear and nonlinear ordinary differential equations with multi-point boundary conditions.

In our experiments, the performance of the proposed LS-SVM algorithms is directly related to the choice of the regularization parameter γ and the kernel parameter σ. The larger the regularization parameter γ is, the smaller the error \(e_{i}\) is, but when γ is a quite large value, the system of equations will be ill-conditioning. Therefore, the chosen value for γ was 1010. The validation set is obtained to be the set of midpoints \(Z=\{z_{i}|z_{i}=(x_{i}+x_{i+1})/2,i =1,\ldots,N-1\}\), where \(\{x_{i}\}_{i=1}^{N}\) are training points [42]. The optimal parameter σ that results in minimum root mean squared error (RMSE) on the validation set is selected and used for evaluating the LS-SVM model on the test set. The RMSE is defined as follows:

$$ \mathrm{RMSE}=\sqrt{\frac{1}{M}\sum _{i=1}^{M}\bigl[y(z_{i})- \hat{y}(z_{i})\bigr]^{2}}. $$
(33)

Example 1

Consider the fourth-order nonlinear ordinary differential equation [51]:

$$ \frac{d^{4}y}{dx^{4}}=-\frac{x^{2}}{1+y^{2}}-72\bigl(1-5x+5x^{2} \bigr)+\frac{x ^{2}}{1+(x-x^{2})^{6}}, \quad x \in [0, 1], $$
(34)

subject to two-point boundary conditions \(y(0)=0\), \(y'(0)=0\), \(y(1)=0\), \(y'(1)=0\). The analytic solution is \(y=x^{3}(1-x)^{3}\).

We train the proposed LS-SVM algorithm for 11 equidistant points in the given interval \([0,1]\). The exact solution and the approximate solution via our proposed LS-SVM algorithm are shown in Fig. 1(a). Furthermore, the error between the analytic solution and the approximate solution is plotted in Fig. 1(b). In spite of using fewer points, we can see that the proposed LS-SVM algorithm could have a much better performance in terms of accuracy. The mean squared error is \(6.5732\times 10^{-15}\) and the maximum absolute error is approximately \(1.1063\times 10^{-7}\).

Figure 1
figure1

Two-point BVP of fourth-order nonlinear ODE (Example 1)

Table 1 lists the results of the exact solution and the approximate solution via our proposed LS-SVM algorithm for 11 testing points at unequal intervals in the domain \([0, 1]\). The absolute errors are shown in Table 1, in which we can see that the maximum absolute error is approximately \(1.1293\times 10^{-7}\).

Table 1 Comparison between the exact solution and the LS-SVM solution (Example 1)

Figure 2 shows the logarithmic relation between the kernel bandwidth and the RMSE in Example 1. The red circle indicates the location of selected kernel bandwidth.

Figure 2
figure2

The logarithmic relation between σ and RMSE (Example 1)

Example 2

Let us consider the fourth-order linear ordinary differential equation [34]:

$$ \frac{d^{4}y}{dx^{4}}=120x, \quad x \in [-1, 1], $$
(35)

subject to two-point boundary conditions \(y(-1)=1\), \(y'(-1)=5\), \(y(1)=3\), \(y'(1)=5\). The analytic solution is \(y=x^{5}+2\).

The proposed LS-SVM model has been trained with 11 equidistant points in the given interval \([-1,1]\). Figure 3(a) shows comparison between the exact solution and the approximate solution via our proposed LS-SVM algorithm, and Fig. 3(b) depicts the error plot between the analytic solution and the approximate solution. From the obtained results, we can see that the mean squared error is \(6.5835\times 10^{-12}\) and the maximum absolute error is approximately \(3.7390\times 10^{-6}\). The error obtained by the proposed LS-SVM algorithm remains low for the training points.

Figure 3
figure3

Two-point BVP of fourth-order linear ODE (Example 2)

Finally, the test results of the exact solution and the approximate solution via our proposed LS-SVM algorithm for 11 equidistant points in the domain \([-1, 1]\) are listed in Table 2. The absolute errors are shown in Table 2, in which we can see that the maximum absolute error is approximately \(3.7071\times 10^{-6}\). It is clear that the proposed LS-SVM algorithm has a better performance in terms of accuracy.

Table 2 Comparison between the exact solution and the LS-SVM solution (Example 2)

Example 3

Consider the fourth-order linear ordinary differential equation [52]:

$$ \frac{d^{4}y}{dx^{4}} + y(x)= \biggl(\biggl(\frac{\pi }{2} \biggr)^{4}+1 \biggr)\cos \biggl(\frac{\pi }{2}x \biggr) , \quad x \in [-1, 1], $$
(36)

subject to two-point boundary conditions \(y(-1)=0\), \(y'(-1)=\pi /2\), \(y(1)=0\), \(y'(1)=-\pi /2\). The analytic solution is \(y=\cos (\pi x/2 )\).

The proposed LS-SVM algorithm for two-point boundary value problems of high-order linear ordinary differential equation has been trained with 11 equidistant points in the given interval \([-1,1]\). Comparison between the exact solution and the approximate solution via our proposed LS-SVM algorithm is depicted in Fig. 4(a). Plot of the error function is cited in Fig. 4(b), from which we can see that the mean squared error is \(2.6426\times 10^{-18}\) and the maximum absolute error is approximately \(2.5670\times 10^{-9}\). The accuracy of the error obtained by the proposed LS-SVM algorithm is \(O(10^{-9})\). The results reveal that the proposed LS-SVM algorithm has higher accuracy, although we only choose 11 equidistant points for training process.

Figure 4
figure4

Two-point BVP of fourth-order linear ODE (Example 3)

Finally, Table 3 incorporates results of the exact solution and the approximate solution via our proposed LS-SVM algorithm for 11 testing points at unequal intervals in the domain \([-1, 1]\). The absolute errors are shown in Table 3, in which we can see that the maximum absolute error is approximately \(2.5543\times 10^{-9}\).

Table 3 Comparison between the exact solution and the LS-SVM solution (Example 3)

Example 4

Consider the third-order nonlinear ordinary differential equation [15]:

$$ \frac{d^{3}y}{dx^{3}}=-y^{2}-\cos (x)+\sin ^{2}(x), \quad x \in [0, 1], $$
(37)

subject to multi-point boundary conditions \(y'(0)=1\), \(y(\frac{1}{2})=\sin ( \frac{1}{2})\), \(y'(1)=\cos (1)\). The analytic solution is \(y=\sin (x)\).

When 11 equidistant points in the interval \([0, 1]\) are used for training, the results are depicted in Fig. 5(a). Figure 5(b) shows the errors between the exact solution and the approximate solution obtained by the proposed LS-SVM algorithm. From the obtained results, although training was performed just for 11 equidistant points in the domain \([0, 1]\), the mean squared error is approximately \(4.3564\times 10^{-7}\). The proposed LS-SVM algorithm obtains a satisfactory result for multi-point boundary value problems of third-order nonlinear ordinary differential equation.

Figure 5
figure5

Multi-point BVP of third-order nonlinear ODE (Example 4)

Finally, Table 4 tabulates results of the exact solution and the approximate solution via our proposed LS-SVM algorithm for 11 testing points at unequal intervals in the domain \([0, 1]\). The absolute errors are shown in Table 4, in which we can see that the mean squared error is approximately \(4.9717\times 10^{-7}\).

Table 4 Comparison between the exact solution and the LS-SVM solution (Example 4)

Example 5

Consider the fourth-order linear ordinary differential equation:

$$ \frac{d^{4}y}{dx^{4}}+\frac{dy}{dx}=4x^{3}+24, \quad x \in [0, 1], $$
(38)

subject to multi-point boundary conditions \(y(0)=0\), \(y'''(0.25)=6\), \(y''(0.5)=3\), \(y(1)=1\). The analytic solution is \(y=x^{4}\).

When 21 equidistant points in the interval \([0, 1]\) are used for training, the approximate solution obtained by the proposed LS-SVM algorithm is compared with the exact solution in Fig. 6(a), and the error is plotted in Fig. 6(b). From which, the mean squared error is approximately \(2.2915\times 10^{-10}\). The proposed LS-SVM algorithm can obtain the desired accuracy, although the training was performed using just a small part points in the domain \([0, 1]\).

Figure 6
figure6

Multi-point BVP of third-order linear ODE (Example 5)

The test results of the exact solution and the approximate solution via our proposed LS-SVM algorithm for 20 equidistant points in the domain \([0, 1]\) are listed in Table 5, and the absolute error is also calculated in Table 5. We can see that the mean squared error is approximately \(2.3557\times 10^{-10}\) and the maximum absolute error is approximately \(2.7702\times 10^{-5}\). The improved LS-SVM algorithm has a good performance for solving multi-point boundary value problems of fourth-order linear ordinary differential equations.

Table 5 Comparison between exact solution and LS-SVM solution (Example 5)

Conclusion

In this paper, the improved LS-SVM algorithms have been developed for solving two-point and multi-point boundary value problems of high-order linear and nonlinear ordinary differential equations. Accuracy of the improved LS-SVM algorithms has been checked by solving a fourth-order nonlinear ordinary differential equation with two-point boundary conditions, two fourth-order linear ordinary differential equations with two-point boundary conditions, a three-order nonlinear ordinary differential equation with multi-point boundary conditions, and a fourth-order linear ordinary differential equation with multi-point boundary conditions. The results obtained by the improved LS-SVM algorithms are compared with the exact solution. It has been noted that our proposed LS-SVM algorithms can solve two-point and multi-point boundary value problems of high-order linear and nonlinear ordinary differential equations with higher accuracy in the tables and graphs. So the improved LS-SVM algorithms in the use of the two-point and multi-point boundary value problems are found to be efficient and straightforward.

References

  1. 1.

    Chawla, M.M., Katti, C.P.: Finite difference methods for two-point boundary value problems involving high order differential equations. BIT Numer. Math. 19, 27–33 (1979)

    MathSciNet  Article  Google Scholar 

  2. 2.

    Mohyud-Din, S.T., Noor, M.A.: Homotopy perturbation method for solving fourth-order boundary value problems. Math. Probl. Eng. 2007, Article ID 98602 (2007)

    MathSciNet  Article  Google Scholar 

  3. 3.

    Noor, M.A., Mohyud-Din, S.T.: Homotopy perturbation method for solving sixth-order boundary value problems. Comput. Math. Appl. 55, 2953–2972 (2008)

    MathSciNet  Article  Google Scholar 

  4. 4.

    Ali, J., Islam, S., Islam, S., Zaman, G.: The solution of multipoint boundary value problems by the optimal homotopy asymptotic method. Comput. Math. Appl. 59, 2000–2006 (2010)

    MathSciNet  Article  Google Scholar 

  5. 5.

    Tatari, M., Dehghan, M.: The use of the Adomian decomposition method for solving multipoint boundary value problems. Phys. Scr. 73, 672–676 (2006)

    Article  Google Scholar 

  6. 6.

    Wazwaz, A.M.: A new algorithm for calculating Adomian polynomials for nonlinear operators. Appl. Math. Comput. 111, 53–69 (2000)

    MathSciNet  MATH  Google Scholar 

  7. 7.

    Wazwaz, A.M.: The numerical solution of fifth-order boundary value problems by the decomposition method. J. Comput. Appl. Math. 136, 259–270 (2001)

    MathSciNet  Article  Google Scholar 

  8. 8.

    Wazwaz, A.M.: The numerical solution of sixth-order boundary value problems by the modified decomposition method. Appl. Math. Comput. 118, 311–325 (2001)

    MathSciNet  MATH  Google Scholar 

  9. 9.

    Wazwaz, A.M.: Approximate solutions to boundary value problems of higher order by the modified decomposition method. Comput. Math. Appl. 40, 679–691 (2000)

    MathSciNet  Article  Google Scholar 

  10. 10.

    Wazwaz, A.M.: The modified decomposition method for analytic treatment of differential equations. Appl. Math. Comput. 173, 165–176 (2006)

    MathSciNet  MATH  Google Scholar 

  11. 11.

    Aziz, I., Siraj-ul-Islam, Nisar, M.: An efficient numerical algorithm based on Haar wavelet for solving a class of linear and nonlinear nonlocal boundary-value problems. Calcolo 53, 621–633 (2016)

    MathSciNet  Article  Google Scholar 

  12. 12.

    Shi, Z., Li, F.: Numerical solution of high-order differential equations by using periodized Shannon wavelets. Appl. Math. Model. 38, 2235–2248 (2014)

    MathSciNet  Article  Google Scholar 

  13. 13.

    Doha, E.H., Bhrawy, A.H., Hafez, R.M.: A Jacobi–Jacobi dual-Petrov–Galerkin method for third- and fifth-order differential equations. Math. Comput. Model. 53, 1820–1832 (2011)

    MathSciNet  Article  Google Scholar 

  14. 14.

    Doha, E.H., Abd-Elhameed, W.M., Bassuony, M.A.: New algorithms for solving high even-order differential equations using third and fourth Chebyshev–Galerkin methods. J. Comput. Phys. 236, 563–579 (2013)

    MathSciNet  Article  Google Scholar 

  15. 15.

    Doha, E.H., Bhrawy, A.H., Hafez, R.M.: On shifted Jacobi spectral method for high-order multi-point boundary value problems. Commun. Nonlinear Sci. Numer. Simul. 17, 3802–3810 (2012)

    MathSciNet  Article  Google Scholar 

  16. 16.

    Saadatmandi, A., Dehghan, M.: The use of sinc-collocation method for solving multi–point boundary value problems. Commun. Nonlinear Sci. Numer. Simul. 17, 593–601 (2012)

    MathSciNet  Article  Google Scholar 

  17. 17.

    Noor, M.A., Mohyud-Din, S.T.: Variational iteration technique for solving higher order boundary value problems. Appl. Math. Comput. 189, 1929–1942 (2007)

    MathSciNet  MATH  Google Scholar 

  18. 18.

    Xu, L.: The variational iteration method for fourth order boundary value problems. Chaos Solitons Fractals 39, 1386–1394 (2009)

    Article  Google Scholar 

  19. 19.

    Noor, M.A., Mohyud-Din, S.T.: Modified variational iteration method for solving fourth-order boundary value problems. J. Appl. Math. Comput. 29, 81–94 (2009)

    MathSciNet  Article  Google Scholar 

  20. 20.

    Hou, M., Han, X.: The multidimensional function approximation based on constructive wavelet RBF neural network. Appl. Soft Comput. 11(2), 2173–2177 (2011)

    Article  Google Scholar 

  21. 21.

    Hou, M., Han, X.: Multivariate numerical approximation using constructive \(L^{2}(R)\) RBF neural network. Neural Comput. Appl. 21(1), 25–34 (2012)

    Article  Google Scholar 

  22. 22.

    Hou, M., Han, X.: Constructive approximation to multivariate function by decay RBF neural network. IEEE Trans. Neural Netw. 21(9), 1517–1523 (2010)

    Article  Google Scholar 

  23. 23.

    Yang, Y., Hou, M., Luo, J.: A novel improved extreme learning machine algorithm in solving ordinary differential equations by Legendre neural network methods. Adv. Differ. Equ. 2018, 469 (2018)

    MathSciNet  Article  Google Scholar 

  24. 24.

    Mall, S., Chakraverty, S.: Application of Legendre neural network for solving ordinary differential equations. Appl. Soft Comput. 43, 347–356 (2016)

    Article  Google Scholar 

  25. 25.

    Rudd, K., Ferrari, S.: A constrained integration (CINT) approach to solving partial differential equations using artificial neural networks. Neurocomputing 155, 277–285 (2015)

    Article  Google Scholar 

  26. 26.

    Sun, H., Hou, M., Yang, Y., Zhang, T., Weng, F., Han, F.: Solving partial differential equation based on Bernstein neural network and extreme learning machine algorithm. Neural Process. Lett. (2018). https://doi.org/10.1007/s11063-018-9911-8

    Article  Google Scholar 

  27. 27.

    Yang, Y., Hou, M., Sun, H., Zhang, T., Weng, F., Luo, J.: Neural network algorithm based on Legendre improved extreme learning machine for solving elliptic partial differential equations. Soft Comput. (2019). https://doi.org/10.1007/s00500-019-03944-1

    Article  Google Scholar 

  28. 28.

    Zuniga-Aguilar, C.J., Romero-Ugalde, H.M., Gomez-Aguilar, J.F., et al.: Solving fractional differential equations of variable-order involving operators with Mittag–Leffler kernel using artificial neural networks. Chaos Solitons Fractals 103, 382–403 (2017)

    MathSciNet  Article  Google Scholar 

  29. 29.

    Rostami, F., Jafarian, A.: A new artificial neural network structure for solving high-order linear fractional differential equations. Int. J. Comput. Math. 95(3), 528–539 (2018)

    MathSciNet  Article  Google Scholar 

  30. 30.

    Pakdaman, M., Ahmadian, A., Effati, S., et al.: Solving differential equations of fractional order using an optimization technique based on training artificial neural network. Appl. Math. Comput. 293, 81–95 (2017)

    MathSciNet  MATH  Google Scholar 

  31. 31.

    Chaharborj, S.S., Chaharborj, S.S., Mahmoudi, Y.: Study of fractional order integro–differential equations by using Chebyshev neural network. J. Math. Stat. 13(1), 1–13 (2017)

    Article  Google Scholar 

  32. 32.

    Zhou, T., Liu, X., Hou, M., Liu, C.: Numerical solution for ruin probability of continuous time model based on neural network algorithm. Neurocomputing 331, 67–76 (2019)

    Article  Google Scholar 

  33. 33.

    Chakraverty, S., Mall, S.: Regression based weight generation algorithm in neural network for solution of initial and boundary value problems. Neural Comput. Appl. 25, 585–594 (2014)

    Article  Google Scholar 

  34. 34.

    Malek, A., Beidokhti, R.S.: Numerical solution for high order differential equations using a hybrid neural network–optimization method. Appl. Math. Comput. 183, 260–271 (2006)

    MathSciNet  MATH  Google Scholar 

  35. 35.

    Mai-Duy, N.: Solving high order ordinary differential equations with radial basis function networks. Int. J. Numer. Methods Eng. 62(6), 824–852 (2005)

    MathSciNet  Article  Google Scholar 

  36. 36.

    Vapnik, V.N.: The Nature of Statistical Learning Theory, 1st edn. Springer, New York (1995)

    Google Scholar 

  37. 37.

    Suykens, J.A.K., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), 293–300 (1999)

    Article  Google Scholar 

  38. 38.

    Yang, Y., Tan, M., Dai, Y.: An improved CS-LSSVM algorithm-based fault pattern recognition of ship power equipments. PLoS ONE 12, 1–10 (2017)

    Google Scholar 

  39. 39.

    Liu, X., Bo, L., Luo, H.: Bearing faults diagnostics based on hybrid LS-SVM and EMD method. Measurement 59, 145–166 (2015)

    Article  Google Scholar 

  40. 40.

    Yu, L., Chen, H., Wang, S., Lai, K.K.: Evolving least squares support vector machines for stock market trend mining. IEEE Trans. Evol. Comput. 13, 87–102 (2009)

    Article  Google Scholar 

  41. 41.

    Junga, H.C., Kimb, J.S., Heo, H.: Prediction of building energy consumption using an improved real coded genetic algorithm based least squares support vector machine approach. Energy Build. 90, 76–84 (2015)

    Article  Google Scholar 

  42. 42.

    Mehrkanoon, S., Falck, T., Suykens, J.A.K.: Approximate solutions to ordinary differential equations using least squares support vector machines. IEEE Trans. Neural Netw. Learn. Syst. 23(9), 1356–1367 (2012)

    Article  Google Scholar 

  43. 43.

    Mehrkanoon, S., Suykens, J.A.K.: Learning solutions to partial differential equations using LS-SVM. Neurocomputing 159, 105–116 (2015)

    Article  Google Scholar 

  44. 44.

    Mehrkanoon, S., Suykens, J.A.K.: LS-SVM approximate solution to linear time varying descriptor systems. Automatica 48, 2502–2511 (2012)

    MathSciNet  Article  Google Scholar 

  45. 45.

    Zhang, G., Wang, S., Wang, Y.: LS-SVM approximate solution for affine nonlinear systems with partially unknown functions. J. Ind. Manag. Optim. 10, 621–636 (2014)

    MathSciNet  Article  Google Scholar 

  46. 46.

    Wang, Q., Wang, K., Chen, S.: Least squares approximation method for the solution of Volterra–Fredholm integral equations. J. Comput. Appl. Math. 272, 141–147 (2014)

    MathSciNet  Article  Google Scholar 

  47. 47.

    Mehrkanoon, S., Suykens, J.A.K.: Deep hybrid neural–kernel networks using random Fourier features. Neurocomputing 298, 46–54 (2018)

    Article  Google Scholar 

  48. 48.

    Suykens, J.A.K., Gestel, T.V., Brabanter, J.D., Moor, B.D., Vandewalle, J.: Least Squares Support Vector Machines. World Scientific, Singapore (2002)

    Google Scholar 

  49. 49.

    Kincaid, D.R., Cheney, E.W.: Numerical Analysis: Mathematics of Scientific Computing, 3rd edn. Brooks/Cole, Pacific Grove (2002)

    Google Scholar 

  50. 50.

    Arfken, G.B., Weber, H.J.: Mathematical Methods for Physicists, 4nd edn. Academic Press, New York (1995) 973.

    Google Scholar 

  51. 51.

    Gamel, M.E., Behiry, S.H., Hashish, H.: Numerical method for solution of special nonlinear fourth-order boundary value problems. Appl. Math. Comput. 145, 717–734 (2003)

    MathSciNet  MATH  Google Scholar 

  52. 52.

    Nurmuhammada, A., Muhammada, M., Moria, M., Sugiharab, M.: Double exponential transformation in the sinc-collocation method for a boundary value problem with fourth-order ordinary differential equation. J. Comput. Appl. Math. 182, 32–50 (2005)

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

The authors sincerely thank all the reviewers and the editors for their careful reading and valuable comments, which improved the quality of this paper.

Availability of data and materials

Not applicable.

Funding

This study was funded by the National Natural Science Foundation of China under Grants 61375063.

Author information

Affiliations

Authors

Contributions

All authors contributed to the draft of the manuscript and all authors read and approved the final manuscript.

Corresponding author

Correspondence to Muzhou Hou.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lu, Y., Yin, Q., Li, H. et al. The LS-SVM algorithms for boundary value problems of high-order ordinary differential equations. Adv Differ Equ 2019, 195 (2019). https://doi.org/10.1186/s13662-019-2131-3

Download citation

Keywords

  • Numerical solutions
  • High-order ordinary differential equations
  • Two-point boundary value problems
  • Multi-point boundary value problems
  • LS-SVM algorithms