update
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
From the website for the L-BFGS-B code (from at
|
||||
http://www.ece.northwestern.edu/~nocedal/lbfgsb.html):
|
||||
|
||||
"""
|
||||
L-BFGS-B is a limited-memory quasi-Newton code for bound-constrained
|
||||
optimization, i.e. for problems where the only constraints are of the
|
||||
form l<= x <= u.
|
||||
"""
|
||||
|
||||
This is a Python wrapper (using F2PY) written by David M. Cooke
|
||||
<cookedm@physics.mcmaster.ca> and released as version 0.9 on April 9, 2004.
|
||||
The wrapper was slightly modified by Joonas Paalasmaa for the 3.0 version
|
||||
in March 2012.
|
||||
|
||||
License of L-BFGS-B (Fortran code)
|
||||
==================================
|
||||
|
||||
The version included here (in lbfgsb.f) is 3.0 (released April 25, 2011). It was
|
||||
written by Ciyou Zhu, Richard Byrd, and Jorge Nocedal <nocedal@ece.nwu.edu>. It
|
||||
carries the following condition for use:
|
||||
|
||||
"""
|
||||
This software is freely available, but we expect that all publications
|
||||
describing work using this software, or all commercial products using it,
|
||||
quote at least one of the references given below. This software is released
|
||||
under the BSD License.
|
||||
|
||||
References
|
||||
* R. H. Byrd, P. Lu and J. Nocedal. A Limited Memory Algorithm for Bound
|
||||
Constrained Optimization, (1995), SIAM Journal on Scientific and
|
||||
Statistical Computing, 16, 5, pp. 1190-1208.
|
||||
* C. Zhu, R. H. Byrd and J. Nocedal. L-BFGS-B: Algorithm 778: L-BFGS-B,
|
||||
FORTRAN routines for large scale bound constrained optimization (1997),
|
||||
ACM Transactions on Mathematical Software, 23, 4, pp. 550 - 560.
|
||||
* J.L. Morales and J. Nocedal. L-BFGS-B: Remark on Algorithm 778: L-BFGS-B,
|
||||
FORTRAN routines for large scale bound constrained optimization (2011),
|
||||
ACM Transactions on Mathematical Software, 38, 1.
|
||||
"""
|
||||
|
||||
The Python wrapper
|
||||
==================
|
||||
|
||||
This code uses F2PY (http://cens.ioc.ee/projects/f2py2e/) to generate
|
||||
the wrapper around the Fortran code.
|
||||
|
||||
The Python code and wrapper are copyrighted 2004 by David M. Cooke
|
||||
<cookedm@physics.mcmaster.ca>.
|
||||
|
||||
Example usage
|
||||
=============
|
||||
|
||||
An example of the usage is given at the bottom of the lbfgsb.py file.
|
||||
Run it with 'python lbfgsb.py'.
|
||||
|
||||
License for the Python wrapper
|
||||
==============================
|
||||
|
||||
Copyright (c) 2004 David M. Cooke <cookedm@physics.mcmaster.ca>
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
of the Software, and to permit persons to whom the Software is furnished to do
|
||||
so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,452 @@
|
||||
"""
|
||||
=====================================================
|
||||
Optimization and root finding (:mod:`scipy.optimize`)
|
||||
=====================================================
|
||||
|
||||
.. currentmodule:: scipy.optimize
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
optimize.cython_optimize
|
||||
|
||||
SciPy ``optimize`` provides functions for minimizing (or maximizing)
|
||||
objective functions, possibly subject to constraints. It includes
|
||||
solvers for nonlinear problems (with support for both local and global
|
||||
optimization algorithms), linear programming, constrained
|
||||
and nonlinear least-squares, root finding, and curve fitting.
|
||||
|
||||
Common functions and objects, shared across different solvers, are:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
show_options - Show specific options optimization solvers.
|
||||
OptimizeResult - The optimization result returned by some optimizers.
|
||||
OptimizeWarning - The optimization encountered problems.
|
||||
|
||||
|
||||
Optimization
|
||||
============
|
||||
|
||||
Scalar functions optimization
|
||||
-----------------------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
minimize_scalar - Interface for minimizers of univariate functions
|
||||
|
||||
The `minimize_scalar` function supports the following methods:
|
||||
|
||||
.. toctree::
|
||||
|
||||
optimize.minimize_scalar-brent
|
||||
optimize.minimize_scalar-bounded
|
||||
optimize.minimize_scalar-golden
|
||||
|
||||
Local (multivariate) optimization
|
||||
---------------------------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
minimize - Interface for minimizers of multivariate functions.
|
||||
|
||||
The `minimize` function supports the following methods:
|
||||
|
||||
.. toctree::
|
||||
|
||||
optimize.minimize-neldermead
|
||||
optimize.minimize-powell
|
||||
optimize.minimize-cg
|
||||
optimize.minimize-bfgs
|
||||
optimize.minimize-newtoncg
|
||||
optimize.minimize-lbfgsb
|
||||
optimize.minimize-tnc
|
||||
optimize.minimize-cobyla
|
||||
optimize.minimize-cobyqa
|
||||
optimize.minimize-slsqp
|
||||
optimize.minimize-trustconstr
|
||||
optimize.minimize-dogleg
|
||||
optimize.minimize-trustncg
|
||||
optimize.minimize-trustkrylov
|
||||
optimize.minimize-trustexact
|
||||
|
||||
Constraints are passed to `minimize` function as a single object or
|
||||
as a list of objects from the following classes:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
NonlinearConstraint - Class defining general nonlinear constraints.
|
||||
LinearConstraint - Class defining general linear constraints.
|
||||
|
||||
Simple bound constraints are handled separately and there is a special class
|
||||
for them:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
Bounds - Bound constraints.
|
||||
|
||||
Quasi-Newton strategies implementing `HessianUpdateStrategy`
|
||||
interface can be used to approximate the Hessian in `minimize`
|
||||
function (available only for the 'trust-constr' method). Available
|
||||
quasi-Newton methods implementing this interface are:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
BFGS - Broyden-Fletcher-Goldfarb-Shanno (BFGS) Hessian update strategy.
|
||||
SR1 - Symmetric-rank-1 Hessian update strategy.
|
||||
|
||||
.. _global_optimization:
|
||||
|
||||
Global optimization
|
||||
-------------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
basinhopping - Basinhopping stochastic optimizer.
|
||||
brute - Brute force searching optimizer.
|
||||
differential_evolution - Stochastic optimizer using differential evolution.
|
||||
|
||||
shgo - Simplicial homology global optimizer.
|
||||
dual_annealing - Dual annealing stochastic optimizer.
|
||||
direct - DIRECT (Dividing Rectangles) optimizer.
|
||||
|
||||
Least-squares and curve fitting
|
||||
===============================
|
||||
|
||||
Nonlinear least-squares
|
||||
-----------------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
least_squares - Solve a nonlinear least-squares problem with bounds on the variables.
|
||||
|
||||
Linear least-squares
|
||||
--------------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
nnls - Linear least-squares problem with non-negativity constraint.
|
||||
lsq_linear - Linear least-squares problem with bound constraints.
|
||||
isotonic_regression - Least squares problem of isotonic regression via PAVA.
|
||||
|
||||
Curve fitting
|
||||
-------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
curve_fit -- Fit curve to a set of points.
|
||||
|
||||
Root finding
|
||||
============
|
||||
|
||||
Scalar functions
|
||||
----------------
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
root_scalar - Unified interface for nonlinear solvers of scalar functions.
|
||||
brentq - quadratic interpolation Brent method.
|
||||
brenth - Brent method, modified by Harris with hyperbolic extrapolation.
|
||||
ridder - Ridder's method.
|
||||
bisect - Bisection method.
|
||||
newton - Newton's method (also Secant and Halley's methods).
|
||||
toms748 - Alefeld, Potra & Shi Algorithm 748.
|
||||
RootResults - The root finding result returned by some root finders.
|
||||
|
||||
The `root_scalar` function supports the following methods:
|
||||
|
||||
.. toctree::
|
||||
|
||||
optimize.root_scalar-brentq
|
||||
optimize.root_scalar-brenth
|
||||
optimize.root_scalar-bisect
|
||||
optimize.root_scalar-ridder
|
||||
optimize.root_scalar-newton
|
||||
optimize.root_scalar-toms748
|
||||
optimize.root_scalar-secant
|
||||
optimize.root_scalar-halley
|
||||
|
||||
|
||||
|
||||
The table below lists situations and appropriate methods, along with
|
||||
*asymptotic* convergence rates per iteration (and per function evaluation)
|
||||
for successful convergence to a simple root(*).
|
||||
Bisection is the slowest of them all, adding one bit of accuracy for each
|
||||
function evaluation, but is guaranteed to converge.
|
||||
The other bracketing methods all (eventually) increase the number of accurate
|
||||
bits by about 50% for every function evaluation.
|
||||
The derivative-based methods, all built on `newton`, can converge quite quickly
|
||||
if the initial value is close to the root. They can also be applied to
|
||||
functions defined on (a subset of) the complex plane.
|
||||
|
||||
+-------------+----------+----------+-----------+-------------+-------------+----------------+
|
||||
| Domain of f | Bracket? | Derivatives? | Solvers | Convergence |
|
||||
+ + +----------+-----------+ +-------------+----------------+
|
||||
| | | `fprime` | `fprime2` | | Guaranteed? | Rate(s)(*) |
|
||||
+=============+==========+==========+===========+=============+=============+================+
|
||||
| `R` | Yes | N/A | N/A | - bisection | - Yes | - 1 "Linear" |
|
||||
| | | | | - brentq | - Yes | - >=1, <= 1.62 |
|
||||
| | | | | - brenth | - Yes | - >=1, <= 1.62 |
|
||||
| | | | | - ridder | - Yes | - 2.0 (1.41) |
|
||||
| | | | | - toms748 | - Yes | - 2.7 (1.65) |
|
||||
+-------------+----------+----------+-----------+-------------+-------------+----------------+
|
||||
| `R` or `C` | No | No | No | secant | No | 1.62 (1.62) |
|
||||
+-------------+----------+----------+-----------+-------------+-------------+----------------+
|
||||
| `R` or `C` | No | Yes | No | newton | No | 2.00 (1.41) |
|
||||
+-------------+----------+----------+-----------+-------------+-------------+----------------+
|
||||
| `R` or `C` | No | Yes | Yes | halley | No | 3.00 (1.44) |
|
||||
+-------------+----------+----------+-----------+-------------+-------------+----------------+
|
||||
|
||||
.. seealso::
|
||||
|
||||
`scipy.optimize.cython_optimize` -- Typed Cython versions of root finding functions
|
||||
|
||||
Fixed point finding:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
fixed_point - Single-variable fixed-point solver.
|
||||
|
||||
Multidimensional
|
||||
----------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
root - Unified interface for nonlinear solvers of multivariate functions.
|
||||
|
||||
The `root` function supports the following methods:
|
||||
|
||||
.. toctree::
|
||||
|
||||
optimize.root-hybr
|
||||
optimize.root-lm
|
||||
optimize.root-broyden1
|
||||
optimize.root-broyden2
|
||||
optimize.root-anderson
|
||||
optimize.root-linearmixing
|
||||
optimize.root-diagbroyden
|
||||
optimize.root-excitingmixing
|
||||
optimize.root-krylov
|
||||
optimize.root-dfsane
|
||||
|
||||
Linear programming / MILP
|
||||
=========================
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
milp -- Mixed integer linear programming.
|
||||
linprog -- Unified interface for minimizers of linear programming problems.
|
||||
|
||||
The `linprog` function supports the following methods:
|
||||
|
||||
.. toctree::
|
||||
|
||||
optimize.linprog-simplex
|
||||
optimize.linprog-interior-point
|
||||
optimize.linprog-revised_simplex
|
||||
optimize.linprog-highs-ipm
|
||||
optimize.linprog-highs-ds
|
||||
optimize.linprog-highs
|
||||
|
||||
The simplex, interior-point, and revised simplex methods support callback
|
||||
functions, such as:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
linprog_verbose_callback -- Sample callback function for linprog (simplex).
|
||||
|
||||
Assignment problems
|
||||
===================
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
linear_sum_assignment -- Solves the linear-sum assignment problem.
|
||||
quadratic_assignment -- Solves the quadratic assignment problem.
|
||||
|
||||
The `quadratic_assignment` function supports the following methods:
|
||||
|
||||
.. toctree::
|
||||
|
||||
optimize.qap-faq
|
||||
optimize.qap-2opt
|
||||
|
||||
Utilities
|
||||
=========
|
||||
|
||||
Finite-difference approximation
|
||||
-------------------------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
approx_fprime - Approximate the gradient of a scalar function.
|
||||
check_grad - Check the supplied derivative using finite differences.
|
||||
|
||||
|
||||
Line search
|
||||
-----------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
bracket - Bracket a minimum, given two starting points.
|
||||
line_search - Return a step that satisfies the strong Wolfe conditions.
|
||||
|
||||
Hessian approximation
|
||||
---------------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
LbfgsInvHessProduct - Linear operator for L-BFGS approximate inverse Hessian.
|
||||
HessianUpdateStrategy - Interface for implementing Hessian update strategies
|
||||
|
||||
Benchmark problems
|
||||
------------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
rosen - The Rosenbrock function.
|
||||
rosen_der - The derivative of the Rosenbrock function.
|
||||
rosen_hess - The Hessian matrix of the Rosenbrock function.
|
||||
rosen_hess_prod - Product of the Rosenbrock Hessian with a vector.
|
||||
|
||||
Legacy functions
|
||||
================
|
||||
|
||||
The functions below are not recommended for use in new scripts;
|
||||
all of these methods are accessible via a newer, more consistent
|
||||
interfaces, provided by the interfaces above.
|
||||
|
||||
Optimization
|
||||
------------
|
||||
|
||||
General-purpose multivariate methods:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
fmin - Nelder-Mead Simplex algorithm.
|
||||
fmin_powell - Powell's (modified) conjugate direction method.
|
||||
fmin_cg - Non-linear (Polak-Ribiere) conjugate gradient algorithm.
|
||||
fmin_bfgs - Quasi-Newton method (Broydon-Fletcher-Goldfarb-Shanno).
|
||||
fmin_ncg - Line-search Newton Conjugate Gradient.
|
||||
|
||||
Constrained multivariate methods:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
fmin_l_bfgs_b - Zhu, Byrd, and Nocedal's constrained optimizer.
|
||||
fmin_tnc - Truncated Newton code.
|
||||
fmin_cobyla - Constrained optimization by linear approximation.
|
||||
fmin_slsqp - Minimization using sequential least-squares programming.
|
||||
|
||||
Univariate (scalar) minimization methods:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
fminbound - Bounded minimization of a scalar function.
|
||||
brent - 1-D function minimization using Brent method.
|
||||
golden - 1-D function minimization using Golden Section method.
|
||||
|
||||
Least-squares
|
||||
-------------
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
leastsq - Minimize the sum of squares of M equations in N unknowns.
|
||||
|
||||
Root finding
|
||||
------------
|
||||
|
||||
General nonlinear solvers:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
fsolve - Non-linear multivariable equation solver.
|
||||
broyden1 - Broyden's first method.
|
||||
broyden2 - Broyden's second method.
|
||||
NoConvergence - Exception raised when nonlinear solver does not converge.
|
||||
|
||||
Large-scale nonlinear solvers:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
newton_krylov
|
||||
anderson
|
||||
|
||||
BroydenFirst
|
||||
InverseJacobian
|
||||
KrylovJacobian
|
||||
|
||||
Simple iteration solvers:
|
||||
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
excitingmixing
|
||||
linearmixing
|
||||
diagbroyden
|
||||
|
||||
""" # noqa: E501
|
||||
|
||||
from ._optimize import *
|
||||
from ._minimize import *
|
||||
from ._root import *
|
||||
from ._root_scalar import *
|
||||
from ._minpack_py import *
|
||||
from ._zeros_py import *
|
||||
from ._lbfgsb_py import fmin_l_bfgs_b, LbfgsInvHessProduct
|
||||
from ._tnc import fmin_tnc
|
||||
from ._cobyla_py import fmin_cobyla
|
||||
from ._nonlin import *
|
||||
from ._slsqp_py import fmin_slsqp
|
||||
from ._nnls import nnls
|
||||
from ._basinhopping import basinhopping
|
||||
from ._linprog import linprog, linprog_verbose_callback
|
||||
from ._lsap import linear_sum_assignment
|
||||
from ._differentialevolution import differential_evolution
|
||||
from ._lsq import least_squares, lsq_linear
|
||||
from ._isotonic import isotonic_regression
|
||||
from ._constraints import (NonlinearConstraint,
|
||||
LinearConstraint,
|
||||
Bounds)
|
||||
from ._hessian_update_strategy import HessianUpdateStrategy, BFGS, SR1
|
||||
from ._shgo import shgo
|
||||
from ._dual_annealing import dual_annealing
|
||||
from ._qap import quadratic_assignment
|
||||
from ._direct_py import direct
|
||||
from ._milp import milp
|
||||
|
||||
# Deprecated namespaces, to be removed in v2.0.0
|
||||
from . import (
|
||||
cobyla, lbfgsb, linesearch, minpack, minpack2, moduleTNC, nonlin, optimize,
|
||||
slsqp, tnc, zeros
|
||||
)
|
||||
|
||||
__all__ = [s for s in dir() if not s.startswith('_')]
|
||||
|
||||
from scipy._lib._testutils import PytestTester
|
||||
test = PytestTester(__name__)
|
||||
del PytestTester
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
.CondaPkg/env/Lib/site-packages/scipy/optimize/__pycache__/_differentiable_functions.cpython-312.pyc
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,753 @@
|
||||
"""
|
||||
basinhopping: The basinhopping global optimization algorithm
|
||||
"""
|
||||
import numpy as np
|
||||
import math
|
||||
import inspect
|
||||
import scipy.optimize
|
||||
from scipy._lib._util import check_random_state
|
||||
|
||||
__all__ = ['basinhopping']
|
||||
|
||||
|
||||
_params = (inspect.Parameter('res_new', kind=inspect.Parameter.KEYWORD_ONLY),
|
||||
inspect.Parameter('res_old', kind=inspect.Parameter.KEYWORD_ONLY))
|
||||
_new_accept_test_signature = inspect.Signature(parameters=_params)
|
||||
|
||||
|
||||
class Storage:
|
||||
"""
|
||||
Class used to store the lowest energy structure
|
||||
"""
|
||||
def __init__(self, minres):
|
||||
self._add(minres)
|
||||
|
||||
def _add(self, minres):
|
||||
self.minres = minres
|
||||
self.minres.x = np.copy(minres.x)
|
||||
|
||||
def update(self, minres):
|
||||
if minres.success and (minres.fun < self.minres.fun
|
||||
or not self.minres.success):
|
||||
self._add(minres)
|
||||
return True
|
||||
else:
|
||||
return False
|
||||
|
||||
def get_lowest(self):
|
||||
return self.minres
|
||||
|
||||
|
||||
class BasinHoppingRunner:
|
||||
"""This class implements the core of the basinhopping algorithm.
|
||||
|
||||
x0 : ndarray
|
||||
The starting coordinates.
|
||||
minimizer : callable
|
||||
The local minimizer, with signature ``result = minimizer(x)``.
|
||||
The return value is an `optimize.OptimizeResult` object.
|
||||
step_taking : callable
|
||||
This function displaces the coordinates randomly. Signature should
|
||||
be ``x_new = step_taking(x)``. Note that `x` may be modified in-place.
|
||||
accept_tests : list of callables
|
||||
Each test is passed the kwargs `f_new`, `x_new`, `f_old` and
|
||||
`x_old`. These tests will be used to judge whether or not to accept
|
||||
the step. The acceptable return values are True, False, or ``"force
|
||||
accept"``. If any of the tests return False then the step is rejected.
|
||||
If ``"force accept"``, then this will override any other tests in
|
||||
order to accept the step. This can be used, for example, to forcefully
|
||||
escape from a local minimum that ``basinhopping`` is trapped in.
|
||||
disp : bool, optional
|
||||
Display status messages.
|
||||
|
||||
"""
|
||||
def __init__(self, x0, minimizer, step_taking, accept_tests, disp=False):
|
||||
self.x = np.copy(x0)
|
||||
self.minimizer = minimizer
|
||||
self.step_taking = step_taking
|
||||
self.accept_tests = accept_tests
|
||||
self.disp = disp
|
||||
|
||||
self.nstep = 0
|
||||
|
||||
# initialize return object
|
||||
self.res = scipy.optimize.OptimizeResult()
|
||||
self.res.minimization_failures = 0
|
||||
|
||||
# do initial minimization
|
||||
minres = minimizer(self.x)
|
||||
if not minres.success:
|
||||
self.res.minimization_failures += 1
|
||||
if self.disp:
|
||||
print("warning: basinhopping: local minimization failure")
|
||||
self.x = np.copy(minres.x)
|
||||
self.energy = minres.fun
|
||||
self.incumbent_minres = minres # best minimize result found so far
|
||||
if self.disp:
|
||||
print("basinhopping step %d: f %g" % (self.nstep, self.energy))
|
||||
|
||||
# initialize storage class
|
||||
self.storage = Storage(minres)
|
||||
|
||||
if hasattr(minres, "nfev"):
|
||||
self.res.nfev = minres.nfev
|
||||
if hasattr(minres, "njev"):
|
||||
self.res.njev = minres.njev
|
||||
if hasattr(minres, "nhev"):
|
||||
self.res.nhev = minres.nhev
|
||||
|
||||
def _monte_carlo_step(self):
|
||||
"""Do one Monte Carlo iteration
|
||||
|
||||
Randomly displace the coordinates, minimize, and decide whether
|
||||
or not to accept the new coordinates.
|
||||
"""
|
||||
# Take a random step. Make a copy of x because the step_taking
|
||||
# algorithm might change x in place
|
||||
x_after_step = np.copy(self.x)
|
||||
x_after_step = self.step_taking(x_after_step)
|
||||
|
||||
# do a local minimization
|
||||
minres = self.minimizer(x_after_step)
|
||||
x_after_quench = minres.x
|
||||
energy_after_quench = minres.fun
|
||||
if not minres.success:
|
||||
self.res.minimization_failures += 1
|
||||
if self.disp:
|
||||
print("warning: basinhopping: local minimization failure")
|
||||
if hasattr(minres, "nfev"):
|
||||
self.res.nfev += minres.nfev
|
||||
if hasattr(minres, "njev"):
|
||||
self.res.njev += minres.njev
|
||||
if hasattr(minres, "nhev"):
|
||||
self.res.nhev += minres.nhev
|
||||
|
||||
# accept the move based on self.accept_tests. If any test is False,
|
||||
# then reject the step. If any test returns the special string
|
||||
# 'force accept', then accept the step regardless. This can be used
|
||||
# to forcefully escape from a local minimum if normal basin hopping
|
||||
# steps are not sufficient.
|
||||
accept = True
|
||||
for test in self.accept_tests:
|
||||
if inspect.signature(test) == _new_accept_test_signature:
|
||||
testres = test(res_new=minres, res_old=self.incumbent_minres)
|
||||
else:
|
||||
testres = test(f_new=energy_after_quench, x_new=x_after_quench,
|
||||
f_old=self.energy, x_old=self.x)
|
||||
|
||||
if testres == 'force accept':
|
||||
accept = True
|
||||
break
|
||||
elif testres is None:
|
||||
raise ValueError("accept_tests must return True, False, or "
|
||||
"'force accept'")
|
||||
elif not testres:
|
||||
accept = False
|
||||
|
||||
# Report the result of the acceptance test to the take step class.
|
||||
# This is for adaptive step taking
|
||||
if hasattr(self.step_taking, "report"):
|
||||
self.step_taking.report(accept, f_new=energy_after_quench,
|
||||
x_new=x_after_quench, f_old=self.energy,
|
||||
x_old=self.x)
|
||||
|
||||
return accept, minres
|
||||
|
||||
def one_cycle(self):
|
||||
"""Do one cycle of the basinhopping algorithm
|
||||
"""
|
||||
self.nstep += 1
|
||||
new_global_min = False
|
||||
|
||||
accept, minres = self._monte_carlo_step()
|
||||
|
||||
if accept:
|
||||
self.energy = minres.fun
|
||||
self.x = np.copy(minres.x)
|
||||
self.incumbent_minres = minres # best minimize result found so far
|
||||
new_global_min = self.storage.update(minres)
|
||||
|
||||
# print some information
|
||||
if self.disp:
|
||||
self.print_report(minres.fun, accept)
|
||||
if new_global_min:
|
||||
print("found new global minimum on step %d with function"
|
||||
" value %g" % (self.nstep, self.energy))
|
||||
|
||||
# save some variables as BasinHoppingRunner attributes
|
||||
self.xtrial = minres.x
|
||||
self.energy_trial = minres.fun
|
||||
self.accept = accept
|
||||
|
||||
return new_global_min
|
||||
|
||||
def print_report(self, energy_trial, accept):
|
||||
"""print a status update"""
|
||||
minres = self.storage.get_lowest()
|
||||
print("basinhopping step %d: f %g trial_f %g accepted %d "
|
||||
" lowest_f %g" % (self.nstep, self.energy, energy_trial,
|
||||
accept, minres.fun))
|
||||
|
||||
|
||||
class AdaptiveStepsize:
|
||||
"""
|
||||
Class to implement adaptive stepsize.
|
||||
|
||||
This class wraps the step taking class and modifies the stepsize to
|
||||
ensure the true acceptance rate is as close as possible to the target.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
takestep : callable
|
||||
The step taking routine. Must contain modifiable attribute
|
||||
takestep.stepsize
|
||||
accept_rate : float, optional
|
||||
The target step acceptance rate
|
||||
interval : int, optional
|
||||
Interval for how often to update the stepsize
|
||||
factor : float, optional
|
||||
The step size is multiplied or divided by this factor upon each
|
||||
update.
|
||||
verbose : bool, optional
|
||||
Print information about each update
|
||||
|
||||
"""
|
||||
def __init__(self, takestep, accept_rate=0.5, interval=50, factor=0.9,
|
||||
verbose=True):
|
||||
self.takestep = takestep
|
||||
self.target_accept_rate = accept_rate
|
||||
self.interval = interval
|
||||
self.factor = factor
|
||||
self.verbose = verbose
|
||||
|
||||
self.nstep = 0
|
||||
self.nstep_tot = 0
|
||||
self.naccept = 0
|
||||
|
||||
def __call__(self, x):
|
||||
return self.take_step(x)
|
||||
|
||||
def _adjust_step_size(self):
|
||||
old_stepsize = self.takestep.stepsize
|
||||
accept_rate = float(self.naccept) / self.nstep
|
||||
if accept_rate > self.target_accept_rate:
|
||||
# We're accepting too many steps. This generally means we're
|
||||
# trapped in a basin. Take bigger steps.
|
||||
self.takestep.stepsize /= self.factor
|
||||
else:
|
||||
# We're not accepting enough steps. Take smaller steps.
|
||||
self.takestep.stepsize *= self.factor
|
||||
if self.verbose:
|
||||
print(f"adaptive stepsize: acceptance rate {accept_rate:f} target "
|
||||
f"{self.target_accept_rate:f} new stepsize "
|
||||
f"{self.takestep.stepsize:g} old stepsize {old_stepsize:g}")
|
||||
|
||||
def take_step(self, x):
|
||||
self.nstep += 1
|
||||
self.nstep_tot += 1
|
||||
if self.nstep % self.interval == 0:
|
||||
self._adjust_step_size()
|
||||
return self.takestep(x)
|
||||
|
||||
def report(self, accept, **kwargs):
|
||||
"called by basinhopping to report the result of the step"
|
||||
if accept:
|
||||
self.naccept += 1
|
||||
|
||||
|
||||
class RandomDisplacement:
|
||||
"""Add a random displacement of maximum size `stepsize` to each coordinate.
|
||||
|
||||
Calling this updates `x` in-place.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
stepsize : float, optional
|
||||
Maximum stepsize in any dimension
|
||||
random_gen : {None, int, `numpy.random.Generator`,
|
||||
`numpy.random.RandomState`}, optional
|
||||
|
||||
If `seed` is None (or `np.random`), the `numpy.random.RandomState`
|
||||
singleton is used.
|
||||
If `seed` is an int, a new ``RandomState`` instance is used,
|
||||
seeded with `seed`.
|
||||
If `seed` is already a ``Generator`` or ``RandomState`` instance then
|
||||
that instance is used.
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self, stepsize=0.5, random_gen=None):
|
||||
self.stepsize = stepsize
|
||||
self.random_gen = check_random_state(random_gen)
|
||||
|
||||
def __call__(self, x):
|
||||
x += self.random_gen.uniform(-self.stepsize, self.stepsize,
|
||||
np.shape(x))
|
||||
return x
|
||||
|
||||
|
||||
class MinimizerWrapper:
|
||||
"""
|
||||
wrap a minimizer function as a minimizer class
|
||||
"""
|
||||
def __init__(self, minimizer, func=None, **kwargs):
|
||||
self.minimizer = minimizer
|
||||
self.func = func
|
||||
self.kwargs = kwargs
|
||||
|
||||
def __call__(self, x0):
|
||||
if self.func is None:
|
||||
return self.minimizer(x0, **self.kwargs)
|
||||
else:
|
||||
return self.minimizer(self.func, x0, **self.kwargs)
|
||||
|
||||
|
||||
class Metropolis:
|
||||
"""Metropolis acceptance criterion.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
T : float
|
||||
The "temperature" parameter for the accept or reject criterion.
|
||||
random_gen : {None, int, `numpy.random.Generator`,
|
||||
`numpy.random.RandomState`}, optional
|
||||
|
||||
If `seed` is None (or `np.random`), the `numpy.random.RandomState`
|
||||
singleton is used.
|
||||
If `seed` is an int, a new ``RandomState`` instance is used,
|
||||
seeded with `seed`.
|
||||
If `seed` is already a ``Generator`` or ``RandomState`` instance then
|
||||
that instance is used.
|
||||
Random number generator used for acceptance test.
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self, T, random_gen=None):
|
||||
# Avoid ZeroDivisionError since "MBH can be regarded as a special case
|
||||
# of the BH framework with the Metropolis criterion, where temperature
|
||||
# T = 0." (Reject all steps that increase energy.)
|
||||
self.beta = 1.0 / T if T != 0 else float('inf')
|
||||
self.random_gen = check_random_state(random_gen)
|
||||
|
||||
def accept_reject(self, res_new, res_old):
|
||||
"""
|
||||
Assuming the local search underlying res_new was successful:
|
||||
If new energy is lower than old, it will always be accepted.
|
||||
If new is higher than old, there is a chance it will be accepted,
|
||||
less likely for larger differences.
|
||||
"""
|
||||
with np.errstate(invalid='ignore'):
|
||||
# The energy values being fed to Metropolis are 1-length arrays, and if
|
||||
# they are equal, their difference is 0, which gets multiplied by beta,
|
||||
# which is inf, and array([0]) * float('inf') causes
|
||||
#
|
||||
# RuntimeWarning: invalid value encountered in multiply
|
||||
#
|
||||
# Ignore this warning so when the algorithm is on a flat plane, it always
|
||||
# accepts the step, to try to move off the plane.
|
||||
prod = -(res_new.fun - res_old.fun) * self.beta
|
||||
w = math.exp(min(0, prod))
|
||||
|
||||
rand = self.random_gen.uniform()
|
||||
return w >= rand and (res_new.success or not res_old.success)
|
||||
|
||||
def __call__(self, *, res_new, res_old):
|
||||
"""
|
||||
f_new and f_old are mandatory in kwargs
|
||||
"""
|
||||
return bool(self.accept_reject(res_new, res_old))
|
||||
|
||||
|
||||
def basinhopping(func, x0, niter=100, T=1.0, stepsize=0.5,
|
||||
minimizer_kwargs=None, take_step=None, accept_test=None,
|
||||
callback=None, interval=50, disp=False, niter_success=None,
|
||||
seed=None, *, target_accept_rate=0.5, stepwise_factor=0.9):
|
||||
"""Find the global minimum of a function using the basin-hopping algorithm.
|
||||
|
||||
Basin-hopping is a two-phase method that combines a global stepping
|
||||
algorithm with local minimization at each step. Designed to mimic
|
||||
the natural process of energy minimization of clusters of atoms, it works
|
||||
well for similar problems with "funnel-like, but rugged" energy landscapes
|
||||
[5]_.
|
||||
|
||||
As the step-taking, step acceptance, and minimization methods are all
|
||||
customizable, this function can also be used to implement other two-phase
|
||||
methods.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable ``f(x, *args)``
|
||||
Function to be optimized. ``args`` can be passed as an optional item
|
||||
in the dict `minimizer_kwargs`
|
||||
x0 : array_like
|
||||
Initial guess.
|
||||
niter : integer, optional
|
||||
The number of basin-hopping iterations. There will be a total of
|
||||
``niter + 1`` runs of the local minimizer.
|
||||
T : float, optional
|
||||
The "temperature" parameter for the acceptance or rejection criterion.
|
||||
Higher "temperatures" mean that larger jumps in function value will be
|
||||
accepted. For best results `T` should be comparable to the
|
||||
separation (in function value) between local minima.
|
||||
stepsize : float, optional
|
||||
Maximum step size for use in the random displacement.
|
||||
minimizer_kwargs : dict, optional
|
||||
Extra keyword arguments to be passed to the local minimizer
|
||||
`scipy.optimize.minimize` Some important options could be:
|
||||
|
||||
method : str
|
||||
The minimization method (e.g. ``"L-BFGS-B"``)
|
||||
args : tuple
|
||||
Extra arguments passed to the objective function (`func`) and
|
||||
its derivatives (Jacobian, Hessian).
|
||||
|
||||
take_step : callable ``take_step(x)``, optional
|
||||
Replace the default step-taking routine with this routine. The default
|
||||
step-taking routine is a random displacement of the coordinates, but
|
||||
other step-taking algorithms may be better for some systems.
|
||||
`take_step` can optionally have the attribute ``take_step.stepsize``.
|
||||
If this attribute exists, then `basinhopping` will adjust
|
||||
``take_step.stepsize`` in order to try to optimize the global minimum
|
||||
search.
|
||||
accept_test : callable, ``accept_test(f_new=f_new, x_new=x_new, f_old=fold, x_old=x_old)``, optional
|
||||
Define a test which will be used to judge whether to accept the
|
||||
step. This will be used in addition to the Metropolis test based on
|
||||
"temperature" `T`. The acceptable return values are True,
|
||||
False, or ``"force accept"``. If any of the tests return False
|
||||
then the step is rejected. If the latter, then this will override any
|
||||
other tests in order to accept the step. This can be used, for example,
|
||||
to forcefully escape from a local minimum that `basinhopping` is
|
||||
trapped in.
|
||||
callback : callable, ``callback(x, f, accept)``, optional
|
||||
A callback function which will be called for all minima found. ``x``
|
||||
and ``f`` are the coordinates and function value of the trial minimum,
|
||||
and ``accept`` is whether that minimum was accepted. This can
|
||||
be used, for example, to save the lowest N minima found. Also,
|
||||
`callback` can be used to specify a user defined stop criterion by
|
||||
optionally returning True to stop the `basinhopping` routine.
|
||||
interval : integer, optional
|
||||
interval for how often to update the `stepsize`
|
||||
disp : bool, optional
|
||||
Set to True to print status messages
|
||||
niter_success : integer, optional
|
||||
Stop the run if the global minimum candidate remains the same for this
|
||||
number of iterations.
|
||||
seed : {None, int, `numpy.random.Generator`, `numpy.random.RandomState`}, optional
|
||||
|
||||
If `seed` is None (or `np.random`), the `numpy.random.RandomState`
|
||||
singleton is used.
|
||||
If `seed` is an int, a new ``RandomState`` instance is used,
|
||||
seeded with `seed`.
|
||||
If `seed` is already a ``Generator`` or ``RandomState`` instance then
|
||||
that instance is used.
|
||||
Specify `seed` for repeatable minimizations. The random numbers
|
||||
generated with this seed only affect the default Metropolis
|
||||
`accept_test` and the default `take_step`. If you supply your own
|
||||
`take_step` and `accept_test`, and these functions use random
|
||||
number generation, then those functions are responsible for the state
|
||||
of their random number generator.
|
||||
target_accept_rate : float, optional
|
||||
The target acceptance rate that is used to adjust the `stepsize`.
|
||||
If the current acceptance rate is greater than the target,
|
||||
then the `stepsize` is increased. Otherwise, it is decreased.
|
||||
Range is (0, 1). Default is 0.5.
|
||||
|
||||
.. versionadded:: 1.8.0
|
||||
|
||||
stepwise_factor : float, optional
|
||||
The `stepsize` is multiplied or divided by this stepwise factor upon
|
||||
each update. Range is (0, 1). Default is 0.9.
|
||||
|
||||
.. versionadded:: 1.8.0
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : OptimizeResult
|
||||
The optimization result represented as a `OptimizeResult` object.
|
||||
Important attributes are: ``x`` the solution array, ``fun`` the value
|
||||
of the function at the solution, and ``message`` which describes the
|
||||
cause of the termination. The ``OptimizeResult`` object returned by the
|
||||
selected minimizer at the lowest minimum is also contained within this
|
||||
object and can be accessed through the ``lowest_optimization_result``
|
||||
attribute. See `OptimizeResult` for a description of other attributes.
|
||||
|
||||
See Also
|
||||
--------
|
||||
minimize :
|
||||
The local minimization function called once for each basinhopping step.
|
||||
`minimizer_kwargs` is passed to this routine.
|
||||
|
||||
Notes
|
||||
-----
|
||||
Basin-hopping is a stochastic algorithm which attempts to find the global
|
||||
minimum of a smooth scalar function of one or more variables [1]_ [2]_ [3]_
|
||||
[4]_. The algorithm in its current form was described by David Wales and
|
||||
Jonathan Doye [2]_ http://www-wales.ch.cam.ac.uk/.
|
||||
|
||||
The algorithm is iterative with each cycle composed of the following
|
||||
features
|
||||
|
||||
1) random perturbation of the coordinates
|
||||
|
||||
2) local minimization
|
||||
|
||||
3) accept or reject the new coordinates based on the minimized function
|
||||
value
|
||||
|
||||
The acceptance test used here is the Metropolis criterion of standard Monte
|
||||
Carlo algorithms, although there are many other possibilities [3]_.
|
||||
|
||||
This global minimization method has been shown to be extremely efficient
|
||||
for a wide variety of problems in physics and chemistry. It is
|
||||
particularly useful when the function has many minima separated by large
|
||||
barriers. See the `Cambridge Cluster Database
|
||||
<https://www-wales.ch.cam.ac.uk/CCD.html>`_ for databases of molecular
|
||||
systems that have been optimized primarily using basin-hopping. This
|
||||
database includes minimization problems exceeding 300 degrees of freedom.
|
||||
|
||||
See the free software program `GMIN <https://www-wales.ch.cam.ac.uk/GMIN>`_
|
||||
for a Fortran implementation of basin-hopping. This implementation has many
|
||||
variations of the procedure described above, including more
|
||||
advanced step taking algorithms and alternate acceptance criterion.
|
||||
|
||||
For stochastic global optimization there is no way to determine if the true
|
||||
global minimum has actually been found. Instead, as a consistency check,
|
||||
the algorithm can be run from a number of different random starting points
|
||||
to ensure the lowest minimum found in each example has converged to the
|
||||
global minimum. For this reason, `basinhopping` will by default simply
|
||||
run for the number of iterations `niter` and return the lowest minimum
|
||||
found. It is left to the user to ensure that this is in fact the global
|
||||
minimum.
|
||||
|
||||
Choosing `stepsize`: This is a crucial parameter in `basinhopping` and
|
||||
depends on the problem being solved. The step is chosen uniformly in the
|
||||
region from x0-stepsize to x0+stepsize, in each dimension. Ideally, it
|
||||
should be comparable to the typical separation (in argument values) between
|
||||
local minima of the function being optimized. `basinhopping` will, by
|
||||
default, adjust `stepsize` to find an optimal value, but this may take
|
||||
many iterations. You will get quicker results if you set a sensible
|
||||
initial value for ``stepsize``.
|
||||
|
||||
Choosing `T`: The parameter `T` is the "temperature" used in the
|
||||
Metropolis criterion. Basinhopping steps are always accepted if
|
||||
``func(xnew) < func(xold)``. Otherwise, they are accepted with
|
||||
probability::
|
||||
|
||||
exp( -(func(xnew) - func(xold)) / T )
|
||||
|
||||
So, for best results, `T` should to be comparable to the typical
|
||||
difference (in function values) between local minima. (The height of
|
||||
"walls" between local minima is irrelevant.)
|
||||
|
||||
If `T` is 0, the algorithm becomes Monotonic Basin-Hopping, in which all
|
||||
steps that increase energy are rejected.
|
||||
|
||||
.. versionadded:: 0.12.0
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Wales, David J. 2003, Energy Landscapes, Cambridge University Press,
|
||||
Cambridge, UK.
|
||||
.. [2] Wales, D J, and Doye J P K, Global Optimization by Basin-Hopping and
|
||||
the Lowest Energy Structures of Lennard-Jones Clusters Containing up to
|
||||
110 Atoms. Journal of Physical Chemistry A, 1997, 101, 5111.
|
||||
.. [3] Li, Z. and Scheraga, H. A., Monte Carlo-minimization approach to the
|
||||
multiple-minima problem in protein folding, Proc. Natl. Acad. Sci. USA,
|
||||
1987, 84, 6611.
|
||||
.. [4] Wales, D. J. and Scheraga, H. A., Global optimization of clusters,
|
||||
crystals, and biomolecules, Science, 1999, 285, 1368.
|
||||
.. [5] Olson, B., Hashmi, I., Molloy, K., and Shehu1, A., Basin Hopping as
|
||||
a General and Versatile Optimization Framework for the Characterization
|
||||
of Biological Macromolecules, Advances in Artificial Intelligence,
|
||||
Volume 2012 (2012), Article ID 674832, :doi:`10.1155/2012/674832`
|
||||
|
||||
Examples
|
||||
--------
|
||||
The following example is a 1-D minimization problem, with many
|
||||
local minima superimposed on a parabola.
|
||||
|
||||
>>> import numpy as np
|
||||
>>> from scipy.optimize import basinhopping
|
||||
>>> func = lambda x: np.cos(14.5 * x - 0.3) + (x + 0.2) * x
|
||||
>>> x0 = [1.]
|
||||
|
||||
Basinhopping, internally, uses a local minimization algorithm. We will use
|
||||
the parameter `minimizer_kwargs` to tell basinhopping which algorithm to
|
||||
use and how to set up that minimizer. This parameter will be passed to
|
||||
`scipy.optimize.minimize`.
|
||||
|
||||
>>> minimizer_kwargs = {"method": "BFGS"}
|
||||
>>> ret = basinhopping(func, x0, minimizer_kwargs=minimizer_kwargs,
|
||||
... niter=200)
|
||||
>>> # the global minimum is:
|
||||
>>> ret.x, ret.fun
|
||||
-0.1951, -1.0009
|
||||
|
||||
Next consider a 2-D minimization problem. Also, this time, we
|
||||
will use gradient information to significantly speed up the search.
|
||||
|
||||
>>> def func2d(x):
|
||||
... f = np.cos(14.5 * x[0] - 0.3) + (x[1] + 0.2) * x[1] + (x[0] +
|
||||
... 0.2) * x[0]
|
||||
... df = np.zeros(2)
|
||||
... df[0] = -14.5 * np.sin(14.5 * x[0] - 0.3) + 2. * x[0] + 0.2
|
||||
... df[1] = 2. * x[1] + 0.2
|
||||
... return f, df
|
||||
|
||||
We'll also use a different local minimization algorithm. Also, we must tell
|
||||
the minimizer that our function returns both energy and gradient (Jacobian).
|
||||
|
||||
>>> minimizer_kwargs = {"method":"L-BFGS-B", "jac":True}
|
||||
>>> x0 = [1.0, 1.0]
|
||||
>>> ret = basinhopping(func2d, x0, minimizer_kwargs=minimizer_kwargs,
|
||||
... niter=200)
|
||||
>>> print("global minimum: x = [%.4f, %.4f], f(x) = %.4f" % (ret.x[0],
|
||||
... ret.x[1],
|
||||
... ret.fun))
|
||||
global minimum: x = [-0.1951, -0.1000], f(x) = -1.0109
|
||||
|
||||
Here is an example using a custom step-taking routine. Imagine you want
|
||||
the first coordinate to take larger steps than the rest of the coordinates.
|
||||
This can be implemented like so:
|
||||
|
||||
>>> class MyTakeStep:
|
||||
... def __init__(self, stepsize=0.5):
|
||||
... self.stepsize = stepsize
|
||||
... self.rng = np.random.default_rng()
|
||||
... def __call__(self, x):
|
||||
... s = self.stepsize
|
||||
... x[0] += self.rng.uniform(-2.*s, 2.*s)
|
||||
... x[1:] += self.rng.uniform(-s, s, x[1:].shape)
|
||||
... return x
|
||||
|
||||
Since ``MyTakeStep.stepsize`` exists basinhopping will adjust the magnitude
|
||||
of `stepsize` to optimize the search. We'll use the same 2-D function as
|
||||
before
|
||||
|
||||
>>> mytakestep = MyTakeStep()
|
||||
>>> ret = basinhopping(func2d, x0, minimizer_kwargs=minimizer_kwargs,
|
||||
... niter=200, take_step=mytakestep)
|
||||
>>> print("global minimum: x = [%.4f, %.4f], f(x) = %.4f" % (ret.x[0],
|
||||
... ret.x[1],
|
||||
... ret.fun))
|
||||
global minimum: x = [-0.1951, -0.1000], f(x) = -1.0109
|
||||
|
||||
Now, let's do an example using a custom callback function which prints the
|
||||
value of every minimum found
|
||||
|
||||
>>> def print_fun(x, f, accepted):
|
||||
... print("at minimum %.4f accepted %d" % (f, int(accepted)))
|
||||
|
||||
We'll run it for only 10 basinhopping steps this time.
|
||||
|
||||
>>> rng = np.random.default_rng()
|
||||
>>> ret = basinhopping(func2d, x0, minimizer_kwargs=minimizer_kwargs,
|
||||
... niter=10, callback=print_fun, seed=rng)
|
||||
at minimum 0.4159 accepted 1
|
||||
at minimum -0.4317 accepted 1
|
||||
at minimum -1.0109 accepted 1
|
||||
at minimum -0.9073 accepted 1
|
||||
at minimum -0.4317 accepted 0
|
||||
at minimum -0.1021 accepted 1
|
||||
at minimum -0.7425 accepted 1
|
||||
at minimum -0.9073 accepted 1
|
||||
at minimum -0.4317 accepted 0
|
||||
at minimum -0.7425 accepted 1
|
||||
at minimum -0.9073 accepted 1
|
||||
|
||||
The minimum at -1.0109 is actually the global minimum, found already on the
|
||||
8th iteration.
|
||||
|
||||
""" # numpy/numpydoc#87 # noqa: E501
|
||||
if target_accept_rate <= 0. or target_accept_rate >= 1.:
|
||||
raise ValueError('target_accept_rate has to be in range (0, 1)')
|
||||
if stepwise_factor <= 0. or stepwise_factor >= 1.:
|
||||
raise ValueError('stepwise_factor has to be in range (0, 1)')
|
||||
|
||||
x0 = np.array(x0)
|
||||
|
||||
# set up the np.random generator
|
||||
rng = check_random_state(seed)
|
||||
|
||||
# set up minimizer
|
||||
if minimizer_kwargs is None:
|
||||
minimizer_kwargs = dict()
|
||||
wrapped_minimizer = MinimizerWrapper(scipy.optimize.minimize, func,
|
||||
**minimizer_kwargs)
|
||||
|
||||
# set up step-taking algorithm
|
||||
if take_step is not None:
|
||||
if not callable(take_step):
|
||||
raise TypeError("take_step must be callable")
|
||||
# if take_step.stepsize exists then use AdaptiveStepsize to control
|
||||
# take_step.stepsize
|
||||
if hasattr(take_step, "stepsize"):
|
||||
take_step_wrapped = AdaptiveStepsize(
|
||||
take_step, interval=interval,
|
||||
accept_rate=target_accept_rate,
|
||||
factor=stepwise_factor,
|
||||
verbose=disp)
|
||||
else:
|
||||
take_step_wrapped = take_step
|
||||
else:
|
||||
# use default
|
||||
displace = RandomDisplacement(stepsize=stepsize, random_gen=rng)
|
||||
take_step_wrapped = AdaptiveStepsize(displace, interval=interval,
|
||||
accept_rate=target_accept_rate,
|
||||
factor=stepwise_factor,
|
||||
verbose=disp)
|
||||
|
||||
# set up accept tests
|
||||
accept_tests = []
|
||||
if accept_test is not None:
|
||||
if not callable(accept_test):
|
||||
raise TypeError("accept_test must be callable")
|
||||
accept_tests = [accept_test]
|
||||
|
||||
# use default
|
||||
metropolis = Metropolis(T, random_gen=rng)
|
||||
accept_tests.append(metropolis)
|
||||
|
||||
if niter_success is None:
|
||||
niter_success = niter + 2
|
||||
|
||||
bh = BasinHoppingRunner(x0, wrapped_minimizer, take_step_wrapped,
|
||||
accept_tests, disp=disp)
|
||||
|
||||
# The wrapped minimizer is called once during construction of
|
||||
# BasinHoppingRunner, so run the callback
|
||||
if callable(callback):
|
||||
callback(bh.storage.minres.x, bh.storage.minres.fun, True)
|
||||
|
||||
# start main iteration loop
|
||||
count, i = 0, 0
|
||||
message = ["requested number of basinhopping iterations completed"
|
||||
" successfully"]
|
||||
for i in range(niter):
|
||||
new_global_min = bh.one_cycle()
|
||||
|
||||
if callable(callback):
|
||||
# should we pass a copy of x?
|
||||
val = callback(bh.xtrial, bh.energy_trial, bh.accept)
|
||||
if val is not None:
|
||||
if val:
|
||||
message = ["callback function requested stop early by"
|
||||
"returning True"]
|
||||
break
|
||||
|
||||
count += 1
|
||||
if new_global_min:
|
||||
count = 0
|
||||
elif count > niter_success:
|
||||
message = ["success condition satisfied"]
|
||||
break
|
||||
|
||||
# prepare return object
|
||||
res = bh.res
|
||||
res.lowest_optimization_result = bh.storage.get_lowest()
|
||||
res.x = np.copy(res.lowest_optimization_result.x)
|
||||
res.fun = res.lowest_optimization_result.fun
|
||||
res.message = message
|
||||
res.nit = i + 1
|
||||
res.success = res.lowest_optimization_result.success
|
||||
return res
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,666 @@
|
||||
import numpy as np
|
||||
import scipy._lib._elementwise_iterative_method as eim
|
||||
from scipy._lib._util import _RichResult
|
||||
|
||||
_ELIMITS = -1 # used in _bracket_root
|
||||
_ESTOPONESIDE = 2 # used in _bracket_root
|
||||
|
||||
def _bracket_root_iv(func, xl0, xr0, xmin, xmax, factor, args, maxiter):
|
||||
|
||||
if not callable(func):
|
||||
raise ValueError('`func` must be callable.')
|
||||
|
||||
if not np.iterable(args):
|
||||
args = (args,)
|
||||
|
||||
xl0 = np.asarray(xl0)[()]
|
||||
if not np.issubdtype(xl0.dtype, np.number) or np.iscomplex(xl0).any():
|
||||
raise ValueError('`xl0` must be numeric and real.')
|
||||
|
||||
xr0 = xl0 + 1 if xr0 is None else xr0
|
||||
xmin = -np.inf if xmin is None else xmin
|
||||
xmax = np.inf if xmax is None else xmax
|
||||
factor = 2. if factor is None else factor
|
||||
xl0, xr0, xmin, xmax, factor = np.broadcast_arrays(xl0, xr0, xmin, xmax, factor)
|
||||
|
||||
if not np.issubdtype(xr0.dtype, np.number) or np.iscomplex(xr0).any():
|
||||
raise ValueError('`xr0` must be numeric and real.')
|
||||
|
||||
if not np.issubdtype(xmin.dtype, np.number) or np.iscomplex(xmin).any():
|
||||
raise ValueError('`xmin` must be numeric and real.')
|
||||
|
||||
if not np.issubdtype(xmax.dtype, np.number) or np.iscomplex(xmax).any():
|
||||
raise ValueError('`xmax` must be numeric and real.')
|
||||
|
||||
if not np.issubdtype(factor.dtype, np.number) or np.iscomplex(factor).any():
|
||||
raise ValueError('`factor` must be numeric and real.')
|
||||
if not np.all(factor > 1):
|
||||
raise ValueError('All elements of `factor` must be greater than 1.')
|
||||
|
||||
maxiter = np.asarray(maxiter)
|
||||
message = '`maxiter` must be a non-negative integer.'
|
||||
if (not np.issubdtype(maxiter.dtype, np.number) or maxiter.shape != tuple()
|
||||
or np.iscomplex(maxiter)):
|
||||
raise ValueError(message)
|
||||
maxiter_int = int(maxiter[()])
|
||||
if not maxiter == maxiter_int or maxiter < 0:
|
||||
raise ValueError(message)
|
||||
|
||||
return func, xl0, xr0, xmin, xmax, factor, args, maxiter
|
||||
|
||||
|
||||
def _bracket_root(func, xl0, xr0=None, *, xmin=None, xmax=None, factor=None,
|
||||
args=(), maxiter=1000):
|
||||
"""Bracket the root of a monotonic scalar function of one variable
|
||||
|
||||
This function works elementwise when `xl0`, `xr0`, `xmin`, `xmax`, `factor`, and
|
||||
the elements of `args` are broadcastable arrays.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
The function for which the root is to be bracketed.
|
||||
The signature must be::
|
||||
|
||||
func(x: ndarray, *args) -> ndarray
|
||||
|
||||
where each element of ``x`` is a finite real and ``args`` is a tuple,
|
||||
which may contain an arbitrary number of arrays that are broadcastable
|
||||
with `x`. ``func`` must be an elementwise function: each element
|
||||
``func(x)[i]`` must equal ``func(x[i])`` for all indices ``i``.
|
||||
xl0, xr0: float array_like
|
||||
Starting guess of bracket, which need not contain a root. If `xr0` is
|
||||
not provided, ``xr0 = xl0 + 1``. Must be broadcastable with one another.
|
||||
xmin, xmax : float array_like, optional
|
||||
Minimum and maximum allowable endpoints of the bracket, inclusive. Must
|
||||
be broadcastable with `xl0` and `xr0`.
|
||||
factor : float array_like, default: 2
|
||||
The factor used to grow the bracket. See notes for details.
|
||||
args : tuple, optional
|
||||
Additional positional arguments to be passed to `func`. Must be arrays
|
||||
broadcastable with `xl0`, `xr0`, `xmin`, and `xmax`. If the callable to be
|
||||
bracketed requires arguments that are not broadcastable with these
|
||||
arrays, wrap that callable with `func` such that `func` accepts
|
||||
only `x` and broadcastable arrays.
|
||||
maxiter : int, optional
|
||||
The maximum number of iterations of the algorithm to perform.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : _RichResult
|
||||
An instance of `scipy._lib._util._RichResult` with the following
|
||||
attributes. The descriptions are written as though the values will be
|
||||
scalars; however, if `func` returns an array, the outputs will be
|
||||
arrays of the same shape.
|
||||
|
||||
xl, xr : float
|
||||
The lower and upper ends of the bracket, if the algorithm
|
||||
terminated successfully.
|
||||
fl, fr : float
|
||||
The function value at the lower and upper ends of the bracket.
|
||||
nfev : int
|
||||
The number of function evaluations required to find the bracket.
|
||||
This is distinct from the number of times `func` is *called*
|
||||
because the function may evaluated at multiple points in a single
|
||||
call.
|
||||
nit : int
|
||||
The number of iterations of the algorithm that were performed.
|
||||
status : int
|
||||
An integer representing the exit status of the algorithm.
|
||||
|
||||
- ``0`` : The algorithm produced a valid bracket.
|
||||
- ``-1`` : The bracket expanded to the allowable limits without finding a bracket.
|
||||
- ``-2`` : The maximum number of iterations was reached.
|
||||
- ``-3`` : A non-finite value was encountered.
|
||||
- ``-4`` : Iteration was terminated by `callback`.
|
||||
- ``-5``: The initial bracket does not satisfy `xmin <= xl0 < xr0 < xmax`.
|
||||
- ``1`` : The algorithm is proceeding normally (in `callback` only).
|
||||
- ``2`` : A bracket was found in the opposite search direction (in `callback` only).
|
||||
|
||||
success : bool
|
||||
``True`` when the algorithm terminated successfully (status ``0``).
|
||||
|
||||
Notes
|
||||
-----
|
||||
This function generalizes an algorithm found in pieces throughout
|
||||
`scipy.stats`. The strategy is to iteratively grow the bracket `(l, r)`
|
||||
until ``func(l) < 0 < func(r)``. The bracket grows to the left as follows.
|
||||
|
||||
- If `xmin` is not provided, the distance between `xl0` and `l` is iteratively
|
||||
increased by `factor`.
|
||||
- If `xmin` is provided, the distance between `xmin` and `l` is iteratively
|
||||
decreased by `factor`. Note that this also *increases* the bracket size.
|
||||
|
||||
Growth of the bracket to the right is analogous.
|
||||
|
||||
Growth of the bracket in one direction stops when the endpoint is no longer
|
||||
finite, the function value at the endpoint is no longer finite, or the
|
||||
endpoint reaches its limiting value (`xmin` or `xmax`). Iteration terminates
|
||||
when the bracket stops growing in both directions, the bracket surrounds
|
||||
the root, or a root is found (accidentally).
|
||||
|
||||
If two brackets are found - that is, a bracket is found on both sides in
|
||||
the same iteration, the smaller of the two is returned.
|
||||
If roots of the function are found, both `l` and `r` are set to the
|
||||
leftmost root.
|
||||
|
||||
""" # noqa: E501
|
||||
# Todo:
|
||||
# - find bracket with sign change in specified direction
|
||||
# - Add tolerance
|
||||
# - allow factor < 1?
|
||||
|
||||
callback = None # works; I just don't want to test it
|
||||
temp = _bracket_root_iv(func, xl0, xr0, xmin, xmax, factor, args, maxiter)
|
||||
func, xl0, xr0, xmin, xmax, factor, args, maxiter = temp
|
||||
|
||||
xs = (xl0, xr0)
|
||||
temp = eim._initialize(func, xs, args)
|
||||
func, xs, fs, args, shape, dtype, xp = temp # line split for PEP8
|
||||
xl0, xr0 = xs
|
||||
xmin = np.broadcast_to(xmin, shape).astype(dtype, copy=False).ravel()
|
||||
xmax = np.broadcast_to(xmax, shape).astype(dtype, copy=False).ravel()
|
||||
invalid_bracket = ~((xmin <= xl0) & (xl0 < xr0) & (xr0 <= xmax))
|
||||
|
||||
# The approach is to treat the left and right searches as though they were
|
||||
# (almost) totally independent one-sided bracket searches. (The interaction
|
||||
# is considered when checking for termination and preparing the result
|
||||
# object.)
|
||||
# `x` is the "moving" end of the bracket
|
||||
x = np.concatenate(xs)
|
||||
f = np.concatenate(fs)
|
||||
invalid_bracket = np.concatenate((invalid_bracket, invalid_bracket))
|
||||
n = len(x) // 2
|
||||
|
||||
# `x_last` is the previous location of the moving end of the bracket. If
|
||||
# the signs of `f` and `f_last` are different, `x` and `x_last` form a
|
||||
# bracket.
|
||||
x_last = np.concatenate((x[n:], x[:n]))
|
||||
f_last = np.concatenate((f[n:], f[:n]))
|
||||
# `x0` is the "fixed" end of the bracket.
|
||||
x0 = x_last
|
||||
# We don't need to retain the corresponding function value, since the
|
||||
# fixed end of the bracket is only needed to compute the new value of the
|
||||
# moving end; it is never returned.
|
||||
limit = np.concatenate((xmin, xmax))
|
||||
|
||||
factor = np.broadcast_to(factor, shape).astype(dtype, copy=False).ravel()
|
||||
factor = np.concatenate((factor, factor))
|
||||
|
||||
active = np.arange(2*n)
|
||||
args = [np.concatenate((arg, arg)) for arg in args]
|
||||
|
||||
# This is needed due to inner workings of `eim._loop`.
|
||||
# We're abusing it a tiny bit.
|
||||
shape = shape + (2,)
|
||||
|
||||
# `d` is for "distance".
|
||||
# For searches without a limit, the distance between the fixed end of the
|
||||
# bracket `x0` and the moving end `x` will grow by `factor` each iteration.
|
||||
# For searches with a limit, the distance between the `limit` and moving
|
||||
# end of the bracket `x` will shrink by `factor` each iteration.
|
||||
i = np.isinf(limit)
|
||||
ni = ~i
|
||||
d = np.zeros_like(x)
|
||||
d[i] = x[i] - x0[i]
|
||||
d[ni] = limit[ni] - x[ni]
|
||||
|
||||
status = np.full_like(x, eim._EINPROGRESS, dtype=int) # in progress
|
||||
status[invalid_bracket] = eim._EINPUTERR
|
||||
nit, nfev = 0, 1 # one function evaluation per side performed above
|
||||
|
||||
work = _RichResult(x=x, x0=x0, f=f, limit=limit, factor=factor,
|
||||
active=active, d=d, x_last=x_last, f_last=f_last,
|
||||
nit=nit, nfev=nfev, status=status, args=args,
|
||||
xl=None, xr=None, fl=None, fr=None, n=n)
|
||||
res_work_pairs = [('status', 'status'), ('xl', 'xl'), ('xr', 'xr'),
|
||||
('nit', 'nit'), ('nfev', 'nfev'), ('fl', 'fl'),
|
||||
('fr', 'fr'), ('x', 'x'), ('f', 'f'),
|
||||
('x_last', 'x_last'), ('f_last', 'f_last')]
|
||||
|
||||
def pre_func_eval(work):
|
||||
# Initialize moving end of bracket
|
||||
x = np.zeros_like(work.x)
|
||||
|
||||
# Unlimited brackets grow by `factor` by increasing distance from fixed
|
||||
# end to moving end.
|
||||
i = np.isinf(work.limit) # indices of unlimited brackets
|
||||
work.d[i] *= work.factor[i]
|
||||
x[i] = work.x0[i] + work.d[i]
|
||||
|
||||
# Limited brackets grow by decreasing the distance from the limit to
|
||||
# the moving end.
|
||||
ni = ~i # indices of limited brackets
|
||||
work.d[ni] /= work.factor[ni]
|
||||
x[ni] = work.limit[ni] - work.d[ni]
|
||||
|
||||
return x
|
||||
|
||||
def post_func_eval(x, f, work):
|
||||
# Keep track of the previous location of the moving end so that we can
|
||||
# return a narrower bracket. (The alternative is to remember the
|
||||
# original fixed end, but then the bracket would be wider than needed.)
|
||||
work.x_last = work.x
|
||||
work.f_last = work.f
|
||||
work.x = x
|
||||
work.f = f
|
||||
|
||||
def check_termination(work):
|
||||
# Condition 0: initial bracket is invalid
|
||||
stop = (work.status == eim._EINPUTERR)
|
||||
|
||||
# Condition 1: a valid bracket (or the root itself) has been found
|
||||
sf = np.sign(work.f)
|
||||
sf_last = np.sign(work.f_last)
|
||||
i = ((sf_last == -sf) | (sf_last == 0) | (sf == 0)) & ~stop
|
||||
work.status[i] = eim._ECONVERGED
|
||||
stop[i] = True
|
||||
|
||||
# Condition 2: the other side's search found a valid bracket.
|
||||
# (If we just found a bracket with the rightward search, we can stop
|
||||
# the leftward search, and vice-versa.)
|
||||
# To do this, we need to set the status of the other side's search;
|
||||
# this is tricky because `work.status` contains only the *active*
|
||||
# elements, so we don't immediately know the index of the element we
|
||||
# need to set - or even if it's still there. (That search may have
|
||||
# terminated already, e.g. by reaching its `limit`.)
|
||||
# To facilitate this, `work.active` contains a unit integer index of
|
||||
# each search. Index `k` (`k < n)` and `k + n` correspond with a
|
||||
# leftward and rightward search, respectively. Elements are removed
|
||||
# from `work.active` just as they are removed from `work.status`, so
|
||||
# we use `work.active` to help find the right location in
|
||||
# `work.status`.
|
||||
# Get the integer indices of the elements that can also stop
|
||||
also_stop = (work.active[i] + work.n) % (2*work.n)
|
||||
# Check whether they are still active.
|
||||
# To start, we need to find out where in `work.active` they would
|
||||
# appear if they are indeed there.
|
||||
j = np.searchsorted(work.active, also_stop)
|
||||
# If the location exceeds the length of the `work.active`, they are
|
||||
# not there.
|
||||
j = j[j < len(work.active)]
|
||||
# Check whether they are still there.
|
||||
j = j[also_stop == work.active[j]]
|
||||
# Now convert these to boolean indices to use with `work.status`.
|
||||
i = np.zeros_like(stop)
|
||||
i[j] = True # boolean indices of elements that can also stop
|
||||
i = i & ~stop
|
||||
work.status[i] = _ESTOPONESIDE
|
||||
stop[i] = True
|
||||
|
||||
# Condition 3: moving end of bracket reaches limit
|
||||
i = (work.x == work.limit) & ~stop
|
||||
work.status[i] = _ELIMITS
|
||||
stop[i] = True
|
||||
|
||||
# Condition 4: non-finite value encountered
|
||||
i = ~(np.isfinite(work.x) & np.isfinite(work.f)) & ~stop
|
||||
work.status[i] = eim._EVALUEERR
|
||||
stop[i] = True
|
||||
|
||||
return stop
|
||||
|
||||
def post_termination_check(work):
|
||||
pass
|
||||
|
||||
def customize_result(res, shape):
|
||||
n = len(res['x']) // 2
|
||||
|
||||
# To avoid ambiguity, below we refer to `xl0`, the initial left endpoint
|
||||
# as `a` and `xr0`, the initial right endpoint, as `b`.
|
||||
# Because we treat the two one-sided searches as though they were
|
||||
# independent, what we keep track of in `work` and what we want to
|
||||
# return in `res` look quite different. Combine the results from the
|
||||
# two one-sided searches before reporting the results to the user.
|
||||
# - "a" refers to the leftward search (the moving end started at `a`)
|
||||
# - "b" refers to the rightward search (the moving end started at `b`)
|
||||
# - "l" refers to the left end of the bracket (closer to -oo)
|
||||
# - "r" refers to the right end of the bracket (closer to +oo)
|
||||
xal = res['x'][:n]
|
||||
xar = res['x_last'][:n]
|
||||
xbl = res['x_last'][n:]
|
||||
xbr = res['x'][n:]
|
||||
|
||||
fal = res['f'][:n]
|
||||
far = res['f_last'][:n]
|
||||
fbl = res['f_last'][n:]
|
||||
fbr = res['f'][n:]
|
||||
|
||||
# Initialize the brackets and corresponding function values to return
|
||||
# to the user. Brackets may not be valid (e.g. there is no root,
|
||||
# there weren't enough iterations, NaN encountered), but we still need
|
||||
# to return something. One option would be all NaNs, but what I've
|
||||
# chosen here is the left- and right-most points at which the function
|
||||
# has been evaluated. This gives the user some information about what
|
||||
# interval of the real line has been searched and shows that there is
|
||||
# no sign change between the two ends.
|
||||
xl = xal.copy()
|
||||
fl = fal.copy()
|
||||
xr = xbr.copy()
|
||||
fr = fbr.copy()
|
||||
|
||||
# `status` indicates whether the bracket is valid or not. If so,
|
||||
# we want to adjust the bracket we return to be the narrowest possible
|
||||
# given the points at which we evaluated the function.
|
||||
# For example if bracket "a" is valid and smaller than bracket "b" OR
|
||||
# if bracket "a" is valid and bracket "b" is not valid, we want to
|
||||
# return bracket "a" (and vice versa).
|
||||
sa = res['status'][:n]
|
||||
sb = res['status'][n:]
|
||||
|
||||
da = xar - xal
|
||||
db = xbr - xbl
|
||||
|
||||
i1 = ((da <= db) & (sa == 0)) | ((sa == 0) & (sb != 0))
|
||||
i2 = ((db <= da) & (sb == 0)) | ((sb == 0) & (sa != 0))
|
||||
|
||||
xr[i1] = xar[i1]
|
||||
fr[i1] = far[i1]
|
||||
xl[i2] = xbl[i2]
|
||||
fl[i2] = fbl[i2]
|
||||
|
||||
# Finish assembling the result object
|
||||
res['xl'] = xl
|
||||
res['xr'] = xr
|
||||
res['fl'] = fl
|
||||
res['fr'] = fr
|
||||
|
||||
res['nit'] = np.maximum(res['nit'][:n], res['nit'][n:])
|
||||
res['nfev'] = res['nfev'][:n] + res['nfev'][n:]
|
||||
# If the status on one side is zero, the status is zero. In any case,
|
||||
# report the status from one side only.
|
||||
res['status'] = np.choose(sa == 0, (sb, sa))
|
||||
res['success'] = (res['status'] == 0)
|
||||
|
||||
del res['x']
|
||||
del res['f']
|
||||
del res['x_last']
|
||||
del res['f_last']
|
||||
|
||||
return shape[:-1]
|
||||
|
||||
return eim._loop(work, callback, shape, maxiter, func, args, dtype,
|
||||
pre_func_eval, post_func_eval, check_termination,
|
||||
post_termination_check, customize_result, res_work_pairs,
|
||||
xp)
|
||||
|
||||
|
||||
def _bracket_minimum_iv(func, xm0, xl0, xr0, xmin, xmax, factor, args, maxiter):
|
||||
|
||||
if not callable(func):
|
||||
raise ValueError('`func` must be callable.')
|
||||
|
||||
if not np.iterable(args):
|
||||
args = (args,)
|
||||
|
||||
xm0 = np.asarray(xm0)[()]
|
||||
if not np.issubdtype(xm0.dtype, np.number) or np.iscomplex(xm0).any():
|
||||
raise ValueError('`xm0` must be numeric and real.')
|
||||
|
||||
xmin = -np.inf if xmin is None else xmin
|
||||
xmax = np.inf if xmax is None else xmax
|
||||
|
||||
# If xl0 (xr0) is not supplied, fill with a dummy value for the sake
|
||||
# of broadcasting. We need to wait until xmin (xmax) has been validated
|
||||
# to compute the default values.
|
||||
xl0_not_supplied = False
|
||||
if xl0 is None:
|
||||
xl0 = np.nan
|
||||
xl0_not_supplied = True
|
||||
|
||||
xr0_not_supplied = False
|
||||
if xr0 is None:
|
||||
xr0 = np.nan
|
||||
xr0_not_supplied = True
|
||||
|
||||
factor = 2.0 if factor is None else factor
|
||||
xl0, xm0, xr0, xmin, xmax, factor = np.broadcast_arrays(
|
||||
xl0, xm0, xr0, xmin, xmax, factor
|
||||
)
|
||||
|
||||
if not np.issubdtype(xl0.dtype, np.number) or np.iscomplex(xl0).any():
|
||||
raise ValueError('`xl0` must be numeric and real.')
|
||||
|
||||
if not np.issubdtype(xr0.dtype, np.number) or np.iscomplex(xr0).any():
|
||||
raise ValueError('`xr0` must be numeric and real.')
|
||||
|
||||
if not np.issubdtype(xmin.dtype, np.number) or np.iscomplex(xmin).any():
|
||||
raise ValueError('`xmin` must be numeric and real.')
|
||||
|
||||
if not np.issubdtype(xmax.dtype, np.number) or np.iscomplex(xmax).any():
|
||||
raise ValueError('`xmax` must be numeric and real.')
|
||||
|
||||
if not np.issubdtype(factor.dtype, np.number) or np.iscomplex(factor).any():
|
||||
raise ValueError('`factor` must be numeric and real.')
|
||||
if not np.all(factor > 1):
|
||||
raise ValueError('All elements of `factor` must be greater than 1.')
|
||||
|
||||
# Calculate default values of xl0 and/or xr0 if they have not been supplied
|
||||
# by the user. We need to be careful to ensure xl0 and xr0 are not outside
|
||||
# of (xmin, xmax).
|
||||
if xl0_not_supplied:
|
||||
xl0 = xm0 - np.minimum((xm0 - xmin)/16, 0.5)
|
||||
if xr0_not_supplied:
|
||||
xr0 = xm0 + np.minimum((xmax - xm0)/16, 0.5)
|
||||
|
||||
maxiter = np.asarray(maxiter)
|
||||
message = '`maxiter` must be a non-negative integer.'
|
||||
if (not np.issubdtype(maxiter.dtype, np.number) or maxiter.shape != tuple()
|
||||
or np.iscomplex(maxiter)):
|
||||
raise ValueError(message)
|
||||
maxiter_int = int(maxiter[()])
|
||||
if not maxiter == maxiter_int or maxiter < 0:
|
||||
raise ValueError(message)
|
||||
|
||||
return func, xm0, xl0, xr0, xmin, xmax, factor, args, maxiter
|
||||
|
||||
|
||||
def _bracket_minimum(func, xm0, *, xl0=None, xr0=None, xmin=None, xmax=None,
|
||||
factor=None, args=(), maxiter=1000):
|
||||
"""Bracket the minimum of a unimodal scalar function of one variable
|
||||
|
||||
This function works elementwise when `xm0`, `xl0`, `xr0`, `xmin`, `xmax`,
|
||||
and the elements of `args` are broadcastable arrays.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
The function for which the minimum is to be bracketed.
|
||||
The signature must be::
|
||||
|
||||
func(x: ndarray, *args) -> ndarray
|
||||
|
||||
where each element of ``x`` is a finite real and ``args`` is a tuple,
|
||||
which may contain an arbitrary number of arrays that are broadcastable
|
||||
with ``x``. `func` must be an elementwise function: each element
|
||||
``func(x)[i]`` must equal ``func(x[i])`` for all indices `i`.
|
||||
xm0: float array_like
|
||||
Starting guess for middle point of bracket.
|
||||
xl0, xr0: float array_like, optional
|
||||
Starting guesses for left and right endpoints of the bracket. Must be
|
||||
broadcastable with one another and with `xm0`.
|
||||
xmin, xmax : float array_like, optional
|
||||
Minimum and maximum allowable endpoints of the bracket, inclusive. Must
|
||||
be broadcastable with `xl0`, `xm0`, and `xr0`.
|
||||
factor : float array_like, optional
|
||||
Controls expansion of bracket endpoint in downhill direction. Works
|
||||
differently in the cases where a limit is set in the downhill direction
|
||||
with `xmax` or `xmin`. See Notes.
|
||||
args : tuple, optional
|
||||
Additional positional arguments to be passed to `func`. Must be arrays
|
||||
broadcastable with `xl0`, `xm0`, `xr0`, `xmin`, and `xmax`. If the
|
||||
callable to be bracketed requires arguments that are not broadcastable
|
||||
with these arrays, wrap that callable with `func` such that `func`
|
||||
accepts only ``x`` and broadcastable arrays.
|
||||
maxiter : int, optional
|
||||
The maximum number of iterations of the algorithm to perform. The number
|
||||
of function evaluations is three greater than the number of iterations.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : _RichResult
|
||||
An instance of `scipy._lib._util._RichResult` with the following
|
||||
attributes. The descriptions are written as though the values will be
|
||||
scalars; however, if `func` returns an array, the outputs will be
|
||||
arrays of the same shape.
|
||||
|
||||
xl, xm, xr : float
|
||||
The left, middle, and right points of the bracket, if the algorithm
|
||||
terminated successfully.
|
||||
fl, fm, fr : float
|
||||
The function value at the left, middle, and right points of the bracket.
|
||||
nfev : int
|
||||
The number of function evaluations required to find the bracket.
|
||||
nit : int
|
||||
The number of iterations of the algorithm that were performed.
|
||||
status : int
|
||||
An integer representing the exit status of the algorithm.
|
||||
|
||||
- ``0`` : The algorithm produced a valid bracket.
|
||||
- ``-1`` : The bracket expanded to the allowable limits. Assuming
|
||||
unimodality, this implies the endpoint at the limit is a
|
||||
minimizer.
|
||||
- ``-2`` : The maximum number of iterations was reached.
|
||||
- ``-3`` : A non-finite value was encountered.
|
||||
- ``-4`` : ``None`` shall pass.
|
||||
- ``-5`` : The initial bracket does not satisfy
|
||||
`xmin <= xl0 < xm0 < xr0 <= xmax`.
|
||||
|
||||
success : bool
|
||||
``True`` when the algorithm terminated successfully (status ``0``).
|
||||
|
||||
Notes
|
||||
-----
|
||||
Similar to `scipy.optimize.bracket`, this function seeks to find real
|
||||
points ``xl < xm < xr`` such that ``f(xl) >= f(xm)`` and ``f(xr) >= f(xm)``,
|
||||
where at least one of the inequalities is strict. Unlike `scipy.optimize.bracket`,
|
||||
this function can operate in a vectorized manner on array input, so long as
|
||||
the input arrays are broadcastable with each other. Also unlike
|
||||
`scipy.optimize.bracket`, users may specify minimum and maximum endpoints
|
||||
for the desired bracket.
|
||||
|
||||
Given an initial trio of points ``xl = xl0``, ``xm = xm0``, ``xr = xr0``,
|
||||
the algorithm checks if these points already give a valid bracket. If not,
|
||||
a new endpoint, ``w`` is chosen in the "downhill" direction, ``xm`` becomes the new
|
||||
opposite endpoint, and either `xl` or `xr` becomes the new middle point,
|
||||
depending on which direction is downhill. The algorithm repeats from here.
|
||||
|
||||
The new endpoint `w` is chosen differently depending on whether or not a
|
||||
boundary `xmin` or `xmax` has been set in the downhill direction. Without
|
||||
loss of generality, suppose the downhill direction is to the right, so that
|
||||
``f(xl) > f(xm) > f(xr)``. If there is no boundary to the right, then `w`
|
||||
is chosen to be ``xr + factor * (xr - xm)`` where `factor` is controlled by
|
||||
the user (defaults to 2.0) so that step sizes increase in geometric proportion.
|
||||
If there is a boundary, `xmax` in this case, then `w` is chosen to be
|
||||
``xmax - (xmax - xr)/factor``, with steps slowing to a stop at
|
||||
`xmax`. This cautious approach ensures that a minimum near but distinct from
|
||||
the boundary isn't missed while also detecting whether or not the `xmax` is
|
||||
a minimizer when `xmax` is reached after a finite number of steps.
|
||||
""" # noqa: E501
|
||||
callback = None # works; I just don't want to test it
|
||||
|
||||
temp = _bracket_minimum_iv(func, xm0, xl0, xr0, xmin, xmax, factor, args, maxiter)
|
||||
func, xm0, xl0, xr0, xmin, xmax, factor, args, maxiter = temp
|
||||
|
||||
xs = (xl0, xm0, xr0)
|
||||
temp = eim._initialize(func, xs, args)
|
||||
func, xs, fs, args, shape, dtype, xp = temp
|
||||
|
||||
xl0, xm0, xr0 = xs
|
||||
fl0, fm0, fr0 = fs
|
||||
xmin = np.broadcast_to(xmin, shape).astype(dtype, copy=False).ravel()
|
||||
xmax = np.broadcast_to(xmax, shape).astype(dtype, copy=False).ravel()
|
||||
invalid_bracket = ~((xmin <= xl0) & (xl0 < xm0) & (xm0 < xr0) & (xr0 <= xmax))
|
||||
# We will modify factor later on so make a copy. np.broadcast_to returns
|
||||
# a read-only view.
|
||||
factor = np.broadcast_to(factor, shape).astype(dtype, copy=True).ravel()
|
||||
|
||||
# To simplify the logic, swap xl and xr if f(xl) < f(xr). We should always be
|
||||
# marching downhill in the direction from xl to xr.
|
||||
comp = fl0 < fr0
|
||||
xl0[comp], xr0[comp] = xr0[comp], xl0[comp]
|
||||
fl0[comp], fr0[comp] = fr0[comp], fl0[comp]
|
||||
# We only need the boundary in the direction we're traveling.
|
||||
limit = np.where(comp, xmin, xmax)
|
||||
|
||||
unlimited = np.isinf(limit)
|
||||
limited = ~unlimited
|
||||
step = np.empty_like(xl0)
|
||||
|
||||
step[unlimited] = (xr0[unlimited] - xm0[unlimited])
|
||||
step[limited] = (limit[limited] - xr0[limited])
|
||||
|
||||
# Step size is divided by factor for case where there is a limit.
|
||||
factor[limited] = 1 / factor[limited]
|
||||
|
||||
status = np.full_like(xl0, eim._EINPROGRESS, dtype=int)
|
||||
status[invalid_bracket] = eim._EINPUTERR
|
||||
nit, nfev = 0, 3
|
||||
|
||||
work = _RichResult(xl=xl0, xm=xm0, xr=xr0, xr0=xr0, fl=fl0, fm=fm0, fr=fr0,
|
||||
step=step, limit=limit, limited=limited, factor=factor, nit=nit,
|
||||
nfev=nfev, status=status, args=args)
|
||||
|
||||
res_work_pairs = [('status', 'status'), ('xl', 'xl'), ('xm', 'xm'), ('xr', 'xr'),
|
||||
('nit', 'nit'), ('nfev', 'nfev'), ('fl', 'fl'), ('fm', 'fm'),
|
||||
('fr', 'fr')]
|
||||
|
||||
def pre_func_eval(work):
|
||||
work.step *= work.factor
|
||||
x = np.empty_like(work.xr)
|
||||
x[~work.limited] = work.xr0[~work.limited] + work.step[~work.limited]
|
||||
x[work.limited] = work.limit[work.limited] - work.step[work.limited]
|
||||
# Since the new bracket endpoint is calculated from an offset with the
|
||||
# limit, it may be the case that the new endpoint equals the old endpoint,
|
||||
# when the old endpoint is sufficiently close to the limit. We use the
|
||||
# limit itself as the new endpoint in these cases.
|
||||
x[work.limited] = np.where(
|
||||
x[work.limited] == work.xr[work.limited],
|
||||
work.limit[work.limited],
|
||||
x[work.limited],
|
||||
)
|
||||
return x
|
||||
|
||||
def post_func_eval(x, f, work):
|
||||
work.xl, work.xm, work.xr = work.xm, work.xr, x
|
||||
work.fl, work.fm, work.fr = work.fm, work.fr, f
|
||||
|
||||
def check_termination(work):
|
||||
# Condition 0: Initial bracket is invalid.
|
||||
stop = (work.status == eim._EINPUTERR)
|
||||
|
||||
# Condition 1: A valid bracket has been found.
|
||||
i = (
|
||||
(work.fl >= work.fm) & (work.fr > work.fm)
|
||||
| (work.fl > work.fm) & (work.fr >= work.fm)
|
||||
) & ~stop
|
||||
work.status[i] = eim._ECONVERGED
|
||||
stop[i] = True
|
||||
|
||||
# Condition 2: Moving end of bracket reaches limit.
|
||||
i = (work.xr == work.limit) & ~stop
|
||||
work.status[i] = _ELIMITS
|
||||
stop[i] = True
|
||||
|
||||
# Condition 3: non-finite value encountered
|
||||
i = ~(np.isfinite(work.xr) & np.isfinite(work.fr)) & ~stop
|
||||
work.status[i] = eim._EVALUEERR
|
||||
stop[i] = True
|
||||
|
||||
return stop
|
||||
|
||||
def post_termination_check(work):
|
||||
pass
|
||||
|
||||
def customize_result(res, shape):
|
||||
# Reorder entries of xl and xr if they were swapped due to f(xl0) < f(xr0).
|
||||
comp = res['xl'] > res['xr']
|
||||
res['xl'][comp], res['xr'][comp] = res['xr'][comp], res['xl'][comp]
|
||||
res['fl'][comp], res['fr'][comp] = res['fr'][comp], res['fl'][comp]
|
||||
return shape
|
||||
|
||||
return eim._loop(work, callback, shape,
|
||||
maxiter, func, args, dtype,
|
||||
pre_func_eval, post_func_eval,
|
||||
check_termination, post_termination_check,
|
||||
customize_result, res_work_pairs, xp)
|
||||
@@ -0,0 +1,549 @@
|
||||
import math
|
||||
import numpy as np
|
||||
import scipy._lib._elementwise_iterative_method as eim
|
||||
from scipy._lib._util import _RichResult
|
||||
from scipy._lib._array_api import xp_clip, xp_minimum, xp_sign
|
||||
|
||||
# TODO:
|
||||
# - (maybe?) don't use fancy indexing assignment
|
||||
# - figure out how to replace the new `try`/`except`s
|
||||
|
||||
|
||||
def _chandrupatla(func, a, b, *, args=(), xatol=None, xrtol=None,
|
||||
fatol=None, frtol=0, maxiter=None, callback=None):
|
||||
"""Find the root of an elementwise function using Chandrupatla's algorithm.
|
||||
|
||||
For each element of the output of `func`, `chandrupatla` seeks the scalar
|
||||
root that makes the element 0. This function allows for `a`, `b`, and the
|
||||
output of `func` to be of any broadcastable shapes.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
The function whose root is desired. The signature must be::
|
||||
|
||||
func(x: ndarray, *args) -> ndarray
|
||||
|
||||
where each element of ``x`` is a finite real and ``args`` is a tuple,
|
||||
which may contain an arbitrary number of components of any type(s).
|
||||
``func`` must be an elementwise function: each element ``func(x)[i]``
|
||||
must equal ``func(x[i])`` for all indices ``i``. `_chandrupatla`
|
||||
seeks an array ``x`` such that ``func(x)`` is an array of zeros.
|
||||
a, b : array_like
|
||||
The lower and upper bounds of the root of the function. Must be
|
||||
broadcastable with one another.
|
||||
args : tuple, optional
|
||||
Additional positional arguments to be passed to `func`.
|
||||
xatol, xrtol, fatol, frtol : float, optional
|
||||
Absolute and relative tolerances on the root and function value.
|
||||
See Notes for details.
|
||||
maxiter : int, optional
|
||||
The maximum number of iterations of the algorithm to perform.
|
||||
The default is the maximum possible number of bisections within
|
||||
the (normal) floating point numbers of the relevant dtype.
|
||||
callback : callable, optional
|
||||
An optional user-supplied function to be called before the first
|
||||
iteration and after each iteration.
|
||||
Called as ``callback(res)``, where ``res`` is a ``_RichResult``
|
||||
similar to that returned by `_chandrupatla` (but containing the current
|
||||
iterate's values of all variables). If `callback` raises a
|
||||
``StopIteration``, the algorithm will terminate immediately and
|
||||
`_chandrupatla` will return a result.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : _RichResult
|
||||
An instance of `scipy._lib._util._RichResult` with the following
|
||||
attributes. The descriptions are written as though the values will be
|
||||
scalars; however, if `func` returns an array, the outputs will be
|
||||
arrays of the same shape.
|
||||
|
||||
x : float
|
||||
The root of the function, if the algorithm terminated successfully.
|
||||
nfev : int
|
||||
The number of times the function was called to find the root.
|
||||
nit : int
|
||||
The number of iterations of Chandrupatla's algorithm performed.
|
||||
status : int
|
||||
An integer representing the exit status of the algorithm.
|
||||
``0`` : The algorithm converged to the specified tolerances.
|
||||
``-1`` : The algorithm encountered an invalid bracket.
|
||||
``-2`` : The maximum number of iterations was reached.
|
||||
``-3`` : A non-finite value was encountered.
|
||||
``-4`` : Iteration was terminated by `callback`.
|
||||
``1`` : The algorithm is proceeding normally (in `callback` only).
|
||||
success : bool
|
||||
``True`` when the algorithm terminated successfully (status ``0``).
|
||||
fun : float
|
||||
The value of `func` evaluated at `x`.
|
||||
xl, xr : float
|
||||
The lower and upper ends of the bracket.
|
||||
fl, fr : float
|
||||
The function value at the lower and upper ends of the bracket.
|
||||
|
||||
Notes
|
||||
-----
|
||||
Implemented based on Chandrupatla's original paper [1]_.
|
||||
|
||||
If ``xl`` and ``xr`` are the left and right ends of the bracket,
|
||||
``xmin = xl if abs(func(xl)) <= abs(func(xr)) else xr``,
|
||||
and ``fmin0 = min(func(a), func(b))``, then the algorithm is considered to
|
||||
have converged when ``abs(xr - xl) < xatol + abs(xmin) * xrtol`` or
|
||||
``fun(xmin) <= fatol + abs(fmin0) * frtol``. This is equivalent to the
|
||||
termination condition described in [1]_ with ``xrtol = 4e-10``,
|
||||
``xatol = 1e-5``, and ``fatol = frtol = 0``. The default values are
|
||||
``xatol = 4*tiny``, ``xrtol = 4*eps``, ``frtol = 0``, and ``fatol = tiny``,
|
||||
where ``eps`` and ``tiny`` are the precision and smallest normal number
|
||||
of the result ``dtype`` of function inputs and outputs.
|
||||
|
||||
References
|
||||
----------
|
||||
|
||||
.. [1] Chandrupatla, Tirupathi R.
|
||||
"A new hybrid quadratic/bisection algorithm for finding the zero of a
|
||||
nonlinear function without using derivatives".
|
||||
Advances in Engineering Software, 28(3), 145-149.
|
||||
https://doi.org/10.1016/s0965-9978(96)00051-8
|
||||
|
||||
See Also
|
||||
--------
|
||||
brentq, brenth, ridder, bisect, newton
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> from scipy import optimize
|
||||
>>> def f(x, c):
|
||||
... return x**3 - 2*x - c
|
||||
>>> c = 5
|
||||
>>> res = optimize._chandrupatla._chandrupatla(f, 0, 3, args=(c,))
|
||||
>>> res.x
|
||||
2.0945514818937463
|
||||
|
||||
>>> c = [3, 4, 5]
|
||||
>>> res = optimize._chandrupatla._chandrupatla(f, 0, 3, args=(c,))
|
||||
>>> res.x
|
||||
array([1.8932892 , 2. , 2.09455148])
|
||||
|
||||
"""
|
||||
res = _chandrupatla_iv(func, args, xatol, xrtol,
|
||||
fatol, frtol, maxiter, callback)
|
||||
func, args, xatol, xrtol, fatol, frtol, maxiter, callback = res
|
||||
|
||||
# Initialization
|
||||
temp = eim._initialize(func, (a, b), args)
|
||||
func, xs, fs, args, shape, dtype, xp = temp
|
||||
x1, x2 = xs
|
||||
f1, f2 = fs
|
||||
status = xp.full_like(x1, eim._EINPROGRESS, dtype=xp.int32) # in progress
|
||||
nit, nfev = 0, 2 # two function evaluations performed above
|
||||
finfo = xp.finfo(dtype)
|
||||
xatol = 4*finfo.smallest_normal if xatol is None else xatol
|
||||
xrtol = 4*finfo.eps if xrtol is None else xrtol
|
||||
fatol = finfo.smallest_normal if fatol is None else fatol
|
||||
frtol = frtol * xp_minimum(xp.abs(f1), xp.abs(f2))
|
||||
maxiter = (math.log2(finfo.max) - math.log2(finfo.smallest_normal)
|
||||
if maxiter is None else maxiter)
|
||||
work = _RichResult(x1=x1, f1=f1, x2=x2, f2=f2, x3=None, f3=None, t=0.5,
|
||||
xatol=xatol, xrtol=xrtol, fatol=fatol, frtol=frtol,
|
||||
nit=nit, nfev=nfev, status=status)
|
||||
res_work_pairs = [('status', 'status'), ('x', 'xmin'), ('fun', 'fmin'),
|
||||
('nit', 'nit'), ('nfev', 'nfev'), ('xl', 'x1'),
|
||||
('fl', 'f1'), ('xr', 'x2'), ('fr', 'f2')]
|
||||
|
||||
def pre_func_eval(work):
|
||||
# [1] Figure 1 (first box)
|
||||
x = work.x1 + work.t * (work.x2 - work.x1)
|
||||
return x
|
||||
|
||||
def post_func_eval(x, f, work):
|
||||
# [1] Figure 1 (first diamond and boxes)
|
||||
# Note: y/n are reversed in figure; compare to BASIC in appendix
|
||||
work.x3, work.f3 = (xp.asarray(work.x2, copy=True),
|
||||
xp.asarray(work.f2, copy=True))
|
||||
j = xp.sign(f) == xp.sign(work.f1)
|
||||
nj = ~j
|
||||
work.x3[j], work.f3[j] = work.x1[j], work.f1[j]
|
||||
work.x2[nj], work.f2[nj] = work.x1[nj], work.f1[nj]
|
||||
work.x1, work.f1 = x, f
|
||||
|
||||
def check_termination(work):
|
||||
# [1] Figure 1 (second diamond)
|
||||
# Check for all terminal conditions and record statuses.
|
||||
|
||||
# See [1] Section 4 (first two sentences)
|
||||
i = xp.abs(work.f1) < xp.abs(work.f2)
|
||||
work.xmin = xp.where(i, work.x1, work.x2)
|
||||
work.fmin = xp.where(i, work.f1, work.f2)
|
||||
stop = xp.zeros_like(work.x1, dtype=xp.bool) # termination condition met
|
||||
|
||||
# If function value tolerance is met, report successful convergence,
|
||||
# regardless of other conditions. Note that `frtol` has been redefined
|
||||
# as `frtol = frtol * minimum(f1, f2)`, where `f1` and `f2` are the
|
||||
# function evaluated at the original ends of the bracket.
|
||||
i = xp.abs(work.fmin) <= work.fatol + work.frtol
|
||||
work.status[i] = eim._ECONVERGED
|
||||
stop[i] = True
|
||||
|
||||
# If the bracket is no longer valid, report failure (unless a function
|
||||
# tolerance is met, as detected above).
|
||||
i = (xp_sign(work.f1) == xp_sign(work.f2)) & ~stop
|
||||
NaN = xp.asarray(xp.nan, dtype=work.xmin.dtype)
|
||||
work.xmin[i], work.fmin[i], work.status[i] = NaN, NaN, eim._ESIGNERR
|
||||
stop[i] = True
|
||||
|
||||
# If the abscissae are non-finite or either function value is NaN,
|
||||
# report failure.
|
||||
x_nonfinite = ~(xp.isfinite(work.x1) & xp.isfinite(work.x2))
|
||||
f_nan = xp.isnan(work.f1) & xp.isnan(work.f2)
|
||||
i = (x_nonfinite | f_nan) & ~stop
|
||||
work.xmin[i], work.fmin[i], work.status[i] = NaN, NaN, eim._EVALUEERR
|
||||
stop[i] = True
|
||||
|
||||
# This is the convergence criterion used in bisect. Chandrupatla's
|
||||
# criterion is equivalent to this except with a factor of 4 on `xrtol`.
|
||||
work.dx = xp.abs(work.x2 - work.x1)
|
||||
work.tol = xp.abs(work.xmin) * work.xrtol + work.xatol
|
||||
i = work.dx < work.tol
|
||||
work.status[i] = eim._ECONVERGED
|
||||
stop[i] = True
|
||||
|
||||
return stop
|
||||
|
||||
def post_termination_check(work):
|
||||
# [1] Figure 1 (third diamond and boxes / Equation 1)
|
||||
xi1 = (work.x1 - work.x2) / (work.x3 - work.x2)
|
||||
phi1 = (work.f1 - work.f2) / (work.f3 - work.f2)
|
||||
alpha = (work.x3 - work.x1) / (work.x2 - work.x1)
|
||||
j = ((1 - xp.sqrt(1 - xi1)) < phi1) & (phi1 < xp.sqrt(xi1))
|
||||
|
||||
f1j, f2j, f3j, alphaj = work.f1[j], work.f2[j], work.f3[j], alpha[j]
|
||||
t = xp.full_like(alpha, 0.5)
|
||||
t[j] = (f1j / (f1j - f2j) * f3j / (f3j - f2j)
|
||||
- alphaj * f1j / (f3j - f1j) * f2j / (f2j - f3j))
|
||||
|
||||
# [1] Figure 1 (last box; see also BASIC in appendix with comment
|
||||
# "Adjust T Away from the Interval Boundary")
|
||||
tl = 0.5 * work.tol / work.dx
|
||||
work.t = xp_clip(t, tl, 1 - tl)
|
||||
|
||||
def customize_result(res, shape):
|
||||
xl, xr, fl, fr = res['xl'], res['xr'], res['fl'], res['fr']
|
||||
i = res['xl'] < res['xr']
|
||||
res['xl'] = xp.where(i, xl, xr)
|
||||
res['xr'] = xp.where(i, xr, xl)
|
||||
res['fl'] = xp.where(i, fl, fr)
|
||||
res['fr'] = xp.where(i, fr, fl)
|
||||
return shape
|
||||
|
||||
return eim._loop(work, callback, shape, maxiter, func, args, dtype,
|
||||
pre_func_eval, post_func_eval, check_termination,
|
||||
post_termination_check, customize_result, res_work_pairs,
|
||||
xp=xp)
|
||||
|
||||
|
||||
def _chandrupatla_iv(func, args, xatol, xrtol,
|
||||
fatol, frtol, maxiter, callback):
|
||||
# Input validation for `_chandrupatla`
|
||||
|
||||
if not callable(func):
|
||||
raise ValueError('`func` must be callable.')
|
||||
|
||||
if not np.iterable(args):
|
||||
args = (args,)
|
||||
|
||||
# tolerances are floats, not arrays; OK to use NumPy
|
||||
tols = np.asarray([xatol if xatol is not None else 1,
|
||||
xrtol if xrtol is not None else 1,
|
||||
fatol if fatol is not None else 1,
|
||||
frtol if frtol is not None else 1])
|
||||
if (not np.issubdtype(tols.dtype, np.number) or np.any(tols < 0)
|
||||
or np.any(np.isnan(tols)) or tols.shape != (4,)):
|
||||
raise ValueError('Tolerances must be non-negative scalars.')
|
||||
|
||||
if maxiter is not None:
|
||||
maxiter_int = int(maxiter)
|
||||
if maxiter != maxiter_int or maxiter < 0:
|
||||
raise ValueError('`maxiter` must be a non-negative integer.')
|
||||
|
||||
if callback is not None and not callable(callback):
|
||||
raise ValueError('`callback` must be callable.')
|
||||
|
||||
return func, args, xatol, xrtol, fatol, frtol, maxiter, callback
|
||||
|
||||
|
||||
def _chandrupatla_minimize(func, x1, x2, x3, *, args=(), xatol=None,
|
||||
xrtol=None, fatol=None, frtol=None, maxiter=100,
|
||||
callback=None):
|
||||
"""Find the minimizer of an elementwise function.
|
||||
|
||||
For each element of the output of `func`, `_chandrupatla_minimize` seeks
|
||||
the scalar minimizer that minimizes the element. This function allows for
|
||||
`x1`, `x2`, `x3`, and the elements of `args` to be arrays of any
|
||||
broadcastable shapes.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
The function whose minimizer is desired. The signature must be::
|
||||
|
||||
func(x: ndarray, *args) -> ndarray
|
||||
|
||||
where each element of ``x`` is a finite real and ``args`` is a tuple,
|
||||
which may contain an arbitrary number of arrays that are broadcastable
|
||||
with `x`. ``func`` must be an elementwise function: each element
|
||||
``func(x)[i]`` must equal ``func(x[i])`` for all indices ``i``.
|
||||
`_chandrupatla` seeks an array ``x`` such that ``func(x)`` is an array
|
||||
of minima.
|
||||
x1, x2, x3 : array_like
|
||||
The abscissae of a standard scalar minimization bracket. A bracket is
|
||||
valid if ``x1 < x2 < x3`` and ``func(x1) > func(x2) <= func(x3)``.
|
||||
Must be broadcastable with one another and `args`.
|
||||
args : tuple, optional
|
||||
Additional positional arguments to be passed to `func`. Must be arrays
|
||||
broadcastable with `x1`, `x2`, and `x3`. If the callable to be
|
||||
differentiated requires arguments that are not broadcastable with `x`,
|
||||
wrap that callable with `func` such that `func` accepts only `x` and
|
||||
broadcastable arrays.
|
||||
xatol, xrtol, fatol, frtol : float, optional
|
||||
Absolute and relative tolerances on the minimizer and function value.
|
||||
See Notes for details.
|
||||
maxiter : int, optional
|
||||
The maximum number of iterations of the algorithm to perform.
|
||||
callback : callable, optional
|
||||
An optional user-supplied function to be called before the first
|
||||
iteration and after each iteration.
|
||||
Called as ``callback(res)``, where ``res`` is a ``_RichResult``
|
||||
similar to that returned by `_chandrupatla_minimize` (but containing
|
||||
the current iterate's values of all variables). If `callback` raises a
|
||||
``StopIteration``, the algorithm will terminate immediately and
|
||||
`_chandrupatla_minimize` will return a result.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : _RichResult
|
||||
An instance of `scipy._lib._util._RichResult` with the following
|
||||
attributes. (The descriptions are written as though the values will be
|
||||
scalars; however, if `func` returns an array, the outputs will be
|
||||
arrays of the same shape.)
|
||||
|
||||
success : bool
|
||||
``True`` when the algorithm terminated successfully (status ``0``).
|
||||
status : int
|
||||
An integer representing the exit status of the algorithm.
|
||||
``0`` : The algorithm converged to the specified tolerances.
|
||||
``-1`` : The algorithm encountered an invalid bracket.
|
||||
``-2`` : The maximum number of iterations was reached.
|
||||
``-3`` : A non-finite value was encountered.
|
||||
``-4`` : Iteration was terminated by `callback`.
|
||||
``1`` : The algorithm is proceeding normally (in `callback` only).
|
||||
x : float
|
||||
The minimizer of the function, if the algorithm terminated
|
||||
successfully.
|
||||
fun : float
|
||||
The value of `func` evaluated at `x`.
|
||||
nfev : int
|
||||
The number of points at which `func` was evaluated.
|
||||
nit : int
|
||||
The number of iterations of the algorithm that were performed.
|
||||
xl, xm, xr : float
|
||||
The final three-point bracket.
|
||||
fl, fm, fr : float
|
||||
The function value at the bracket points.
|
||||
|
||||
Notes
|
||||
-----
|
||||
Implemented based on Chandrupatla's original paper [1]_.
|
||||
|
||||
If ``x1 < x2 < x3`` are the points of the bracket and ``f1 > f2 <= f3``
|
||||
are the values of ``func`` at those points, then the algorithm is
|
||||
considered to have converged when ``x3 - x1 <= abs(x2)*xrtol + xatol``
|
||||
or ``(f1 - 2*f2 + f3)/2 <= abs(f2)*frtol + fatol``. Note that first of
|
||||
these differs from the termination conditions described in [1]_. The
|
||||
default values of `xrtol` is the square root of the precision of the
|
||||
appropriate dtype, and ``xatol = fatol = frtol`` is the smallest normal
|
||||
number of the appropriate dtype.
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Chandrupatla, Tirupathi R. (1998).
|
||||
"An efficient quadratic fit-sectioning algorithm for minimization
|
||||
without derivatives".
|
||||
Computer Methods in Applied Mechanics and Engineering, 152 (1-2),
|
||||
211-217. https://doi.org/10.1016/S0045-7825(97)00190-4
|
||||
|
||||
See Also
|
||||
--------
|
||||
golden, brent, bounded
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> from scipy.optimize._chandrupatla import _chandrupatla_minimize
|
||||
>>> def f(x, args=1):
|
||||
... return (x - args)**2
|
||||
>>> res = _chandrupatla_minimize(f, -5, 0, 5)
|
||||
>>> res.x
|
||||
1.0
|
||||
>>> c = [1, 1.5, 2]
|
||||
>>> res = _chandrupatla_minimize(f, -5, 0, 5, args=(c,))
|
||||
>>> res.x
|
||||
array([1. , 1.5, 2. ])
|
||||
"""
|
||||
res = _chandrupatla_iv(func, args, xatol, xrtol,
|
||||
fatol, frtol, maxiter, callback)
|
||||
func, args, xatol, xrtol, fatol, frtol, maxiter, callback = res
|
||||
|
||||
# Initialization
|
||||
xs = (x1, x2, x3)
|
||||
temp = eim._initialize(func, xs, args)
|
||||
func, xs, fs, args, shape, dtype, xp = temp # line split for PEP8
|
||||
x1, x2, x3 = xs
|
||||
f1, f2, f3 = fs
|
||||
phi = dtype.type(0.5 + 0.5*5**0.5) # golden ratio
|
||||
status = np.full_like(x1, eim._EINPROGRESS, dtype=int) # in progress
|
||||
nit, nfev = 0, 3 # three function evaluations performed above
|
||||
fatol = np.finfo(dtype).tiny if fatol is None else fatol
|
||||
frtol = np.finfo(dtype).tiny if frtol is None else frtol
|
||||
xatol = np.finfo(dtype).tiny if xatol is None else xatol
|
||||
xrtol = np.sqrt(np.finfo(dtype).eps) if xrtol is None else xrtol
|
||||
|
||||
# Ensure that x1 < x2 < x3 initially.
|
||||
xs, fs = np.vstack((x1, x2, x3)), np.vstack((f1, f2, f3))
|
||||
i = np.argsort(xs, axis=0)
|
||||
x1, x2, x3 = np.take_along_axis(xs, i, axis=0)
|
||||
f1, f2, f3 = np.take_along_axis(fs, i, axis=0)
|
||||
q0 = x3.copy() # "At the start, q0 is set at x3..." ([1] after (7))
|
||||
|
||||
work = _RichResult(x1=x1, f1=f1, x2=x2, f2=f2, x3=x3, f3=f3, phi=phi,
|
||||
xatol=xatol, xrtol=xrtol, fatol=fatol, frtol=frtol,
|
||||
nit=nit, nfev=nfev, status=status, q0=q0, args=args)
|
||||
res_work_pairs = [('status', 'status'),
|
||||
('x', 'x2'), ('fun', 'f2'),
|
||||
('nit', 'nit'), ('nfev', 'nfev'),
|
||||
('xl', 'x1'), ('xm', 'x2'), ('xr', 'x3'),
|
||||
('fl', 'f1'), ('fm', 'f2'), ('fr', 'f3')]
|
||||
|
||||
def pre_func_eval(work):
|
||||
# `_check_termination` is called first -> `x3 - x2 > x2 - x1`
|
||||
# But let's calculate a few terms that we'll reuse
|
||||
x21 = work.x2 - work.x1
|
||||
x32 = work.x3 - work.x2
|
||||
|
||||
# [1] Section 3. "The quadratic minimum point Q1 is calculated using
|
||||
# the relations developed in the previous section." [1] Section 2 (5/6)
|
||||
A = x21 * (work.f3 - work.f2)
|
||||
B = x32 * (work.f1 - work.f2)
|
||||
C = A / (A + B)
|
||||
# q1 = C * (work.x1 + work.x2) / 2 + (1 - C) * (work.x2 + work.x3) / 2
|
||||
q1 = 0.5 * (C*(work.x1 - work.x3) + work.x2 + work.x3) # much faster
|
||||
# this is an array, so multiplying by 0.5 does not change dtype
|
||||
|
||||
# "If Q1 and Q0 are sufficiently close... Q1 is accepted if it is
|
||||
# sufficiently away from the inside point x2"
|
||||
i = abs(q1 - work.q0) < 0.5 * abs(x21) # [1] (7)
|
||||
xi = q1[i]
|
||||
# Later, after (9), "If the point Q1 is in a +/- xtol neighborhood of
|
||||
# x2, the new point is chosen in the larger interval at a distance
|
||||
# tol away from x2."
|
||||
# See also QBASIC code after "Accept Ql adjust if close to X2".
|
||||
j = abs(q1[i] - work.x2[i]) <= work.xtol[i]
|
||||
xi[j] = work.x2[i][j] + np.sign(x32[i][j]) * work.xtol[i][j]
|
||||
|
||||
# "If condition (7) is not satisfied, golden sectioning of the larger
|
||||
# interval is carried out to introduce the new point."
|
||||
# (For simplicity, we go ahead and calculate it for all points, but we
|
||||
# change the elements for which the condition was satisfied.)
|
||||
x = work.x2 + (2 - work.phi) * x32
|
||||
x[i] = xi
|
||||
|
||||
# "We define Q0 as the value of Q1 at the previous iteration."
|
||||
work.q0 = q1
|
||||
return x
|
||||
|
||||
def post_func_eval(x, f, work):
|
||||
# Standard logic for updating a three-point bracket based on a new
|
||||
# point. In QBASIC code, see "IF SGN(X-X2) = SGN(X3-X2) THEN...".
|
||||
# There is an awful lot of data copying going on here; this would
|
||||
# probably benefit from code optimization or implementation in Pythran.
|
||||
i = np.sign(x - work.x2) == np.sign(work.x3 - work.x2)
|
||||
xi, x1i, x2i, x3i = x[i], work.x1[i], work.x2[i], work.x3[i],
|
||||
fi, f1i, f2i, f3i = f[i], work.f1[i], work.f2[i], work.f3[i]
|
||||
j = fi > f2i
|
||||
x3i[j], f3i[j] = xi[j], fi[j]
|
||||
j = ~j
|
||||
x1i[j], f1i[j], x2i[j], f2i[j] = x2i[j], f2i[j], xi[j], fi[j]
|
||||
|
||||
ni = ~i
|
||||
xni, x1ni, x2ni, x3ni = x[ni], work.x1[ni], work.x2[ni], work.x3[ni],
|
||||
fni, f1ni, f2ni, f3ni = f[ni], work.f1[ni], work.f2[ni], work.f3[ni]
|
||||
j = fni > f2ni
|
||||
x1ni[j], f1ni[j] = xni[j], fni[j]
|
||||
j = ~j
|
||||
x3ni[j], f3ni[j], x2ni[j], f2ni[j] = x2ni[j], f2ni[j], xni[j], fni[j]
|
||||
|
||||
work.x1[i], work.x2[i], work.x3[i] = x1i, x2i, x3i
|
||||
work.f1[i], work.f2[i], work.f3[i] = f1i, f2i, f3i
|
||||
work.x1[ni], work.x2[ni], work.x3[ni] = x1ni, x2ni, x3ni,
|
||||
work.f1[ni], work.f2[ni], work.f3[ni] = f1ni, f2ni, f3ni
|
||||
|
||||
def check_termination(work):
|
||||
# Check for all terminal conditions and record statuses.
|
||||
stop = np.zeros_like(work.x1, dtype=bool) # termination condition met
|
||||
|
||||
# Bracket is invalid; stop and don't return minimizer/minimum
|
||||
i = ((work.f2 > work.f1) | (work.f2 > work.f3))
|
||||
work.x2[i], work.f2[i] = np.nan, np.nan
|
||||
stop[i], work.status[i] = True, eim._ESIGNERR
|
||||
|
||||
# Non-finite values; stop and don't return minimizer/minimum
|
||||
finite = np.isfinite(work.x1+work.x2+work.x3+work.f1+work.f2+work.f3)
|
||||
i = ~(finite | stop)
|
||||
work.x2[i], work.f2[i] = np.nan, np.nan
|
||||
stop[i], work.status[i] = True, eim._EVALUEERR
|
||||
|
||||
# [1] Section 3 "Points 1 and 3 are interchanged if necessary to make
|
||||
# the (x2, x3) the larger interval."
|
||||
# Note: I had used np.choose; this is much faster. This would be a good
|
||||
# place to save e.g. `work.x3 - work.x2` for reuse, but I tried and
|
||||
# didn't notice a speed boost, so let's keep it simple.
|
||||
i = abs(work.x3 - work.x2) < abs(work.x2 - work.x1)
|
||||
temp = work.x1[i]
|
||||
work.x1[i] = work.x3[i]
|
||||
work.x3[i] = temp
|
||||
temp = work.f1[i]
|
||||
work.f1[i] = work.f3[i]
|
||||
work.f3[i] = temp
|
||||
|
||||
# [1] Section 3 (bottom of page 212)
|
||||
# "We set a tolerance value xtol..."
|
||||
work.xtol = abs(work.x2) * work.xrtol + work.xatol # [1] (8)
|
||||
# "The convergence based on interval is achieved when..."
|
||||
# Note: Equality allowed in case of `xtol=0`
|
||||
i = abs(work.x3 - work.x2) <= 2 * work.xtol # [1] (9)
|
||||
|
||||
# "We define ftol using..."
|
||||
ftol = abs(work.f2) * work.frtol + work.fatol # [1] (10)
|
||||
# "The convergence based on function values is achieved when..."
|
||||
# Note 1: modify in place to incorporate tolerance on function value.
|
||||
# Note 2: factor of 2 is not in the text; see QBASIC start of DO loop
|
||||
i |= (work.f1 - 2 * work.f2 + work.f3) <= 2*ftol # [1] (11)
|
||||
i &= ~stop
|
||||
stop[i], work.status[i] = True, eim._ECONVERGED
|
||||
|
||||
return stop
|
||||
|
||||
def post_termination_check(work):
|
||||
pass
|
||||
|
||||
def customize_result(res, shape):
|
||||
xl, xr, fl, fr = res['xl'], res['xr'], res['fl'], res['fr']
|
||||
i = res['xl'] < res['xr']
|
||||
res['xl'] = np.choose(i, (xr, xl))
|
||||
res['xr'] = np.choose(i, (xl, xr))
|
||||
res['fl'] = np.choose(i, (fr, fl))
|
||||
res['fr'] = np.choose(i, (fl, fr))
|
||||
return shape
|
||||
|
||||
return eim._loop(work, callback, shape, maxiter, func, args, dtype,
|
||||
pre_func_eval, post_func_eval, check_termination,
|
||||
post_termination_check, customize_result, res_work_pairs,
|
||||
xp=xp)
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,316 @@
|
||||
"""
|
||||
Interface to Constrained Optimization By Linear Approximation
|
||||
|
||||
Functions
|
||||
---------
|
||||
.. autosummary::
|
||||
:toctree: generated/
|
||||
|
||||
fmin_cobyla
|
||||
|
||||
"""
|
||||
|
||||
import functools
|
||||
from threading import RLock
|
||||
|
||||
import numpy as np
|
||||
from scipy.optimize import _cobyla as cobyla
|
||||
from ._optimize import (OptimizeResult, _check_unknown_options,
|
||||
_prepare_scalar_function)
|
||||
try:
|
||||
from itertools import izip
|
||||
except ImportError:
|
||||
izip = zip
|
||||
|
||||
__all__ = ['fmin_cobyla']
|
||||
|
||||
# Workaround as _cobyla.minimize is not threadsafe
|
||||
# due to an unknown f2py bug and can segfault,
|
||||
# see gh-9658.
|
||||
_module_lock = RLock()
|
||||
def synchronized(func):
|
||||
@functools.wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
with _module_lock:
|
||||
return func(*args, **kwargs)
|
||||
return wrapper
|
||||
|
||||
@synchronized
|
||||
def fmin_cobyla(func, x0, cons, args=(), consargs=None, rhobeg=1.0,
|
||||
rhoend=1e-4, maxfun=1000, disp=None, catol=2e-4,
|
||||
*, callback=None):
|
||||
"""
|
||||
Minimize a function using the Constrained Optimization By Linear
|
||||
Approximation (COBYLA) method. This method wraps a FORTRAN
|
||||
implementation of the algorithm.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
Function to minimize. In the form func(x, \\*args).
|
||||
x0 : ndarray
|
||||
Initial guess.
|
||||
cons : sequence
|
||||
Constraint functions; must all be ``>=0`` (a single function
|
||||
if only 1 constraint). Each function takes the parameters `x`
|
||||
as its first argument, and it can return either a single number or
|
||||
an array or list of numbers.
|
||||
args : tuple, optional
|
||||
Extra arguments to pass to function.
|
||||
consargs : tuple, optional
|
||||
Extra arguments to pass to constraint functions (default of None means
|
||||
use same extra arguments as those passed to func).
|
||||
Use ``()`` for no extra arguments.
|
||||
rhobeg : float, optional
|
||||
Reasonable initial changes to the variables.
|
||||
rhoend : float, optional
|
||||
Final accuracy in the optimization (not precisely guaranteed). This
|
||||
is a lower bound on the size of the trust region.
|
||||
disp : {0, 1, 2, 3}, optional
|
||||
Controls the frequency of output; 0 implies no output.
|
||||
maxfun : int, optional
|
||||
Maximum number of function evaluations.
|
||||
catol : float, optional
|
||||
Absolute tolerance for constraint violations.
|
||||
callback : callable, optional
|
||||
Called after each iteration, as ``callback(x)``, where ``x`` is the
|
||||
current parameter vector.
|
||||
|
||||
Returns
|
||||
-------
|
||||
x : ndarray
|
||||
The argument that minimises `f`.
|
||||
|
||||
See also
|
||||
--------
|
||||
minimize: Interface to minimization algorithms for multivariate
|
||||
functions. See the 'COBYLA' `method` in particular.
|
||||
|
||||
Notes
|
||||
-----
|
||||
This algorithm is based on linear approximations to the objective
|
||||
function and each constraint. We briefly describe the algorithm.
|
||||
|
||||
Suppose the function is being minimized over k variables. At the
|
||||
jth iteration the algorithm has k+1 points v_1, ..., v_(k+1),
|
||||
an approximate solution x_j, and a radius RHO_j.
|
||||
(i.e., linear plus a constant) approximations to the objective
|
||||
function and constraint functions such that their function values
|
||||
agree with the linear approximation on the k+1 points v_1,.., v_(k+1).
|
||||
This gives a linear program to solve (where the linear approximations
|
||||
of the constraint functions are constrained to be non-negative).
|
||||
|
||||
However, the linear approximations are likely only good
|
||||
approximations near the current simplex, so the linear program is
|
||||
given the further requirement that the solution, which
|
||||
will become x_(j+1), must be within RHO_j from x_j. RHO_j only
|
||||
decreases, never increases. The initial RHO_j is rhobeg and the
|
||||
final RHO_j is rhoend. In this way COBYLA's iterations behave
|
||||
like a trust region algorithm.
|
||||
|
||||
Additionally, the linear program may be inconsistent, or the
|
||||
approximation may give poor improvement. For details about
|
||||
how these issues are resolved, as well as how the points v_i are
|
||||
updated, refer to the source code or the references below.
|
||||
|
||||
|
||||
References
|
||||
----------
|
||||
Powell M.J.D. (1994), "A direct search optimization method that models
|
||||
the objective and constraint functions by linear interpolation.", in
|
||||
Advances in Optimization and Numerical Analysis, eds. S. Gomez and
|
||||
J-P Hennart, Kluwer Academic (Dordrecht), pp. 51-67
|
||||
|
||||
Powell M.J.D. (1998), "Direct search algorithms for optimization
|
||||
calculations", Acta Numerica 7, 287-336
|
||||
|
||||
Powell M.J.D. (2007), "A view of algorithms for optimization without
|
||||
derivatives", Cambridge University Technical Report DAMTP 2007/NA03
|
||||
|
||||
|
||||
Examples
|
||||
--------
|
||||
Minimize the objective function f(x,y) = x*y subject
|
||||
to the constraints x**2 + y**2 < 1 and y > 0::
|
||||
|
||||
>>> def objective(x):
|
||||
... return x[0]*x[1]
|
||||
...
|
||||
>>> def constr1(x):
|
||||
... return 1 - (x[0]**2 + x[1]**2)
|
||||
...
|
||||
>>> def constr2(x):
|
||||
... return x[1]
|
||||
...
|
||||
>>> from scipy.optimize import fmin_cobyla
|
||||
>>> fmin_cobyla(objective, [0.0, 0.1], [constr1, constr2], rhoend=1e-7)
|
||||
array([-0.70710685, 0.70710671])
|
||||
|
||||
The exact solution is (-sqrt(2)/2, sqrt(2)/2).
|
||||
|
||||
|
||||
|
||||
"""
|
||||
err = "cons must be a sequence of callable functions or a single"\
|
||||
" callable function."
|
||||
try:
|
||||
len(cons)
|
||||
except TypeError as e:
|
||||
if callable(cons):
|
||||
cons = [cons]
|
||||
else:
|
||||
raise TypeError(err) from e
|
||||
else:
|
||||
for thisfunc in cons:
|
||||
if not callable(thisfunc):
|
||||
raise TypeError(err)
|
||||
|
||||
if consargs is None:
|
||||
consargs = args
|
||||
|
||||
# build constraints
|
||||
con = tuple({'type': 'ineq', 'fun': c, 'args': consargs} for c in cons)
|
||||
|
||||
# options
|
||||
opts = {'rhobeg': rhobeg,
|
||||
'tol': rhoend,
|
||||
'disp': disp,
|
||||
'maxiter': maxfun,
|
||||
'catol': catol,
|
||||
'callback': callback}
|
||||
|
||||
sol = _minimize_cobyla(func, x0, args, constraints=con,
|
||||
**opts)
|
||||
if disp and not sol['success']:
|
||||
print(f"COBYLA failed to find a solution: {sol.message}")
|
||||
return sol['x']
|
||||
|
||||
|
||||
@synchronized
|
||||
def _minimize_cobyla(fun, x0, args=(), constraints=(),
|
||||
rhobeg=1.0, tol=1e-4, maxiter=1000,
|
||||
disp=False, catol=2e-4, callback=None, bounds=None,
|
||||
**unknown_options):
|
||||
"""
|
||||
Minimize a scalar function of one or more variables using the
|
||||
Constrained Optimization BY Linear Approximation (COBYLA) algorithm.
|
||||
|
||||
Options
|
||||
-------
|
||||
rhobeg : float
|
||||
Reasonable initial changes to the variables.
|
||||
tol : float
|
||||
Final accuracy in the optimization (not precisely guaranteed).
|
||||
This is a lower bound on the size of the trust region.
|
||||
disp : bool
|
||||
Set to True to print convergence messages. If False,
|
||||
`verbosity` is ignored as set to 0.
|
||||
maxiter : int
|
||||
Maximum number of function evaluations.
|
||||
catol : float
|
||||
Tolerance (absolute) for constraint violations
|
||||
|
||||
"""
|
||||
_check_unknown_options(unknown_options)
|
||||
maxfun = maxiter
|
||||
rhoend = tol
|
||||
iprint = int(bool(disp))
|
||||
|
||||
# check constraints
|
||||
if isinstance(constraints, dict):
|
||||
constraints = (constraints, )
|
||||
|
||||
if bounds:
|
||||
i_lb = np.isfinite(bounds.lb)
|
||||
if np.any(i_lb):
|
||||
def lb_constraint(x, *args, **kwargs):
|
||||
return x[i_lb] - bounds.lb[i_lb]
|
||||
|
||||
constraints.append({'type': 'ineq', 'fun': lb_constraint})
|
||||
|
||||
i_ub = np.isfinite(bounds.ub)
|
||||
if np.any(i_ub):
|
||||
def ub_constraint(x):
|
||||
return bounds.ub[i_ub] - x[i_ub]
|
||||
|
||||
constraints.append({'type': 'ineq', 'fun': ub_constraint})
|
||||
|
||||
for ic, con in enumerate(constraints):
|
||||
# check type
|
||||
try:
|
||||
ctype = con['type'].lower()
|
||||
except KeyError as e:
|
||||
raise KeyError('Constraint %d has no type defined.' % ic) from e
|
||||
except TypeError as e:
|
||||
raise TypeError('Constraints must be defined using a '
|
||||
'dictionary.') from e
|
||||
except AttributeError as e:
|
||||
raise TypeError("Constraint's type must be a string.") from e
|
||||
else:
|
||||
if ctype != 'ineq':
|
||||
raise ValueError("Constraints of type '%s' not handled by "
|
||||
"COBYLA." % con['type'])
|
||||
|
||||
# check function
|
||||
if 'fun' not in con:
|
||||
raise KeyError('Constraint %d has no function defined.' % ic)
|
||||
|
||||
# check extra arguments
|
||||
if 'args' not in con:
|
||||
con['args'] = ()
|
||||
|
||||
# m is the total number of constraint values
|
||||
# it takes into account that some constraints may be vector-valued
|
||||
cons_lengths = []
|
||||
for c in constraints:
|
||||
f = c['fun'](x0, *c['args'])
|
||||
try:
|
||||
cons_length = len(f)
|
||||
except TypeError:
|
||||
cons_length = 1
|
||||
cons_lengths.append(cons_length)
|
||||
m = sum(cons_lengths)
|
||||
|
||||
# create the ScalarFunction, cobyla doesn't require derivative function
|
||||
def _jac(x, *args):
|
||||
return None
|
||||
|
||||
sf = _prepare_scalar_function(fun, x0, args=args, jac=_jac)
|
||||
|
||||
def calcfc(x, con):
|
||||
f = sf.fun(x)
|
||||
i = 0
|
||||
for size, c in izip(cons_lengths, constraints):
|
||||
con[i: i + size] = c['fun'](x, *c['args'])
|
||||
i += size
|
||||
return f
|
||||
|
||||
def wrapped_callback(x):
|
||||
if callback is not None:
|
||||
callback(np.copy(x))
|
||||
|
||||
info = np.zeros(4, np.float64)
|
||||
xopt, info = cobyla.minimize(calcfc, m=m, x=np.copy(x0), rhobeg=rhobeg,
|
||||
rhoend=rhoend, iprint=iprint, maxfun=maxfun,
|
||||
dinfo=info, callback=wrapped_callback)
|
||||
|
||||
if info[3] > catol:
|
||||
# Check constraint violation
|
||||
info[0] = 4
|
||||
|
||||
return OptimizeResult(x=xopt,
|
||||
status=int(info[0]),
|
||||
success=info[0] == 1,
|
||||
message={1: 'Optimization terminated successfully.',
|
||||
2: 'Maximum number of function evaluations '
|
||||
'has been exceeded.',
|
||||
3: 'Rounding errors are becoming damaging '
|
||||
'in COBYLA subroutine.',
|
||||
4: 'Did not converge to a solution '
|
||||
'satisfying the constraints. See '
|
||||
'`maxcv` for magnitude of violation.',
|
||||
5: 'NaN result encountered.'
|
||||
}.get(info[0], 'Unknown exit status.'),
|
||||
nfev=int(info[1]),
|
||||
fun=info[2],
|
||||
maxcv=info[3])
|
||||
@@ -0,0 +1,62 @@
|
||||
import numpy as np
|
||||
|
||||
from ._optimize import _check_unknown_options
|
||||
|
||||
|
||||
def _minimize_cobyqa(fun, x0, args=(), bounds=None, constraints=(),
|
||||
callback=None, disp=False, maxfev=None, maxiter=None,
|
||||
f_target=-np.inf, feasibility_tol=1e-8,
|
||||
initial_tr_radius=1.0, final_tr_radius=1e-6, scale=False,
|
||||
**unknown_options):
|
||||
"""
|
||||
Minimize a scalar function of one or more variables using the
|
||||
Constrained Optimization BY Quadratic Approximations (COBYQA) algorithm [1]_.
|
||||
|
||||
.. versionadded:: 1.14.0
|
||||
|
||||
Options
|
||||
-------
|
||||
disp : bool
|
||||
Set to True to print information about the optimization procedure.
|
||||
maxfev : int
|
||||
Maximum number of function evaluations.
|
||||
maxiter : int
|
||||
Maximum number of iterations.
|
||||
f_target : float
|
||||
Target value for the objective function. The optimization procedure is
|
||||
terminated when the objective function value of a feasible point (see
|
||||
`feasibility_tol` below) is less than or equal to this target.
|
||||
feasibility_tol : float
|
||||
Absolute tolerance for the constraint violation.
|
||||
initial_tr_radius : float
|
||||
Initial trust-region radius. Typically, this value should be in the
|
||||
order of one tenth of the greatest expected change to the variables.
|
||||
final_tr_radius : float
|
||||
Final trust-region radius. It should indicate the accuracy required in
|
||||
the final values of the variables. If provided, this option overrides
|
||||
the value of `tol` in the `minimize` function.
|
||||
scale : bool
|
||||
Set to True to scale the variables according to the bounds. If True and
|
||||
if all the lower and upper bounds are finite, the variables are scaled
|
||||
to be within the range :math:`[-1, 1]`. If any of the lower or upper
|
||||
bounds is infinite, the variables are not scaled.
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] COBYQA
|
||||
https://www.cobyqa.com/stable/
|
||||
"""
|
||||
from .._lib.cobyqa import minimize # import here to avoid circular imports
|
||||
|
||||
_check_unknown_options(unknown_options)
|
||||
options = {
|
||||
'disp': bool(disp),
|
||||
'maxfev': int(maxfev) if maxfev is not None else 500 * len(x0),
|
||||
'maxiter': int(maxiter) if maxiter is not None else 1000 * len(x0),
|
||||
'target': float(f_target),
|
||||
'feasibility_tol': float(feasibility_tol),
|
||||
'radius_init': float(initial_tr_radius),
|
||||
'radius_final': float(final_tr_radius),
|
||||
'scale': bool(scale),
|
||||
}
|
||||
return minimize(fun, x0, args, bounds, constraints, callback, options)
|
||||
@@ -0,0 +1,590 @@
|
||||
"""Constraints definition for minimize."""
|
||||
import numpy as np
|
||||
from ._hessian_update_strategy import BFGS
|
||||
from ._differentiable_functions import (
|
||||
VectorFunction, LinearVectorFunction, IdentityVectorFunction)
|
||||
from ._optimize import OptimizeWarning
|
||||
from warnings import warn, catch_warnings, simplefilter, filterwarnings
|
||||
from scipy.sparse import issparse
|
||||
|
||||
|
||||
def _arr_to_scalar(x):
|
||||
# If x is a numpy array, return x.item(). This will
|
||||
# fail if the array has more than one element.
|
||||
return x.item() if isinstance(x, np.ndarray) else x
|
||||
|
||||
|
||||
class NonlinearConstraint:
|
||||
"""Nonlinear constraint on the variables.
|
||||
|
||||
The constraint has the general inequality form::
|
||||
|
||||
lb <= fun(x) <= ub
|
||||
|
||||
Here the vector of independent variables x is passed as ndarray of shape
|
||||
(n,) and ``fun`` returns a vector with m components.
|
||||
|
||||
It is possible to use equal bounds to represent an equality constraint or
|
||||
infinite bounds to represent a one-sided constraint.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
fun : callable
|
||||
The function defining the constraint.
|
||||
The signature is ``fun(x) -> array_like, shape (m,)``.
|
||||
lb, ub : array_like
|
||||
Lower and upper bounds on the constraint. Each array must have the
|
||||
shape (m,) or be a scalar, in the latter case a bound will be the same
|
||||
for all components of the constraint. Use ``np.inf`` with an
|
||||
appropriate sign to specify a one-sided constraint.
|
||||
Set components of `lb` and `ub` equal to represent an equality
|
||||
constraint. Note that you can mix constraints of different types:
|
||||
interval, one-sided or equality, by setting different components of
|
||||
`lb` and `ub` as necessary.
|
||||
jac : {callable, '2-point', '3-point', 'cs'}, optional
|
||||
Method of computing the Jacobian matrix (an m-by-n matrix,
|
||||
where element (i, j) is the partial derivative of f[i] with
|
||||
respect to x[j]). The keywords {'2-point', '3-point',
|
||||
'cs'} select a finite difference scheme for the numerical estimation.
|
||||
A callable must have the following signature:
|
||||
``jac(x) -> {ndarray, sparse matrix}, shape (m, n)``.
|
||||
Default is '2-point'.
|
||||
hess : {callable, '2-point', '3-point', 'cs', HessianUpdateStrategy, None}, optional
|
||||
Method for computing the Hessian matrix. The keywords
|
||||
{'2-point', '3-point', 'cs'} select a finite difference scheme for
|
||||
numerical estimation. Alternatively, objects implementing
|
||||
`HessianUpdateStrategy` interface can be used to approximate the
|
||||
Hessian. Currently available implementations are:
|
||||
|
||||
- `BFGS` (default option)
|
||||
- `SR1`
|
||||
|
||||
A callable must return the Hessian matrix of ``dot(fun, v)`` and
|
||||
must have the following signature:
|
||||
``hess(x, v) -> {LinearOperator, sparse matrix, array_like}, shape (n, n)``.
|
||||
Here ``v`` is ndarray with shape (m,) containing Lagrange multipliers.
|
||||
keep_feasible : array_like of bool, optional
|
||||
Whether to keep the constraint components feasible throughout
|
||||
iterations. A single value set this property for all components.
|
||||
Default is False. Has no effect for equality constraints.
|
||||
finite_diff_rel_step: None or array_like, optional
|
||||
Relative step size for the finite difference approximation. Default is
|
||||
None, which will select a reasonable value automatically depending
|
||||
on a finite difference scheme.
|
||||
finite_diff_jac_sparsity: {None, array_like, sparse matrix}, optional
|
||||
Defines the sparsity structure of the Jacobian matrix for finite
|
||||
difference estimation, its shape must be (m, n). If the Jacobian has
|
||||
only few non-zero elements in *each* row, providing the sparsity
|
||||
structure will greatly speed up the computations. A zero entry means
|
||||
that a corresponding element in the Jacobian is identically zero.
|
||||
If provided, forces the use of 'lsmr' trust-region solver.
|
||||
If None (default) then dense differencing will be used.
|
||||
|
||||
Notes
|
||||
-----
|
||||
Finite difference schemes {'2-point', '3-point', 'cs'} may be used for
|
||||
approximating either the Jacobian or the Hessian. We, however, do not allow
|
||||
its use for approximating both simultaneously. Hence whenever the Jacobian
|
||||
is estimated via finite-differences, we require the Hessian to be estimated
|
||||
using one of the quasi-Newton strategies.
|
||||
|
||||
The scheme 'cs' is potentially the most accurate, but requires the function
|
||||
to correctly handles complex inputs and be analytically continuable to the
|
||||
complex plane. The scheme '3-point' is more accurate than '2-point' but
|
||||
requires twice as many operations.
|
||||
|
||||
Examples
|
||||
--------
|
||||
Constrain ``x[0] < sin(x[1]) + 1.9``
|
||||
|
||||
>>> from scipy.optimize import NonlinearConstraint
|
||||
>>> import numpy as np
|
||||
>>> con = lambda x: x[0] - np.sin(x[1])
|
||||
>>> nlc = NonlinearConstraint(con, -np.inf, 1.9)
|
||||
|
||||
"""
|
||||
def __init__(self, fun, lb, ub, jac='2-point', hess=BFGS(),
|
||||
keep_feasible=False, finite_diff_rel_step=None,
|
||||
finite_diff_jac_sparsity=None):
|
||||
self.fun = fun
|
||||
self.lb = lb
|
||||
self.ub = ub
|
||||
self.finite_diff_rel_step = finite_diff_rel_step
|
||||
self.finite_diff_jac_sparsity = finite_diff_jac_sparsity
|
||||
self.jac = jac
|
||||
self.hess = hess
|
||||
self.keep_feasible = keep_feasible
|
||||
|
||||
|
||||
class LinearConstraint:
|
||||
"""Linear constraint on the variables.
|
||||
|
||||
The constraint has the general inequality form::
|
||||
|
||||
lb <= A.dot(x) <= ub
|
||||
|
||||
Here the vector of independent variables x is passed as ndarray of shape
|
||||
(n,) and the matrix A has shape (m, n).
|
||||
|
||||
It is possible to use equal bounds to represent an equality constraint or
|
||||
infinite bounds to represent a one-sided constraint.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
A : {array_like, sparse matrix}, shape (m, n)
|
||||
Matrix defining the constraint.
|
||||
lb, ub : dense array_like, optional
|
||||
Lower and upper limits on the constraint. Each array must have the
|
||||
shape (m,) or be a scalar, in the latter case a bound will be the same
|
||||
for all components of the constraint. Use ``np.inf`` with an
|
||||
appropriate sign to specify a one-sided constraint.
|
||||
Set components of `lb` and `ub` equal to represent an equality
|
||||
constraint. Note that you can mix constraints of different types:
|
||||
interval, one-sided or equality, by setting different components of
|
||||
`lb` and `ub` as necessary. Defaults to ``lb = -np.inf``
|
||||
and ``ub = np.inf`` (no limits).
|
||||
keep_feasible : dense array_like of bool, optional
|
||||
Whether to keep the constraint components feasible throughout
|
||||
iterations. A single value set this property for all components.
|
||||
Default is False. Has no effect for equality constraints.
|
||||
"""
|
||||
def _input_validation(self):
|
||||
if self.A.ndim != 2:
|
||||
message = "`A` must have exactly two dimensions."
|
||||
raise ValueError(message)
|
||||
|
||||
try:
|
||||
shape = self.A.shape[0:1]
|
||||
self.lb = np.broadcast_to(self.lb, shape)
|
||||
self.ub = np.broadcast_to(self.ub, shape)
|
||||
self.keep_feasible = np.broadcast_to(self.keep_feasible, shape)
|
||||
except ValueError:
|
||||
message = ("`lb`, `ub`, and `keep_feasible` must be broadcastable "
|
||||
"to shape `A.shape[0:1]`")
|
||||
raise ValueError(message)
|
||||
|
||||
def __init__(self, A, lb=-np.inf, ub=np.inf, keep_feasible=False):
|
||||
if not issparse(A):
|
||||
# In some cases, if the constraint is not valid, this emits a
|
||||
# VisibleDeprecationWarning about ragged nested sequences
|
||||
# before eventually causing an error. `scipy.optimize.milp` would
|
||||
# prefer that this just error out immediately so it can handle it
|
||||
# rather than concerning the user.
|
||||
with catch_warnings():
|
||||
simplefilter("error")
|
||||
self.A = np.atleast_2d(A).astype(np.float64)
|
||||
else:
|
||||
self.A = A
|
||||
if issparse(lb) or issparse(ub):
|
||||
raise ValueError("Constraint limits must be dense arrays.")
|
||||
self.lb = np.atleast_1d(lb).astype(np.float64)
|
||||
self.ub = np.atleast_1d(ub).astype(np.float64)
|
||||
|
||||
if issparse(keep_feasible):
|
||||
raise ValueError("`keep_feasible` must be a dense array.")
|
||||
self.keep_feasible = np.atleast_1d(keep_feasible).astype(bool)
|
||||
self._input_validation()
|
||||
|
||||
def residual(self, x):
|
||||
"""
|
||||
Calculate the residual between the constraint function and the limits
|
||||
|
||||
For a linear constraint of the form::
|
||||
|
||||
lb <= A@x <= ub
|
||||
|
||||
the lower and upper residuals between ``A@x`` and the limits are values
|
||||
``sl`` and ``sb`` such that::
|
||||
|
||||
lb + sl == A@x == ub - sb
|
||||
|
||||
When all elements of ``sl`` and ``sb`` are positive, all elements of
|
||||
the constraint are satisfied; a negative element in ``sl`` or ``sb``
|
||||
indicates that the corresponding element of the constraint is not
|
||||
satisfied.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
x: array_like
|
||||
Vector of independent variables
|
||||
|
||||
Returns
|
||||
-------
|
||||
sl, sb : array-like
|
||||
The lower and upper residuals
|
||||
"""
|
||||
return self.A@x - self.lb, self.ub - self.A@x
|
||||
|
||||
|
||||
class Bounds:
|
||||
"""Bounds constraint on the variables.
|
||||
|
||||
The constraint has the general inequality form::
|
||||
|
||||
lb <= x <= ub
|
||||
|
||||
It is possible to use equal bounds to represent an equality constraint or
|
||||
infinite bounds to represent a one-sided constraint.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
lb, ub : dense array_like, optional
|
||||
Lower and upper bounds on independent variables. `lb`, `ub`, and
|
||||
`keep_feasible` must be the same shape or broadcastable.
|
||||
Set components of `lb` and `ub` equal
|
||||
to fix a variable. Use ``np.inf`` with an appropriate sign to disable
|
||||
bounds on all or some variables. Note that you can mix constraints of
|
||||
different types: interval, one-sided or equality, by setting different
|
||||
components of `lb` and `ub` as necessary. Defaults to ``lb = -np.inf``
|
||||
and ``ub = np.inf`` (no bounds).
|
||||
keep_feasible : dense array_like of bool, optional
|
||||
Whether to keep the constraint components feasible throughout
|
||||
iterations. Must be broadcastable with `lb` and `ub`.
|
||||
Default is False. Has no effect for equality constraints.
|
||||
"""
|
||||
def _input_validation(self):
|
||||
try:
|
||||
res = np.broadcast_arrays(self.lb, self.ub, self.keep_feasible)
|
||||
self.lb, self.ub, self.keep_feasible = res
|
||||
except ValueError:
|
||||
message = "`lb`, `ub`, and `keep_feasible` must be broadcastable."
|
||||
raise ValueError(message)
|
||||
|
||||
def __init__(self, lb=-np.inf, ub=np.inf, keep_feasible=False):
|
||||
if issparse(lb) or issparse(ub):
|
||||
raise ValueError("Lower and upper bounds must be dense arrays.")
|
||||
self.lb = np.atleast_1d(lb)
|
||||
self.ub = np.atleast_1d(ub)
|
||||
|
||||
if issparse(keep_feasible):
|
||||
raise ValueError("`keep_feasible` must be a dense array.")
|
||||
self.keep_feasible = np.atleast_1d(keep_feasible).astype(bool)
|
||||
self._input_validation()
|
||||
|
||||
def __repr__(self):
|
||||
start = f"{type(self).__name__}({self.lb!r}, {self.ub!r}"
|
||||
if np.any(self.keep_feasible):
|
||||
end = f", keep_feasible={self.keep_feasible!r})"
|
||||
else:
|
||||
end = ")"
|
||||
return start + end
|
||||
|
||||
def residual(self, x):
|
||||
"""Calculate the residual (slack) between the input and the bounds
|
||||
|
||||
For a bound constraint of the form::
|
||||
|
||||
lb <= x <= ub
|
||||
|
||||
the lower and upper residuals between `x` and the bounds are values
|
||||
``sl`` and ``sb`` such that::
|
||||
|
||||
lb + sl == x == ub - sb
|
||||
|
||||
When all elements of ``sl`` and ``sb`` are positive, all elements of
|
||||
``x`` lie within the bounds; a negative element in ``sl`` or ``sb``
|
||||
indicates that the corresponding element of ``x`` is out of bounds.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
x: array_like
|
||||
Vector of independent variables
|
||||
|
||||
Returns
|
||||
-------
|
||||
sl, sb : array-like
|
||||
The lower and upper residuals
|
||||
"""
|
||||
return x - self.lb, self.ub - x
|
||||
|
||||
|
||||
class PreparedConstraint:
|
||||
"""Constraint prepared from a user defined constraint.
|
||||
|
||||
On creation it will check whether a constraint definition is valid and
|
||||
the initial point is feasible. If created successfully, it will contain
|
||||
the attributes listed below.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
constraint : {NonlinearConstraint, LinearConstraint`, Bounds}
|
||||
Constraint to check and prepare.
|
||||
x0 : array_like
|
||||
Initial vector of independent variables.
|
||||
sparse_jacobian : bool or None, optional
|
||||
If bool, then the Jacobian of the constraint will be converted
|
||||
to the corresponded format if necessary. If None (default), such
|
||||
conversion is not made.
|
||||
finite_diff_bounds : 2-tuple, optional
|
||||
Lower and upper bounds on the independent variables for the finite
|
||||
difference approximation, if applicable. Defaults to no bounds.
|
||||
|
||||
Attributes
|
||||
----------
|
||||
fun : {VectorFunction, LinearVectorFunction, IdentityVectorFunction}
|
||||
Function defining the constraint wrapped by one of the convenience
|
||||
classes.
|
||||
bounds : 2-tuple
|
||||
Contains lower and upper bounds for the constraints --- lb and ub.
|
||||
These are converted to ndarray and have a size equal to the number of
|
||||
the constraints.
|
||||
keep_feasible : ndarray
|
||||
Array indicating which components must be kept feasible with a size
|
||||
equal to the number of the constraints.
|
||||
"""
|
||||
def __init__(self, constraint, x0, sparse_jacobian=None,
|
||||
finite_diff_bounds=(-np.inf, np.inf)):
|
||||
if isinstance(constraint, NonlinearConstraint):
|
||||
fun = VectorFunction(constraint.fun, x0,
|
||||
constraint.jac, constraint.hess,
|
||||
constraint.finite_diff_rel_step,
|
||||
constraint.finite_diff_jac_sparsity,
|
||||
finite_diff_bounds, sparse_jacobian)
|
||||
elif isinstance(constraint, LinearConstraint):
|
||||
fun = LinearVectorFunction(constraint.A, x0, sparse_jacobian)
|
||||
elif isinstance(constraint, Bounds):
|
||||
fun = IdentityVectorFunction(x0, sparse_jacobian)
|
||||
else:
|
||||
raise ValueError("`constraint` of an unknown type is passed.")
|
||||
|
||||
m = fun.m
|
||||
|
||||
lb = np.asarray(constraint.lb, dtype=float)
|
||||
ub = np.asarray(constraint.ub, dtype=float)
|
||||
keep_feasible = np.asarray(constraint.keep_feasible, dtype=bool)
|
||||
|
||||
lb = np.broadcast_to(lb, m)
|
||||
ub = np.broadcast_to(ub, m)
|
||||
keep_feasible = np.broadcast_to(keep_feasible, m)
|
||||
|
||||
if keep_feasible.shape != (m,):
|
||||
raise ValueError("`keep_feasible` has a wrong shape.")
|
||||
|
||||
mask = keep_feasible & (lb != ub)
|
||||
f0 = fun.f
|
||||
if np.any(f0[mask] < lb[mask]) or np.any(f0[mask] > ub[mask]):
|
||||
raise ValueError("`x0` is infeasible with respect to some "
|
||||
"inequality constraint with `keep_feasible` "
|
||||
"set to True.")
|
||||
|
||||
self.fun = fun
|
||||
self.bounds = (lb, ub)
|
||||
self.keep_feasible = keep_feasible
|
||||
|
||||
def violation(self, x):
|
||||
"""How much the constraint is exceeded by.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
x : array-like
|
||||
Vector of independent variables
|
||||
|
||||
Returns
|
||||
-------
|
||||
excess : array-like
|
||||
How much the constraint is exceeded by, for each of the
|
||||
constraints specified by `PreparedConstraint.fun`.
|
||||
"""
|
||||
with catch_warnings():
|
||||
# Ignore the following warning, it's not important when
|
||||
# figuring out total violation
|
||||
# UserWarning: delta_grad == 0.0. Check if the approximated
|
||||
# function is linear
|
||||
filterwarnings("ignore", "delta_grad", UserWarning)
|
||||
ev = self.fun.fun(np.asarray(x))
|
||||
|
||||
excess_lb = np.maximum(self.bounds[0] - ev, 0)
|
||||
excess_ub = np.maximum(ev - self.bounds[1], 0)
|
||||
|
||||
return excess_lb + excess_ub
|
||||
|
||||
|
||||
def new_bounds_to_old(lb, ub, n):
|
||||
"""Convert the new bounds representation to the old one.
|
||||
|
||||
The new representation is a tuple (lb, ub) and the old one is a list
|
||||
containing n tuples, ith containing lower and upper bound on a ith
|
||||
variable.
|
||||
If any of the entries in lb/ub are -np.inf/np.inf they are replaced by
|
||||
None.
|
||||
"""
|
||||
lb = np.broadcast_to(lb, n)
|
||||
ub = np.broadcast_to(ub, n)
|
||||
|
||||
lb = [float(x) if x > -np.inf else None for x in lb]
|
||||
ub = [float(x) if x < np.inf else None for x in ub]
|
||||
|
||||
return list(zip(lb, ub))
|
||||
|
||||
|
||||
def old_bound_to_new(bounds):
|
||||
"""Convert the old bounds representation to the new one.
|
||||
|
||||
The new representation is a tuple (lb, ub) and the old one is a list
|
||||
containing n tuples, ith containing lower and upper bound on a ith
|
||||
variable.
|
||||
If any of the entries in lb/ub are None they are replaced by
|
||||
-np.inf/np.inf.
|
||||
"""
|
||||
lb, ub = zip(*bounds)
|
||||
|
||||
# Convert occurrences of None to -inf or inf, and replace occurrences of
|
||||
# any numpy array x with x.item(). Then wrap the results in numpy arrays.
|
||||
lb = np.array([float(_arr_to_scalar(x)) if x is not None else -np.inf
|
||||
for x in lb])
|
||||
ub = np.array([float(_arr_to_scalar(x)) if x is not None else np.inf
|
||||
for x in ub])
|
||||
|
||||
return lb, ub
|
||||
|
||||
|
||||
def strict_bounds(lb, ub, keep_feasible, n_vars):
|
||||
"""Remove bounds which are not asked to be kept feasible."""
|
||||
strict_lb = np.resize(lb, n_vars).astype(float)
|
||||
strict_ub = np.resize(ub, n_vars).astype(float)
|
||||
keep_feasible = np.resize(keep_feasible, n_vars)
|
||||
strict_lb[~keep_feasible] = -np.inf
|
||||
strict_ub[~keep_feasible] = np.inf
|
||||
return strict_lb, strict_ub
|
||||
|
||||
|
||||
def new_constraint_to_old(con, x0):
|
||||
"""
|
||||
Converts new-style constraint objects to old-style constraint dictionaries.
|
||||
"""
|
||||
if isinstance(con, NonlinearConstraint):
|
||||
if (con.finite_diff_jac_sparsity is not None or
|
||||
con.finite_diff_rel_step is not None or
|
||||
not isinstance(con.hess, BFGS) or # misses user specified BFGS
|
||||
con.keep_feasible):
|
||||
warn("Constraint options `finite_diff_jac_sparsity`, "
|
||||
"`finite_diff_rel_step`, `keep_feasible`, and `hess`"
|
||||
"are ignored by this method.",
|
||||
OptimizeWarning, stacklevel=3)
|
||||
|
||||
fun = con.fun
|
||||
if callable(con.jac):
|
||||
jac = con.jac
|
||||
else:
|
||||
jac = None
|
||||
|
||||
else: # LinearConstraint
|
||||
if np.any(con.keep_feasible):
|
||||
warn("Constraint option `keep_feasible` is ignored by this method.",
|
||||
OptimizeWarning, stacklevel=3)
|
||||
|
||||
A = con.A
|
||||
if issparse(A):
|
||||
A = A.toarray()
|
||||
def fun(x):
|
||||
return np.dot(A, x)
|
||||
def jac(x):
|
||||
return A
|
||||
|
||||
# FIXME: when bugs in VectorFunction/LinearVectorFunction are worked out,
|
||||
# use pcon.fun.fun and pcon.fun.jac. Until then, get fun/jac above.
|
||||
pcon = PreparedConstraint(con, x0)
|
||||
lb, ub = pcon.bounds
|
||||
|
||||
i_eq = lb == ub
|
||||
i_bound_below = np.logical_xor(lb != -np.inf, i_eq)
|
||||
i_bound_above = np.logical_xor(ub != np.inf, i_eq)
|
||||
i_unbounded = np.logical_and(lb == -np.inf, ub == np.inf)
|
||||
|
||||
if np.any(i_unbounded):
|
||||
warn("At least one constraint is unbounded above and below. Such "
|
||||
"constraints are ignored.",
|
||||
OptimizeWarning, stacklevel=3)
|
||||
|
||||
ceq = []
|
||||
if np.any(i_eq):
|
||||
def f_eq(x):
|
||||
y = np.array(fun(x)).flatten()
|
||||
return y[i_eq] - lb[i_eq]
|
||||
ceq = [{"type": "eq", "fun": f_eq}]
|
||||
|
||||
if jac is not None:
|
||||
def j_eq(x):
|
||||
dy = jac(x)
|
||||
if issparse(dy):
|
||||
dy = dy.toarray()
|
||||
dy = np.atleast_2d(dy)
|
||||
return dy[i_eq, :]
|
||||
ceq[0]["jac"] = j_eq
|
||||
|
||||
cineq = []
|
||||
n_bound_below = np.sum(i_bound_below)
|
||||
n_bound_above = np.sum(i_bound_above)
|
||||
if n_bound_below + n_bound_above:
|
||||
def f_ineq(x):
|
||||
y = np.zeros(n_bound_below + n_bound_above)
|
||||
y_all = np.array(fun(x)).flatten()
|
||||
y[:n_bound_below] = y_all[i_bound_below] - lb[i_bound_below]
|
||||
y[n_bound_below:] = -(y_all[i_bound_above] - ub[i_bound_above])
|
||||
return y
|
||||
cineq = [{"type": "ineq", "fun": f_ineq}]
|
||||
|
||||
if jac is not None:
|
||||
def j_ineq(x):
|
||||
dy = np.zeros((n_bound_below + n_bound_above, len(x0)))
|
||||
dy_all = jac(x)
|
||||
if issparse(dy_all):
|
||||
dy_all = dy_all.toarray()
|
||||
dy_all = np.atleast_2d(dy_all)
|
||||
dy[:n_bound_below, :] = dy_all[i_bound_below]
|
||||
dy[n_bound_below:, :] = -dy_all[i_bound_above]
|
||||
return dy
|
||||
cineq[0]["jac"] = j_ineq
|
||||
|
||||
old_constraints = ceq + cineq
|
||||
|
||||
if len(old_constraints) > 1:
|
||||
warn("Equality and inequality constraints are specified in the same "
|
||||
"element of the constraint list. For efficient use with this "
|
||||
"method, equality and inequality constraints should be specified "
|
||||
"in separate elements of the constraint list. ",
|
||||
OptimizeWarning, stacklevel=3)
|
||||
return old_constraints
|
||||
|
||||
|
||||
def old_constraint_to_new(ic, con):
|
||||
"""
|
||||
Converts old-style constraint dictionaries to new-style constraint objects.
|
||||
"""
|
||||
# check type
|
||||
try:
|
||||
ctype = con['type'].lower()
|
||||
except KeyError as e:
|
||||
raise KeyError('Constraint %d has no type defined.' % ic) from e
|
||||
except TypeError as e:
|
||||
raise TypeError(
|
||||
'Constraints must be a sequence of dictionaries.'
|
||||
) from e
|
||||
except AttributeError as e:
|
||||
raise TypeError("Constraint's type must be a string.") from e
|
||||
else:
|
||||
if ctype not in ['eq', 'ineq']:
|
||||
raise ValueError("Unknown constraint type '%s'." % con['type'])
|
||||
if 'fun' not in con:
|
||||
raise ValueError('Constraint %d has no function defined.' % ic)
|
||||
|
||||
lb = 0
|
||||
if ctype == 'eq':
|
||||
ub = 0
|
||||
else:
|
||||
ub = np.inf
|
||||
|
||||
jac = '2-point'
|
||||
if 'args' in con:
|
||||
args = con['args']
|
||||
def fun(x):
|
||||
return con["fun"](x, *args)
|
||||
if 'jac' in con:
|
||||
def jac(x):
|
||||
return con["jac"](x, *args)
|
||||
else:
|
||||
fun = con['fun']
|
||||
if 'jac' in con:
|
||||
jac = con['jac']
|
||||
|
||||
return NonlinearConstraint(fun, lb, ub, jac)
|
||||
@@ -0,0 +1,728 @@
|
||||
import numpy as np
|
||||
|
||||
"""
|
||||
# 2023 - ported from minpack2.dcsrch, dcstep (Fortran) to Python
|
||||
c MINPACK-1 Project. June 1983.
|
||||
c Argonne National Laboratory.
|
||||
c Jorge J. More' and David J. Thuente.
|
||||
c
|
||||
c MINPACK-2 Project. November 1993.
|
||||
c Argonne National Laboratory and University of Minnesota.
|
||||
c Brett M. Averick, Richard G. Carter, and Jorge J. More'.
|
||||
"""
|
||||
|
||||
# NOTE this file was linted by black on first commit, and can be kept that way.
|
||||
|
||||
|
||||
class DCSRCH:
|
||||
"""
|
||||
Parameters
|
||||
----------
|
||||
phi : callable phi(alpha)
|
||||
Function at point `alpha`
|
||||
derphi : callable phi'(alpha)
|
||||
Objective function derivative. Returns a scalar.
|
||||
ftol : float
|
||||
A nonnegative tolerance for the sufficient decrease condition.
|
||||
gtol : float
|
||||
A nonnegative tolerance for the curvature condition.
|
||||
xtol : float
|
||||
A nonnegative relative tolerance for an acceptable step. The
|
||||
subroutine exits with a warning if the relative difference between
|
||||
sty and stx is less than xtol.
|
||||
stpmin : float
|
||||
A nonnegative lower bound for the step.
|
||||
stpmax :
|
||||
A nonnegative upper bound for the step.
|
||||
|
||||
Notes
|
||||
-----
|
||||
|
||||
This subroutine finds a step that satisfies a sufficient
|
||||
decrease condition and a curvature condition.
|
||||
|
||||
Each call of the subroutine updates an interval with
|
||||
endpoints stx and sty. The interval is initially chosen
|
||||
so that it contains a minimizer of the modified function
|
||||
|
||||
psi(stp) = f(stp) - f(0) - ftol*stp*f'(0).
|
||||
|
||||
If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the
|
||||
interval is chosen so that it contains a minimizer of f.
|
||||
|
||||
The algorithm is designed to find a step that satisfies
|
||||
the sufficient decrease condition
|
||||
|
||||
f(stp) <= f(0) + ftol*stp*f'(0),
|
||||
|
||||
and the curvature condition
|
||||
|
||||
abs(f'(stp)) <= gtol*abs(f'(0)).
|
||||
|
||||
If ftol is less than gtol and if, for example, the function
|
||||
is bounded below, then there is always a step which satisfies
|
||||
both conditions.
|
||||
|
||||
If no step can be found that satisfies both conditions, then
|
||||
the algorithm stops with a warning. In this case stp only
|
||||
satisfies the sufficient decrease condition.
|
||||
|
||||
A typical invocation of dcsrch has the following outline:
|
||||
|
||||
Evaluate the function at stp = 0.0d0; store in f.
|
||||
Evaluate the gradient at stp = 0.0d0; store in g.
|
||||
Choose a starting step stp.
|
||||
|
||||
task = 'START'
|
||||
10 continue
|
||||
call dcsrch(stp,f,g,ftol,gtol,xtol,task,stpmin,stpmax,
|
||||
isave,dsave)
|
||||
if (task .eq. 'FG') then
|
||||
Evaluate the function and the gradient at stp
|
||||
go to 10
|
||||
end if
|
||||
|
||||
NOTE: The user must not alter work arrays between calls.
|
||||
|
||||
The subroutine statement is
|
||||
|
||||
subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax,
|
||||
task,isave,dsave)
|
||||
where
|
||||
|
||||
stp is a double precision variable.
|
||||
On entry stp is the current estimate of a satisfactory
|
||||
step. On initial entry, a positive initial estimate
|
||||
must be provided.
|
||||
On exit stp is the current estimate of a satisfactory step
|
||||
if task = 'FG'. If task = 'CONV' then stp satisfies
|
||||
the sufficient decrease and curvature condition.
|
||||
|
||||
f is a double precision variable.
|
||||
On initial entry f is the value of the function at 0.
|
||||
On subsequent entries f is the value of the
|
||||
function at stp.
|
||||
On exit f is the value of the function at stp.
|
||||
|
||||
g is a double precision variable.
|
||||
On initial entry g is the derivative of the function at 0.
|
||||
On subsequent entries g is the derivative of the
|
||||
function at stp.
|
||||
On exit g is the derivative of the function at stp.
|
||||
|
||||
ftol is a double precision variable.
|
||||
On entry ftol specifies a nonnegative tolerance for the
|
||||
sufficient decrease condition.
|
||||
On exit ftol is unchanged.
|
||||
|
||||
gtol is a double precision variable.
|
||||
On entry gtol specifies a nonnegative tolerance for the
|
||||
curvature condition.
|
||||
On exit gtol is unchanged.
|
||||
|
||||
xtol is a double precision variable.
|
||||
On entry xtol specifies a nonnegative relative tolerance
|
||||
for an acceptable step. The subroutine exits with a
|
||||
warning if the relative difference between sty and stx
|
||||
is less than xtol.
|
||||
|
||||
On exit xtol is unchanged.
|
||||
|
||||
task is a character variable of length at least 60.
|
||||
On initial entry task must be set to 'START'.
|
||||
On exit task indicates the required action:
|
||||
|
||||
If task(1:2) = 'FG' then evaluate the function and
|
||||
derivative at stp and call dcsrch again.
|
||||
|
||||
If task(1:4) = 'CONV' then the search is successful.
|
||||
|
||||
If task(1:4) = 'WARN' then the subroutine is not able
|
||||
to satisfy the convergence conditions. The exit value of
|
||||
stp contains the best point found during the search.
|
||||
|
||||
If task(1:5) = 'ERROR' then there is an error in the
|
||||
input arguments.
|
||||
|
||||
On exit with convergence, a warning or an error, the
|
||||
variable task contains additional information.
|
||||
|
||||
stpmin is a double precision variable.
|
||||
On entry stpmin is a nonnegative lower bound for the step.
|
||||
On exit stpmin is unchanged.
|
||||
|
||||
stpmax is a double precision variable.
|
||||
On entry stpmax is a nonnegative upper bound for the step.
|
||||
On exit stpmax is unchanged.
|
||||
|
||||
isave is an integer work array of dimension 2.
|
||||
|
||||
dsave is a double precision work array of dimension 13.
|
||||
|
||||
Subprograms called
|
||||
|
||||
MINPACK-2 ... dcstep
|
||||
MINPACK-1 Project. June 1983.
|
||||
Argonne National Laboratory.
|
||||
Jorge J. More' and David J. Thuente.
|
||||
|
||||
MINPACK-2 Project. November 1993.
|
||||
Argonne National Laboratory and University of Minnesota.
|
||||
Brett M. Averick, Richard G. Carter, and Jorge J. More'.
|
||||
"""
|
||||
|
||||
def __init__(self, phi, derphi, ftol, gtol, xtol, stpmin, stpmax):
|
||||
self.stage = None
|
||||
self.ginit = None
|
||||
self.gtest = None
|
||||
self.gx = None
|
||||
self.gy = None
|
||||
self.finit = None
|
||||
self.fx = None
|
||||
self.fy = None
|
||||
self.stx = None
|
||||
self.sty = None
|
||||
self.stmin = None
|
||||
self.stmax = None
|
||||
self.width = None
|
||||
self.width1 = None
|
||||
|
||||
# leave all assessment of tolerances/limits to the first call of
|
||||
# this object
|
||||
self.ftol = ftol
|
||||
self.gtol = gtol
|
||||
self.xtol = xtol
|
||||
self.stpmin = stpmin
|
||||
self.stpmax = stpmax
|
||||
|
||||
self.phi = phi
|
||||
self.derphi = derphi
|
||||
|
||||
def __call__(self, alpha1, phi0=None, derphi0=None, maxiter=100):
|
||||
"""
|
||||
Parameters
|
||||
----------
|
||||
alpha1 : float
|
||||
alpha1 is the current estimate of a satisfactory
|
||||
step. A positive initial estimate must be provided.
|
||||
phi0 : float
|
||||
the value of `phi` at 0 (if known).
|
||||
derphi0 : float
|
||||
the derivative of `derphi` at 0 (if known).
|
||||
maxiter : int
|
||||
|
||||
Returns
|
||||
-------
|
||||
alpha : float
|
||||
Step size, or None if no suitable step was found.
|
||||
phi : float
|
||||
Value of `phi` at the new point `alpha`.
|
||||
phi0 : float
|
||||
Value of `phi` at `alpha=0`.
|
||||
task : bytes
|
||||
On exit task indicates status information.
|
||||
|
||||
If task[:4] == b'CONV' then the search is successful.
|
||||
|
||||
If task[:4] == b'WARN' then the subroutine is not able
|
||||
to satisfy the convergence conditions. The exit value of
|
||||
stp contains the best point found during the search.
|
||||
|
||||
If task[:5] == b'ERROR' then there is an error in the
|
||||
input arguments.
|
||||
"""
|
||||
if phi0 is None:
|
||||
phi0 = self.phi(0.0)
|
||||
if derphi0 is None:
|
||||
derphi0 = self.derphi(0.0)
|
||||
|
||||
phi1 = phi0
|
||||
derphi1 = derphi0
|
||||
|
||||
task = b"START"
|
||||
for i in range(maxiter):
|
||||
stp, phi1, derphi1, task = self._iterate(
|
||||
alpha1, phi1, derphi1, task
|
||||
)
|
||||
|
||||
if not np.isfinite(stp):
|
||||
task = b"WARN"
|
||||
stp = None
|
||||
break
|
||||
|
||||
if task[:2] == b"FG":
|
||||
alpha1 = stp
|
||||
phi1 = self.phi(stp)
|
||||
derphi1 = self.derphi(stp)
|
||||
else:
|
||||
break
|
||||
else:
|
||||
# maxiter reached, the line search did not converge
|
||||
stp = None
|
||||
task = b"WARNING: dcsrch did not converge within max iterations"
|
||||
|
||||
if task[:5] == b"ERROR" or task[:4] == b"WARN":
|
||||
stp = None # failed
|
||||
|
||||
return stp, phi1, phi0, task
|
||||
|
||||
def _iterate(self, stp, f, g, task):
|
||||
"""
|
||||
Parameters
|
||||
----------
|
||||
stp : float
|
||||
The current estimate of a satisfactory step. On initial entry, a
|
||||
positive initial estimate must be provided.
|
||||
f : float
|
||||
On first call f is the value of the function at 0. On subsequent
|
||||
entries f should be the value of the function at stp.
|
||||
g : float
|
||||
On initial entry g is the derivative of the function at 0. On
|
||||
subsequent entries g is the derivative of the function at stp.
|
||||
task : bytes
|
||||
On initial entry task must be set to 'START'.
|
||||
|
||||
On exit with convergence, a warning or an error, the
|
||||
variable task contains additional information.
|
||||
|
||||
|
||||
Returns
|
||||
-------
|
||||
stp, f, g, task: tuple
|
||||
|
||||
stp : float
|
||||
the current estimate of a satisfactory step if task = 'FG'. If
|
||||
task = 'CONV' then stp satisfies the sufficient decrease and
|
||||
curvature condition.
|
||||
f : float
|
||||
the value of the function at stp.
|
||||
g : float
|
||||
the derivative of the function at stp.
|
||||
task : bytes
|
||||
On exit task indicates the required action:
|
||||
|
||||
If task(1:2) == b'FG' then evaluate the function and
|
||||
derivative at stp and call dcsrch again.
|
||||
|
||||
If task(1:4) == b'CONV' then the search is successful.
|
||||
|
||||
If task(1:4) == b'WARN' then the subroutine is not able
|
||||
to satisfy the convergence conditions. The exit value of
|
||||
stp contains the best point found during the search.
|
||||
|
||||
If task(1:5) == b'ERROR' then there is an error in the
|
||||
input arguments.
|
||||
"""
|
||||
p5 = 0.5
|
||||
p66 = 0.66
|
||||
xtrapl = 1.1
|
||||
xtrapu = 4.0
|
||||
|
||||
if task[:5] == b"START":
|
||||
if stp < self.stpmin:
|
||||
task = b"ERROR: STP .LT. STPMIN"
|
||||
if stp > self.stpmax:
|
||||
task = b"ERROR: STP .GT. STPMAX"
|
||||
if g >= 0:
|
||||
task = b"ERROR: INITIAL G .GE. ZERO"
|
||||
if self.ftol < 0:
|
||||
task = b"ERROR: FTOL .LT. ZERO"
|
||||
if self.gtol < 0:
|
||||
task = b"ERROR: GTOL .LT. ZERO"
|
||||
if self.xtol < 0:
|
||||
task = b"ERROR: XTOL .LT. ZERO"
|
||||
if self.stpmin < 0:
|
||||
task = b"ERROR: STPMIN .LT. ZERO"
|
||||
if self.stpmax < self.stpmin:
|
||||
task = b"ERROR: STPMAX .LT. STPMIN"
|
||||
|
||||
if task[:5] == b"ERROR":
|
||||
return stp, f, g, task
|
||||
|
||||
# Initialize local variables.
|
||||
|
||||
self.brackt = False
|
||||
self.stage = 1
|
||||
self.finit = f
|
||||
self.ginit = g
|
||||
self.gtest = self.ftol * self.ginit
|
||||
self.width = self.stpmax - self.stpmin
|
||||
self.width1 = self.width / p5
|
||||
|
||||
# The variables stx, fx, gx contain the values of the step,
|
||||
# function, and derivative at the best step.
|
||||
# The variables sty, fy, gy contain the value of the step,
|
||||
# function, and derivative at sty.
|
||||
# The variables stp, f, g contain the values of the step,
|
||||
# function, and derivative at stp.
|
||||
|
||||
self.stx = 0.0
|
||||
self.fx = self.finit
|
||||
self.gx = self.ginit
|
||||
self.sty = 0.0
|
||||
self.fy = self.finit
|
||||
self.gy = self.ginit
|
||||
self.stmin = 0
|
||||
self.stmax = stp + xtrapu * stp
|
||||
task = b"FG"
|
||||
return stp, f, g, task
|
||||
|
||||
# in the original Fortran this was a location to restore variables
|
||||
# we don't need to do that because they're attributes.
|
||||
|
||||
# If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the
|
||||
# algorithm enters the second stage.
|
||||
ftest = self.finit + stp * self.gtest
|
||||
|
||||
if self.stage == 1 and f <= ftest and g >= 0:
|
||||
self.stage = 2
|
||||
|
||||
# test for warnings
|
||||
if self.brackt and (stp <= self.stmin or stp >= self.stmax):
|
||||
task = b"WARNING: ROUNDING ERRORS PREVENT PROGRESS"
|
||||
if self.brackt and self.stmax - self.stmin <= self.xtol * self.stmax:
|
||||
task = b"WARNING: XTOL TEST SATISFIED"
|
||||
if stp == self.stpmax and f <= ftest and g <= self.gtest:
|
||||
task = b"WARNING: STP = STPMAX"
|
||||
if stp == self.stpmin and (f > ftest or g >= self.gtest):
|
||||
task = b"WARNING: STP = STPMIN"
|
||||
|
||||
# test for convergence
|
||||
if f <= ftest and abs(g) <= self.gtol * -self.ginit:
|
||||
task = b"CONVERGENCE"
|
||||
|
||||
# test for termination
|
||||
if task[:4] == b"WARN" or task[:4] == b"CONV":
|
||||
return stp, f, g, task
|
||||
|
||||
# A modified function is used to predict the step during the
|
||||
# first stage if a lower function value has been obtained but
|
||||
# the decrease is not sufficient.
|
||||
if self.stage == 1 and f <= self.fx and f > ftest:
|
||||
# Define the modified function and derivative values.
|
||||
fm = f - stp * self.gtest
|
||||
fxm = self.fx - self.stx * self.gtest
|
||||
fym = self.fy - self.sty * self.gtest
|
||||
gm = g - self.gtest
|
||||
gxm = self.gx - self.gtest
|
||||
gym = self.gy - self.gtest
|
||||
|
||||
# Call dcstep to update stx, sty, and to compute the new step.
|
||||
# dcstep can have several operations which can produce NaN
|
||||
# e.g. inf/inf. Filter these out.
|
||||
with np.errstate(invalid="ignore", over="ignore"):
|
||||
tup = dcstep(
|
||||
self.stx,
|
||||
fxm,
|
||||
gxm,
|
||||
self.sty,
|
||||
fym,
|
||||
gym,
|
||||
stp,
|
||||
fm,
|
||||
gm,
|
||||
self.brackt,
|
||||
self.stmin,
|
||||
self.stmax,
|
||||
)
|
||||
self.stx, fxm, gxm, self.sty, fym, gym, stp, self.brackt = tup
|
||||
|
||||
# Reset the function and derivative values for f
|
||||
self.fx = fxm + self.stx * self.gtest
|
||||
self.fy = fym + self.sty * self.gtest
|
||||
self.gx = gxm + self.gtest
|
||||
self.gy = gym + self.gtest
|
||||
|
||||
else:
|
||||
# Call dcstep to update stx, sty, and to compute the new step.
|
||||
# dcstep can have several operations which can produce NaN
|
||||
# e.g. inf/inf. Filter these out.
|
||||
|
||||
with np.errstate(invalid="ignore", over="ignore"):
|
||||
tup = dcstep(
|
||||
self.stx,
|
||||
self.fx,
|
||||
self.gx,
|
||||
self.sty,
|
||||
self.fy,
|
||||
self.gy,
|
||||
stp,
|
||||
f,
|
||||
g,
|
||||
self.brackt,
|
||||
self.stmin,
|
||||
self.stmax,
|
||||
)
|
||||
(
|
||||
self.stx,
|
||||
self.fx,
|
||||
self.gx,
|
||||
self.sty,
|
||||
self.fy,
|
||||
self.gy,
|
||||
stp,
|
||||
self.brackt,
|
||||
) = tup
|
||||
|
||||
# Decide if a bisection step is needed
|
||||
if self.brackt:
|
||||
if abs(self.sty - self.stx) >= p66 * self.width1:
|
||||
stp = self.stx + p5 * (self.sty - self.stx)
|
||||
self.width1 = self.width
|
||||
self.width = abs(self.sty - self.stx)
|
||||
|
||||
# Set the minimum and maximum steps allowed for stp.
|
||||
if self.brackt:
|
||||
self.stmin = min(self.stx, self.sty)
|
||||
self.stmax = max(self.stx, self.sty)
|
||||
else:
|
||||
self.stmin = stp + xtrapl * (stp - self.stx)
|
||||
self.stmax = stp + xtrapu * (stp - self.stx)
|
||||
|
||||
# Force the step to be within the bounds stpmax and stpmin.
|
||||
stp = np.clip(stp, self.stpmin, self.stpmax)
|
||||
|
||||
# If further progress is not possible, let stp be the best
|
||||
# point obtained during the search.
|
||||
if (
|
||||
self.brackt
|
||||
and (stp <= self.stmin or stp >= self.stmax)
|
||||
or (
|
||||
self.brackt
|
||||
and self.stmax - self.stmin <= self.xtol * self.stmax
|
||||
)
|
||||
):
|
||||
stp = self.stx
|
||||
|
||||
# Obtain another function and derivative
|
||||
task = b"FG"
|
||||
return stp, f, g, task
|
||||
|
||||
|
||||
def dcstep(stx, fx, dx, sty, fy, dy, stp, fp, dp, brackt, stpmin, stpmax):
|
||||
"""
|
||||
Subroutine dcstep
|
||||
|
||||
This subroutine computes a safeguarded step for a search
|
||||
procedure and updates an interval that contains a step that
|
||||
satisfies a sufficient decrease and a curvature condition.
|
||||
|
||||
The parameter stx contains the step with the least function
|
||||
value. If brackt is set to .true. then a minimizer has
|
||||
been bracketed in an interval with endpoints stx and sty.
|
||||
The parameter stp contains the current step.
|
||||
The subroutine assumes that if brackt is set to .true. then
|
||||
|
||||
min(stx,sty) < stp < max(stx,sty),
|
||||
|
||||
and that the derivative at stx is negative in the direction
|
||||
of the step.
|
||||
|
||||
The subroutine statement is
|
||||
|
||||
subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt,
|
||||
stpmin,stpmax)
|
||||
|
||||
where
|
||||
|
||||
stx is a double precision variable.
|
||||
On entry stx is the best step obtained so far and is an
|
||||
endpoint of the interval that contains the minimizer.
|
||||
On exit stx is the updated best step.
|
||||
|
||||
fx is a double precision variable.
|
||||
On entry fx is the function at stx.
|
||||
On exit fx is the function at stx.
|
||||
|
||||
dx is a double precision variable.
|
||||
On entry dx is the derivative of the function at
|
||||
stx. The derivative must be negative in the direction of
|
||||
the step, that is, dx and stp - stx must have opposite
|
||||
signs.
|
||||
On exit dx is the derivative of the function at stx.
|
||||
|
||||
sty is a double precision variable.
|
||||
On entry sty is the second endpoint of the interval that
|
||||
contains the minimizer.
|
||||
On exit sty is the updated endpoint of the interval that
|
||||
contains the minimizer.
|
||||
|
||||
fy is a double precision variable.
|
||||
On entry fy is the function at sty.
|
||||
On exit fy is the function at sty.
|
||||
|
||||
dy is a double precision variable.
|
||||
On entry dy is the derivative of the function at sty.
|
||||
On exit dy is the derivative of the function at the exit sty.
|
||||
|
||||
stp is a double precision variable.
|
||||
On entry stp is the current step. If brackt is set to .true.
|
||||
then on input stp must be between stx and sty.
|
||||
On exit stp is a new trial step.
|
||||
|
||||
fp is a double precision variable.
|
||||
On entry fp is the function at stp
|
||||
On exit fp is unchanged.
|
||||
|
||||
dp is a double precision variable.
|
||||
On entry dp is the derivative of the function at stp.
|
||||
On exit dp is unchanged.
|
||||
|
||||
brackt is an logical variable.
|
||||
On entry brackt specifies if a minimizer has been bracketed.
|
||||
Initially brackt must be set to .false.
|
||||
On exit brackt specifies if a minimizer has been bracketed.
|
||||
When a minimizer is bracketed brackt is set to .true.
|
||||
|
||||
stpmin is a double precision variable.
|
||||
On entry stpmin is a lower bound for the step.
|
||||
On exit stpmin is unchanged.
|
||||
|
||||
stpmax is a double precision variable.
|
||||
On entry stpmax is an upper bound for the step.
|
||||
On exit stpmax is unchanged.
|
||||
|
||||
MINPACK-1 Project. June 1983
|
||||
Argonne National Laboratory.
|
||||
Jorge J. More' and David J. Thuente.
|
||||
|
||||
MINPACK-2 Project. November 1993.
|
||||
Argonne National Laboratory and University of Minnesota.
|
||||
Brett M. Averick and Jorge J. More'.
|
||||
|
||||
"""
|
||||
sgn_dp = np.sign(dp)
|
||||
sgn_dx = np.sign(dx)
|
||||
|
||||
# sgnd = dp * (dx / abs(dx))
|
||||
sgnd = sgn_dp * sgn_dx
|
||||
|
||||
# First case: A higher function value. The minimum is bracketed.
|
||||
# If the cubic step is closer to stx than the quadratic step, the
|
||||
# cubic step is taken, otherwise the average of the cubic and
|
||||
# quadratic steps is taken.
|
||||
if fp > fx:
|
||||
theta = 3.0 * (fx - fp) / (stp - stx) + dx + dp
|
||||
s = max(abs(theta), abs(dx), abs(dp))
|
||||
gamma = s * np.sqrt((theta / s) ** 2 - (dx / s) * (dp / s))
|
||||
if stp < stx:
|
||||
gamma *= -1
|
||||
p = (gamma - dx) + theta
|
||||
q = ((gamma - dx) + gamma) + dp
|
||||
r = p / q
|
||||
stpc = stx + r * (stp - stx)
|
||||
stpq = stx + ((dx / ((fx - fp) / (stp - stx) + dx)) / 2.0) * (stp - stx)
|
||||
if abs(stpc - stx) <= abs(stpq - stx):
|
||||
stpf = stpc
|
||||
else:
|
||||
stpf = stpc + (stpq - stpc) / 2.0
|
||||
brackt = True
|
||||
elif sgnd < 0.0:
|
||||
# Second case: A lower function value and derivatives of opposite
|
||||
# sign. The minimum is bracketed. If the cubic step is farther from
|
||||
# stp than the secant step, the cubic step is taken, otherwise the
|
||||
# secant step is taken.
|
||||
theta = 3 * (fx - fp) / (stp - stx) + dx + dp
|
||||
s = max(abs(theta), abs(dx), abs(dp))
|
||||
gamma = s * np.sqrt((theta / s) ** 2 - (dx / s) * (dp / s))
|
||||
if stp > stx:
|
||||
gamma *= -1
|
||||
p = (gamma - dp) + theta
|
||||
q = ((gamma - dp) + gamma) + dx
|
||||
r = p / q
|
||||
stpc = stp + r * (stx - stp)
|
||||
stpq = stp + (dp / (dp - dx)) * (stx - stp)
|
||||
if abs(stpc - stp) > abs(stpq - stp):
|
||||
stpf = stpc
|
||||
else:
|
||||
stpf = stpq
|
||||
brackt = True
|
||||
elif abs(dp) < abs(dx):
|
||||
# Third case: A lower function value, derivatives of the same sign,
|
||||
# and the magnitude of the derivative decreases.
|
||||
|
||||
# The cubic step is computed only if the cubic tends to infinity
|
||||
# in the direction of the step or if the minimum of the cubic
|
||||
# is beyond stp. Otherwise the cubic step is defined to be the
|
||||
# secant step.
|
||||
theta = 3 * (fx - fp) / (stp - stx) + dx + dp
|
||||
s = max(abs(theta), abs(dx), abs(dp))
|
||||
|
||||
# The case gamma = 0 only arises if the cubic does not tend
|
||||
# to infinity in the direction of the step.
|
||||
gamma = s * np.sqrt(max(0, (theta / s) ** 2 - (dx / s) * (dp / s)))
|
||||
if stp > stx:
|
||||
gamma = -gamma
|
||||
p = (gamma - dp) + theta
|
||||
q = (gamma + (dx - dp)) + gamma
|
||||
r = p / q
|
||||
if r < 0 and gamma != 0:
|
||||
stpc = stp + r * (stx - stp)
|
||||
elif stp > stx:
|
||||
stpc = stpmax
|
||||
else:
|
||||
stpc = stpmin
|
||||
stpq = stp + (dp / (dp - dx)) * (stx - stp)
|
||||
|
||||
if brackt:
|
||||
# A minimizer has been bracketed. If the cubic step is
|
||||
# closer to stp than the secant step, the cubic step is
|
||||
# taken, otherwise the secant step is taken.
|
||||
if abs(stpc - stp) < abs(stpq - stp):
|
||||
stpf = stpc
|
||||
else:
|
||||
stpf = stpq
|
||||
|
||||
if stp > stx:
|
||||
stpf = min(stp + 0.66 * (sty - stp), stpf)
|
||||
else:
|
||||
stpf = max(stp + 0.66 * (sty - stp), stpf)
|
||||
else:
|
||||
# A minimizer has not been bracketed. If the cubic step is
|
||||
# farther from stp than the secant step, the cubic step is
|
||||
# taken, otherwise the secant step is taken.
|
||||
if abs(stpc - stp) > abs(stpq - stp):
|
||||
stpf = stpc
|
||||
else:
|
||||
stpf = stpq
|
||||
stpf = np.clip(stpf, stpmin, stpmax)
|
||||
|
||||
else:
|
||||
# Fourth case: A lower function value, derivatives of the same sign,
|
||||
# and the magnitude of the derivative does not decrease. If the
|
||||
# minimum is not bracketed, the step is either stpmin or stpmax,
|
||||
# otherwise the cubic step is taken.
|
||||
if brackt:
|
||||
theta = 3.0 * (fp - fy) / (sty - stp) + dy + dp
|
||||
s = max(abs(theta), abs(dy), abs(dp))
|
||||
gamma = s * np.sqrt((theta / s) ** 2 - (dy / s) * (dp / s))
|
||||
if stp > sty:
|
||||
gamma = -gamma
|
||||
p = (gamma - dp) + theta
|
||||
q = ((gamma - dp) + gamma) + dy
|
||||
r = p / q
|
||||
stpc = stp + r * (sty - stp)
|
||||
stpf = stpc
|
||||
elif stp > stx:
|
||||
stpf = stpmax
|
||||
else:
|
||||
stpf = stpmin
|
||||
|
||||
# Update the interval which contains a minimizer.
|
||||
if fp > fx:
|
||||
sty = stp
|
||||
fy = fp
|
||||
dy = dp
|
||||
else:
|
||||
if sgnd < 0:
|
||||
sty = stx
|
||||
fy = fx
|
||||
dy = dx
|
||||
stx = stp
|
||||
fx = fp
|
||||
dx = dp
|
||||
|
||||
# Compute the new step.
|
||||
stp = stpf
|
||||
|
||||
return stx, fx, dx, sty, fy, dy, stp, brackt
|
||||
@@ -0,0 +1,693 @@
|
||||
import numpy as np
|
||||
import scipy.sparse as sps
|
||||
from ._numdiff import approx_derivative, group_columns
|
||||
from ._hessian_update_strategy import HessianUpdateStrategy
|
||||
from scipy.sparse.linalg import LinearOperator
|
||||
from scipy._lib._array_api import atleast_nd, array_namespace
|
||||
|
||||
|
||||
FD_METHODS = ('2-point', '3-point', 'cs')
|
||||
|
||||
|
||||
def _wrapper_fun(fun, args=()):
|
||||
ncalls = [0]
|
||||
|
||||
def wrapped(x):
|
||||
ncalls[0] += 1
|
||||
# Send a copy because the user may overwrite it.
|
||||
# Overwriting results in undefined behaviour because
|
||||
# fun(self.x) will change self.x, with the two no longer linked.
|
||||
fx = fun(np.copy(x), *args)
|
||||
# Make sure the function returns a true scalar
|
||||
if not np.isscalar(fx):
|
||||
try:
|
||||
fx = np.asarray(fx).item()
|
||||
except (TypeError, ValueError) as e:
|
||||
raise ValueError(
|
||||
"The user-provided objective function "
|
||||
"must return a scalar value."
|
||||
) from e
|
||||
return fx
|
||||
return wrapped, ncalls
|
||||
|
||||
|
||||
def _wrapper_grad(grad, fun=None, args=(), finite_diff_options=None):
|
||||
ncalls = [0]
|
||||
|
||||
if callable(grad):
|
||||
def wrapped(x, **kwds):
|
||||
# kwds present to give function same signature as numdiff variant
|
||||
ncalls[0] += 1
|
||||
return np.atleast_1d(grad(np.copy(x), *args))
|
||||
return wrapped, ncalls
|
||||
|
||||
elif grad in FD_METHODS:
|
||||
def wrapped1(x, f0=None):
|
||||
ncalls[0] += 1
|
||||
return approx_derivative(
|
||||
fun, x, f0=f0, **finite_diff_options
|
||||
)
|
||||
|
||||
return wrapped1, ncalls
|
||||
|
||||
|
||||
def _wrapper_hess(hess, grad=None, x0=None, args=(), finite_diff_options=None):
|
||||
if callable(hess):
|
||||
H = hess(np.copy(x0), *args)
|
||||
ncalls = [1]
|
||||
|
||||
if sps.issparse(H):
|
||||
def wrapped(x, **kwds):
|
||||
ncalls[0] += 1
|
||||
return sps.csr_matrix(hess(np.copy(x), *args))
|
||||
|
||||
H = sps.csr_matrix(H)
|
||||
|
||||
elif isinstance(H, LinearOperator):
|
||||
def wrapped(x, **kwds):
|
||||
ncalls[0] += 1
|
||||
return hess(np.copy(x), *args)
|
||||
|
||||
else: # dense
|
||||
def wrapped(x, **kwds):
|
||||
ncalls[0] += 1
|
||||
return np.atleast_2d(np.asarray(hess(np.copy(x), *args)))
|
||||
|
||||
H = np.atleast_2d(np.asarray(H))
|
||||
|
||||
return wrapped, ncalls, H
|
||||
elif hess in FD_METHODS:
|
||||
ncalls = [0]
|
||||
|
||||
def wrapped1(x, f0=None):
|
||||
return approx_derivative(
|
||||
grad, x, f0=f0, **finite_diff_options
|
||||
)
|
||||
|
||||
return wrapped1, ncalls, None
|
||||
|
||||
|
||||
class ScalarFunction:
|
||||
"""Scalar function and its derivatives.
|
||||
|
||||
This class defines a scalar function F: R^n->R and methods for
|
||||
computing or approximating its first and second derivatives.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
fun : callable
|
||||
evaluates the scalar function. Must be of the form ``fun(x, *args)``,
|
||||
where ``x`` is the argument in the form of a 1-D array and ``args`` is
|
||||
a tuple of any additional fixed parameters needed to completely specify
|
||||
the function. Should return a scalar.
|
||||
x0 : array-like
|
||||
Provides an initial set of variables for evaluating fun. Array of real
|
||||
elements of size (n,), where 'n' is the number of independent
|
||||
variables.
|
||||
args : tuple, optional
|
||||
Any additional fixed parameters needed to completely specify the scalar
|
||||
function.
|
||||
grad : {callable, '2-point', '3-point', 'cs'}
|
||||
Method for computing the gradient vector.
|
||||
If it is a callable, it should be a function that returns the gradient
|
||||
vector:
|
||||
|
||||
``grad(x, *args) -> array_like, shape (n,)``
|
||||
|
||||
where ``x`` is an array with shape (n,) and ``args`` is a tuple with
|
||||
the fixed parameters.
|
||||
Alternatively, the keywords {'2-point', '3-point', 'cs'} can be used
|
||||
to select a finite difference scheme for numerical estimation of the
|
||||
gradient with a relative step size. These finite difference schemes
|
||||
obey any specified `bounds`.
|
||||
hess : {callable, '2-point', '3-point', 'cs', HessianUpdateStrategy}
|
||||
Method for computing the Hessian matrix. If it is callable, it should
|
||||
return the Hessian matrix:
|
||||
|
||||
``hess(x, *args) -> {LinearOperator, spmatrix, array}, (n, n)``
|
||||
|
||||
where x is a (n,) ndarray and `args` is a tuple with the fixed
|
||||
parameters. Alternatively, the keywords {'2-point', '3-point', 'cs'}
|
||||
select a finite difference scheme for numerical estimation. Or, objects
|
||||
implementing `HessianUpdateStrategy` interface can be used to
|
||||
approximate the Hessian.
|
||||
Whenever the gradient is estimated via finite-differences, the Hessian
|
||||
cannot be estimated with options {'2-point', '3-point', 'cs'} and needs
|
||||
to be estimated using one of the quasi-Newton strategies.
|
||||
finite_diff_rel_step : None or array_like
|
||||
Relative step size to use. The absolute step size is computed as
|
||||
``h = finite_diff_rel_step * sign(x0) * max(1, abs(x0))``, possibly
|
||||
adjusted to fit into the bounds. For ``method='3-point'`` the sign
|
||||
of `h` is ignored. If None then finite_diff_rel_step is selected
|
||||
automatically,
|
||||
finite_diff_bounds : tuple of array_like
|
||||
Lower and upper bounds on independent variables. Defaults to no bounds,
|
||||
(-np.inf, np.inf). Each bound must match the size of `x0` or be a
|
||||
scalar, in the latter case the bound will be the same for all
|
||||
variables. Use it to limit the range of function evaluation.
|
||||
epsilon : None or array_like, optional
|
||||
Absolute step size to use, possibly adjusted to fit into the bounds.
|
||||
For ``method='3-point'`` the sign of `epsilon` is ignored. By default
|
||||
relative steps are used, only if ``epsilon is not None`` are absolute
|
||||
steps used.
|
||||
|
||||
Notes
|
||||
-----
|
||||
This class implements a memoization logic. There are methods `fun`,
|
||||
`grad`, hess` and corresponding attributes `f`, `g` and `H`. The following
|
||||
things should be considered:
|
||||
|
||||
1. Use only public methods `fun`, `grad` and `hess`.
|
||||
2. After one of the methods is called, the corresponding attribute
|
||||
will be set. However, a subsequent call with a different argument
|
||||
of *any* of the methods may overwrite the attribute.
|
||||
"""
|
||||
def __init__(self, fun, x0, args, grad, hess, finite_diff_rel_step,
|
||||
finite_diff_bounds, epsilon=None):
|
||||
if not callable(grad) and grad not in FD_METHODS:
|
||||
raise ValueError(
|
||||
f"`grad` must be either callable or one of {FD_METHODS}."
|
||||
)
|
||||
|
||||
if not (callable(hess) or hess in FD_METHODS
|
||||
or isinstance(hess, HessianUpdateStrategy)):
|
||||
raise ValueError(
|
||||
f"`hess` must be either callable, HessianUpdateStrategy"
|
||||
f" or one of {FD_METHODS}."
|
||||
)
|
||||
|
||||
if grad in FD_METHODS and hess in FD_METHODS:
|
||||
raise ValueError("Whenever the gradient is estimated via "
|
||||
"finite-differences, we require the Hessian "
|
||||
"to be estimated using one of the "
|
||||
"quasi-Newton strategies.")
|
||||
|
||||
self.xp = xp = array_namespace(x0)
|
||||
_x = atleast_nd(x0, ndim=1, xp=xp)
|
||||
_dtype = xp.float64
|
||||
if xp.isdtype(_x.dtype, "real floating"):
|
||||
_dtype = _x.dtype
|
||||
|
||||
# original arguments
|
||||
self._wrapped_fun, self._nfev = _wrapper_fun(fun, args=args)
|
||||
self._orig_fun = fun
|
||||
self._orig_grad = grad
|
||||
self._orig_hess = hess
|
||||
self._args = args
|
||||
|
||||
# promotes to floating
|
||||
self.x = xp.astype(_x, _dtype)
|
||||
self.x_dtype = _dtype
|
||||
self.n = self.x.size
|
||||
self.f_updated = False
|
||||
self.g_updated = False
|
||||
self.H_updated = False
|
||||
|
||||
self._lowest_x = None
|
||||
self._lowest_f = np.inf
|
||||
|
||||
finite_diff_options = {}
|
||||
if grad in FD_METHODS:
|
||||
finite_diff_options["method"] = grad
|
||||
finite_diff_options["rel_step"] = finite_diff_rel_step
|
||||
finite_diff_options["abs_step"] = epsilon
|
||||
finite_diff_options["bounds"] = finite_diff_bounds
|
||||
if hess in FD_METHODS:
|
||||
finite_diff_options["method"] = hess
|
||||
finite_diff_options["rel_step"] = finite_diff_rel_step
|
||||
finite_diff_options["abs_step"] = epsilon
|
||||
finite_diff_options["as_linear_operator"] = True
|
||||
|
||||
# Initial function evaluation
|
||||
self._update_fun()
|
||||
|
||||
# Initial gradient evaluation
|
||||
self._wrapped_grad, self._ngev = _wrapper_grad(
|
||||
grad,
|
||||
fun=self._wrapped_fun,
|
||||
args=args,
|
||||
finite_diff_options=finite_diff_options
|
||||
)
|
||||
self._update_grad()
|
||||
|
||||
# Hessian evaluation
|
||||
if callable(hess):
|
||||
self._wrapped_hess, self._nhev, self.H = _wrapper_hess(
|
||||
hess, x0=x0, args=args
|
||||
)
|
||||
self.H_updated = True
|
||||
elif hess in FD_METHODS:
|
||||
self._wrapped_hess, self._nhev, self.H = _wrapper_hess(
|
||||
hess,
|
||||
grad=self._wrapped_grad,
|
||||
x0=x0,
|
||||
finite_diff_options=finite_diff_options
|
||||
)
|
||||
self._update_grad()
|
||||
self.H = self._wrapped_hess(self.x, f0=self.g)
|
||||
self.H_updated = True
|
||||
elif isinstance(hess, HessianUpdateStrategy):
|
||||
self.H = hess
|
||||
self.H.initialize(self.n, 'hess')
|
||||
self.H_updated = True
|
||||
self.x_prev = None
|
||||
self.g_prev = None
|
||||
self._nhev = [0]
|
||||
|
||||
@property
|
||||
def nfev(self):
|
||||
return self._nfev[0]
|
||||
|
||||
@property
|
||||
def ngev(self):
|
||||
return self._ngev[0]
|
||||
|
||||
@property
|
||||
def nhev(self):
|
||||
return self._nhev[0]
|
||||
|
||||
def _update_x(self, x):
|
||||
if isinstance(self._orig_hess, HessianUpdateStrategy):
|
||||
self._update_grad()
|
||||
self.x_prev = self.x
|
||||
self.g_prev = self.g
|
||||
# ensure that self.x is a copy of x. Don't store a reference
|
||||
# otherwise the memoization doesn't work properly.
|
||||
|
||||
_x = atleast_nd(x, ndim=1, xp=self.xp)
|
||||
self.x = self.xp.astype(_x, self.x_dtype)
|
||||
self.f_updated = False
|
||||
self.g_updated = False
|
||||
self.H_updated = False
|
||||
self._update_hess()
|
||||
else:
|
||||
# ensure that self.x is a copy of x. Don't store a reference
|
||||
# otherwise the memoization doesn't work properly.
|
||||
_x = atleast_nd(x, ndim=1, xp=self.xp)
|
||||
self.x = self.xp.astype(_x, self.x_dtype)
|
||||
self.f_updated = False
|
||||
self.g_updated = False
|
||||
self.H_updated = False
|
||||
|
||||
def _update_fun(self):
|
||||
if not self.f_updated:
|
||||
fx = self._wrapped_fun(self.x)
|
||||
if fx < self._lowest_f:
|
||||
self._lowest_x = self.x
|
||||
self._lowest_f = fx
|
||||
|
||||
self.f = fx
|
||||
self.f_updated = True
|
||||
|
||||
def _update_grad(self):
|
||||
if not self.g_updated:
|
||||
if self._orig_grad in FD_METHODS:
|
||||
self._update_fun()
|
||||
self.g = self._wrapped_grad(self.x, f0=self.f)
|
||||
self.g_updated = True
|
||||
|
||||
def _update_hess(self):
|
||||
if not self.H_updated:
|
||||
if self._orig_hess in FD_METHODS:
|
||||
self._update_grad()
|
||||
self.H = self._wrapped_hess(self.x, f0=self.g)
|
||||
elif isinstance(self._orig_hess, HessianUpdateStrategy):
|
||||
self._update_grad()
|
||||
self.H.update(self.x - self.x_prev, self.g - self.g_prev)
|
||||
else: # should be callable(hess)
|
||||
self.H = self._wrapped_hess(self.x)
|
||||
|
||||
self.H_updated = True
|
||||
|
||||
def fun(self, x):
|
||||
if not np.array_equal(x, self.x):
|
||||
self._update_x(x)
|
||||
self._update_fun()
|
||||
return self.f
|
||||
|
||||
def grad(self, x):
|
||||
if not np.array_equal(x, self.x):
|
||||
self._update_x(x)
|
||||
self._update_grad()
|
||||
return self.g
|
||||
|
||||
def hess(self, x):
|
||||
if not np.array_equal(x, self.x):
|
||||
self._update_x(x)
|
||||
self._update_hess()
|
||||
return self.H
|
||||
|
||||
def fun_and_grad(self, x):
|
||||
if not np.array_equal(x, self.x):
|
||||
self._update_x(x)
|
||||
self._update_fun()
|
||||
self._update_grad()
|
||||
return self.f, self.g
|
||||
|
||||
|
||||
class VectorFunction:
|
||||
"""Vector function and its derivatives.
|
||||
|
||||
This class defines a vector function F: R^n->R^m and methods for
|
||||
computing or approximating its first and second derivatives.
|
||||
|
||||
Notes
|
||||
-----
|
||||
This class implements a memoization logic. There are methods `fun`,
|
||||
`jac`, hess` and corresponding attributes `f`, `J` and `H`. The following
|
||||
things should be considered:
|
||||
|
||||
1. Use only public methods `fun`, `jac` and `hess`.
|
||||
2. After one of the methods is called, the corresponding attribute
|
||||
will be set. However, a subsequent call with a different argument
|
||||
of *any* of the methods may overwrite the attribute.
|
||||
"""
|
||||
def __init__(self, fun, x0, jac, hess,
|
||||
finite_diff_rel_step, finite_diff_jac_sparsity,
|
||||
finite_diff_bounds, sparse_jacobian):
|
||||
if not callable(jac) and jac not in FD_METHODS:
|
||||
raise ValueError(f"`jac` must be either callable or one of {FD_METHODS}.")
|
||||
|
||||
if not (callable(hess) or hess in FD_METHODS
|
||||
or isinstance(hess, HessianUpdateStrategy)):
|
||||
raise ValueError("`hess` must be either callable,"
|
||||
f"HessianUpdateStrategy or one of {FD_METHODS}.")
|
||||
|
||||
if jac in FD_METHODS and hess in FD_METHODS:
|
||||
raise ValueError("Whenever the Jacobian is estimated via "
|
||||
"finite-differences, we require the Hessian to "
|
||||
"be estimated using one of the quasi-Newton "
|
||||
"strategies.")
|
||||
|
||||
self.xp = xp = array_namespace(x0)
|
||||
_x = atleast_nd(x0, ndim=1, xp=xp)
|
||||
_dtype = xp.float64
|
||||
if xp.isdtype(_x.dtype, "real floating"):
|
||||
_dtype = _x.dtype
|
||||
|
||||
# promotes to floating
|
||||
self.x = xp.astype(_x, _dtype)
|
||||
self.x_dtype = _dtype
|
||||
|
||||
self.n = self.x.size
|
||||
self.nfev = 0
|
||||
self.njev = 0
|
||||
self.nhev = 0
|
||||
self.f_updated = False
|
||||
self.J_updated = False
|
||||
self.H_updated = False
|
||||
|
||||
finite_diff_options = {}
|
||||
if jac in FD_METHODS:
|
||||
finite_diff_options["method"] = jac
|
||||
finite_diff_options["rel_step"] = finite_diff_rel_step
|
||||
if finite_diff_jac_sparsity is not None:
|
||||
sparsity_groups = group_columns(finite_diff_jac_sparsity)
|
||||
finite_diff_options["sparsity"] = (finite_diff_jac_sparsity,
|
||||
sparsity_groups)
|
||||
finite_diff_options["bounds"] = finite_diff_bounds
|
||||
self.x_diff = np.copy(self.x)
|
||||
if hess in FD_METHODS:
|
||||
finite_diff_options["method"] = hess
|
||||
finite_diff_options["rel_step"] = finite_diff_rel_step
|
||||
finite_diff_options["as_linear_operator"] = True
|
||||
self.x_diff = np.copy(self.x)
|
||||
if jac in FD_METHODS and hess in FD_METHODS:
|
||||
raise ValueError("Whenever the Jacobian is estimated via "
|
||||
"finite-differences, we require the Hessian to "
|
||||
"be estimated using one of the quasi-Newton "
|
||||
"strategies.")
|
||||
|
||||
# Function evaluation
|
||||
def fun_wrapped(x):
|
||||
self.nfev += 1
|
||||
return np.atleast_1d(fun(x))
|
||||
|
||||
def update_fun():
|
||||
self.f = fun_wrapped(self.x)
|
||||
|
||||
self._update_fun_impl = update_fun
|
||||
update_fun()
|
||||
|
||||
self.v = np.zeros_like(self.f)
|
||||
self.m = self.v.size
|
||||
|
||||
# Jacobian Evaluation
|
||||
if callable(jac):
|
||||
self.J = jac(self.x)
|
||||
self.J_updated = True
|
||||
self.njev += 1
|
||||
|
||||
if (sparse_jacobian or
|
||||
sparse_jacobian is None and sps.issparse(self.J)):
|
||||
def jac_wrapped(x):
|
||||
self.njev += 1
|
||||
return sps.csr_matrix(jac(x))
|
||||
self.J = sps.csr_matrix(self.J)
|
||||
self.sparse_jacobian = True
|
||||
|
||||
elif sps.issparse(self.J):
|
||||
def jac_wrapped(x):
|
||||
self.njev += 1
|
||||
return jac(x).toarray()
|
||||
self.J = self.J.toarray()
|
||||
self.sparse_jacobian = False
|
||||
|
||||
else:
|
||||
def jac_wrapped(x):
|
||||
self.njev += 1
|
||||
return np.atleast_2d(jac(x))
|
||||
self.J = np.atleast_2d(self.J)
|
||||
self.sparse_jacobian = False
|
||||
|
||||
def update_jac():
|
||||
self.J = jac_wrapped(self.x)
|
||||
|
||||
elif jac in FD_METHODS:
|
||||
self.J = approx_derivative(fun_wrapped, self.x, f0=self.f,
|
||||
**finite_diff_options)
|
||||
self.J_updated = True
|
||||
|
||||
if (sparse_jacobian or
|
||||
sparse_jacobian is None and sps.issparse(self.J)):
|
||||
def update_jac():
|
||||
self._update_fun()
|
||||
self.J = sps.csr_matrix(
|
||||
approx_derivative(fun_wrapped, self.x, f0=self.f,
|
||||
**finite_diff_options))
|
||||
self.J = sps.csr_matrix(self.J)
|
||||
self.sparse_jacobian = True
|
||||
|
||||
elif sps.issparse(self.J):
|
||||
def update_jac():
|
||||
self._update_fun()
|
||||
self.J = approx_derivative(fun_wrapped, self.x, f0=self.f,
|
||||
**finite_diff_options).toarray()
|
||||
self.J = self.J.toarray()
|
||||
self.sparse_jacobian = False
|
||||
|
||||
else:
|
||||
def update_jac():
|
||||
self._update_fun()
|
||||
self.J = np.atleast_2d(
|
||||
approx_derivative(fun_wrapped, self.x, f0=self.f,
|
||||
**finite_diff_options))
|
||||
self.J = np.atleast_2d(self.J)
|
||||
self.sparse_jacobian = False
|
||||
|
||||
self._update_jac_impl = update_jac
|
||||
|
||||
# Define Hessian
|
||||
if callable(hess):
|
||||
self.H = hess(self.x, self.v)
|
||||
self.H_updated = True
|
||||
self.nhev += 1
|
||||
|
||||
if sps.issparse(self.H):
|
||||
def hess_wrapped(x, v):
|
||||
self.nhev += 1
|
||||
return sps.csr_matrix(hess(x, v))
|
||||
self.H = sps.csr_matrix(self.H)
|
||||
|
||||
elif isinstance(self.H, LinearOperator):
|
||||
def hess_wrapped(x, v):
|
||||
self.nhev += 1
|
||||
return hess(x, v)
|
||||
|
||||
else:
|
||||
def hess_wrapped(x, v):
|
||||
self.nhev += 1
|
||||
return np.atleast_2d(np.asarray(hess(x, v)))
|
||||
self.H = np.atleast_2d(np.asarray(self.H))
|
||||
|
||||
def update_hess():
|
||||
self.H = hess_wrapped(self.x, self.v)
|
||||
elif hess in FD_METHODS:
|
||||
def jac_dot_v(x, v):
|
||||
return jac_wrapped(x).T.dot(v)
|
||||
|
||||
def update_hess():
|
||||
self._update_jac()
|
||||
self.H = approx_derivative(jac_dot_v, self.x,
|
||||
f0=self.J.T.dot(self.v),
|
||||
args=(self.v,),
|
||||
**finite_diff_options)
|
||||
update_hess()
|
||||
self.H_updated = True
|
||||
elif isinstance(hess, HessianUpdateStrategy):
|
||||
self.H = hess
|
||||
self.H.initialize(self.n, 'hess')
|
||||
self.H_updated = True
|
||||
self.x_prev = None
|
||||
self.J_prev = None
|
||||
|
||||
def update_hess():
|
||||
self._update_jac()
|
||||
# When v is updated before x was updated, then x_prev and
|
||||
# J_prev are None and we need this check.
|
||||
if self.x_prev is not None and self.J_prev is not None:
|
||||
delta_x = self.x - self.x_prev
|
||||
delta_g = self.J.T.dot(self.v) - self.J_prev.T.dot(self.v)
|
||||
self.H.update(delta_x, delta_g)
|
||||
|
||||
self._update_hess_impl = update_hess
|
||||
|
||||
if isinstance(hess, HessianUpdateStrategy):
|
||||
def update_x(x):
|
||||
self._update_jac()
|
||||
self.x_prev = self.x
|
||||
self.J_prev = self.J
|
||||
_x = atleast_nd(x, ndim=1, xp=self.xp)
|
||||
self.x = self.xp.astype(_x, self.x_dtype)
|
||||
self.f_updated = False
|
||||
self.J_updated = False
|
||||
self.H_updated = False
|
||||
self._update_hess()
|
||||
else:
|
||||
def update_x(x):
|
||||
_x = atleast_nd(x, ndim=1, xp=self.xp)
|
||||
self.x = self.xp.astype(_x, self.x_dtype)
|
||||
self.f_updated = False
|
||||
self.J_updated = False
|
||||
self.H_updated = False
|
||||
|
||||
self._update_x_impl = update_x
|
||||
|
||||
def _update_v(self, v):
|
||||
if not np.array_equal(v, self.v):
|
||||
self.v = v
|
||||
self.H_updated = False
|
||||
|
||||
def _update_x(self, x):
|
||||
if not np.array_equal(x, self.x):
|
||||
self._update_x_impl(x)
|
||||
|
||||
def _update_fun(self):
|
||||
if not self.f_updated:
|
||||
self._update_fun_impl()
|
||||
self.f_updated = True
|
||||
|
||||
def _update_jac(self):
|
||||
if not self.J_updated:
|
||||
self._update_jac_impl()
|
||||
self.J_updated = True
|
||||
|
||||
def _update_hess(self):
|
||||
if not self.H_updated:
|
||||
self._update_hess_impl()
|
||||
self.H_updated = True
|
||||
|
||||
def fun(self, x):
|
||||
self._update_x(x)
|
||||
self._update_fun()
|
||||
return self.f
|
||||
|
||||
def jac(self, x):
|
||||
self._update_x(x)
|
||||
self._update_jac()
|
||||
return self.J
|
||||
|
||||
def hess(self, x, v):
|
||||
# v should be updated before x.
|
||||
self._update_v(v)
|
||||
self._update_x(x)
|
||||
self._update_hess()
|
||||
return self.H
|
||||
|
||||
|
||||
class LinearVectorFunction:
|
||||
"""Linear vector function and its derivatives.
|
||||
|
||||
Defines a linear function F = A x, where x is N-D vector and
|
||||
A is m-by-n matrix. The Jacobian is constant and equals to A. The Hessian
|
||||
is identically zero and it is returned as a csr matrix.
|
||||
"""
|
||||
def __init__(self, A, x0, sparse_jacobian):
|
||||
if sparse_jacobian or sparse_jacobian is None and sps.issparse(A):
|
||||
self.J = sps.csr_matrix(A)
|
||||
self.sparse_jacobian = True
|
||||
elif sps.issparse(A):
|
||||
self.J = A.toarray()
|
||||
self.sparse_jacobian = False
|
||||
else:
|
||||
# np.asarray makes sure A is ndarray and not matrix
|
||||
self.J = np.atleast_2d(np.asarray(A))
|
||||
self.sparse_jacobian = False
|
||||
|
||||
self.m, self.n = self.J.shape
|
||||
|
||||
self.xp = xp = array_namespace(x0)
|
||||
_x = atleast_nd(x0, ndim=1, xp=xp)
|
||||
_dtype = xp.float64
|
||||
if xp.isdtype(_x.dtype, "real floating"):
|
||||
_dtype = _x.dtype
|
||||
|
||||
# promotes to floating
|
||||
self.x = xp.astype(_x, _dtype)
|
||||
self.x_dtype = _dtype
|
||||
|
||||
self.f = self.J.dot(self.x)
|
||||
self.f_updated = True
|
||||
|
||||
self.v = np.zeros(self.m, dtype=float)
|
||||
self.H = sps.csr_matrix((self.n, self.n))
|
||||
|
||||
def _update_x(self, x):
|
||||
if not np.array_equal(x, self.x):
|
||||
_x = atleast_nd(x, ndim=1, xp=self.xp)
|
||||
self.x = self.xp.astype(_x, self.x_dtype)
|
||||
self.f_updated = False
|
||||
|
||||
def fun(self, x):
|
||||
self._update_x(x)
|
||||
if not self.f_updated:
|
||||
self.f = self.J.dot(x)
|
||||
self.f_updated = True
|
||||
return self.f
|
||||
|
||||
def jac(self, x):
|
||||
self._update_x(x)
|
||||
return self.J
|
||||
|
||||
def hess(self, x, v):
|
||||
self._update_x(x)
|
||||
self.v = v
|
||||
return self.H
|
||||
|
||||
|
||||
class IdentityVectorFunction(LinearVectorFunction):
|
||||
"""Identity vector function and its derivatives.
|
||||
|
||||
The Jacobian is the identity matrix, returned as a dense array when
|
||||
`sparse_jacobian=False` and as a csr matrix otherwise. The Hessian is
|
||||
identically zero and it is returned as a csr matrix.
|
||||
"""
|
||||
def __init__(self, x0, sparse_jacobian):
|
||||
n = len(x0)
|
||||
if sparse_jacobian or sparse_jacobian is None:
|
||||
A = sps.eye(n, format='csr')
|
||||
sparse_jacobian = True
|
||||
else:
|
||||
A = np.eye(n)
|
||||
sparse_jacobian = False
|
||||
super().__init__(A, x0, sparse_jacobian)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,856 @@
|
||||
# mypy: disable-error-code="attr-defined"
|
||||
import numpy as np
|
||||
import scipy._lib._elementwise_iterative_method as eim
|
||||
from scipy._lib._util import _RichResult
|
||||
|
||||
_EERRORINCREASE = -1 # used in _differentiate
|
||||
|
||||
def _differentiate_iv(func, x, args, atol, rtol, maxiter, order, initial_step,
|
||||
step_factor, step_direction, preserve_shape, callback):
|
||||
# Input validation for `_differentiate`
|
||||
|
||||
if not callable(func):
|
||||
raise ValueError('`func` must be callable.')
|
||||
|
||||
# x has more complex IV that is taken care of during initialization
|
||||
x = np.asarray(x)
|
||||
dtype = x.dtype if np.issubdtype(x.dtype, np.inexact) else np.float64
|
||||
|
||||
if not np.iterable(args):
|
||||
args = (args,)
|
||||
|
||||
if atol is None:
|
||||
atol = np.finfo(dtype).tiny
|
||||
|
||||
if rtol is None:
|
||||
rtol = np.sqrt(np.finfo(dtype).eps)
|
||||
|
||||
message = 'Tolerances and step parameters must be non-negative scalars.'
|
||||
tols = np.asarray([atol, rtol, initial_step, step_factor])
|
||||
if (not np.issubdtype(tols.dtype, np.number)
|
||||
or np.any(tols < 0)
|
||||
or tols.shape != (4,)):
|
||||
raise ValueError(message)
|
||||
initial_step, step_factor = tols[2:].astype(dtype)
|
||||
|
||||
maxiter_int = int(maxiter)
|
||||
if maxiter != maxiter_int or maxiter <= 0:
|
||||
raise ValueError('`maxiter` must be a positive integer.')
|
||||
|
||||
order_int = int(order)
|
||||
if order_int != order or order <= 0:
|
||||
raise ValueError('`order` must be a positive integer.')
|
||||
|
||||
step_direction = np.sign(step_direction).astype(dtype)
|
||||
x, step_direction = np.broadcast_arrays(x, step_direction)
|
||||
x, step_direction = x[()], step_direction[()]
|
||||
|
||||
message = '`preserve_shape` must be True or False.'
|
||||
if preserve_shape not in {True, False}:
|
||||
raise ValueError(message)
|
||||
|
||||
if callback is not None and not callable(callback):
|
||||
raise ValueError('`callback` must be callable.')
|
||||
|
||||
return (func, x, args, atol, rtol, maxiter_int, order_int, initial_step,
|
||||
step_factor, step_direction, preserve_shape, callback)
|
||||
|
||||
|
||||
def _differentiate(func, x, *, args=(), atol=None, rtol=None, maxiter=10,
|
||||
order=8, initial_step=0.5, step_factor=2.0,
|
||||
step_direction=0, preserve_shape=False, callback=None):
|
||||
"""Evaluate the derivative of an elementwise scalar function numerically.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
The function whose derivative is desired. The signature must be::
|
||||
|
||||
func(x: ndarray, *fargs) -> ndarray
|
||||
|
||||
where each element of ``x`` is a finite real number and ``fargs`` is a tuple,
|
||||
which may contain an arbitrary number of arrays that are broadcastable
|
||||
with `x`. ``func`` must be an elementwise function: each element
|
||||
``func(x)[i]`` must equal ``func(x[i])`` for all indices ``i``.
|
||||
x : array_like
|
||||
Abscissae at which to evaluate the derivative.
|
||||
args : tuple, optional
|
||||
Additional positional arguments to be passed to `func`. Must be arrays
|
||||
broadcastable with `x`. If the callable to be differentiated requires
|
||||
arguments that are not broadcastable with `x`, wrap that callable with
|
||||
`func`. See Examples.
|
||||
atol, rtol : float, optional
|
||||
Absolute and relative tolerances for the stopping condition: iteration
|
||||
will stop when ``res.error < atol + rtol * abs(res.df)``. The default
|
||||
`atol` is the smallest normal number of the appropriate dtype, and
|
||||
the default `rtol` is the square root of the precision of the
|
||||
appropriate dtype.
|
||||
order : int, default: 8
|
||||
The (positive integer) order of the finite difference formula to be
|
||||
used. Odd integers will be rounded up to the next even integer.
|
||||
initial_step : float, default: 0.5
|
||||
The (absolute) initial step size for the finite difference derivative
|
||||
approximation.
|
||||
step_factor : float, default: 2.0
|
||||
The factor by which the step size is *reduced* in each iteration; i.e.
|
||||
the step size in iteration 1 is ``initial_step/step_factor``. If
|
||||
``step_factor < 1``, subsequent steps will be greater than the initial
|
||||
step; this may be useful if steps smaller than some threshold are
|
||||
undesirable (e.g. due to subtractive cancellation error).
|
||||
maxiter : int, default: 10
|
||||
The maximum number of iterations of the algorithm to perform. See
|
||||
notes.
|
||||
step_direction : array_like
|
||||
An array representing the direction of the finite difference steps (for
|
||||
use when `x` lies near to the boundary of the domain of the function.)
|
||||
Must be broadcastable with `x` and all `args`.
|
||||
Where 0 (default), central differences are used; where negative (e.g.
|
||||
-1), steps are non-positive; and where positive (e.g. 1), all steps are
|
||||
non-negative.
|
||||
preserve_shape : bool, default: False
|
||||
In the following, "arguments of `func`" refers to the array ``x`` and
|
||||
any arrays within ``fargs``. Let ``shape`` be the broadcasted shape
|
||||
of `x` and all elements of `args` (which is conceptually
|
||||
distinct from ``fargs`` passed into `f`).
|
||||
|
||||
- When ``preserve_shape=False`` (default), `f` must accept arguments
|
||||
of *any* broadcastable shapes.
|
||||
|
||||
- When ``preserve_shape=True``, `f` must accept arguments of shape
|
||||
``shape`` *or* ``shape + (n,)``, where ``(n,)`` is the number of
|
||||
abscissae at which the function is being evaluated.
|
||||
|
||||
In either case, for each scalar element ``xi`` within `x`, the array
|
||||
returned by `f` must include the scalar ``f(xi)`` at the same index.
|
||||
Consequently, the shape of the output is always the shape of the input
|
||||
``x``.
|
||||
|
||||
See Examples.
|
||||
callback : callable, optional
|
||||
An optional user-supplied function to be called before the first
|
||||
iteration and after each iteration.
|
||||
Called as ``callback(res)``, where ``res`` is a ``_RichResult``
|
||||
similar to that returned by `_differentiate` (but containing the
|
||||
current iterate's values of all variables). If `callback` raises a
|
||||
``StopIteration``, the algorithm will terminate immediately and
|
||||
`_differentiate` will return a result.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : _RichResult
|
||||
An instance of `scipy._lib._util._RichResult` with the following
|
||||
attributes. (The descriptions are written as though the values will be
|
||||
scalars; however, if `func` returns an array, the outputs will be
|
||||
arrays of the same shape.)
|
||||
|
||||
success : bool
|
||||
``True`` when the algorithm terminated successfully (status ``0``).
|
||||
status : int
|
||||
An integer representing the exit status of the algorithm.
|
||||
``0`` : The algorithm converged to the specified tolerances.
|
||||
``-1`` : The error estimate increased, so iteration was terminated.
|
||||
``-2`` : The maximum number of iterations was reached.
|
||||
``-3`` : A non-finite value was encountered.
|
||||
``-4`` : Iteration was terminated by `callback`.
|
||||
``1`` : The algorithm is proceeding normally (in `callback` only).
|
||||
df : float
|
||||
The derivative of `func` at `x`, if the algorithm terminated
|
||||
successfully.
|
||||
error : float
|
||||
An estimate of the error: the magnitude of the difference between
|
||||
the current estimate of the derivative and the estimate in the
|
||||
previous iteration.
|
||||
nit : int
|
||||
The number of iterations performed.
|
||||
nfev : int
|
||||
The number of points at which `func` was evaluated.
|
||||
x : float
|
||||
The value at which the derivative of `func` was evaluated
|
||||
(after broadcasting with `args` and `step_direction`).
|
||||
|
||||
Notes
|
||||
-----
|
||||
The implementation was inspired by jacobi [1]_, numdifftools [2]_, and
|
||||
DERIVEST [3]_, but the implementation follows the theory of Taylor series
|
||||
more straightforwardly (and arguably naively so).
|
||||
In the first iteration, the derivative is estimated using a finite
|
||||
difference formula of order `order` with maximum step size `initial_step`.
|
||||
Each subsequent iteration, the maximum step size is reduced by
|
||||
`step_factor`, and the derivative is estimated again until a termination
|
||||
condition is reached. The error estimate is the magnitude of the difference
|
||||
between the current derivative approximation and that of the previous
|
||||
iteration.
|
||||
|
||||
The stencils of the finite difference formulae are designed such that
|
||||
abscissae are "nested": after `func` is evaluated at ``order + 1``
|
||||
points in the first iteration, `func` is evaluated at only two new points
|
||||
in each subsequent iteration; ``order - 1`` previously evaluated function
|
||||
values required by the finite difference formula are reused, and two
|
||||
function values (evaluations at the points furthest from `x`) are unused.
|
||||
|
||||
Step sizes are absolute. When the step size is small relative to the
|
||||
magnitude of `x`, precision is lost; for example, if `x` is ``1e20``, the
|
||||
default initial step size of ``0.5`` cannot be resolved. Accordingly,
|
||||
consider using larger initial step sizes for large magnitudes of `x`.
|
||||
|
||||
The default tolerances are challenging to satisfy at points where the
|
||||
true derivative is exactly zero. If the derivative may be exactly zero,
|
||||
consider specifying an absolute tolerance (e.g. ``atol=1e-16``) to
|
||||
improve convergence.
|
||||
|
||||
References
|
||||
----------
|
||||
[1]_ Hans Dembinski (@HDembinski). jacobi.
|
||||
https://github.com/HDembinski/jacobi
|
||||
[2]_ Per A. Brodtkorb and John D'Errico. numdifftools.
|
||||
https://numdifftools.readthedocs.io/en/latest/
|
||||
[3]_ John D'Errico. DERIVEST: Adaptive Robust Numerical Differentiation.
|
||||
https://www.mathworks.com/matlabcentral/fileexchange/13490-adaptive-robust-numerical-differentiation
|
||||
[4]_ Numerical Differentition. Wikipedia.
|
||||
https://en.wikipedia.org/wiki/Numerical_differentiation
|
||||
|
||||
Examples
|
||||
--------
|
||||
Evaluate the derivative of ``np.exp`` at several points ``x``.
|
||||
|
||||
>>> import numpy as np
|
||||
>>> from scipy.optimize._differentiate import _differentiate
|
||||
>>> f = np.exp
|
||||
>>> df = np.exp # true derivative
|
||||
>>> x = np.linspace(1, 2, 5)
|
||||
>>> res = _differentiate(f, x)
|
||||
>>> res.df # approximation of the derivative
|
||||
array([2.71828183, 3.49034296, 4.48168907, 5.75460268, 7.3890561 ])
|
||||
>>> res.error # estimate of the error
|
||||
array(
|
||||
[7.12940817e-12, 9.16688947e-12, 1.17594823e-11, 1.50972568e-11, 1.93942640e-11]
|
||||
)
|
||||
>>> abs(res.df - df(x)) # true error
|
||||
array(
|
||||
[3.06421555e-14, 3.01980663e-14, 5.06261699e-14, 6.30606678e-14, 8.34887715e-14]
|
||||
)
|
||||
|
||||
Show the convergence of the approximation as the step size is reduced.
|
||||
Each iteration, the step size is reduced by `step_factor`, so for
|
||||
sufficiently small initial step, each iteration reduces the error by a
|
||||
factor of ``1/step_factor**order`` until finite precision arithmetic
|
||||
inhibits further improvement.
|
||||
|
||||
>>> iter = list(range(1, 12)) # maximum iterations
|
||||
>>> hfac = 2 # step size reduction per iteration
|
||||
>>> hdir = [-1, 0, 1] # compare left-, central-, and right- steps
|
||||
>>> order = 4 # order of differentiation formula
|
||||
>>> x = 1
|
||||
>>> ref = df(x)
|
||||
>>> errors = [] # true error
|
||||
>>> for i in iter:
|
||||
... res = _differentiate(f, x, maxiter=i, step_factor=hfac,
|
||||
... step_direction=hdir, order=order,
|
||||
... atol=0, rtol=0) # prevent early termination
|
||||
... errors.append(abs(res.df - ref))
|
||||
>>> errors = np.array(errors)
|
||||
>>> plt.semilogy(iter, errors[:, 0], label='left differences')
|
||||
>>> plt.semilogy(iter, errors[:, 1], label='central differences')
|
||||
>>> plt.semilogy(iter, errors[:, 2], label='right differences')
|
||||
>>> plt.xlabel('iteration')
|
||||
>>> plt.ylabel('error')
|
||||
>>> plt.legend()
|
||||
>>> plt.show()
|
||||
>>> (errors[1, 1] / errors[0, 1], 1 / hfac**order)
|
||||
(0.06215223140159822, 0.0625)
|
||||
|
||||
The implementation is vectorized over `x`, `step_direction`, and `args`.
|
||||
The function is evaluated once before the first iteration to perform input
|
||||
validation and standardization, and once per iteration thereafter.
|
||||
|
||||
>>> def f(x, p):
|
||||
... print('here')
|
||||
... f.nit += 1
|
||||
... return x**p
|
||||
>>> f.nit = 0
|
||||
>>> def df(x, p):
|
||||
... return p*x**(p-1)
|
||||
>>> x = np.arange(1, 5)
|
||||
>>> p = np.arange(1, 6).reshape((-1, 1))
|
||||
>>> hdir = np.arange(-1, 2).reshape((-1, 1, 1))
|
||||
>>> res = _differentiate(f, x, args=(p,), step_direction=hdir, maxiter=1)
|
||||
>>> np.allclose(res.df, df(x, p))
|
||||
True
|
||||
>>> res.df.shape
|
||||
(3, 5, 4)
|
||||
>>> f.nit
|
||||
2
|
||||
|
||||
By default, `preserve_shape` is False, and therefore the callable
|
||||
`f` may be called with arrays of any broadcastable shapes.
|
||||
For example:
|
||||
|
||||
>>> shapes = []
|
||||
>>> def f(x, c):
|
||||
... shape = np.broadcast_shapes(x.shape, c.shape)
|
||||
... shapes.append(shape)
|
||||
... return np.sin(c*x)
|
||||
>>>
|
||||
>>> c = [1, 5, 10, 20]
|
||||
>>> res = _differentiate(f, 0, args=(c,))
|
||||
>>> shapes
|
||||
[(4,), (4, 8), (4, 2), (3, 2), (2, 2), (1, 2)]
|
||||
|
||||
To understand where these shapes are coming from - and to better
|
||||
understand how `_differentiate` computes accurate results - note that
|
||||
higher values of ``c`` correspond with higher frequency sinusoids.
|
||||
The higher frequency sinusoids make the function's derivative change
|
||||
faster, so more function evaluations are required to achieve the target
|
||||
accuracy:
|
||||
|
||||
>>> res.nfev
|
||||
array([11, 13, 15, 17])
|
||||
|
||||
The initial ``shape``, ``(4,)``, corresponds with evaluating the
|
||||
function at a single abscissa and all four frequencies; this is used
|
||||
for input validation and to determine the size and dtype of the arrays
|
||||
that store results. The next shape corresponds with evaluating the
|
||||
function at an initial grid of abscissae and all four frequencies.
|
||||
Successive calls to the function evaluate the function at two more
|
||||
abscissae, increasing the effective order of the approximation by two.
|
||||
However, in later function evaluations, the function is evaluated at
|
||||
fewer frequencies because the corresponding derivative has already
|
||||
converged to the required tolerance. This saves function evaluations to
|
||||
improve performance, but it requires the function to accept arguments of
|
||||
any shape.
|
||||
|
||||
"Vector-valued" functions are unlikely to satisfy this requirement.
|
||||
For example, consider
|
||||
|
||||
>>> def f(x):
|
||||
... return [x, np.sin(3*x), x+np.sin(10*x), np.sin(20*x)*(x-1)**2]
|
||||
|
||||
This integrand is not compatible with `_differentiate` as written; for instance,
|
||||
the shape of the output will not be the same as the shape of ``x``. Such a
|
||||
function *could* be converted to a compatible form with the introduction of
|
||||
additional parameters, but this would be inconvenient. In such cases,
|
||||
a simpler solution would be to use `preserve_shape`.
|
||||
|
||||
>>> shapes = []
|
||||
>>> def f(x):
|
||||
... shapes.append(x.shape)
|
||||
... x0, x1, x2, x3 = x
|
||||
... return [x0, np.sin(3*x1), x2+np.sin(10*x2), np.sin(20*x3)*(x3-1)**2]
|
||||
>>>
|
||||
>>> x = np.zeros(4)
|
||||
>>> res = _differentiate(f, x, preserve_shape=True)
|
||||
>>> shapes
|
||||
[(4,), (4, 8), (4, 2), (4, 2), (4, 2), (4, 2)]
|
||||
|
||||
Here, the shape of ``x`` is ``(4,)``. With ``preserve_shape=True``, the
|
||||
function may be called with argument ``x`` of shape ``(4,)`` or ``(4, n)``,
|
||||
and this is what we observe.
|
||||
|
||||
"""
|
||||
# TODO (followup):
|
||||
# - investigate behavior at saddle points
|
||||
# - array initial_step / step_factor?
|
||||
# - multivariate functions?
|
||||
|
||||
res = _differentiate_iv(func, x, args, atol, rtol, maxiter, order, initial_step,
|
||||
step_factor, step_direction, preserve_shape, callback)
|
||||
(func, x, args, atol, rtol, maxiter, order,
|
||||
h0, fac, hdir, preserve_shape, callback) = res
|
||||
|
||||
# Initialization
|
||||
# Since f(x) (no step) is not needed for central differences, it may be
|
||||
# possible to eliminate this function evaluation. However, it's useful for
|
||||
# input validation and standardization, and everything else is designed to
|
||||
# reduce function calls, so let's keep it simple.
|
||||
temp = eim._initialize(func, (x,), args, preserve_shape=preserve_shape)
|
||||
func, xs, fs, args, shape, dtype, xp = temp
|
||||
x, f = xs[0], fs[0]
|
||||
df = np.full_like(f, np.nan)
|
||||
# Ideally we'd broadcast the shape of `hdir` in `_elementwise_algo_init`, but
|
||||
# it's simpler to do it here than to generalize `_elementwise_algo_init` further.
|
||||
# `hdir` and `x` are already broadcasted in `_differentiate_iv`, so we know
|
||||
# that `hdir` can be broadcasted to the final shape.
|
||||
hdir = np.broadcast_to(hdir, shape).flatten()
|
||||
|
||||
status = np.full_like(x, eim._EINPROGRESS, dtype=int) # in progress
|
||||
nit, nfev = 0, 1 # one function evaluations performed above
|
||||
# Boolean indices of left, central, right, and (all) one-sided steps
|
||||
il = hdir < 0
|
||||
ic = hdir == 0
|
||||
ir = hdir > 0
|
||||
io = il | ir
|
||||
|
||||
# Most of these attributes are reasonably obvious, but:
|
||||
# - `fs` holds all the function values of all active `x`. The zeroth
|
||||
# axis corresponds with active points `x`, the first axis corresponds
|
||||
# with the different steps (in the order described in
|
||||
# `_differentiate_weights`).
|
||||
# - `terms` (which could probably use a better name) is half the `order`,
|
||||
# which is always even.
|
||||
work = _RichResult(x=x, df=df, fs=f[:, np.newaxis], error=np.nan, h=h0,
|
||||
df_last=np.nan, error_last=np.nan, h0=h0, fac=fac,
|
||||
atol=atol, rtol=rtol, nit=nit, nfev=nfev,
|
||||
status=status, dtype=dtype, terms=(order+1)//2,
|
||||
hdir=hdir, il=il, ic=ic, ir=ir, io=io)
|
||||
# This is the correspondence between terms in the `work` object and the
|
||||
# final result. In this case, the mapping is trivial. Note that `success`
|
||||
# is prepended automatically.
|
||||
res_work_pairs = [('status', 'status'), ('df', 'df'), ('error', 'error'),
|
||||
('nit', 'nit'), ('nfev', 'nfev'), ('x', 'x')]
|
||||
|
||||
def pre_func_eval(work):
|
||||
"""Determine the abscissae at which the function needs to be evaluated.
|
||||
|
||||
See `_differentiate_weights` for a description of the stencil (pattern
|
||||
of the abscissae).
|
||||
|
||||
In the first iteration, there is only one stored function value in
|
||||
`work.fs`, `f(x)`, so we need to evaluate at `order` new points. In
|
||||
subsequent iterations, we evaluate at two new points. Note that
|
||||
`work.x` is always flattened into a 1D array after broadcasting with
|
||||
all `args`, so we add a new axis at the end and evaluate all point
|
||||
in one call to the function.
|
||||
|
||||
For improvement:
|
||||
- Consider measuring the step size actually taken, since `(x + h) - x`
|
||||
is not identically equal to `h` with floating point arithmetic.
|
||||
- Adjust the step size automatically if `x` is too big to resolve the
|
||||
step.
|
||||
- We could probably save some work if there are no central difference
|
||||
steps or no one-sided steps.
|
||||
"""
|
||||
n = work.terms # half the order
|
||||
h = work.h # step size
|
||||
c = work.fac # step reduction factor
|
||||
d = c**0.5 # square root of step reduction factor (one-sided stencil)
|
||||
# Note - no need to be careful about dtypes until we allocate `x_eval`
|
||||
|
||||
if work.nit == 0:
|
||||
hc = h / c**np.arange(n)
|
||||
hc = np.concatenate((-hc[::-1], hc))
|
||||
else:
|
||||
hc = np.asarray([-h, h]) / c**(n-1)
|
||||
|
||||
if work.nit == 0:
|
||||
hr = h / d**np.arange(2*n)
|
||||
else:
|
||||
hr = np.asarray([h, h/d]) / c**(n-1)
|
||||
|
||||
n_new = 2*n if work.nit == 0 else 2 # number of new abscissae
|
||||
x_eval = np.zeros((len(work.hdir), n_new), dtype=work.dtype)
|
||||
il, ic, ir = work.il, work.ic, work.ir
|
||||
x_eval[ir] = work.x[ir, np.newaxis] + hr
|
||||
x_eval[ic] = work.x[ic, np.newaxis] + hc
|
||||
x_eval[il] = work.x[il, np.newaxis] - hr
|
||||
return x_eval
|
||||
|
||||
def post_func_eval(x, f, work):
|
||||
""" Estimate the derivative and error from the function evaluations
|
||||
|
||||
As in `pre_func_eval`: in the first iteration, there is only one stored
|
||||
function value in `work.fs`, `f(x)`, so we need to add the `order` new
|
||||
points. In subsequent iterations, we add two new points. The tricky
|
||||
part is getting the order to match that of the weights, which is
|
||||
described in `_differentiate_weights`.
|
||||
|
||||
For improvement:
|
||||
- Change the order of the weights (and steps in `pre_func_eval`) to
|
||||
simplify `work_fc` concatenation and eliminate `fc` concatenation.
|
||||
- It would be simple to do one-step Richardson extrapolation with `df`
|
||||
and `df_last` to increase the order of the estimate and/or improve
|
||||
the error estimate.
|
||||
- Process the function evaluations in a more numerically favorable
|
||||
way. For instance, combining the pairs of central difference evals
|
||||
into a second-order approximation and using Richardson extrapolation
|
||||
to produce a higher order approximation seemed to retain accuracy up
|
||||
to very high order.
|
||||
- Alternatively, we could use `polyfit` like Jacobi. An advantage of
|
||||
fitting polynomial to more points than necessary is improved noise
|
||||
tolerance.
|
||||
"""
|
||||
n = work.terms
|
||||
n_new = n if work.nit == 0 else 1
|
||||
il, ic, io = work.il, work.ic, work.io
|
||||
|
||||
# Central difference
|
||||
# `work_fc` is *all* the points at which the function has been evaluated
|
||||
# `fc` is the points we're using *this iteration* to produce the estimate
|
||||
work_fc = (f[ic, :n_new], work.fs[ic, :], f[ic, -n_new:])
|
||||
work_fc = np.concatenate(work_fc, axis=-1)
|
||||
if work.nit == 0:
|
||||
fc = work_fc
|
||||
else:
|
||||
fc = (work_fc[:, :n], work_fc[:, n:n+1], work_fc[:, -n:])
|
||||
fc = np.concatenate(fc, axis=-1)
|
||||
|
||||
# One-sided difference
|
||||
work_fo = np.concatenate((work.fs[io, :], f[io, :]), axis=-1)
|
||||
if work.nit == 0:
|
||||
fo = work_fo
|
||||
else:
|
||||
fo = np.concatenate((work_fo[:, 0:1], work_fo[:, -2*n:]), axis=-1)
|
||||
|
||||
work.fs = np.zeros((len(ic), work.fs.shape[-1] + 2*n_new))
|
||||
work.fs[ic] = work_fc
|
||||
work.fs[io] = work_fo
|
||||
|
||||
wc, wo = _differentiate_weights(work, n)
|
||||
work.df_last = work.df.copy()
|
||||
work.df[ic] = fc @ wc / work.h
|
||||
work.df[io] = fo @ wo / work.h
|
||||
work.df[il] *= -1
|
||||
|
||||
work.h /= work.fac
|
||||
work.error_last = work.error
|
||||
# Simple error estimate - the difference in derivative estimates between
|
||||
# this iteration and the last. This is typically conservative because if
|
||||
# convergence has begin, the true error is much closer to the difference
|
||||
# between the current estimate and the *next* error estimate. However,
|
||||
# we could use Richarson extrapolation to produce an error estimate that
|
||||
# is one order higher, and take the difference between that and
|
||||
# `work.df` (which would just be constant factor that depends on `fac`.)
|
||||
work.error = abs(work.df - work.df_last)
|
||||
|
||||
def check_termination(work):
|
||||
"""Terminate due to convergence, non-finite values, or error increase"""
|
||||
stop = np.zeros_like(work.df).astype(bool)
|
||||
|
||||
i = work.error < work.atol + work.rtol*abs(work.df)
|
||||
work.status[i] = eim._ECONVERGED
|
||||
stop[i] = True
|
||||
|
||||
if work.nit > 0:
|
||||
i = ~((np.isfinite(work.x) & np.isfinite(work.df)) | stop)
|
||||
work.df[i], work.status[i] = np.nan, eim._EVALUEERR
|
||||
stop[i] = True
|
||||
|
||||
# With infinite precision, there is a step size below which
|
||||
# all smaller step sizes will reduce the error. But in floating point
|
||||
# arithmetic, catastrophic cancellation will begin to cause the error
|
||||
# to increase again. This heuristic tries to avoid step sizes that are
|
||||
# too small. There may be more theoretically sound approaches for
|
||||
# detecting a step size that minimizes the total error, but this
|
||||
# heuristic seems simple and effective.
|
||||
i = (work.error > work.error_last*10) & ~stop
|
||||
work.status[i] = _EERRORINCREASE
|
||||
stop[i] = True
|
||||
|
||||
return stop
|
||||
|
||||
def post_termination_check(work):
|
||||
return
|
||||
|
||||
def customize_result(res, shape):
|
||||
return shape
|
||||
|
||||
return eim._loop(work, callback, shape, maxiter, func, args, dtype,
|
||||
pre_func_eval, post_func_eval, check_termination,
|
||||
post_termination_check, customize_result, res_work_pairs,
|
||||
xp, preserve_shape)
|
||||
|
||||
|
||||
def _differentiate_weights(work, n):
|
||||
# This produces the weights of the finite difference formula for a given
|
||||
# stencil. In experiments, use of a second-order central difference formula
|
||||
# with Richardson extrapolation was more accurate numerically, but it was
|
||||
# more complicated, and it would have become even more complicated when
|
||||
# adding support for one-sided differences. However, now that all the
|
||||
# function evaluation values are stored, they can be processed in whatever
|
||||
# way is desired to produce the derivative estimate. We leave alternative
|
||||
# approaches to future work. To be more self-contained, here is the theory
|
||||
# for deriving the weights below.
|
||||
#
|
||||
# Recall that the Taylor expansion of a univariate, scalar-values function
|
||||
# about a point `x` may be expressed as:
|
||||
# f(x + h) = f(x) + f'(x)*h + f''(x)/2!*h**2 + O(h**3)
|
||||
# Suppose we evaluate f(x), f(x+h), and f(x-h). We have:
|
||||
# f(x) = f(x)
|
||||
# f(x + h) = f(x) + f'(x)*h + f''(x)/2!*h**2 + O(h**3)
|
||||
# f(x - h) = f(x) - f'(x)*h + f''(x)/2!*h**2 + O(h**3)
|
||||
# We can solve for weights `wi` such that:
|
||||
# w1*f(x) = w1*(f(x))
|
||||
# + w2*f(x + h) = w2*(f(x) + f'(x)*h + f''(x)/2!*h**2) + O(h**3)
|
||||
# + w3*f(x - h) = w3*(f(x) - f'(x)*h + f''(x)/2!*h**2) + O(h**3)
|
||||
# = 0 + f'(x)*h + 0 + O(h**3)
|
||||
# Then
|
||||
# f'(x) ~ (w1*f(x) + w2*f(x+h) + w3*f(x-h))/h
|
||||
# is a finite difference derivative approximation with error O(h**2),
|
||||
# and so it is said to be a "second-order" approximation. Under certain
|
||||
# conditions (e.g. well-behaved function, `h` sufficiently small), the
|
||||
# error in the approximation will decrease with h**2; that is, if `h` is
|
||||
# reduced by a factor of 2, the error is reduced by a factor of 4.
|
||||
#
|
||||
# By default, we use eighth-order formulae. Our central-difference formula
|
||||
# uses abscissae:
|
||||
# x-h/c**3, x-h/c**2, x-h/c, x-h, x, x+h, x+h/c, x+h/c**2, x+h/c**3
|
||||
# where `c` is the step factor. (Typically, the step factor is greater than
|
||||
# one, so the outermost points - as written above - are actually closest to
|
||||
# `x`.) This "stencil" is chosen so that each iteration, the step can be
|
||||
# reduced by the factor `c`, and most of the function evaluations can be
|
||||
# reused with the new step size. For example, in the next iteration, we
|
||||
# will have:
|
||||
# x-h/c**4, x-h/c**3, x-h/c**2, x-h/c, x, x+h/c, x+h/c**2, x+h/c**3, x+h/c**4
|
||||
# We do not reuse `x-h` and `x+h` for the new derivative estimate.
|
||||
# While this would increase the order of the formula and thus the
|
||||
# theoretical convergence rate, it is also less stable numerically.
|
||||
# (As noted above, there are other ways of processing the values that are
|
||||
# more stable. Thus, even now we store `f(x-h)` and `f(x+h)` in `work.fs`
|
||||
# to simplify future development of this sort of improvement.)
|
||||
#
|
||||
# The (right) one-sided formula is produced similarly using abscissae
|
||||
# x, x+h, x+h/d, x+h/d**2, ..., x+h/d**6, x+h/d**7, x+h/d**7
|
||||
# where `d` is the square root of `c`. (The left one-sided formula simply
|
||||
# uses -h.) When the step size is reduced by factor `c = d**2`, we have
|
||||
# abscissae:
|
||||
# x, x+h/d**2, x+h/d**3..., x+h/d**8, x+h/d**9, x+h/d**9
|
||||
# `d` is chosen as the square root of `c` so that the rate of the step-size
|
||||
# reduction is the same per iteration as in the central difference case.
|
||||
# Note that because the central difference formulas are inherently of even
|
||||
# order, for simplicity, we use only even-order formulas for one-sided
|
||||
# differences, too.
|
||||
|
||||
# It's possible for the user to specify `fac` in, say, double precision but
|
||||
# `x` and `args` in single precision. `fac` gets converted to single
|
||||
# precision, but we should always use double precision for the intermediate
|
||||
# calculations here to avoid additional error in the weights.
|
||||
fac = work.fac.astype(np.float64)
|
||||
|
||||
# Note that if the user switches back to floating point precision with
|
||||
# `x` and `args`, then `fac` will not necessarily equal the (lower
|
||||
# precision) cached `_differentiate_weights.fac`, and the weights will
|
||||
# need to be recalculated. This could be fixed, but it's late, and of
|
||||
# low consequence.
|
||||
if fac != _differentiate_weights.fac:
|
||||
_differentiate_weights.central = []
|
||||
_differentiate_weights.right = []
|
||||
_differentiate_weights.fac = fac
|
||||
|
||||
if len(_differentiate_weights.central) != 2*n + 1:
|
||||
# Central difference weights. Consider refactoring this; it could
|
||||
# probably be more compact.
|
||||
i = np.arange(-n, n + 1)
|
||||
p = np.abs(i) - 1. # center point has power `p` -1, but sign `s` is 0
|
||||
s = np.sign(i)
|
||||
|
||||
h = s / fac ** p
|
||||
A = np.vander(h, increasing=True).T
|
||||
b = np.zeros(2*n + 1)
|
||||
b[1] = 1
|
||||
weights = np.linalg.solve(A, b)
|
||||
|
||||
# Enforce identities to improve accuracy
|
||||
weights[n] = 0
|
||||
for i in range(n):
|
||||
weights[-i-1] = -weights[i]
|
||||
|
||||
# Cache the weights. We only need to calculate them once unless
|
||||
# the step factor changes.
|
||||
_differentiate_weights.central = weights
|
||||
|
||||
# One-sided difference weights. The left one-sided weights (with
|
||||
# negative steps) are simply the negative of the right one-sided
|
||||
# weights, so no need to compute them separately.
|
||||
i = np.arange(2*n + 1)
|
||||
p = i - 1.
|
||||
s = np.sign(i)
|
||||
|
||||
h = s / np.sqrt(fac) ** p
|
||||
A = np.vander(h, increasing=True).T
|
||||
b = np.zeros(2 * n + 1)
|
||||
b[1] = 1
|
||||
weights = np.linalg.solve(A, b)
|
||||
|
||||
_differentiate_weights.right = weights
|
||||
|
||||
return (_differentiate_weights.central.astype(work.dtype, copy=False),
|
||||
_differentiate_weights.right.astype(work.dtype, copy=False))
|
||||
_differentiate_weights.central = []
|
||||
_differentiate_weights.right = []
|
||||
_differentiate_weights.fac = None
|
||||
|
||||
|
||||
def _jacobian(func, x, *, atol=None, rtol=None, maxiter=10,
|
||||
order=8, initial_step=0.5, step_factor=2.0):
|
||||
r"""Evaluate the Jacobian of a function numerically.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
The function whose Jacobian is desired. The signature must be::
|
||||
|
||||
func(x: ndarray) -> ndarray
|
||||
|
||||
where each element of ``x`` is a finite real. If the function to be
|
||||
differentiated accepts additional, arguments wrap it (e.g. using
|
||||
`functools.partial` or ``lambda``) and pass the wrapped callable
|
||||
into `_jacobian`. See Notes regarding vectorization and the dimensionality
|
||||
of the input and output.
|
||||
x : array_like
|
||||
Points at which to evaluate the Jacobian. Must have at least one dimension.
|
||||
See Notes regarding the dimensionality and vectorization.
|
||||
atol, rtol : float, optional
|
||||
Absolute and relative tolerances for the stopping condition: iteration
|
||||
will stop for each element of the Jacobian when
|
||||
``res.error < atol + rtol * abs(res.df)``. The default `atol` is the
|
||||
smallest normal number of the appropriate dtype, and the default `rtol`
|
||||
is the square root of the precision of the appropriate dtype.
|
||||
order : int, default: 8
|
||||
The (positive integer) order of the finite difference formula to be
|
||||
used. Odd integers will be rounded up to the next even integer.
|
||||
initial_step : float, default: 0.5
|
||||
The (absolute) initial step size for the finite difference derivative
|
||||
approximation.
|
||||
step_factor : float, default: 2.0
|
||||
The factor by which the step size is *reduced* in each iteration; i.e.
|
||||
the step size in iteration 1 is ``initial_step/step_factor``. If
|
||||
``step_factor < 1``, subsequent steps will be greater than the initial
|
||||
step; this may be useful if steps smaller than some threshold are
|
||||
undesirable (e.g. due to subtractive cancellation error).
|
||||
maxiter : int, default: 10
|
||||
The maximum number of iterations of the algorithm to perform.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : _RichResult
|
||||
An instance of `scipy._lib._util._RichResult` with the following
|
||||
attributes.
|
||||
|
||||
success : bool array
|
||||
``True`` when the algorithm terminated successfully (status ``0``).
|
||||
status : int array
|
||||
An integer representing the exit status of the algorithm.
|
||||
``0`` : The algorithm converged to the specified tolerances.
|
||||
``-1`` : The error estimate increased, so iteration was terminated.
|
||||
``-2`` : The maximum number of iterations was reached.
|
||||
``-3`` : A non-finite value was encountered.
|
||||
``-4`` : Iteration was terminated by `callback`.
|
||||
``1`` : The algorithm is proceeding normally (in `callback` only).
|
||||
df : float array
|
||||
The Jacobian of `func` at `x`, if the algorithm terminated
|
||||
successfully.
|
||||
error : float array
|
||||
An estimate of the error: the magnitude of the difference between
|
||||
the current estimate of the derivative and the estimate in the
|
||||
previous iteration.
|
||||
nit : int array
|
||||
The number of iterations performed.
|
||||
nfev : int array
|
||||
The number of points at which `func` was evaluated.
|
||||
x : float array
|
||||
The value at which the derivative of `func` was evaluated.
|
||||
|
||||
See Also
|
||||
--------
|
||||
_differentiate
|
||||
|
||||
Notes
|
||||
-----
|
||||
Suppose we wish to evaluate the Jacobian of a function
|
||||
:math:`f: \mathbf{R^m} \rightarrow \mathbf{R^n}`, and assign to variables
|
||||
``m`` and ``n`` the positive integer values of :math:`m` and :math:`n`,
|
||||
respectively. If we wish to evaluate the Jacobian at a single point,
|
||||
then:
|
||||
|
||||
- argument `x` must be an array of shape ``(m,)``
|
||||
- argument `func` must be vectorized to accept an array of shape ``(m, p)``.
|
||||
The first axis represents the :math:`m` inputs of :math:`f`; the second
|
||||
is for evaluating the function at multiple points in a single call.
|
||||
- argument `func` must return an array of shape ``(n, p)``. The first
|
||||
axis represents the :math:`n` outputs of :math:`f`; the second
|
||||
is for the result of evaluating the function at multiple points.
|
||||
- attribute ``df`` of the result object will be an array of shape ``(n, m)``,
|
||||
the Jacobian.
|
||||
|
||||
This function is also vectorized in the sense that the Jacobian can be
|
||||
evaluated at ``k`` points in a single call. In this case, `x` would be an
|
||||
array of shape ``(m, k)``, `func` would accept an array of shape
|
||||
``(m, k, p)`` and return an array of shape ``(n, k, p)``, and the ``df``
|
||||
attribute of the result would have shape ``(n, m, k)``.
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Jacobian matrix and determinant, *Wikipedia*,
|
||||
https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant
|
||||
|
||||
Examples
|
||||
--------
|
||||
The Rosenbrock function maps from :math:`\mathbf{R}^m \righarrow \mathbf{R}`;
|
||||
the SciPy implementation `scipy.optimize.rosen` is vectorized to accept an
|
||||
array of shape ``(m, p)`` and return an array of shape ``m``. Suppose we wish
|
||||
to evaluate the Jacobian (AKA the gradient because the function returns a scalar)
|
||||
at ``[0.5, 0.5, 0.5]``.
|
||||
|
||||
>>> import numpy as np
|
||||
>>> from scipy.optimize._differentiate import _jacobian as jacobian
|
||||
>>> from scipy.optimize import rosen, rosen_der
|
||||
>>> m = 3
|
||||
>>> x = np.full(m, 0.5)
|
||||
>>> res = jacobian(rosen, x)
|
||||
>>> ref = rosen_der(x) # reference value of the gradient
|
||||
>>> res.df, ref
|
||||
(array([-51., -1., 50.]), array([-51., -1., 50.]))
|
||||
|
||||
As an example of a function with multiple outputs, consider Example 4
|
||||
from [1]_.
|
||||
|
||||
>>> def f(x):
|
||||
... x1, x2, x3 = x ...
|
||||
... return [x1, 5*x3, 4*x2**2 - 2*x3, x3*np.sin(x1)]
|
||||
|
||||
The true Jacobian is given by:
|
||||
|
||||
>>> def df(x):
|
||||
... x1, x2, x3 = x
|
||||
... one = np.ones_like(x1)
|
||||
... return [[one, 0*one, 0*one],
|
||||
... [0*one, 0*one, 5*one],
|
||||
... [0*one, 8*x2, -2*one],
|
||||
... [x3*np.cos(x1), 0*one, np.sin(x1)]]
|
||||
|
||||
Evaluate the Jacobian at an arbitrary point.
|
||||
|
||||
>>> rng = np.random.default_rng(389252938452)
|
||||
>>> x = rng.random(size=3)
|
||||
>>> res = jacobian(f, x)
|
||||
>>> ref = df(x)
|
||||
>>> res.df.shape == (4, 3)
|
||||
True
|
||||
>>> np.allclose(res.df, ref)
|
||||
True
|
||||
|
||||
Evaluate the Jacobian at 10 arbitrary points in a single call.
|
||||
|
||||
>>> x = rng.random(size=(3, 10))
|
||||
>>> res = jacobian(f, x)
|
||||
>>> ref = df(x)
|
||||
>>> res.df.shape == (4, 3, 10)
|
||||
True
|
||||
>>> np.allclose(res.df, ref)
|
||||
True
|
||||
|
||||
"""
|
||||
x = np.asarray(x)
|
||||
int_dtype = np.issubdtype(x.dtype, np.integer)
|
||||
x0 = np.asarray(x, dtype=float) if int_dtype else x
|
||||
|
||||
if x0.ndim < 1:
|
||||
message = "Argument `x` must be at least 1-D."
|
||||
raise ValueError(message)
|
||||
|
||||
m = x0.shape[0]
|
||||
i = np.arange(m)
|
||||
|
||||
def wrapped(x):
|
||||
p = () if x.ndim == x0.ndim else (x.shape[-1],) # number of abscissae
|
||||
new_dims = (1,) if x.ndim == x0.ndim else (1, -1)
|
||||
new_shape = (m, m) + x0.shape[1:] + p
|
||||
xph = np.expand_dims(x0, new_dims)
|
||||
xph = np.broadcast_to(xph, new_shape).copy()
|
||||
xph[i, i] = x
|
||||
return func(xph)
|
||||
|
||||
res = _differentiate(wrapped, x, atol=atol, rtol=rtol,
|
||||
maxiter=maxiter, order=order, initial_step=initial_step,
|
||||
step_factor=step_factor, preserve_shape=True)
|
||||
del res.x # the user knows `x`, and the way it gets broadcasted is meaningless here
|
||||
return res
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,278 @@
|
||||
from __future__ import annotations
|
||||
from typing import ( # noqa: UP035
|
||||
Any, Callable, Iterable, TYPE_CHECKING
|
||||
)
|
||||
|
||||
import numpy as np
|
||||
from scipy.optimize import OptimizeResult
|
||||
from ._constraints import old_bound_to_new, Bounds
|
||||
from ._direct import direct as _direct # type: ignore
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import numpy.typing as npt
|
||||
|
||||
__all__ = ['direct']
|
||||
|
||||
ERROR_MESSAGES = (
|
||||
"Number of function evaluations done is larger than maxfun={}",
|
||||
"Number of iterations is larger than maxiter={}",
|
||||
"u[i] < l[i] for some i",
|
||||
"maxfun is too large",
|
||||
"Initialization failed",
|
||||
"There was an error in the creation of the sample points",
|
||||
"An error occurred while the function was sampled",
|
||||
"Maximum number of levels has been reached.",
|
||||
"Forced stop",
|
||||
"Invalid arguments",
|
||||
"Out of memory",
|
||||
)
|
||||
|
||||
SUCCESS_MESSAGES = (
|
||||
("The best function value found is within a relative error={} "
|
||||
"of the (known) global optimum f_min"),
|
||||
("The volume of the hyperrectangle containing the lowest function value "
|
||||
"found is below vol_tol={}"),
|
||||
("The side length measure of the hyperrectangle containing the lowest "
|
||||
"function value found is below len_tol={}"),
|
||||
)
|
||||
|
||||
|
||||
def direct(
|
||||
func: Callable[[npt.ArrayLike, tuple[Any]], float],
|
||||
bounds: Iterable | Bounds,
|
||||
*,
|
||||
args: tuple = (),
|
||||
eps: float = 1e-4,
|
||||
maxfun: int | None = None,
|
||||
maxiter: int = 1000,
|
||||
locally_biased: bool = True,
|
||||
f_min: float = -np.inf,
|
||||
f_min_rtol: float = 1e-4,
|
||||
vol_tol: float = 1e-16,
|
||||
len_tol: float = 1e-6,
|
||||
callback: Callable[[npt.ArrayLike], None] | None = None
|
||||
) -> OptimizeResult:
|
||||
"""
|
||||
Finds the global minimum of a function using the
|
||||
DIRECT algorithm.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
The objective function to be minimized.
|
||||
``func(x, *args) -> float``
|
||||
where ``x`` is an 1-D array with shape (n,) and ``args`` is a tuple of
|
||||
the fixed parameters needed to completely specify the function.
|
||||
bounds : sequence or `Bounds`
|
||||
Bounds for variables. There are two ways to specify the bounds:
|
||||
|
||||
1. Instance of `Bounds` class.
|
||||
2. ``(min, max)`` pairs for each element in ``x``.
|
||||
|
||||
args : tuple, optional
|
||||
Any additional fixed parameters needed to
|
||||
completely specify the objective function.
|
||||
eps : float, optional
|
||||
Minimal required difference of the objective function values
|
||||
between the current best hyperrectangle and the next potentially
|
||||
optimal hyperrectangle to be divided. In consequence, `eps` serves as a
|
||||
tradeoff between local and global search: the smaller, the more local
|
||||
the search becomes. Default is 1e-4.
|
||||
maxfun : int or None, optional
|
||||
Approximate upper bound on objective function evaluations.
|
||||
If `None`, will be automatically set to ``1000 * N`` where ``N``
|
||||
represents the number of dimensions. Will be capped if necessary to
|
||||
limit DIRECT's RAM usage to app. 1GiB. This will only occur for very
|
||||
high dimensional problems and excessive `max_fun`. Default is `None`.
|
||||
maxiter : int, optional
|
||||
Maximum number of iterations. Default is 1000.
|
||||
locally_biased : bool, optional
|
||||
If `True` (default), use the locally biased variant of the
|
||||
algorithm known as DIRECT_L. If `False`, use the original unbiased
|
||||
DIRECT algorithm. For hard problems with many local minima,
|
||||
`False` is recommended.
|
||||
f_min : float, optional
|
||||
Function value of the global optimum. Set this value only if the
|
||||
global optimum is known. Default is ``-np.inf``, so that this
|
||||
termination criterion is deactivated.
|
||||
f_min_rtol : float, optional
|
||||
Terminate the optimization once the relative error between the
|
||||
current best minimum `f` and the supplied global minimum `f_min`
|
||||
is smaller than `f_min_rtol`. This parameter is only used if
|
||||
`f_min` is also set. Must lie between 0 and 1. Default is 1e-4.
|
||||
vol_tol : float, optional
|
||||
Terminate the optimization once the volume of the hyperrectangle
|
||||
containing the lowest function value is smaller than `vol_tol`
|
||||
of the complete search space. Must lie between 0 and 1.
|
||||
Default is 1e-16.
|
||||
len_tol : float, optional
|
||||
If `locally_biased=True`, terminate the optimization once half of
|
||||
the normalized maximal side length of the hyperrectangle containing
|
||||
the lowest function value is smaller than `len_tol`.
|
||||
If `locally_biased=False`, terminate the optimization once half of
|
||||
the normalized diagonal of the hyperrectangle containing the lowest
|
||||
function value is smaller than `len_tol`. Must lie between 0 and 1.
|
||||
Default is 1e-6.
|
||||
callback : callable, optional
|
||||
A callback function with signature ``callback(xk)`` where ``xk``
|
||||
represents the best function value found so far.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : OptimizeResult
|
||||
The optimization result represented as a ``OptimizeResult`` object.
|
||||
Important attributes are: ``x`` the solution array, ``success`` a
|
||||
Boolean flag indicating if the optimizer exited successfully and
|
||||
``message`` which describes the cause of the termination. See
|
||||
`OptimizeResult` for a description of other attributes.
|
||||
|
||||
Notes
|
||||
-----
|
||||
DIviding RECTangles (DIRECT) is a deterministic global
|
||||
optimization algorithm capable of minimizing a black box function with
|
||||
its variables subject to lower and upper bound constraints by sampling
|
||||
potential solutions in the search space [1]_. The algorithm starts by
|
||||
normalising the search space to an n-dimensional unit hypercube.
|
||||
It samples the function at the center of this hypercube and at 2n
|
||||
(n is the number of variables) more points, 2 in each coordinate
|
||||
direction. Using these function values, DIRECT then divides the
|
||||
domain into hyperrectangles, each having exactly one of the sampling
|
||||
points as its center. In each iteration, DIRECT chooses, using the `eps`
|
||||
parameter which defaults to 1e-4, some of the existing hyperrectangles
|
||||
to be further divided. This division process continues until either the
|
||||
maximum number of iterations or maximum function evaluations allowed
|
||||
are exceeded, or the hyperrectangle containing the minimal value found
|
||||
so far becomes small enough. If `f_min` is specified, the optimization
|
||||
will stop once this function value is reached within a relative tolerance.
|
||||
The locally biased variant of DIRECT (originally called DIRECT_L) [2]_ is
|
||||
used by default. It makes the search more locally biased and more
|
||||
efficient for cases with only a few local minima.
|
||||
|
||||
A note about termination criteria: `vol_tol` refers to the volume of the
|
||||
hyperrectangle containing the lowest function value found so far. This
|
||||
volume decreases exponentially with increasing dimensionality of the
|
||||
problem. Therefore `vol_tol` should be decreased to avoid premature
|
||||
termination of the algorithm for higher dimensions. This does not hold
|
||||
for `len_tol`: it refers either to half of the maximal side length
|
||||
(for ``locally_biased=True``) or half of the diagonal of the
|
||||
hyperrectangle (for ``locally_biased=False``).
|
||||
|
||||
This code is based on the DIRECT 2.0.4 Fortran code by Gablonsky et al. at
|
||||
https://ctk.math.ncsu.edu/SOFTWARE/DIRECTv204.tar.gz .
|
||||
This original version was initially converted via f2c and then cleaned up
|
||||
and reorganized by Steven G. Johnson, August 2007, for the NLopt project.
|
||||
The `direct` function wraps the C implementation.
|
||||
|
||||
.. versionadded:: 1.9.0
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Jones, D.R., Perttunen, C.D. & Stuckman, B.E. Lipschitzian
|
||||
optimization without the Lipschitz constant. J Optim Theory Appl
|
||||
79, 157-181 (1993).
|
||||
.. [2] Gablonsky, J., Kelley, C. A Locally-Biased form of the DIRECT
|
||||
Algorithm. Journal of Global Optimization 21, 27-37 (2001).
|
||||
|
||||
Examples
|
||||
--------
|
||||
The following example is a 2-D problem with four local minima: minimizing
|
||||
the Styblinski-Tang function
|
||||
(https://en.wikipedia.org/wiki/Test_functions_for_optimization).
|
||||
|
||||
>>> from scipy.optimize import direct, Bounds
|
||||
>>> def styblinski_tang(pos):
|
||||
... x, y = pos
|
||||
... return 0.5 * (x**4 - 16*x**2 + 5*x + y**4 - 16*y**2 + 5*y)
|
||||
>>> bounds = Bounds([-4., -4.], [4., 4.])
|
||||
>>> result = direct(styblinski_tang, bounds)
|
||||
>>> result.x, result.fun, result.nfev
|
||||
array([-2.90321597, -2.90321597]), -78.3323279095383, 2011
|
||||
|
||||
The correct global minimum was found but with a huge number of function
|
||||
evaluations (2011). Loosening the termination tolerances `vol_tol` and
|
||||
`len_tol` can be used to stop DIRECT earlier.
|
||||
|
||||
>>> result = direct(styblinski_tang, bounds, len_tol=1e-3)
|
||||
>>> result.x, result.fun, result.nfev
|
||||
array([-2.9044353, -2.9044353]), -78.33230330754142, 207
|
||||
|
||||
"""
|
||||
# convert bounds to new Bounds class if necessary
|
||||
if not isinstance(bounds, Bounds):
|
||||
if isinstance(bounds, list) or isinstance(bounds, tuple):
|
||||
lb, ub = old_bound_to_new(bounds)
|
||||
bounds = Bounds(lb, ub)
|
||||
else:
|
||||
message = ("bounds must be a sequence or "
|
||||
"instance of Bounds class")
|
||||
raise ValueError(message)
|
||||
|
||||
lb = np.ascontiguousarray(bounds.lb, dtype=np.float64)
|
||||
ub = np.ascontiguousarray(bounds.ub, dtype=np.float64)
|
||||
|
||||
# validate bounds
|
||||
# check that lower bounds are smaller than upper bounds
|
||||
if not np.all(lb < ub):
|
||||
raise ValueError('Bounds are not consistent min < max')
|
||||
# check for infs
|
||||
if (np.any(np.isinf(lb)) or np.any(np.isinf(ub))):
|
||||
raise ValueError("Bounds must not be inf.")
|
||||
|
||||
# validate tolerances
|
||||
if (vol_tol < 0 or vol_tol > 1):
|
||||
raise ValueError("vol_tol must be between 0 and 1.")
|
||||
if (len_tol < 0 or len_tol > 1):
|
||||
raise ValueError("len_tol must be between 0 and 1.")
|
||||
if (f_min_rtol < 0 or f_min_rtol > 1):
|
||||
raise ValueError("f_min_rtol must be between 0 and 1.")
|
||||
|
||||
# validate maxfun and maxiter
|
||||
if maxfun is None:
|
||||
maxfun = 1000 * lb.shape[0]
|
||||
if not isinstance(maxfun, int):
|
||||
raise ValueError("maxfun must be of type int.")
|
||||
if maxfun < 0:
|
||||
raise ValueError("maxfun must be > 0.")
|
||||
if not isinstance(maxiter, int):
|
||||
raise ValueError("maxiter must be of type int.")
|
||||
if maxiter < 0:
|
||||
raise ValueError("maxiter must be > 0.")
|
||||
|
||||
# validate boolean parameters
|
||||
if not isinstance(locally_biased, bool):
|
||||
raise ValueError("locally_biased must be True or False.")
|
||||
|
||||
def _func_wrap(x, args=None):
|
||||
x = np.asarray(x)
|
||||
if args is None:
|
||||
f = func(x)
|
||||
else:
|
||||
f = func(x, *args)
|
||||
# always return a float
|
||||
return np.asarray(f).item()
|
||||
|
||||
# TODO: fix disp argument
|
||||
x, fun, ret_code, nfev, nit = _direct(
|
||||
_func_wrap,
|
||||
np.asarray(lb), np.asarray(ub),
|
||||
args,
|
||||
False, eps, maxfun, maxiter,
|
||||
locally_biased,
|
||||
f_min, f_min_rtol,
|
||||
vol_tol, len_tol, callback
|
||||
)
|
||||
|
||||
format_val = (maxfun, maxiter, f_min_rtol, vol_tol, len_tol)
|
||||
if ret_code > 2:
|
||||
message = SUCCESS_MESSAGES[ret_code - 3].format(
|
||||
format_val[ret_code - 1])
|
||||
elif 0 < ret_code <= 2:
|
||||
message = ERROR_MESSAGES[ret_code - 1].format(format_val[ret_code - 1])
|
||||
elif 0 > ret_code > -100:
|
||||
message = ERROR_MESSAGES[abs(ret_code) + 1]
|
||||
else:
|
||||
message = ERROR_MESSAGES[ret_code + 99]
|
||||
|
||||
return OptimizeResult(x=np.asarray(x), fun=fun, status=ret_code,
|
||||
success=ret_code > 2, message=message,
|
||||
nfev=nfev, nit=nit)
|
||||
@@ -0,0 +1,732 @@
|
||||
# Dual Annealing implementation.
|
||||
# Copyright (c) 2018 Sylvain Gubian <sylvain.gubian@pmi.com>,
|
||||
# Yang Xiang <yang.xiang@pmi.com>
|
||||
# Author: Sylvain Gubian, Yang Xiang, PMP S.A.
|
||||
|
||||
"""
|
||||
A Dual Annealing global optimization algorithm
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
from scipy.optimize import OptimizeResult
|
||||
from scipy.optimize import minimize, Bounds
|
||||
from scipy.special import gammaln
|
||||
from scipy._lib._util import check_random_state
|
||||
from scipy.optimize._constraints import new_bounds_to_old
|
||||
|
||||
__all__ = ['dual_annealing']
|
||||
|
||||
|
||||
class VisitingDistribution:
|
||||
"""
|
||||
Class used to generate new coordinates based on the distorted
|
||||
Cauchy-Lorentz distribution. Depending on the steps within the strategy
|
||||
chain, the class implements the strategy for generating new location
|
||||
changes.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
lb : array_like
|
||||
A 1-D NumPy ndarray containing lower bounds of the generated
|
||||
components. Neither NaN or inf are allowed.
|
||||
ub : array_like
|
||||
A 1-D NumPy ndarray containing upper bounds for the generated
|
||||
components. Neither NaN or inf are allowed.
|
||||
visiting_param : float
|
||||
Parameter for visiting distribution. Default value is 2.62.
|
||||
Higher values give the visiting distribution a heavier tail, this
|
||||
makes the algorithm jump to a more distant region.
|
||||
The value range is (1, 3]. Its value is fixed for the life of the
|
||||
object.
|
||||
rand_gen : {`~numpy.random.RandomState`, `~numpy.random.Generator`}
|
||||
A `~numpy.random.RandomState`, `~numpy.random.Generator` object
|
||||
for using the current state of the created random generator container.
|
||||
|
||||
"""
|
||||
TAIL_LIMIT = 1.e8
|
||||
MIN_VISIT_BOUND = 1.e-10
|
||||
|
||||
def __init__(self, lb, ub, visiting_param, rand_gen):
|
||||
# if you wish to make _visiting_param adjustable during the life of
|
||||
# the object then _factor2, _factor3, _factor5, _d1, _factor6 will
|
||||
# have to be dynamically calculated in `visit_fn`. They're factored
|
||||
# out here so they don't need to be recalculated all the time.
|
||||
self._visiting_param = visiting_param
|
||||
self.rand_gen = rand_gen
|
||||
self.lower = lb
|
||||
self.upper = ub
|
||||
self.bound_range = ub - lb
|
||||
|
||||
# these are invariant numbers unless visiting_param changes
|
||||
self._factor2 = np.exp((4.0 - self._visiting_param) * np.log(
|
||||
self._visiting_param - 1.0))
|
||||
self._factor3 = np.exp((2.0 - self._visiting_param) * np.log(2.0)
|
||||
/ (self._visiting_param - 1.0))
|
||||
self._factor4_p = np.sqrt(np.pi) * self._factor2 / (self._factor3 * (
|
||||
3.0 - self._visiting_param))
|
||||
|
||||
self._factor5 = 1.0 / (self._visiting_param - 1.0) - 0.5
|
||||
self._d1 = 2.0 - self._factor5
|
||||
self._factor6 = np.pi * (1.0 - self._factor5) / np.sin(
|
||||
np.pi * (1.0 - self._factor5)) / np.exp(gammaln(self._d1))
|
||||
|
||||
def visiting(self, x, step, temperature):
|
||||
""" Based on the step in the strategy chain, new coordinates are
|
||||
generated by changing all components is the same time or only
|
||||
one of them, the new values are computed with visit_fn method
|
||||
"""
|
||||
dim = x.size
|
||||
if step < dim:
|
||||
# Changing all coordinates with a new visiting value
|
||||
visits = self.visit_fn(temperature, dim)
|
||||
upper_sample, lower_sample = self.rand_gen.uniform(size=2)
|
||||
visits[visits > self.TAIL_LIMIT] = self.TAIL_LIMIT * upper_sample
|
||||
visits[visits < -self.TAIL_LIMIT] = -self.TAIL_LIMIT * lower_sample
|
||||
x_visit = visits + x
|
||||
a = x_visit - self.lower
|
||||
b = np.fmod(a, self.bound_range) + self.bound_range
|
||||
x_visit = np.fmod(b, self.bound_range) + self.lower
|
||||
x_visit[np.fabs(
|
||||
x_visit - self.lower) < self.MIN_VISIT_BOUND] += 1.e-10
|
||||
else:
|
||||
# Changing only one coordinate at a time based on strategy
|
||||
# chain step
|
||||
x_visit = np.copy(x)
|
||||
visit = self.visit_fn(temperature, 1)[0]
|
||||
if visit > self.TAIL_LIMIT:
|
||||
visit = self.TAIL_LIMIT * self.rand_gen.uniform()
|
||||
elif visit < -self.TAIL_LIMIT:
|
||||
visit = -self.TAIL_LIMIT * self.rand_gen.uniform()
|
||||
index = step - dim
|
||||
x_visit[index] = visit + x[index]
|
||||
a = x_visit[index] - self.lower[index]
|
||||
b = np.fmod(a, self.bound_range[index]) + self.bound_range[index]
|
||||
x_visit[index] = np.fmod(b, self.bound_range[
|
||||
index]) + self.lower[index]
|
||||
if np.fabs(x_visit[index] - self.lower[
|
||||
index]) < self.MIN_VISIT_BOUND:
|
||||
x_visit[index] += self.MIN_VISIT_BOUND
|
||||
return x_visit
|
||||
|
||||
def visit_fn(self, temperature, dim):
|
||||
""" Formula Visita from p. 405 of reference [2] """
|
||||
x, y = self.rand_gen.normal(size=(dim, 2)).T
|
||||
|
||||
factor1 = np.exp(np.log(temperature) / (self._visiting_param - 1.0))
|
||||
factor4 = self._factor4_p * factor1
|
||||
|
||||
# sigmax
|
||||
x *= np.exp(-(self._visiting_param - 1.0) * np.log(
|
||||
self._factor6 / factor4) / (3.0 - self._visiting_param))
|
||||
|
||||
den = np.exp((self._visiting_param - 1.0) * np.log(np.fabs(y)) /
|
||||
(3.0 - self._visiting_param))
|
||||
|
||||
return x / den
|
||||
|
||||
|
||||
class EnergyState:
|
||||
"""
|
||||
Class used to record the energy state. At any time, it knows what is the
|
||||
currently used coordinates and the most recent best location.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
lower : array_like
|
||||
A 1-D NumPy ndarray containing lower bounds for generating an initial
|
||||
random components in the `reset` method.
|
||||
upper : array_like
|
||||
A 1-D NumPy ndarray containing upper bounds for generating an initial
|
||||
random components in the `reset` method
|
||||
components. Neither NaN or inf are allowed.
|
||||
callback : callable, ``callback(x, f, context)``, optional
|
||||
A callback function which will be called for all minima found.
|
||||
``x`` and ``f`` are the coordinates and function value of the
|
||||
latest minimum found, and `context` has value in [0, 1, 2]
|
||||
"""
|
||||
# Maximum number of trials for generating a valid starting point
|
||||
MAX_REINIT_COUNT = 1000
|
||||
|
||||
def __init__(self, lower, upper, callback=None):
|
||||
self.ebest = None
|
||||
self.current_energy = None
|
||||
self.current_location = None
|
||||
self.xbest = None
|
||||
self.lower = lower
|
||||
self.upper = upper
|
||||
self.callback = callback
|
||||
|
||||
def reset(self, func_wrapper, rand_gen, x0=None):
|
||||
"""
|
||||
Initialize current location is the search domain. If `x0` is not
|
||||
provided, a random location within the bounds is generated.
|
||||
"""
|
||||
if x0 is None:
|
||||
self.current_location = rand_gen.uniform(self.lower, self.upper,
|
||||
size=len(self.lower))
|
||||
else:
|
||||
self.current_location = np.copy(x0)
|
||||
init_error = True
|
||||
reinit_counter = 0
|
||||
while init_error:
|
||||
self.current_energy = func_wrapper.fun(self.current_location)
|
||||
if self.current_energy is None:
|
||||
raise ValueError('Objective function is returning None')
|
||||
if (not np.isfinite(self.current_energy) or np.isnan(
|
||||
self.current_energy)):
|
||||
if reinit_counter >= EnergyState.MAX_REINIT_COUNT:
|
||||
init_error = False
|
||||
message = (
|
||||
'Stopping algorithm because function '
|
||||
'create NaN or (+/-) infinity values even with '
|
||||
'trying new random parameters'
|
||||
)
|
||||
raise ValueError(message)
|
||||
self.current_location = rand_gen.uniform(self.lower,
|
||||
self.upper,
|
||||
size=self.lower.size)
|
||||
reinit_counter += 1
|
||||
else:
|
||||
init_error = False
|
||||
# If first time reset, initialize ebest and xbest
|
||||
if self.ebest is None and self.xbest is None:
|
||||
self.ebest = self.current_energy
|
||||
self.xbest = np.copy(self.current_location)
|
||||
# Otherwise, we keep them in case of reannealing reset
|
||||
|
||||
def update_best(self, e, x, context):
|
||||
self.ebest = e
|
||||
self.xbest = np.copy(x)
|
||||
if self.callback is not None:
|
||||
val = self.callback(x, e, context)
|
||||
if val is not None:
|
||||
if val:
|
||||
return ('Callback function requested to stop early by '
|
||||
'returning True')
|
||||
|
||||
def update_current(self, e, x):
|
||||
self.current_energy = e
|
||||
self.current_location = np.copy(x)
|
||||
|
||||
|
||||
class StrategyChain:
|
||||
"""
|
||||
Class that implements within a Markov chain the strategy for location
|
||||
acceptance and local search decision making.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
acceptance_param : float
|
||||
Parameter for acceptance distribution. It is used to control the
|
||||
probability of acceptance. The lower the acceptance parameter, the
|
||||
smaller the probability of acceptance. Default value is -5.0 with
|
||||
a range (-1e4, -5].
|
||||
visit_dist : VisitingDistribution
|
||||
Instance of `VisitingDistribution` class.
|
||||
func_wrapper : ObjectiveFunWrapper
|
||||
Instance of `ObjectiveFunWrapper` class.
|
||||
minimizer_wrapper: LocalSearchWrapper
|
||||
Instance of `LocalSearchWrapper` class.
|
||||
rand_gen : {None, int, `numpy.random.Generator`,
|
||||
`numpy.random.RandomState`}, optional
|
||||
|
||||
If `seed` is None (or `np.random`), the `numpy.random.RandomState`
|
||||
singleton is used.
|
||||
If `seed` is an int, a new ``RandomState`` instance is used,
|
||||
seeded with `seed`.
|
||||
If `seed` is already a ``Generator`` or ``RandomState`` instance then
|
||||
that instance is used.
|
||||
energy_state: EnergyState
|
||||
Instance of `EnergyState` class.
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self, acceptance_param, visit_dist, func_wrapper,
|
||||
minimizer_wrapper, rand_gen, energy_state):
|
||||
# Local strategy chain minimum energy and location
|
||||
self.emin = energy_state.current_energy
|
||||
self.xmin = np.array(energy_state.current_location)
|
||||
# Global optimizer state
|
||||
self.energy_state = energy_state
|
||||
# Acceptance parameter
|
||||
self.acceptance_param = acceptance_param
|
||||
# Visiting distribution instance
|
||||
self.visit_dist = visit_dist
|
||||
# Wrapper to objective function
|
||||
self.func_wrapper = func_wrapper
|
||||
# Wrapper to the local minimizer
|
||||
self.minimizer_wrapper = minimizer_wrapper
|
||||
self.not_improved_idx = 0
|
||||
self.not_improved_max_idx = 1000
|
||||
self._rand_gen = rand_gen
|
||||
self.temperature_step = 0
|
||||
self.K = 100 * len(energy_state.current_location)
|
||||
|
||||
def accept_reject(self, j, e, x_visit):
|
||||
r = self._rand_gen.uniform()
|
||||
pqv_temp = 1.0 - ((1.0 - self.acceptance_param) *
|
||||
(e - self.energy_state.current_energy) / self.temperature_step)
|
||||
if pqv_temp <= 0.:
|
||||
pqv = 0.
|
||||
else:
|
||||
pqv = np.exp(np.log(pqv_temp) / (
|
||||
1. - self.acceptance_param))
|
||||
|
||||
if r <= pqv:
|
||||
# We accept the new location and update state
|
||||
self.energy_state.update_current(e, x_visit)
|
||||
self.xmin = np.copy(self.energy_state.current_location)
|
||||
|
||||
# No improvement for a long time
|
||||
if self.not_improved_idx >= self.not_improved_max_idx:
|
||||
if j == 0 or self.energy_state.current_energy < self.emin:
|
||||
self.emin = self.energy_state.current_energy
|
||||
self.xmin = np.copy(self.energy_state.current_location)
|
||||
|
||||
def run(self, step, temperature):
|
||||
self.temperature_step = temperature / float(step + 1)
|
||||
self.not_improved_idx += 1
|
||||
for j in range(self.energy_state.current_location.size * 2):
|
||||
if j == 0:
|
||||
if step == 0:
|
||||
self.energy_state_improved = True
|
||||
else:
|
||||
self.energy_state_improved = False
|
||||
x_visit = self.visit_dist.visiting(
|
||||
self.energy_state.current_location, j, temperature)
|
||||
# Calling the objective function
|
||||
e = self.func_wrapper.fun(x_visit)
|
||||
if e < self.energy_state.current_energy:
|
||||
# We have got a better energy value
|
||||
self.energy_state.update_current(e, x_visit)
|
||||
if e < self.energy_state.ebest:
|
||||
val = self.energy_state.update_best(e, x_visit, 0)
|
||||
if val is not None:
|
||||
if val:
|
||||
return val
|
||||
self.energy_state_improved = True
|
||||
self.not_improved_idx = 0
|
||||
else:
|
||||
# We have not improved but do we accept the new location?
|
||||
self.accept_reject(j, e, x_visit)
|
||||
if self.func_wrapper.nfev >= self.func_wrapper.maxfun:
|
||||
return ('Maximum number of function call reached '
|
||||
'during annealing')
|
||||
# End of StrategyChain loop
|
||||
|
||||
def local_search(self):
|
||||
# Decision making for performing a local search
|
||||
# based on strategy chain results
|
||||
# If energy has been improved or no improvement since too long,
|
||||
# performing a local search with the best strategy chain location
|
||||
if self.energy_state_improved:
|
||||
# Global energy has improved, let's see if LS improves further
|
||||
e, x = self.minimizer_wrapper.local_search(self.energy_state.xbest,
|
||||
self.energy_state.ebest)
|
||||
if e < self.energy_state.ebest:
|
||||
self.not_improved_idx = 0
|
||||
val = self.energy_state.update_best(e, x, 1)
|
||||
if val is not None:
|
||||
if val:
|
||||
return val
|
||||
self.energy_state.update_current(e, x)
|
||||
if self.func_wrapper.nfev >= self.func_wrapper.maxfun:
|
||||
return ('Maximum number of function call reached '
|
||||
'during local search')
|
||||
# Check probability of a need to perform a LS even if no improvement
|
||||
do_ls = False
|
||||
if self.K < 90 * len(self.energy_state.current_location):
|
||||
pls = np.exp(self.K * (
|
||||
self.energy_state.ebest - self.energy_state.current_energy) /
|
||||
self.temperature_step)
|
||||
if pls >= self._rand_gen.uniform():
|
||||
do_ls = True
|
||||
# Global energy not improved, let's see what LS gives
|
||||
# on the best strategy chain location
|
||||
if self.not_improved_idx >= self.not_improved_max_idx:
|
||||
do_ls = True
|
||||
if do_ls:
|
||||
e, x = self.minimizer_wrapper.local_search(self.xmin, self.emin)
|
||||
self.xmin = np.copy(x)
|
||||
self.emin = e
|
||||
self.not_improved_idx = 0
|
||||
self.not_improved_max_idx = self.energy_state.current_location.size
|
||||
if e < self.energy_state.ebest:
|
||||
val = self.energy_state.update_best(
|
||||
self.emin, self.xmin, 2)
|
||||
if val is not None:
|
||||
if val:
|
||||
return val
|
||||
self.energy_state.update_current(e, x)
|
||||
if self.func_wrapper.nfev >= self.func_wrapper.maxfun:
|
||||
return ('Maximum number of function call reached '
|
||||
'during dual annealing')
|
||||
|
||||
|
||||
class ObjectiveFunWrapper:
|
||||
|
||||
def __init__(self, func, maxfun=1e7, *args):
|
||||
self.func = func
|
||||
self.args = args
|
||||
# Number of objective function evaluations
|
||||
self.nfev = 0
|
||||
# Number of gradient function evaluation if used
|
||||
self.ngev = 0
|
||||
# Number of hessian of the objective function if used
|
||||
self.nhev = 0
|
||||
self.maxfun = maxfun
|
||||
|
||||
def fun(self, x):
|
||||
self.nfev += 1
|
||||
return self.func(x, *self.args)
|
||||
|
||||
|
||||
class LocalSearchWrapper:
|
||||
"""
|
||||
Class used to wrap around the minimizer used for local search
|
||||
Default local minimizer is SciPy minimizer L-BFGS-B
|
||||
"""
|
||||
|
||||
LS_MAXITER_RATIO = 6
|
||||
LS_MAXITER_MIN = 100
|
||||
LS_MAXITER_MAX = 1000
|
||||
|
||||
def __init__(self, search_bounds, func_wrapper, *args, **kwargs):
|
||||
self.func_wrapper = func_wrapper
|
||||
self.kwargs = kwargs
|
||||
self.jac = self.kwargs.get('jac', None)
|
||||
self.hess = self.kwargs.get('hess', None)
|
||||
self.hessp = self.kwargs.get('hessp', None)
|
||||
self.kwargs.pop("args", None)
|
||||
self.minimizer = minimize
|
||||
bounds_list = list(zip(*search_bounds))
|
||||
self.lower = np.array(bounds_list[0])
|
||||
self.upper = np.array(bounds_list[1])
|
||||
|
||||
# If no minimizer specified, use SciPy minimize with 'L-BFGS-B' method
|
||||
if not self.kwargs:
|
||||
n = len(self.lower)
|
||||
ls_max_iter = min(max(n * self.LS_MAXITER_RATIO,
|
||||
self.LS_MAXITER_MIN),
|
||||
self.LS_MAXITER_MAX)
|
||||
self.kwargs['method'] = 'L-BFGS-B'
|
||||
self.kwargs['options'] = {
|
||||
'maxiter': ls_max_iter,
|
||||
}
|
||||
self.kwargs['bounds'] = list(zip(self.lower, self.upper))
|
||||
else:
|
||||
if callable(self.jac):
|
||||
def wrapped_jac(x):
|
||||
return self.jac(x, *args)
|
||||
self.kwargs['jac'] = wrapped_jac
|
||||
if callable(self.hess):
|
||||
def wrapped_hess(x):
|
||||
return self.hess(x, *args)
|
||||
self.kwargs['hess'] = wrapped_hess
|
||||
if callable(self.hessp):
|
||||
def wrapped_hessp(x, p):
|
||||
return self.hessp(x, p, *args)
|
||||
self.kwargs['hessp'] = wrapped_hessp
|
||||
|
||||
def local_search(self, x, e):
|
||||
# Run local search from the given x location where energy value is e
|
||||
x_tmp = np.copy(x)
|
||||
mres = self.minimizer(self.func_wrapper.fun, x, **self.kwargs)
|
||||
if 'njev' in mres:
|
||||
self.func_wrapper.ngev += mres.njev
|
||||
if 'nhev' in mres:
|
||||
self.func_wrapper.nhev += mres.nhev
|
||||
# Check if is valid value
|
||||
is_finite = np.all(np.isfinite(mres.x)) and np.isfinite(mres.fun)
|
||||
in_bounds = np.all(mres.x >= self.lower) and np.all(
|
||||
mres.x <= self.upper)
|
||||
is_valid = is_finite and in_bounds
|
||||
|
||||
# Use the new point only if it is valid and return a better results
|
||||
if is_valid and mres.fun < e:
|
||||
return mres.fun, mres.x
|
||||
else:
|
||||
return e, x_tmp
|
||||
|
||||
|
||||
def dual_annealing(func, bounds, args=(), maxiter=1000,
|
||||
minimizer_kwargs=None, initial_temp=5230.,
|
||||
restart_temp_ratio=2.e-5, visit=2.62, accept=-5.0,
|
||||
maxfun=1e7, seed=None, no_local_search=False,
|
||||
callback=None, x0=None):
|
||||
"""
|
||||
Find the global minimum of a function using Dual Annealing.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
func : callable
|
||||
The objective function to be minimized. Must be in the form
|
||||
``f(x, *args)``, where ``x`` is the argument in the form of a 1-D array
|
||||
and ``args`` is a tuple of any additional fixed parameters needed to
|
||||
completely specify the function.
|
||||
bounds : sequence or `Bounds`
|
||||
Bounds for variables. There are two ways to specify the bounds:
|
||||
|
||||
1. Instance of `Bounds` class.
|
||||
2. Sequence of ``(min, max)`` pairs for each element in `x`.
|
||||
|
||||
args : tuple, optional
|
||||
Any additional fixed parameters needed to completely specify the
|
||||
objective function.
|
||||
maxiter : int, optional
|
||||
The maximum number of global search iterations. Default value is 1000.
|
||||
minimizer_kwargs : dict, optional
|
||||
Keyword arguments to be passed to the local minimizer
|
||||
(`minimize`). An important option could be ``method`` for the minimizer
|
||||
method to use.
|
||||
If no keyword arguments are provided, the local minimizer defaults to
|
||||
'L-BFGS-B' and uses the already supplied bounds. If `minimizer_kwargs`
|
||||
is specified, then the dict must contain all parameters required to
|
||||
control the local minimization. `args` is ignored in this dict, as it is
|
||||
passed automatically. `bounds` is not automatically passed on to the
|
||||
local minimizer as the method may not support them.
|
||||
initial_temp : float, optional
|
||||
The initial temperature, use higher values to facilitates a wider
|
||||
search of the energy landscape, allowing dual_annealing to escape
|
||||
local minima that it is trapped in. Default value is 5230. Range is
|
||||
(0.01, 5.e4].
|
||||
restart_temp_ratio : float, optional
|
||||
During the annealing process, temperature is decreasing, when it
|
||||
reaches ``initial_temp * restart_temp_ratio``, the reannealing process
|
||||
is triggered. Default value of the ratio is 2e-5. Range is (0, 1).
|
||||
visit : float, optional
|
||||
Parameter for visiting distribution. Default value is 2.62. Higher
|
||||
values give the visiting distribution a heavier tail, this makes
|
||||
the algorithm jump to a more distant region. The value range is (1, 3].
|
||||
accept : float, optional
|
||||
Parameter for acceptance distribution. It is used to control the
|
||||
probability of acceptance. The lower the acceptance parameter, the
|
||||
smaller the probability of acceptance. Default value is -5.0 with
|
||||
a range (-1e4, -5].
|
||||
maxfun : int, optional
|
||||
Soft limit for the number of objective function calls. If the
|
||||
algorithm is in the middle of a local search, this number will be
|
||||
exceeded, the algorithm will stop just after the local search is
|
||||
done. Default value is 1e7.
|
||||
seed : {None, int, `numpy.random.Generator`, `numpy.random.RandomState`}, optional
|
||||
If `seed` is None (or `np.random`), the `numpy.random.RandomState`
|
||||
singleton is used.
|
||||
If `seed` is an int, a new ``RandomState`` instance is used,
|
||||
seeded with `seed`.
|
||||
If `seed` is already a ``Generator`` or ``RandomState`` instance then
|
||||
that instance is used.
|
||||
Specify `seed` for repeatable minimizations. The random numbers
|
||||
generated with this seed only affect the visiting distribution function
|
||||
and new coordinates generation.
|
||||
no_local_search : bool, optional
|
||||
If `no_local_search` is set to True, a traditional Generalized
|
||||
Simulated Annealing will be performed with no local search
|
||||
strategy applied.
|
||||
callback : callable, optional
|
||||
A callback function with signature ``callback(x, f, context)``,
|
||||
which will be called for all minima found.
|
||||
``x`` and ``f`` are the coordinates and function value of the
|
||||
latest minimum found, and ``context`` has value in [0, 1, 2], with the
|
||||
following meaning:
|
||||
|
||||
- 0: minimum detected in the annealing process.
|
||||
- 1: detection occurred in the local search process.
|
||||
- 2: detection done in the dual annealing process.
|
||||
|
||||
If the callback implementation returns True, the algorithm will stop.
|
||||
x0 : ndarray, shape(n,), optional
|
||||
Coordinates of a single N-D starting point.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : OptimizeResult
|
||||
The optimization result represented as a `OptimizeResult` object.
|
||||
Important attributes are: ``x`` the solution array, ``fun`` the value
|
||||
of the function at the solution, and ``message`` which describes the
|
||||
cause of the termination.
|
||||
See `OptimizeResult` for a description of other attributes.
|
||||
|
||||
Notes
|
||||
-----
|
||||
This function implements the Dual Annealing optimization. This stochastic
|
||||
approach derived from [3]_ combines the generalization of CSA (Classical
|
||||
Simulated Annealing) and FSA (Fast Simulated Annealing) [1]_ [2]_ coupled
|
||||
to a strategy for applying a local search on accepted locations [4]_.
|
||||
An alternative implementation of this same algorithm is described in [5]_
|
||||
and benchmarks are presented in [6]_. This approach introduces an advanced
|
||||
method to refine the solution found by the generalized annealing
|
||||
process. This algorithm uses a distorted Cauchy-Lorentz visiting
|
||||
distribution, with its shape controlled by the parameter :math:`q_{v}`
|
||||
|
||||
.. math::
|
||||
|
||||
g_{q_{v}}(\\Delta x(t)) \\propto \\frac{ \\
|
||||
\\left[T_{q_{v}}(t) \\right]^{-\\frac{D}{3-q_{v}}}}{ \\
|
||||
\\left[{1+(q_{v}-1)\\frac{(\\Delta x(t))^{2}} { \\
|
||||
\\left[T_{q_{v}}(t)\\right]^{\\frac{2}{3-q_{v}}}}}\\right]^{ \\
|
||||
\\frac{1}{q_{v}-1}+\\frac{D-1}{2}}}
|
||||
|
||||
Where :math:`t` is the artificial time. This visiting distribution is used
|
||||
to generate a trial jump distance :math:`\\Delta x(t)` of variable
|
||||
:math:`x(t)` under artificial temperature :math:`T_{q_{v}}(t)`.
|
||||
|
||||
From the starting point, after calling the visiting distribution
|
||||
function, the acceptance probability is computed as follows:
|
||||
|
||||
.. math::
|
||||
|
||||
p_{q_{a}} = \\min{\\{1,\\left[1-(1-q_{a}) \\beta \\Delta E \\right]^{ \\
|
||||
\\frac{1}{1-q_{a}}}\\}}
|
||||
|
||||
Where :math:`q_{a}` is a acceptance parameter. For :math:`q_{a}<1`, zero
|
||||
acceptance probability is assigned to the cases where
|
||||
|
||||
.. math::
|
||||
|
||||
[1-(1-q_{a}) \\beta \\Delta E] < 0
|
||||
|
||||
The artificial temperature :math:`T_{q_{v}}(t)` is decreased according to
|
||||
|
||||
.. math::
|
||||
|
||||
T_{q_{v}}(t) = T_{q_{v}}(1) \\frac{2^{q_{v}-1}-1}{\\left( \\
|
||||
1 + t\\right)^{q_{v}-1}-1}
|
||||
|
||||
Where :math:`q_{v}` is the visiting parameter.
|
||||
|
||||
.. versionadded:: 1.2.0
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Tsallis C. Possible generalization of Boltzmann-Gibbs
|
||||
statistics. Journal of Statistical Physics, 52, 479-487 (1998).
|
||||
.. [2] Tsallis C, Stariolo DA. Generalized Simulated Annealing.
|
||||
Physica A, 233, 395-406 (1996).
|
||||
.. [3] Xiang Y, Sun DY, Fan W, Gong XG. Generalized Simulated
|
||||
Annealing Algorithm and Its Application to the Thomson Model.
|
||||
Physics Letters A, 233, 216-220 (1997).
|
||||
.. [4] Xiang Y, Gong XG. Efficiency of Generalized Simulated
|
||||
Annealing. Physical Review E, 62, 4473 (2000).
|
||||
.. [5] Xiang Y, Gubian S, Suomela B, Hoeng J. Generalized
|
||||
Simulated Annealing for Efficient Global Optimization: the GenSA
|
||||
Package for R. The R Journal, Volume 5/1 (2013).
|
||||
.. [6] Mullen, K. Continuous Global Optimization in R. Journal of
|
||||
Statistical Software, 60(6), 1 - 45, (2014).
|
||||
:doi:`10.18637/jss.v060.i06`
|
||||
|
||||
Examples
|
||||
--------
|
||||
The following example is a 10-D problem, with many local minima.
|
||||
The function involved is called Rastrigin
|
||||
(https://en.wikipedia.org/wiki/Rastrigin_function)
|
||||
|
||||
>>> import numpy as np
|
||||
>>> from scipy.optimize import dual_annealing
|
||||
>>> func = lambda x: np.sum(x*x - 10*np.cos(2*np.pi*x)) + 10*np.size(x)
|
||||
>>> lw = [-5.12] * 10
|
||||
>>> up = [5.12] * 10
|
||||
>>> ret = dual_annealing(func, bounds=list(zip(lw, up)))
|
||||
>>> ret.x
|
||||
array([-4.26437714e-09, -3.91699361e-09, -1.86149218e-09, -3.97165720e-09,
|
||||
-6.29151648e-09, -6.53145322e-09, -3.93616815e-09, -6.55623025e-09,
|
||||
-6.05775280e-09, -5.00668935e-09]) # random
|
||||
>>> ret.fun
|
||||
0.000000
|
||||
|
||||
"""
|
||||
|
||||
if isinstance(bounds, Bounds):
|
||||
bounds = new_bounds_to_old(bounds.lb, bounds.ub, len(bounds.lb))
|
||||
|
||||
if x0 is not None and not len(x0) == len(bounds):
|
||||
raise ValueError('Bounds size does not match x0')
|
||||
|
||||
lu = list(zip(*bounds))
|
||||
lower = np.array(lu[0])
|
||||
upper = np.array(lu[1])
|
||||
# Check that restart temperature ratio is correct
|
||||
if restart_temp_ratio <= 0. or restart_temp_ratio >= 1.:
|
||||
raise ValueError('Restart temperature ratio has to be in range (0, 1)')
|
||||
# Checking bounds are valid
|
||||
if (np.any(np.isinf(lower)) or np.any(np.isinf(upper)) or np.any(
|
||||
np.isnan(lower)) or np.any(np.isnan(upper))):
|
||||
raise ValueError('Some bounds values are inf values or nan values')
|
||||
# Checking that bounds are consistent
|
||||
if not np.all(lower < upper):
|
||||
raise ValueError('Bounds are not consistent min < max')
|
||||
# Checking that bounds are the same length
|
||||
if not len(lower) == len(upper):
|
||||
raise ValueError('Bounds do not have the same dimensions')
|
||||
|
||||
# Wrapper for the objective function
|
||||
func_wrapper = ObjectiveFunWrapper(func, maxfun, *args)
|
||||
|
||||
# minimizer_kwargs has to be a dict, not None
|
||||
minimizer_kwargs = minimizer_kwargs or {}
|
||||
|
||||
minimizer_wrapper = LocalSearchWrapper(
|
||||
bounds, func_wrapper, *args, **minimizer_kwargs)
|
||||
|
||||
# Initialization of random Generator for reproducible runs if seed provided
|
||||
rand_state = check_random_state(seed)
|
||||
# Initialization of the energy state
|
||||
energy_state = EnergyState(lower, upper, callback)
|
||||
energy_state.reset(func_wrapper, rand_state, x0)
|
||||
# Minimum value of annealing temperature reached to perform
|
||||
# re-annealing
|
||||
temperature_restart = initial_temp * restart_temp_ratio
|
||||
# VisitingDistribution instance
|
||||
visit_dist = VisitingDistribution(lower, upper, visit, rand_state)
|
||||
# Strategy chain instance
|
||||
strategy_chain = StrategyChain(accept, visit_dist, func_wrapper,
|
||||
minimizer_wrapper, rand_state, energy_state)
|
||||
need_to_stop = False
|
||||
iteration = 0
|
||||
message = []
|
||||
# OptimizeResult object to be returned
|
||||
optimize_res = OptimizeResult()
|
||||
optimize_res.success = True
|
||||
optimize_res.status = 0
|
||||
|
||||
t1 = np.exp((visit - 1) * np.log(2.0)) - 1.0
|
||||
# Run the search loop
|
||||
while not need_to_stop:
|
||||
for i in range(maxiter):
|
||||
# Compute temperature for this step
|
||||
s = float(i) + 2.0
|
||||
t2 = np.exp((visit - 1) * np.log(s)) - 1.0
|
||||
temperature = initial_temp * t1 / t2
|
||||
if iteration >= maxiter:
|
||||
message.append("Maximum number of iteration reached")
|
||||
need_to_stop = True
|
||||
break
|
||||
# Need a re-annealing process?
|
||||
if temperature < temperature_restart:
|
||||
energy_state.reset(func_wrapper, rand_state)
|
||||
break
|
||||
# starting strategy chain
|
||||
val = strategy_chain.run(i, temperature)
|
||||
if val is not None:
|
||||
message.append(val)
|
||||
need_to_stop = True
|
||||
optimize_res.success = False
|
||||
break
|
||||
# Possible local search at the end of the strategy chain
|
||||
if not no_local_search:
|
||||
val = strategy_chain.local_search()
|
||||
if val is not None:
|
||||
message.append(val)
|
||||
need_to_stop = True
|
||||
optimize_res.success = False
|
||||
break
|
||||
iteration += 1
|
||||
|
||||
# Setting the OptimizeResult values
|
||||
optimize_res.x = energy_state.xbest
|
||||
optimize_res.fun = energy_state.ebest
|
||||
optimize_res.nit = iteration
|
||||
optimize_res.nfev = func_wrapper.nfev
|
||||
optimize_res.njev = func_wrapper.ngev
|
||||
optimize_res.nhev = func_wrapper.nhev
|
||||
optimize_res.message = message
|
||||
return optimize_res
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,475 @@
|
||||
"""Hessian update strategies for quasi-Newton optimization methods."""
|
||||
import numpy as np
|
||||
from numpy.linalg import norm
|
||||
from scipy.linalg import get_blas_funcs, issymmetric
|
||||
from warnings import warn
|
||||
|
||||
|
||||
__all__ = ['HessianUpdateStrategy', 'BFGS', 'SR1']
|
||||
|
||||
|
||||
class HessianUpdateStrategy:
|
||||
"""Interface for implementing Hessian update strategies.
|
||||
|
||||
Many optimization methods make use of Hessian (or inverse Hessian)
|
||||
approximations, such as the quasi-Newton methods BFGS, SR1, L-BFGS.
|
||||
Some of these approximations, however, do not actually need to store
|
||||
the entire matrix or can compute the internal matrix product with a
|
||||
given vector in a very efficiently manner. This class serves as an
|
||||
abstract interface between the optimization algorithm and the
|
||||
quasi-Newton update strategies, giving freedom of implementation
|
||||
to store and update the internal matrix as efficiently as possible.
|
||||
Different choices of initialization and update procedure will result
|
||||
in different quasi-Newton strategies.
|
||||
|
||||
Four methods should be implemented in derived classes: ``initialize``,
|
||||
``update``, ``dot`` and ``get_matrix``.
|
||||
|
||||
Notes
|
||||
-----
|
||||
Any instance of a class that implements this interface,
|
||||
can be accepted by the method ``minimize`` and used by
|
||||
the compatible solvers to approximate the Hessian (or
|
||||
inverse Hessian) used by the optimization algorithms.
|
||||
"""
|
||||
|
||||
def initialize(self, n, approx_type):
|
||||
"""Initialize internal matrix.
|
||||
|
||||
Allocate internal memory for storing and updating
|
||||
the Hessian or its inverse.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
n : int
|
||||
Problem dimension.
|
||||
approx_type : {'hess', 'inv_hess'}
|
||||
Selects either the Hessian or the inverse Hessian.
|
||||
When set to 'hess' the Hessian will be stored and updated.
|
||||
When set to 'inv_hess' its inverse will be used instead.
|
||||
"""
|
||||
raise NotImplementedError("The method ``initialize(n, approx_type)``"
|
||||
" is not implemented.")
|
||||
|
||||
def update(self, delta_x, delta_grad):
|
||||
"""Update internal matrix.
|
||||
|
||||
Update Hessian matrix or its inverse (depending on how 'approx_type'
|
||||
is defined) using information about the last evaluated points.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
delta_x : ndarray
|
||||
The difference between two points the gradient
|
||||
function have been evaluated at: ``delta_x = x2 - x1``.
|
||||
delta_grad : ndarray
|
||||
The difference between the gradients:
|
||||
``delta_grad = grad(x2) - grad(x1)``.
|
||||
"""
|
||||
raise NotImplementedError("The method ``update(delta_x, delta_grad)``"
|
||||
" is not implemented.")
|
||||
|
||||
def dot(self, p):
|
||||
"""Compute the product of the internal matrix with the given vector.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
p : array_like
|
||||
1-D array representing a vector.
|
||||
|
||||
Returns
|
||||
-------
|
||||
Hp : array
|
||||
1-D represents the result of multiplying the approximation matrix
|
||||
by vector p.
|
||||
"""
|
||||
raise NotImplementedError("The method ``dot(p)``"
|
||||
" is not implemented.")
|
||||
|
||||
def get_matrix(self):
|
||||
"""Return current internal matrix.
|
||||
|
||||
Returns
|
||||
-------
|
||||
H : ndarray, shape (n, n)
|
||||
Dense matrix containing either the Hessian
|
||||
or its inverse (depending on how 'approx_type'
|
||||
is defined).
|
||||
"""
|
||||
raise NotImplementedError("The method ``get_matrix(p)``"
|
||||
" is not implemented.")
|
||||
|
||||
|
||||
class FullHessianUpdateStrategy(HessianUpdateStrategy):
|
||||
"""Hessian update strategy with full dimensional internal representation.
|
||||
"""
|
||||
_syr = get_blas_funcs('syr', dtype='d') # Symmetric rank 1 update
|
||||
_syr2 = get_blas_funcs('syr2', dtype='d') # Symmetric rank 2 update
|
||||
# Symmetric matrix-vector product
|
||||
_symv = get_blas_funcs('symv', dtype='d')
|
||||
|
||||
def __init__(self, init_scale='auto'):
|
||||
self.init_scale = init_scale
|
||||
# Until initialize is called we can't really use the class,
|
||||
# so it makes sense to set everything to None.
|
||||
self.first_iteration = None
|
||||
self.approx_type = None
|
||||
self.B = None
|
||||
self.H = None
|
||||
|
||||
def initialize(self, n, approx_type):
|
||||
"""Initialize internal matrix.
|
||||
|
||||
Allocate internal memory for storing and updating
|
||||
the Hessian or its inverse.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
n : int
|
||||
Problem dimension.
|
||||
approx_type : {'hess', 'inv_hess'}
|
||||
Selects either the Hessian or the inverse Hessian.
|
||||
When set to 'hess' the Hessian will be stored and updated.
|
||||
When set to 'inv_hess' its inverse will be used instead.
|
||||
"""
|
||||
self.first_iteration = True
|
||||
self.n = n
|
||||
self.approx_type = approx_type
|
||||
if approx_type not in ('hess', 'inv_hess'):
|
||||
raise ValueError("`approx_type` must be 'hess' or 'inv_hess'.")
|
||||
# Create matrix
|
||||
if self.approx_type == 'hess':
|
||||
self.B = np.eye(n, dtype=float)
|
||||
else:
|
||||
self.H = np.eye(n, dtype=float)
|
||||
|
||||
def _auto_scale(self, delta_x, delta_grad):
|
||||
# Heuristic to scale matrix at first iteration.
|
||||
# Described in Nocedal and Wright "Numerical Optimization"
|
||||
# p.143 formula (6.20).
|
||||
s_norm2 = np.dot(delta_x, delta_x)
|
||||
y_norm2 = np.dot(delta_grad, delta_grad)
|
||||
ys = np.abs(np.dot(delta_grad, delta_x))
|
||||
if ys == 0.0 or y_norm2 == 0 or s_norm2 == 0:
|
||||
return 1
|
||||
if self.approx_type == 'hess':
|
||||
return y_norm2 / ys
|
||||
else:
|
||||
return ys / y_norm2
|
||||
|
||||
def _update_implementation(self, delta_x, delta_grad):
|
||||
raise NotImplementedError("The method ``_update_implementation``"
|
||||
" is not implemented.")
|
||||
|
||||
def update(self, delta_x, delta_grad):
|
||||
"""Update internal matrix.
|
||||
|
||||
Update Hessian matrix or its inverse (depending on how 'approx_type'
|
||||
is defined) using information about the last evaluated points.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
delta_x : ndarray
|
||||
The difference between two points the gradient
|
||||
function have been evaluated at: ``delta_x = x2 - x1``.
|
||||
delta_grad : ndarray
|
||||
The difference between the gradients:
|
||||
``delta_grad = grad(x2) - grad(x1)``.
|
||||
"""
|
||||
if np.all(delta_x == 0.0):
|
||||
return
|
||||
if np.all(delta_grad == 0.0):
|
||||
warn('delta_grad == 0.0. Check if the approximated '
|
||||
'function is linear. If the function is linear '
|
||||
'better results can be obtained by defining the '
|
||||
'Hessian as zero instead of using quasi-Newton '
|
||||
'approximations.',
|
||||
UserWarning, stacklevel=2)
|
||||
return
|
||||
if self.first_iteration:
|
||||
# Get user specific scale
|
||||
if isinstance(self.init_scale, str) and self.init_scale == "auto":
|
||||
scale = self._auto_scale(delta_x, delta_grad)
|
||||
else:
|
||||
scale = self.init_scale
|
||||
|
||||
# Check for complex: numpy will silently cast a complex array to
|
||||
# a real one but not so for scalar as it raises a TypeError.
|
||||
# Checking here brings a consistent behavior.
|
||||
replace = False
|
||||
if np.size(scale) == 1:
|
||||
# to account for the legacy behavior having the exact same cast
|
||||
scale = float(scale)
|
||||
elif np.iscomplexobj(scale):
|
||||
raise TypeError("init_scale contains complex elements, "
|
||||
"must be real.")
|
||||
else: # test explicitly for allowed shapes and values
|
||||
replace = True
|
||||
if self.approx_type == 'hess':
|
||||
shape = np.shape(self.B)
|
||||
dtype = self.B.dtype
|
||||
else:
|
||||
shape = np.shape(self.H)
|
||||
dtype = self.H.dtype
|
||||
# copy, will replace the original
|
||||
scale = np.array(scale, dtype=dtype, copy=True)
|
||||
|
||||
# it has to match the shape of the matrix for the multiplication,
|
||||
# no implicit broadcasting is allowed
|
||||
if shape != (init_shape := np.shape(scale)):
|
||||
raise ValueError("If init_scale is an array, it must have the "
|
||||
f"dimensions of the hess/inv_hess: {shape}."
|
||||
f" Got {init_shape}.")
|
||||
if not issymmetric(scale):
|
||||
raise ValueError("If init_scale is an array, it must be"
|
||||
" symmetric (passing scipy.linalg.issymmetric)"
|
||||
" to be an approximation of a hess/inv_hess.")
|
||||
|
||||
# Scale initial matrix with ``scale * np.eye(n)`` or replace
|
||||
# This is not ideal, we could assign the scale directly in
|
||||
# initialize, but we would need to
|
||||
if self.approx_type == 'hess':
|
||||
if replace:
|
||||
self.B = scale
|
||||
else:
|
||||
self.B *= scale
|
||||
else:
|
||||
if replace:
|
||||
self.H = scale
|
||||
else:
|
||||
self.H *= scale
|
||||
self.first_iteration = False
|
||||
self._update_implementation(delta_x, delta_grad)
|
||||
|
||||
def dot(self, p):
|
||||
"""Compute the product of the internal matrix with the given vector.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
p : array_like
|
||||
1-D array representing a vector.
|
||||
|
||||
Returns
|
||||
-------
|
||||
Hp : array
|
||||
1-D represents the result of multiplying the approximation matrix
|
||||
by vector p.
|
||||
"""
|
||||
if self.approx_type == 'hess':
|
||||
return self._symv(1, self.B, p)
|
||||
else:
|
||||
return self._symv(1, self.H, p)
|
||||
|
||||
def get_matrix(self):
|
||||
"""Return the current internal matrix.
|
||||
|
||||
Returns
|
||||
-------
|
||||
M : ndarray, shape (n, n)
|
||||
Dense matrix containing either the Hessian or its inverse
|
||||
(depending on how `approx_type` was defined).
|
||||
"""
|
||||
if self.approx_type == 'hess':
|
||||
M = np.copy(self.B)
|
||||
else:
|
||||
M = np.copy(self.H)
|
||||
li = np.tril_indices_from(M, k=-1)
|
||||
M[li] = M.T[li]
|
||||
return M
|
||||
|
||||
|
||||
class BFGS(FullHessianUpdateStrategy):
|
||||
"""Broyden-Fletcher-Goldfarb-Shanno (BFGS) Hessian update strategy.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
exception_strategy : {'skip_update', 'damp_update'}, optional
|
||||
Define how to proceed when the curvature condition is violated.
|
||||
Set it to 'skip_update' to just skip the update. Or, alternatively,
|
||||
set it to 'damp_update' to interpolate between the actual BFGS
|
||||
result and the unmodified matrix. Both exceptions strategies
|
||||
are explained in [1]_, p.536-537.
|
||||
min_curvature : float
|
||||
This number, scaled by a normalization factor, defines the
|
||||
minimum curvature ``dot(delta_grad, delta_x)`` allowed to go
|
||||
unaffected by the exception strategy. By default is equal to
|
||||
1e-8 when ``exception_strategy = 'skip_update'`` and equal
|
||||
to 0.2 when ``exception_strategy = 'damp_update'``.
|
||||
init_scale : {float, np.array, 'auto'}
|
||||
This parameter can be used to initialize the Hessian or its
|
||||
inverse. When a float is given, the relevant array is initialized
|
||||
to ``np.eye(n) * init_scale``, where ``n`` is the problem dimension.
|
||||
Alternatively, if a precisely ``(n, n)`` shaped, symmetric array is given,
|
||||
this array will be used. Otherwise an error is generated.
|
||||
Set it to 'auto' in order to use an automatic heuristic for choosing
|
||||
the initial scale. The heuristic is described in [1]_, p.143.
|
||||
The default is 'auto'.
|
||||
|
||||
Notes
|
||||
-----
|
||||
The update is based on the description in [1]_, p.140.
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Nocedal, Jorge, and Stephen J. Wright. "Numerical optimization"
|
||||
Second Edition (2006).
|
||||
"""
|
||||
|
||||
def __init__(self, exception_strategy='skip_update', min_curvature=None,
|
||||
init_scale='auto'):
|
||||
if exception_strategy == 'skip_update':
|
||||
if min_curvature is not None:
|
||||
self.min_curvature = min_curvature
|
||||
else:
|
||||
self.min_curvature = 1e-8
|
||||
elif exception_strategy == 'damp_update':
|
||||
if min_curvature is not None:
|
||||
self.min_curvature = min_curvature
|
||||
else:
|
||||
self.min_curvature = 0.2
|
||||
else:
|
||||
raise ValueError("`exception_strategy` must be 'skip_update' "
|
||||
"or 'damp_update'.")
|
||||
|
||||
super().__init__(init_scale)
|
||||
self.exception_strategy = exception_strategy
|
||||
|
||||
def _update_inverse_hessian(self, ys, Hy, yHy, s):
|
||||
"""Update the inverse Hessian matrix.
|
||||
|
||||
BFGS update using the formula:
|
||||
|
||||
``H <- H + ((H*y).T*y + s.T*y)/(s.T*y)^2 * (s*s.T)
|
||||
- 1/(s.T*y) * ((H*y)*s.T + s*(H*y).T)``
|
||||
|
||||
where ``s = delta_x`` and ``y = delta_grad``. This formula is
|
||||
equivalent to (6.17) in [1]_ written in a more efficient way
|
||||
for implementation.
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Nocedal, Jorge, and Stephen J. Wright. "Numerical optimization"
|
||||
Second Edition (2006).
|
||||
"""
|
||||
self.H = self._syr2(-1.0 / ys, s, Hy, a=self.H)
|
||||
self.H = self._syr((ys + yHy) / ys ** 2, s, a=self.H)
|
||||
|
||||
def _update_hessian(self, ys, Bs, sBs, y):
|
||||
"""Update the Hessian matrix.
|
||||
|
||||
BFGS update using the formula:
|
||||
|
||||
``B <- B - (B*s)*(B*s).T/s.T*(B*s) + y*y^T/s.T*y``
|
||||
|
||||
where ``s`` is short for ``delta_x`` and ``y`` is short
|
||||
for ``delta_grad``. Formula (6.19) in [1]_.
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Nocedal, Jorge, and Stephen J. Wright. "Numerical optimization"
|
||||
Second Edition (2006).
|
||||
"""
|
||||
self.B = self._syr(1.0 / ys, y, a=self.B)
|
||||
self.B = self._syr(-1.0 / sBs, Bs, a=self.B)
|
||||
|
||||
def _update_implementation(self, delta_x, delta_grad):
|
||||
# Auxiliary variables w and z
|
||||
if self.approx_type == 'hess':
|
||||
w = delta_x
|
||||
z = delta_grad
|
||||
else:
|
||||
w = delta_grad
|
||||
z = delta_x
|
||||
# Do some common operations
|
||||
wz = np.dot(w, z)
|
||||
Mw = self.dot(w)
|
||||
wMw = Mw.dot(w)
|
||||
# Guarantee that wMw > 0 by reinitializing matrix.
|
||||
# While this is always true in exact arithmetic,
|
||||
# indefinite matrix may appear due to roundoff errors.
|
||||
if wMw <= 0.0:
|
||||
scale = self._auto_scale(delta_x, delta_grad)
|
||||
# Reinitialize matrix
|
||||
if self.approx_type == 'hess':
|
||||
self.B = scale * np.eye(self.n, dtype=float)
|
||||
else:
|
||||
self.H = scale * np.eye(self.n, dtype=float)
|
||||
# Do common operations for new matrix
|
||||
Mw = self.dot(w)
|
||||
wMw = Mw.dot(w)
|
||||
# Check if curvature condition is violated
|
||||
if wz <= self.min_curvature * wMw:
|
||||
# If the option 'skip_update' is set
|
||||
# we just skip the update when the condition
|
||||
# is violated.
|
||||
if self.exception_strategy == 'skip_update':
|
||||
return
|
||||
# If the option 'damp_update' is set we
|
||||
# interpolate between the actual BFGS
|
||||
# result and the unmodified matrix.
|
||||
elif self.exception_strategy == 'damp_update':
|
||||
update_factor = (1-self.min_curvature) / (1 - wz/wMw)
|
||||
z = update_factor*z + (1-update_factor)*Mw
|
||||
wz = np.dot(w, z)
|
||||
# Update matrix
|
||||
if self.approx_type == 'hess':
|
||||
self._update_hessian(wz, Mw, wMw, z)
|
||||
else:
|
||||
self._update_inverse_hessian(wz, Mw, wMw, z)
|
||||
|
||||
|
||||
class SR1(FullHessianUpdateStrategy):
|
||||
"""Symmetric-rank-1 Hessian update strategy.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
min_denominator : float
|
||||
This number, scaled by a normalization factor,
|
||||
defines the minimum denominator magnitude allowed
|
||||
in the update. When the condition is violated we skip
|
||||
the update. By default uses ``1e-8``.
|
||||
init_scale : {float, np.array, 'auto'}, optional
|
||||
This parameter can be used to initialize the Hessian or its
|
||||
inverse. When a float is given, the relevant array is initialized
|
||||
to ``np.eye(n) * init_scale``, where ``n`` is the problem dimension.
|
||||
Alternatively, if a precisely ``(n, n)`` shaped, symmetric array is given,
|
||||
this array will be used. Otherwise an error is generated.
|
||||
Set it to 'auto' in order to use an automatic heuristic for choosing
|
||||
the initial scale. The heuristic is described in [1]_, p.143.
|
||||
The default is 'auto'.
|
||||
|
||||
Notes
|
||||
-----
|
||||
The update is based on the description in [1]_, p.144-146.
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Nocedal, Jorge, and Stephen J. Wright. "Numerical optimization"
|
||||
Second Edition (2006).
|
||||
"""
|
||||
|
||||
def __init__(self, min_denominator=1e-8, init_scale='auto'):
|
||||
self.min_denominator = min_denominator
|
||||
super().__init__(init_scale)
|
||||
|
||||
def _update_implementation(self, delta_x, delta_grad):
|
||||
# Auxiliary variables w and z
|
||||
if self.approx_type == 'hess':
|
||||
w = delta_x
|
||||
z = delta_grad
|
||||
else:
|
||||
w = delta_grad
|
||||
z = delta_x
|
||||
# Do some common operations
|
||||
Mw = self.dot(w)
|
||||
z_minus_Mw = z - Mw
|
||||
denominator = np.dot(w, z_minus_Mw)
|
||||
# If the denominator is too small
|
||||
# we just skip the update.
|
||||
if np.abs(denominator) <= self.min_denominator*norm(w)*norm(z_minus_Mw):
|
||||
return
|
||||
# Update matrix
|
||||
if self.approx_type == 'hess':
|
||||
self.B = self._syr(1/denominator, z_minus_Mw, a=self.B)
|
||||
else:
|
||||
self.H = self._syr(1/denominator, z_minus_Mw, a=self.H)
|
||||
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,106 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from libcpp cimport bool
|
||||
from libcpp.string cimport string
|
||||
|
||||
cdef extern from "HConst.h" nogil:
|
||||
|
||||
const int HIGHS_CONST_I_INF "kHighsIInf"
|
||||
const double HIGHS_CONST_INF "kHighsInf"
|
||||
const double kHighsTiny
|
||||
const double kHighsZero
|
||||
const int kHighsThreadLimit
|
||||
|
||||
cdef enum HighsDebugLevel:
|
||||
HighsDebugLevel_kHighsDebugLevelNone "kHighsDebugLevelNone" = 0
|
||||
HighsDebugLevel_kHighsDebugLevelCheap "kHighsDebugLevelCheap"
|
||||
HighsDebugLevel_kHighsDebugLevelCostly "kHighsDebugLevelCostly"
|
||||
HighsDebugLevel_kHighsDebugLevelExpensive "kHighsDebugLevelExpensive"
|
||||
HighsDebugLevel_kHighsDebugLevelMin "kHighsDebugLevelMin" = HighsDebugLevel_kHighsDebugLevelNone
|
||||
HighsDebugLevel_kHighsDebugLevelMax "kHighsDebugLevelMax" = HighsDebugLevel_kHighsDebugLevelExpensive
|
||||
|
||||
ctypedef enum HighsModelStatus:
|
||||
HighsModelStatusNOTSET "HighsModelStatus::kNotset" = 0
|
||||
HighsModelStatusLOAD_ERROR "HighsModelStatus::kLoadError"
|
||||
HighsModelStatusMODEL_ERROR "HighsModelStatus::kModelError"
|
||||
HighsModelStatusPRESOLVE_ERROR "HighsModelStatus::kPresolveError"
|
||||
HighsModelStatusSOLVE_ERROR "HighsModelStatus::kSolveError"
|
||||
HighsModelStatusPOSTSOLVE_ERROR "HighsModelStatus::kPostsolveError"
|
||||
HighsModelStatusMODEL_EMPTY "HighsModelStatus::kModelEmpty"
|
||||
HighsModelStatusOPTIMAL "HighsModelStatus::kOptimal"
|
||||
HighsModelStatusINFEASIBLE "HighsModelStatus::kInfeasible"
|
||||
HighsModelStatus_UNBOUNDED_OR_INFEASIBLE "HighsModelStatus::kUnboundedOrInfeasible"
|
||||
HighsModelStatusUNBOUNDED "HighsModelStatus::kUnbounded"
|
||||
HighsModelStatusREACHED_DUAL_OBJECTIVE_VALUE_UPPER_BOUND "HighsModelStatus::kObjectiveBound"
|
||||
HighsModelStatusREACHED_OBJECTIVE_TARGET "HighsModelStatus::kObjectiveTarget"
|
||||
HighsModelStatusREACHED_TIME_LIMIT "HighsModelStatus::kTimeLimit"
|
||||
HighsModelStatusREACHED_ITERATION_LIMIT "HighsModelStatus::kIterationLimit"
|
||||
HighsModelStatusUNKNOWN "HighsModelStatus::kUnknown"
|
||||
HighsModelStatusHIGHS_MODEL_STATUS_MIN "HighsModelStatus::kMin" = HighsModelStatusNOTSET
|
||||
HighsModelStatusHIGHS_MODEL_STATUS_MAX "HighsModelStatus::kMax" = HighsModelStatusUNKNOWN
|
||||
|
||||
cdef enum HighsBasisStatus:
|
||||
HighsBasisStatusLOWER "HighsBasisStatus::kLower" = 0, # (slack) variable is at its lower bound [including fixed variables]
|
||||
HighsBasisStatusBASIC "HighsBasisStatus::kBasic" # (slack) variable is basic
|
||||
HighsBasisStatusUPPER "HighsBasisStatus::kUpper" # (slack) variable is at its upper bound
|
||||
HighsBasisStatusZERO "HighsBasisStatus::kZero" # free variable is non-basic and set to zero
|
||||
HighsBasisStatusNONBASIC "HighsBasisStatus::kNonbasic" # nonbasic with no specific bound information - useful for users and postsolve
|
||||
|
||||
cdef enum SolverOption:
|
||||
SOLVER_OPTION_SIMPLEX "SolverOption::SOLVER_OPTION_SIMPLEX" = -1
|
||||
SOLVER_OPTION_CHOOSE "SolverOption::SOLVER_OPTION_CHOOSE"
|
||||
SOLVER_OPTION_IPM "SolverOption::SOLVER_OPTION_IPM"
|
||||
|
||||
cdef enum PrimalDualStatus:
|
||||
PrimalDualStatusSTATUS_NOT_SET "PrimalDualStatus::STATUS_NOT_SET" = -1
|
||||
PrimalDualStatusSTATUS_MIN "PrimalDualStatus::STATUS_MIN" = PrimalDualStatusSTATUS_NOT_SET
|
||||
PrimalDualStatusSTATUS_NO_SOLUTION "PrimalDualStatus::STATUS_NO_SOLUTION"
|
||||
PrimalDualStatusSTATUS_UNKNOWN "PrimalDualStatus::STATUS_UNKNOWN"
|
||||
PrimalDualStatusSTATUS_INFEASIBLE_POINT "PrimalDualStatus::STATUS_INFEASIBLE_POINT"
|
||||
PrimalDualStatusSTATUS_FEASIBLE_POINT "PrimalDualStatus::STATUS_FEASIBLE_POINT"
|
||||
PrimalDualStatusSTATUS_MAX "PrimalDualStatus::STATUS_MAX" = PrimalDualStatusSTATUS_FEASIBLE_POINT
|
||||
|
||||
cdef enum HighsOptionType:
|
||||
HighsOptionTypeBOOL "HighsOptionType::kBool" = 0
|
||||
HighsOptionTypeINT "HighsOptionType::kInt"
|
||||
HighsOptionTypeDOUBLE "HighsOptionType::kDouble"
|
||||
HighsOptionTypeSTRING "HighsOptionType::kString"
|
||||
|
||||
# workaround for lack of enum class support in Cython < 3.x
|
||||
# cdef enum class ObjSense(int):
|
||||
# ObjSenseMINIMIZE "ObjSense::kMinimize" = 1
|
||||
# ObjSenseMAXIMIZE "ObjSense::kMaximize" = -1
|
||||
|
||||
cdef cppclass ObjSense:
|
||||
pass
|
||||
|
||||
cdef ObjSense ObjSenseMINIMIZE "ObjSense::kMinimize"
|
||||
cdef ObjSense ObjSenseMAXIMIZE "ObjSense::kMaximize"
|
||||
|
||||
# cdef enum class MatrixFormat(int):
|
||||
# MatrixFormatkColwise "MatrixFormat::kColwise" = 1
|
||||
# MatrixFormatkRowwise "MatrixFormat::kRowwise"
|
||||
# MatrixFormatkRowwisePartitioned "MatrixFormat::kRowwisePartitioned"
|
||||
|
||||
cdef cppclass MatrixFormat:
|
||||
pass
|
||||
|
||||
cdef MatrixFormat MatrixFormatkColwise "MatrixFormat::kColwise"
|
||||
cdef MatrixFormat MatrixFormatkRowwise "MatrixFormat::kRowwise"
|
||||
cdef MatrixFormat MatrixFormatkRowwisePartitioned "MatrixFormat::kRowwisePartitioned"
|
||||
|
||||
# cdef enum class HighsVarType(int):
|
||||
# kContinuous "HighsVarType::kContinuous"
|
||||
# kInteger "HighsVarType::kInteger"
|
||||
# kSemiContinuous "HighsVarType::kSemiContinuous"
|
||||
# kSemiInteger "HighsVarType::kSemiInteger"
|
||||
# kImplicitInteger "HighsVarType::kImplicitInteger"
|
||||
|
||||
cdef cppclass HighsVarType:
|
||||
pass
|
||||
|
||||
cdef HighsVarType kContinuous "HighsVarType::kContinuous"
|
||||
cdef HighsVarType kInteger "HighsVarType::kInteger"
|
||||
cdef HighsVarType kSemiContinuous "HighsVarType::kSemiContinuous"
|
||||
cdef HighsVarType kSemiInteger "HighsVarType::kSemiInteger"
|
||||
cdef HighsVarType kImplicitInteger "HighsVarType::kImplicitInteger"
|
||||
@@ -0,0 +1,56 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from libc.stdio cimport FILE
|
||||
|
||||
from libcpp cimport bool
|
||||
from libcpp.string cimport string
|
||||
|
||||
from .HighsStatus cimport HighsStatus
|
||||
from .HighsOptions cimport HighsOptions
|
||||
from .HighsInfo cimport HighsInfo
|
||||
from .HighsLp cimport (
|
||||
HighsLp,
|
||||
HighsSolution,
|
||||
HighsBasis,
|
||||
ObjSense,
|
||||
)
|
||||
from .HConst cimport HighsModelStatus
|
||||
|
||||
cdef extern from "Highs.h":
|
||||
# From HiGHS/src/Highs.h
|
||||
cdef cppclass Highs:
|
||||
HighsStatus passHighsOptions(const HighsOptions& options)
|
||||
HighsStatus passModel(const HighsLp& lp)
|
||||
HighsStatus run()
|
||||
HighsStatus setHighsLogfile(FILE* logfile)
|
||||
HighsStatus setHighsOutput(FILE* output)
|
||||
HighsStatus writeHighsOptions(const string filename, const bool report_only_non_default_values = true)
|
||||
|
||||
# split up for cython below
|
||||
#const HighsModelStatus& getModelStatus(const bool scaled_model = False) const
|
||||
const HighsModelStatus & getModelStatus() const
|
||||
|
||||
const HighsInfo& getHighsInfo "getInfo" () const
|
||||
string modelStatusToString(const HighsModelStatus model_status) const
|
||||
#HighsStatus getHighsInfoValue(const string& info, int& value)
|
||||
HighsStatus getHighsInfoValue(const string& info, double& value) const
|
||||
const HighsOptions& getHighsOptions() const
|
||||
|
||||
const HighsLp& getLp() const
|
||||
|
||||
HighsStatus writeSolution(const string filename, const bool pretty) const
|
||||
|
||||
HighsStatus setBasis()
|
||||
const HighsSolution& getSolution() const
|
||||
const HighsBasis& getBasis() const
|
||||
|
||||
bool changeObjectiveSense(const ObjSense sense)
|
||||
|
||||
HighsStatus setHighsOptionValueBool "setOptionValue" (const string & option, const bool value)
|
||||
HighsStatus setHighsOptionValueInt "setOptionValue" (const string & option, const int value)
|
||||
HighsStatus setHighsOptionValueStr "setOptionValue" (const string & option, const string & value)
|
||||
HighsStatus setHighsOptionValueDbl "setOptionValue" (const string & option, const double value)
|
||||
|
||||
string primalDualStatusToString(const int primal_dual_status)
|
||||
|
||||
void resetGlobalScheduler(bool blocking)
|
||||
@@ -0,0 +1,20 @@
|
||||
# cython: language_level=3
|
||||
|
||||
|
||||
cdef extern from "HighsIO.h" nogil:
|
||||
# workaround for lack of enum class support in Cython < 3.x
|
||||
# cdef enum class HighsLogType(int):
|
||||
# kInfo "HighsLogType::kInfo" = 1
|
||||
# kDetailed "HighsLogType::kDetailed"
|
||||
# kVerbose "HighsLogType::kVerbose"
|
||||
# kWarning "HighsLogType::kWarning"
|
||||
# kError "HighsLogType::kError"
|
||||
|
||||
cdef cppclass HighsLogType:
|
||||
pass
|
||||
|
||||
cdef HighsLogType kInfo "HighsLogType::kInfo"
|
||||
cdef HighsLogType kDetailed "HighsLogType::kDetailed"
|
||||
cdef HighsLogType kVerbose "HighsLogType::kVerbose"
|
||||
cdef HighsLogType kWarning "HighsLogType::kWarning"
|
||||
cdef HighsLogType kError "HighsLogType::kError"
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
# cython: language_level=3
|
||||
|
||||
cdef extern from "HighsInfo.h" nogil:
|
||||
# From HiGHS/src/lp_data/HighsInfo.h
|
||||
cdef cppclass HighsInfo:
|
||||
# Inherited from HighsInfoStruct:
|
||||
int mip_node_count
|
||||
int simplex_iteration_count
|
||||
int ipm_iteration_count
|
||||
int crossover_iteration_count
|
||||
int primal_solution_status
|
||||
int dual_solution_status
|
||||
int basis_validity
|
||||
double objective_function_value
|
||||
double mip_dual_bound
|
||||
double mip_gap
|
||||
int num_primal_infeasibilities
|
||||
double max_primal_infeasibility
|
||||
double sum_primal_infeasibilities
|
||||
int num_dual_infeasibilities
|
||||
double max_dual_infeasibility
|
||||
double sum_dual_infeasibilities
|
||||
@@ -0,0 +1,46 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from libcpp cimport bool
|
||||
from libcpp.string cimport string
|
||||
from libcpp.vector cimport vector
|
||||
|
||||
from .HConst cimport HighsBasisStatus, ObjSense, HighsVarType
|
||||
from .HighsSparseMatrix cimport HighsSparseMatrix
|
||||
|
||||
|
||||
cdef extern from "HighsLp.h" nogil:
|
||||
# From HiGHS/src/lp_data/HighsLp.h
|
||||
cdef cppclass HighsLp:
|
||||
int num_col_
|
||||
int num_row_
|
||||
|
||||
vector[double] col_cost_
|
||||
vector[double] col_lower_
|
||||
vector[double] col_upper_
|
||||
vector[double] row_lower_
|
||||
vector[double] row_upper_
|
||||
|
||||
HighsSparseMatrix a_matrix_
|
||||
|
||||
ObjSense sense_
|
||||
double offset_
|
||||
|
||||
string model_name_
|
||||
|
||||
vector[string] row_names_
|
||||
vector[string] col_names_
|
||||
|
||||
vector[HighsVarType] integrality_
|
||||
|
||||
bool isMip() const
|
||||
|
||||
cdef cppclass HighsSolution:
|
||||
vector[double] col_value
|
||||
vector[double] col_dual
|
||||
vector[double] row_value
|
||||
vector[double] row_dual
|
||||
|
||||
cdef cppclass HighsBasis:
|
||||
bool valid_
|
||||
vector[HighsBasisStatus] col_status
|
||||
vector[HighsBasisStatus] row_status
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from .HighsStatus cimport HighsStatus
|
||||
from .HighsLp cimport HighsLp
|
||||
from .HighsOptions cimport HighsOptions
|
||||
|
||||
cdef extern from "HighsLpUtils.h" nogil:
|
||||
# From HiGHS/src/lp_data/HighsLpUtils.h
|
||||
HighsStatus assessLp(HighsLp& lp, const HighsOptions& options)
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from libcpp.string cimport string
|
||||
|
||||
from .HConst cimport HighsModelStatus
|
||||
|
||||
cdef extern from "HighsModelUtils.h" nogil:
|
||||
# From HiGHS/src/lp_data/HighsModelUtils.h
|
||||
string utilHighsModelStatusToString(const HighsModelStatus model_status)
|
||||
string utilBasisStatusToString(const int primal_dual_status)
|
||||
+110
@@ -0,0 +1,110 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from libc.stdio cimport FILE
|
||||
|
||||
from libcpp cimport bool
|
||||
from libcpp.string cimport string
|
||||
from libcpp.vector cimport vector
|
||||
|
||||
from .HConst cimport HighsOptionType
|
||||
|
||||
cdef extern from "HighsOptions.h" nogil:
|
||||
|
||||
cdef cppclass OptionRecord:
|
||||
HighsOptionType type
|
||||
string name
|
||||
string description
|
||||
bool advanced
|
||||
|
||||
cdef cppclass OptionRecordBool(OptionRecord):
|
||||
bool* value
|
||||
bool default_value
|
||||
|
||||
cdef cppclass OptionRecordInt(OptionRecord):
|
||||
int* value
|
||||
int lower_bound
|
||||
int default_value
|
||||
int upper_bound
|
||||
|
||||
cdef cppclass OptionRecordDouble(OptionRecord):
|
||||
double* value
|
||||
double lower_bound
|
||||
double default_value
|
||||
double upper_bound
|
||||
|
||||
cdef cppclass OptionRecordString(OptionRecord):
|
||||
string* value
|
||||
string default_value
|
||||
|
||||
cdef cppclass HighsOptions:
|
||||
# From HighsOptionsStruct:
|
||||
|
||||
# Options read from the command line
|
||||
string model_file
|
||||
string presolve
|
||||
string solver
|
||||
string parallel
|
||||
double time_limit
|
||||
string options_file
|
||||
|
||||
# Options read from the file
|
||||
double infinite_cost
|
||||
double infinite_bound
|
||||
double small_matrix_value
|
||||
double large_matrix_value
|
||||
double primal_feasibility_tolerance
|
||||
double dual_feasibility_tolerance
|
||||
double ipm_optimality_tolerance
|
||||
double dual_objective_value_upper_bound
|
||||
int highs_debug_level
|
||||
int simplex_strategy
|
||||
int simplex_scale_strategy
|
||||
int simplex_crash_strategy
|
||||
int simplex_dual_edge_weight_strategy
|
||||
int simplex_primal_edge_weight_strategy
|
||||
int simplex_iteration_limit
|
||||
int simplex_update_limit
|
||||
int ipm_iteration_limit
|
||||
int highs_min_threads
|
||||
int highs_max_threads
|
||||
int message_level
|
||||
string solution_file
|
||||
bool write_solution_to_file
|
||||
bool write_solution_pretty
|
||||
|
||||
# Advanced options
|
||||
bool run_crossover
|
||||
bool mps_parser_type_free
|
||||
int keep_n_rows
|
||||
int allowed_simplex_matrix_scale_factor
|
||||
int allowed_simplex_cost_scale_factor
|
||||
int simplex_dualise_strategy
|
||||
int simplex_permute_strategy
|
||||
int dual_simplex_cleanup_strategy
|
||||
int simplex_price_strategy
|
||||
int dual_chuzc_sort_strategy
|
||||
bool simplex_initial_condition_check
|
||||
double simplex_initial_condition_tolerance
|
||||
double dual_steepest_edge_weight_log_error_threshhold
|
||||
double dual_simplex_cost_perturbation_multiplier
|
||||
double start_crossover_tolerance
|
||||
bool less_infeasible_DSE_check
|
||||
bool less_infeasible_DSE_choose_row
|
||||
bool use_original_HFactor_logic
|
||||
|
||||
# Options for MIP solver
|
||||
int mip_max_nodes
|
||||
int mip_report_level
|
||||
|
||||
# Switch for MIP solver
|
||||
bool mip
|
||||
|
||||
# Options for HighsPrintMessage and HighsLogMessage
|
||||
FILE* logfile
|
||||
FILE* output
|
||||
int message_level
|
||||
string solution_file
|
||||
bool write_solution_to_file
|
||||
bool write_solution_pretty
|
||||
|
||||
vector[OptionRecord*] records
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from libcpp cimport bool
|
||||
|
||||
from .HighsOptions cimport HighsOptions
|
||||
|
||||
cdef extern from "HighsRuntimeOptions.h" nogil:
|
||||
# From HiGHS/src/lp_data/HighsRuntimeOptions.h
|
||||
bool loadOptions(int argc, char** argv, HighsOptions& options)
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from libcpp.string cimport string
|
||||
|
||||
cdef extern from "HighsStatus.h" nogil:
|
||||
ctypedef enum HighsStatus:
|
||||
HighsStatusError "HighsStatus::kError" = -1
|
||||
HighsStatusOK "HighsStatus::kOk" = 0
|
||||
HighsStatusWarning "HighsStatus::kWarning" = 1
|
||||
|
||||
|
||||
string highsStatusToString(HighsStatus status)
|
||||
+95
@@ -0,0 +1,95 @@
|
||||
# cython: language_level=3
|
||||
|
||||
from libcpp cimport bool
|
||||
|
||||
cdef extern from "SimplexConst.h" nogil:
|
||||
|
||||
cdef enum SimplexAlgorithm:
|
||||
PRIMAL "SimplexAlgorithm::kPrimal" = 0
|
||||
DUAL "SimplexAlgorithm::kDual"
|
||||
|
||||
cdef enum SimplexStrategy:
|
||||
SIMPLEX_STRATEGY_MIN "SimplexStrategy::kSimplexStrategyMin" = 0
|
||||
SIMPLEX_STRATEGY_CHOOSE "SimplexStrategy::kSimplexStrategyChoose" = SIMPLEX_STRATEGY_MIN
|
||||
SIMPLEX_STRATEGY_DUAL "SimplexStrategy::kSimplexStrategyDual"
|
||||
SIMPLEX_STRATEGY_DUAL_PLAIN "SimplexStrategy::kSimplexStrategyDualPlain" = SIMPLEX_STRATEGY_DUAL
|
||||
SIMPLEX_STRATEGY_DUAL_TASKS "SimplexStrategy::kSimplexStrategyDualTasks"
|
||||
SIMPLEX_STRATEGY_DUAL_MULTI "SimplexStrategy::kSimplexStrategyDualMulti"
|
||||
SIMPLEX_STRATEGY_PRIMAL "SimplexStrategy::kSimplexStrategyPrimal"
|
||||
SIMPLEX_STRATEGY_MAX "SimplexStrategy::kSimplexStrategyMax" = SIMPLEX_STRATEGY_PRIMAL
|
||||
SIMPLEX_STRATEGY_NUM "SimplexStrategy::kSimplexStrategyNum"
|
||||
|
||||
cdef enum SimplexCrashStrategy:
|
||||
SIMPLEX_CRASH_STRATEGY_MIN "SimplexCrashStrategy::kSimplexCrashStrategyMin" = 0
|
||||
SIMPLEX_CRASH_STRATEGY_OFF "SimplexCrashStrategy::kSimplexCrashStrategyOff" = SIMPLEX_CRASH_STRATEGY_MIN
|
||||
SIMPLEX_CRASH_STRATEGY_LTSSF_K "SimplexCrashStrategy::kSimplexCrashStrategyLtssfK"
|
||||
SIMPLEX_CRASH_STRATEGY_LTSSF "SimplexCrashStrategy::kSimplexCrashStrategyLtssf" = SIMPLEX_CRASH_STRATEGY_LTSSF_K
|
||||
SIMPLEX_CRASH_STRATEGY_BIXBY "SimplexCrashStrategy::kSimplexCrashStrategyBixby"
|
||||
SIMPLEX_CRASH_STRATEGY_LTSSF_PRI "SimplexCrashStrategy::kSimplexCrashStrategyLtssfPri"
|
||||
SIMPLEX_CRASH_STRATEGY_LTSF_K "SimplexCrashStrategy::kSimplexCrashStrategyLtsfK"
|
||||
SIMPLEX_CRASH_STRATEGY_LTSF_PRI "SimplexCrashStrategy::kSimplexCrashStrategyLtsfPri"
|
||||
SIMPLEX_CRASH_STRATEGY_LTSF "SimplexCrashStrategy::kSimplexCrashStrategyLtsf"
|
||||
SIMPLEX_CRASH_STRATEGY_BIXBY_NO_NONZERO_COL_COSTS "SimplexCrashStrategy::kSimplexCrashStrategyBixbyNoNonzeroColCosts"
|
||||
SIMPLEX_CRASH_STRATEGY_BASIC "SimplexCrashStrategy::kSimplexCrashStrategyBasic"
|
||||
SIMPLEX_CRASH_STRATEGY_TEST_SING "SimplexCrashStrategy::kSimplexCrashStrategyTestSing"
|
||||
SIMPLEX_CRASH_STRATEGY_MAX "SimplexCrashStrategy::kSimplexCrashStrategyMax" = SIMPLEX_CRASH_STRATEGY_TEST_SING
|
||||
|
||||
cdef enum SimplexEdgeWeightStrategy:
|
||||
SIMPLEX_EDGE_WEIGHT_STRATEGY_MIN "SimplexEdgeWeightStrategy::kSimplexEdgeWeightStrategyMin" = -1
|
||||
SIMPLEX_EDGE_WEIGHT_STRATEGY_CHOOSE "SimplexEdgeWeightStrategy::kSimplexEdgeWeightStrategyChoose" = SIMPLEX_EDGE_WEIGHT_STRATEGY_MIN
|
||||
SIMPLEX_EDGE_WEIGHT_STRATEGY_DANTZIG "SimplexEdgeWeightStrategy::kSimplexEdgeWeightStrategyDantzig"
|
||||
SIMPLEX_EDGE_WEIGHT_STRATEGY_DEVEX "SimplexEdgeWeightStrategy::kSimplexEdgeWeightStrategyDevex"
|
||||
SIMPLEX_EDGE_WEIGHT_STRATEGY_STEEPEST_EDGE "SimplexEdgeWeightStrategy::kSimplexEdgeWeightStrategySteepestEdge"
|
||||
SIMPLEX_EDGE_WEIGHT_STRATEGY_STEEPEST_EDGE_UNIT_INITIAL "SimplexEdgeWeightStrategy::kSimplexEdgeWeightStrategySteepestEdgeUnitInitial"
|
||||
SIMPLEX_EDGE_WEIGHT_STRATEGY_MAX "SimplexEdgeWeightStrategy::kSimplexEdgeWeightStrategyMax" = SIMPLEX_EDGE_WEIGHT_STRATEGY_STEEPEST_EDGE_UNIT_INITIAL
|
||||
|
||||
cdef enum SimplexPriceStrategy:
|
||||
SIMPLEX_PRICE_STRATEGY_MIN = 0
|
||||
SIMPLEX_PRICE_STRATEGY_COL = SIMPLEX_PRICE_STRATEGY_MIN
|
||||
SIMPLEX_PRICE_STRATEGY_ROW
|
||||
SIMPLEX_PRICE_STRATEGY_ROW_SWITCH
|
||||
SIMPLEX_PRICE_STRATEGY_ROW_SWITCH_COL_SWITCH
|
||||
SIMPLEX_PRICE_STRATEGY_MAX = SIMPLEX_PRICE_STRATEGY_ROW_SWITCH_COL_SWITCH
|
||||
|
||||
cdef enum SimplexDualChuzcStrategy:
|
||||
SIMPLEX_DUAL_CHUZC_STRATEGY_MIN = 0
|
||||
SIMPLEX_DUAL_CHUZC_STRATEGY_CHOOSE = SIMPLEX_DUAL_CHUZC_STRATEGY_MIN
|
||||
SIMPLEX_DUAL_CHUZC_STRATEGY_QUAD
|
||||
SIMPLEX_DUAL_CHUZC_STRATEGY_HEAP
|
||||
SIMPLEX_DUAL_CHUZC_STRATEGY_BOTH
|
||||
SIMPLEX_DUAL_CHUZC_STRATEGY_MAX = SIMPLEX_DUAL_CHUZC_STRATEGY_BOTH
|
||||
|
||||
cdef enum InvertHint:
|
||||
INVERT_HINT_NO = 0
|
||||
INVERT_HINT_UPDATE_LIMIT_REACHED
|
||||
INVERT_HINT_SYNTHETIC_CLOCK_SAYS_INVERT
|
||||
INVERT_HINT_POSSIBLY_OPTIMAL
|
||||
INVERT_HINT_POSSIBLY_PRIMAL_UNBOUNDED
|
||||
INVERT_HINT_POSSIBLY_DUAL_UNBOUNDED
|
||||
INVERT_HINT_POSSIBLY_SINGULAR_BASIS
|
||||
INVERT_HINT_PRIMAL_INFEASIBLE_IN_PRIMAL_SIMPLEX
|
||||
INVERT_HINT_CHOOSE_COLUMN_FAIL
|
||||
INVERT_HINT_Count
|
||||
|
||||
cdef enum DualEdgeWeightMode:
|
||||
DANTZIG "DualEdgeWeightMode::DANTZIG" = 0
|
||||
DEVEX "DualEdgeWeightMode::DEVEX"
|
||||
STEEPEST_EDGE "DualEdgeWeightMode::STEEPEST_EDGE"
|
||||
Count "DualEdgeWeightMode::Count"
|
||||
|
||||
cdef enum PriceMode:
|
||||
ROW "PriceMode::ROW" = 0
|
||||
COL "PriceMode::COL"
|
||||
|
||||
const int PARALLEL_THREADS_DEFAULT
|
||||
const int DUAL_TASKS_MIN_THREADS
|
||||
const int DUAL_MULTI_MIN_THREADS
|
||||
|
||||
const bool invert_if_row_out_negative
|
||||
|
||||
const int NONBASIC_FLAG_TRUE
|
||||
const int NONBASIC_FLAG_FALSE
|
||||
|
||||
const int NONBASIC_MOVE_UP
|
||||
const int NONBASIC_MOVE_DN
|
||||
const int NONBASIC_MOVE_ZE
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
# cython: language_level=3
|
||||
|
||||
cdef extern from "highs_c_api.h" nogil:
|
||||
int Highs_passLp(void* highs, int numcol, int numrow, int numnz,
|
||||
double* colcost, double* collower, double* colupper,
|
||||
double* rowlower, double* rowupper,
|
||||
int* astart, int* aindex, double* avalue)
|
||||
@@ -0,0 +1,158 @@
|
||||
from __future__ import annotations
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import numpy as np
|
||||
|
||||
from ._optimize import OptimizeResult
|
||||
from ._pava_pybind import pava
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import numpy.typing as npt
|
||||
|
||||
|
||||
__all__ = ["isotonic_regression"]
|
||||
|
||||
|
||||
def isotonic_regression(
|
||||
y: npt.ArrayLike,
|
||||
*,
|
||||
weights: npt.ArrayLike | None = None,
|
||||
increasing: bool = True,
|
||||
) -> OptimizeResult:
|
||||
r"""Nonparametric isotonic regression.
|
||||
|
||||
A (not strictly) monotonically increasing array `x` with the same length
|
||||
as `y` is calculated by the pool adjacent violators algorithm (PAVA), see
|
||||
[1]_. See the Notes section for more details.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
y : (N,) array_like
|
||||
Response variable.
|
||||
weights : (N,) array_like or None
|
||||
Case weights.
|
||||
increasing : bool
|
||||
If True, fit monotonic increasing, i.e. isotonic, regression.
|
||||
If False, fit a monotonic decreasing, i.e. antitonic, regression.
|
||||
Default is True.
|
||||
|
||||
Returns
|
||||
-------
|
||||
res : OptimizeResult
|
||||
The optimization result represented as a ``OptimizeResult`` object.
|
||||
Important attributes are:
|
||||
|
||||
- ``x``: The isotonic regression solution, i.e. an increasing (or
|
||||
decreasing) array of the same length than y, with elements in the
|
||||
range from min(y) to max(y).
|
||||
- ``weights`` : Array with the sum of case weights for each block
|
||||
(or pool) B.
|
||||
- ``blocks``: Array of length B+1 with the indices of the start
|
||||
positions of each block (or pool) B. The j-th block is given by
|
||||
``x[blocks[j]:blocks[j+1]]`` for which all values are the same.
|
||||
|
||||
Notes
|
||||
-----
|
||||
Given data :math:`y` and case weights :math:`w`, the isotonic regression
|
||||
solves the following optimization problem:
|
||||
|
||||
.. math::
|
||||
|
||||
\operatorname{argmin}_{x_i} \sum_i w_i (y_i - x_i)^2 \quad
|
||||
\text{subject to } x_i \leq x_j \text{ whenever } i \leq j \,.
|
||||
|
||||
For every input value :math:`y_i`, it generates a value :math:`x_i` such
|
||||
that :math:`x` is increasing (but not strictly), i.e.
|
||||
:math:`x_i \leq x_{i+1}`. This is accomplished by the PAVA.
|
||||
The solution consists of pools or blocks, i.e. neighboring elements of
|
||||
:math:`x`, e.g. :math:`x_i` and :math:`x_{i+1}`, that all have the same
|
||||
value.
|
||||
|
||||
Most interestingly, the solution stays the same if the squared loss is
|
||||
replaced by the wide class of Bregman functions which are the unique
|
||||
class of strictly consistent scoring functions for the mean, see [2]_
|
||||
and references therein.
|
||||
|
||||
The implemented version of PAVA according to [1]_ has a computational
|
||||
complexity of O(N) with input size N.
|
||||
|
||||
References
|
||||
----------
|
||||
.. [1] Busing, F. M. T. A. (2022).
|
||||
Monotone Regression: A Simple and Fast O(n) PAVA Implementation.
|
||||
Journal of Statistical Software, Code Snippets, 102(1), 1-25.
|
||||
:doi:`10.18637/jss.v102.c01`
|
||||
.. [2] Jordan, A.I., Mühlemann, A. & Ziegel, J.F.
|
||||
Characterizing the optimal solutions to the isotonic regression
|
||||
problem for identifiable functionals.
|
||||
Ann Inst Stat Math 74, 489-514 (2022).
|
||||
:doi:`10.1007/s10463-021-00808-0`
|
||||
|
||||
Examples
|
||||
--------
|
||||
This example demonstrates that ``isotonic_regression`` really solves a
|
||||
constrained optimization problem.
|
||||
|
||||
>>> import numpy as np
|
||||
>>> from scipy.optimize import isotonic_regression, minimize
|
||||
>>> y = [1.5, 1.0, 4.0, 6.0, 5.7, 5.0, 7.8, 9.0, 7.5, 9.5, 9.0]
|
||||
>>> def objective(yhat, y):
|
||||
... return np.sum((yhat - y)**2)
|
||||
>>> def constraint(yhat, y):
|
||||
... # This is for a monotonically increasing regression.
|
||||
... return np.diff(yhat)
|
||||
>>> result = minimize(objective, x0=y, args=(y,),
|
||||
... constraints=[{'type': 'ineq',
|
||||
... 'fun': lambda x: constraint(x, y)}])
|
||||
>>> result.x
|
||||
array([1.25 , 1.25 , 4. , 5.56666667, 5.56666667,
|
||||
5.56666667, 7.8 , 8.25 , 8.25 , 9.25 ,
|
||||
9.25 ])
|
||||
>>> result = isotonic_regression(y)
|
||||
>>> result.x
|
||||
array([1.25 , 1.25 , 4. , 5.56666667, 5.56666667,
|
||||
5.56666667, 7.8 , 8.25 , 8.25 , 9.25 ,
|
||||
9.25 ])
|
||||
|
||||
The big advantage of ``isotonic_regression`` compared to calling
|
||||
``minimize`` is that it is more user friendly, i.e. one does not need to
|
||||
define objective and constraint functions, and that it is orders of
|
||||
magnitudes faster. On commodity hardware (in 2023), for normal distributed
|
||||
input y of length 1000, the minimizer takes about 4 seconds, while
|
||||
``isotonic_regression`` takes about 200 microseconds.
|
||||
"""
|
||||
yarr = np.atleast_1d(y) # Check yarr.ndim == 1 is implicit (pybind11) in pava.
|
||||
order = slice(None) if increasing else slice(None, None, -1)
|
||||
x = np.array(yarr[order], order="C", dtype=np.float64, copy=True)
|
||||
if weights is None:
|
||||
wx = np.ones_like(yarr, dtype=np.float64)
|
||||
else:
|
||||
warr = np.atleast_1d(weights)
|
||||
|
||||
if not (yarr.ndim == warr.ndim == 1 and yarr.shape[0] == warr.shape[0]):
|
||||
raise ValueError(
|
||||
"Input arrays y and w must have one dimension of equal length."
|
||||
)
|
||||
if np.any(warr <= 0):
|
||||
raise ValueError("Weights w must be strictly positive.")
|
||||
|
||||
wx = np.array(warr[order], order="C", dtype=np.float64, copy=True)
|
||||
n = x.shape[0]
|
||||
r = np.full(shape=n + 1, fill_value=-1, dtype=np.intp)
|
||||
x, wx, r, b = pava(x, wx, r)
|
||||
# Now that we know the number of blocks b, we only keep the relevant part
|
||||
# of r and wx.
|
||||
# As information: Due to the pava implementation, after the last block
|
||||
# index, there might be smaller numbers appended to r, e.g.
|
||||
# r = [0, 10, 8, 7] which in the end should be r = [0, 10].
|
||||
r = r[:b + 1]
|
||||
wx = wx[:b]
|
||||
if not increasing:
|
||||
x = x[::-1]
|
||||
wx = wx[::-1]
|
||||
r = r[-1] - r[::-1]
|
||||
return OptimizeResult(
|
||||
x=x,
|
||||
weights=wx,
|
||||
blocks=r,
|
||||
)
|
||||
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user