Optimize: Idea to remove DefaultSettings()
Created by: btracey
Currently, we have DefaultSettingsLocal
and DefaultSettingsGlobal
for users to get easy defaults. In Go generally, and also in Gonum specifically, we try to have the zero value be the default value. This issue is a proposal to make that true
Aside from being un-go-like, there are two gotchas based on the current design. First, we have to choose some default value, and right now that default is DefaultLocalSettings
. If users change the method to a global method, but do not also change the settings, they are going to get weird behavior. Second, because not all of the settings can use the zero value, you cannot naively just create the zero value of settings and modify the settings you care about. We do this in the tests, but we know what we're doing. For a typical user who probably does not want to understand all of the settings and their interactions, this is easy to mess up, in particular the FunctionTolerance
setting could easily lead to mistakes.
There are a couple challenges to having a zero-value default struct.
- We have the
FunctionThreshold
setting, which has a meaningful value for zero - There are two categories of default settings, Local and Global, and how do you make a reasonable value for both of them?
- In particular for the above, Local optimization techniques typically define convergence by having the gradient be a small value. Global optimization techniques typically have some nebulous definition of convergence based on number of function evaluations.
I think there are two parts to the solution. The first is to adopt #488 (closed) . This would replace both the existing FunctionThreshold
and FunctionConverge
with an abstraction that takes in updates, and returns if convergence happens. By default, we could use the existing FunctionConverge
. Using an interface makes it really easy for the user to implement different criteria, for example FunctionThreshold
.
I think the correct solution to the gradient problem is to take to heart the current warning in the documentation Be aware that the default settings of Minimize are to accurately find the minimum.
. Instead of seeing the default gradient tolerance as a setting, we could instead see the gradient tolerance as a limitation of local optimizers. What do I mean by that? Well, currently CmaEsChol
returns a MethodConverged
status when the covariance matrix gets sufficiently small. When it's small, there's nothing reasonable the algorithm can do, it has converged. Similarly, when a local optimizer gets to a location with a small gradient, there is nothing it can do either, it has converged. The proposal is that instead of having defaultGradientAbsTol
be seen as a parameter to Settings
, we instead see it as a parameter to local gradient-based optimizers. BFGS
and friends could have a StopGradient
, much in the way CmaEsChol
has a StopLogDet
, and when they reach a point with sufficiently small gradient, use that to return a terminated Status
. We could set that default value to be something like 1e-10
. If users wanted something less strict (as would often be reasonable), they could use the settings struct to set their own gradient tolerance.
If this is adopted, it would mean that 1) passing a nil
value of Settings
to Minimize
would do roughly the right thing for both local and global optimization 2) Users could change settings by starting with
s = &optimize.Settings{}
s.FunctionEvaluations = 50
and avoid the current gotcha.