Constraints in Bayesian Optimization

Bounds

bayesopt requires finite bounds on all variables. (categorical variables are, by nature, bounded in their possible values.) Pass the lower and upper bounds for real and integer-valued variables in optimizableVariable.

bayesopt uses these bounds to sample points, either uniformly or log-scaled. You set the scaling for sampling in optimizableVariable.

For example, to constrain a variable X1 to values between 1e-6 and 1e3, scaled logarithmically,

xvar = optimizableVariable('X1',[1e-6,1e3],'Transform','log')

bayesopt includes the endpoints in its range. Therefore, you cannot use 0 as a lower bound for a real log-transformed variable.

Tip

To use a zero lower bound in a real log-transformed variable, set the lower bound to 1, then inside the objective function use x-1.

For an integer-valued log-transformed variable, you can use 0 as a lower bound. If a lower bound is 0 for an integer-valued variable, then the software creates the log-scaled space by using the log1p function instead of the log function to include 0 in the variable sampling range. log1p is a function that returns log(1+x) for an input x.

Deterministic Constraints — XConstraintFcn

Sometimes your problem is valid or well-defined only for points in a certain region, called the feasible region. A deterministic constraint is a deterministic function that returns true when a point is feasible, and false when a point is infeasible. So deterministic constraints are not stochastic, and they are not functions of a group of points, but of individual points.

Tip

It is more efficient to use optimizableVariable bounds, instead of deterministic constraints, to confine the optimization to a rectangular region.

Write a deterministic constraint function using the signature

tf = xconstraint(X)

  • X is a width-D table of arbitrary height.

  • tf is a logical column vector, where tf(i) = true exactly when X(i,:) is feasible.

Pass the deterministic constraint function in the bayesopt XConstraintFcn name-value argument. For example,

results = bayesopt(fun,vars,'XConstraintFcn',@xconstraint)

bayesopt evaluates deterministic constraints on thousands of points, and so runs faster when your constraint function is vectorized. See Vectorization.

For example, suppose that the variables named 'x1' and 'x2' are feasible when the norm of the vector [x1 x2] is less than 6, and when x1 <= x2. The following constraint function evaluates these constraints.

function tf = xconstraint(X)
tf1 = sqrt(X.x1.^2 + X.x2.^2) < 6;
tf2 = X.x1 <= X.x2;
tf = tf1 & tf2;

Conditional Constraints — ConditionalVariableFcn

Conditional constraints are functions that enforce one of the following two conditions:

  • When some variables have certain values, other variables are set to given values.

  • When some variables have certain values, other variables have NaN or, for categorical variables, <undefined> values.

Specify a conditional constraint by setting the bayesopt ConditionalVariableFcn name-value argument to a function handle, say @condvariablefcn. The @condvariablefcn function must have the signature

Xnew = condvariablefcn(X)

  • X is a width-D table of arbitrary height.

  • Xnew is a table the same type and size as X.

condvariablefcn sets Xnew to be equal to X, except it also sets the relevant variables in each row of Xnew to the correct values for the constraint.

Note

If you have both conditional constraints and deterministic constraints, bayesopt applies the conditional constraints first. Therefore, if your conditional constraint function can set variables to NaN or <undefined>, ensure that your deterministic constraint function can process these values correctly.

Conditional constraints ensure that variable values are sensible. Therefore, bayesopt applies conditional constraints first so that all passed values are sensible.

Conditional Constraint That Sets a Variable Value

Suppose that you are optimizing a classification using fitcdiscr, and you optimize over both the 'DiscrimType' and 'Gamma' name-value arguments. When 'DiscrimType' is one of the quadratic types, 'Gamma' must be 0 or the solver errors. In that case, use this conditional constraint function:

function XTable = fitcdiscrCVF(XTable)
% Gamma must be 0 if discrim type is a quadratic
XTable.Gamma(ismember(XTable.DiscrimType, {'quadratic',...
        'diagQuadratic','pseudoQuadratic'})) = 0;
end

Conditional Constraint That Sets a Variable to NaN

Suppose that you are optimizing a classification using fitcsvm, and you optimize over both the 'KernelFunction' and 'PolynomialOrder' name-value arguments. When 'KernelFunction' is not 'polynomial', the 'PolynomialOrder' setting does not apply. The following function enforces this conditional constraint.

function Xnew = condvariablefcn(X)
Xnew = X;
Xnew.PolynomialOrder(Xnew.KernelFunction ~= 'polynomial') = NaN;

You can save a line of code as follows:

function X = condvariablefcn(X)
X.PolynomialOrder(X.KernelFunction ~= 'polynomial') = NaN;

In addition, define an objective function that does not pass the 'PolynomialOrder' name-value argument to fitcsvm when the value of 'PolynomialOrder' is NaN.

fun = @(X)mysvmfun(X,predictors,responce,c)

function objective = mysvmfun(X,predictors,response,c)
    args = {predictors,response, ...
        'CVPartition',c, ...
        'KernelFunction',X.KernelFunction};
    if ~isnan(X.PolynomialOrder)
        args = [args,{'PolynomialOrder',X.PolynomialOrder}];
    end
    objective = kfoldLoss(fitcsvm(args{:}));
end

Coupled Constraints

Coupled constraints are constraints that you can evaluate only by calling the objective function. These constraints can be stochastic or deterministic. Return these constraint values from your objective function in the second argument. See Bayesian Optimization Objective Functions.

The objective function returns a numeric vector for the coupled constraints, one entry for each coupled constraint. For each entry, a negative value indicates that the constraint is satisfied (also called feasible). A positive value indicates that the constraint is not satisfied (infeasible).

bayesopt automatically creates a coupled constraint, called the Error constraint, for every run. This constraint enables bayesopt to model points that cause errors in objective function evaluation. For details, see Objective Function Errors and predictError.

If you have coupled constraints in addition to the Error constraint:

  • Include the NumCoupledConstraints name-value argument in your bayesopt call (required). Do not include the Error constraint in this number.

  • If any of your coupled constraints are stochastic, include the AreCoupledConstraintsDeterministic name-value argument and pass false for any stochastic constraint.

Observe the coupled constraint values in each iteration by setting the bayesopt Verbose name-value argument to 1 or 2.

Note

When there are coupled constraints, iterative display and plot functions can give counterintuitive results such as:

  • A minimum objective plot can increase.

  • The optimization can declare a problem infeasible even when it showed an earlier feasible point.

The reason for this behavior is that the decision about whether a point is feasible can change as the optimization progresses. bayesopt determines feasibility with respect to its constraint model, and this model changes as bayesopt evaluates points. So a “minimum objective” plot can increase when the minimal point is later deemed infeasible, and the iterative display can show a feasible point that is later deemed infeasible.

For an example, see Bayesian Optimization with Coupled Constraints.

Bayesian Optimization with Coupled Constraints

Open Script

A coupled constraint is one that can be evaluated only by evaluating the objective function. In this case, the objective function is the cross-validated loss of an SVM model. The coupled constraint is that the number of support vectors is no more than 100. The model details are in Optimize Cross-Validated Classifier Using bayesopt.

Create the data for classification.

rng default
grnpop = mvnrnd([1,0],eye(2),10);
redpop = mvnrnd([0,1],eye(2),10);
redpts = zeros(100,2);
grnpts = redpts;
for i = 1:100
    grnpts(i,:) = mvnrnd(grnpop(randi(10),:),eye(2)*0.02);
    redpts(i,:) = mvnrnd(redpop(randi(10),:),eye(2)*0.02);
end
cdata = [grnpts;redpts];
grp = ones(200,1);
grp(101:200) = -1;
c = cvpartition(200,'KFold',10);
sigma = optimizableVariable('sigma',[1e-5,1e5],'Transform','log');
box = optimizableVariable('box',[1e-5,1e5],'Transform','log');

The objective function is the cross-validation loss of the SVM model for partition c. The coupled constraint is the number of support vectors minus 100.5. This ensures that 100 support vectors give a negative constraint value, but 101 support vectors give a positive value. The model has 200 data points, so the coupled constraint values range from -99.5 (there is always at least one support vector) to 99.5. Positive values mean the constraint is not satisfied.

function [objective,constraint] = mysvmfun(x,cdata,grp,c)
SVMModel = fitcsvm(cdata,grp,'KernelFunction','rbf',...
    'BoxConstraint',x.box,...
    'KernelScale',x.sigma);
cvModel = crossval(SVMModel,'CVPartition',c);
objective = kfoldLoss(cvModel);
constraint = sum(SVMModel.IsSupportVector)-100.5;

Pass the partition c and fitting data cdata and grp to the objective function fun by creating fun as an anonymous function that incorporates this data. See Parameterizing Functions.

fun = @(x)mysvmfun(x,cdata,grp,c);

Set the NumCoupledConstraints to 1 so the optimizer knows that there is a coupled constraint. Set options to plot the constraint model.

results = bayesopt(fun,[sigma,box],'IsObjectiveDeterministic',true,...
    'NumCoupledConstraints',1,'PlotFcn',...
    {@plotMinObjective,@plotConstraintModels},...
    'AcquisitionFunctionName','expected-improvement-plus','Verbose',0);

Most points lead to an infeasible number of support vectors.

See Also

|

Topics