hyperdrive Package

Contains modules and classes supporting hyperparameter tuning.

Hyperparameters are adjustable parameters you choose for model training that guide the training process. The HyperDrive package helps you automate choosing these parameters. For example, you can define the parameter search space as discrete or continuous, and a sampling method over the search space as random, grid, or Bayesian. Also, you can specify a primary metric to optimize in the hyperparameter tuning experiment, and whether to minimize or maximize that metric. You can also define early termination policies in which poorly performing experiment runs are canceled and new ones started. To define a reusable machine learning workflow for HyperDrive, use hyper_drive_step to create a Pipeline.

Modules

error_definition

Error code definitions for HyperDrive SDK.

error_strings

A collection of error strings used throughout the HyperDrive SDK.

exceptions

Exceptions thrown by HyperDrive.

parameter_expressions

Defines functions that can be used in HyperDrive to describe a hyperparameter search space.

These functions are used to specify different types of hyperparameter distributions. The distributions are defined when you configure sampling for a hyperparameter sweep. For example, when you use the RandomParameterSampling class, you can choose to sample from a set of discrete values or a distribution of continuous values. In this case, you could use the choice function to generate a discrete set of values and uniform function to generate a distribution of continuous values.

For examples of using these functions, see the tutorial: https://docs.microsoft.com/azure/machine-learning/how-to-tune-hyperparameters.

Classes

BanditPolicy

Defines an early termination policy based on slack criteria, and a frequency and delay interval for evaluation.

Initialize a BanditPolicy with slack factor, slack_amount, and evaluation interval.

BayesianParameterSampling

Defines Bayesian sampling over a hyperparameter search space.

Bayesian sampling tries to intelligently pick the next sample of hyperparameters, based on how the previous samples performed, such that the new sample improves the reported primary metric.

Initialize BayesianParameterSampling.

EarlyTerminationPolicy

Abstract base class for all early termination policies.

Initialize an early termination policy.

GridParameterSampling

Defines grid sampling over a hyperparameter search space.

Initialize GridParameterSampling.

HyperDriveConfig

Configuration that defines a HyperDrive run.

HyperDrive configuration includes information about hyperparameter space sampling, termination policy, primary metric, resume from configuration, estimator, and the compute target to execute the experiment runs on.

Initialize the HyperDriveConfig.

HyperDriveRun

HyperDriveRun contains the details of a submitted HyperDrive experiment.

This class can be used to manage, check status, and retrieve run details for the HyperDrive run and each of the generated child runs.

Initialize a HyperDrive run.

HyperDriveRunConfig

Configuration that defines a HyperDrive run.

Configuration includes information about parameter space sampling, termination policy, primary metric, estimator and the compute target to execute the experiment runs on.

Initialize the HyperDriveConfig.

HyperParameterSampling

Abstract base class for all hyperparameter sampling algorithms.

This class encapsulates the hyperparameter space, the sampling method, and additional properties for derived sampling classes: BayesianParameterSampling, GridParameterSampling, and RandomParameterSampling.

Initialize HyperParameterSampling.

MedianStoppingPolicy

Defines an early termination policy based on running averages of the primary metric of all runs.

Initialize a MedianStoppingPolicy.

NoTerminationPolicy

Specifies that no early termination policy is applied.

Each run will execute until completion.

Initialize NoTerminationPolicy.

RandomParameterSampling

Defines random sampling over a hyperparameter search space.

Initialize RandomParameterSampling.

TruncationSelectionPolicy

Defines an early termination policy that cancels a given percentage of runs at each evaluation interval.

Initialize a TruncationSelectionPolicy.

Enums

PrimaryMetricGoal

Defines supported metric goals for hyperparameter tuning.

A metric goal is used to determine whether a higher value for a metric is better or worse. Metric goals are used when comparing runs based on the primary metric. For example, you may want to maximize accuracy or minimize error.

The primary metric name and goal are specified in the HyperDriveConfig class when you configure a HyperDrive run.

Functions

choice

Specify a discrete set of options to sample from.

choice(*options)

Parameters

Name Description
options
Required

The list of options to choose from.

Returns

Type Description

The stochastic expression.

lognormal

Specify a value drawn according to exp(normal(mu, sigma)).

The logarithm of the return value is normally distributed. When optimizing, this variable is constrained to be positive.

lognormal(mu, sigma)

Parameters

Name Description
mu
Required

The mean of the normal distribution.

sigma
Required

The standard deviation of the normal distribution.

Returns

Type Description

The stochastic expression.

loguniform

Specify a log uniform distribution.

A value is drawn according to exp(uniform(min_value, max_value)) so that the logarithm of the return value is uniformly distributed. When optimizing, this variable is constrained to the interval [exp(min_value), exp(max_value)]

loguniform(min_value, max_value)

Parameters

Name Description
min_value
Required

The minimum value in the range will be exp(min_value)(inclusive).

max_value
Required

The maximum value in the range will be exp(max_value) (inclusive).

Returns

Type Description

The stochastic expression.

normal

Specify a real value that is normally-distributed with mean mu and standard deviation sigma.

When optimizing, this is an unconstrained variable.

normal(mu, sigma)

Parameters

Name Description
mu
Required

The mean of the normal distribution.

sigma
Required

the standard deviation of the normal distribution.

Returns

Type Description

The stochastic expression.

qlognormal

Specify a value like round(exp(normal(mu, sigma)) / q) * q.

Suitable for a discrete variable with respect to which the objective is smooth and gets smoother with the size of the variable, which is bounded from one side.

qlognormal(mu, sigma, q)

Parameters

Name Description
mu
Required

The mean of the normal distribution.

sigma
Required

The standard deviation of the normal distribution.

q
Required
int

The smoothing factor.

Returns

Type Description

The stochastic expression.

qloguniform

Specify a uniform distribution of the form round(exp(uniform(min_value, max_value) / q) * q.

This is suitable for a discrete variable with respect to which the objective is "smooth", and gets smoother with the size of the value, but which should be bounded both above and below.

qloguniform(min_value, max_value, q)

Parameters

Name Description
min_value
Required

The minimum value in the range (inclusive).

max_value
Required

The maximum value in the range (inclusive).

q
Required
int

The smoothing factor.

Returns

Type Description

The stochastic expression.

qnormal

Specify a value like round(normal(mu, sigma) / q) * q.

Suitable for a discrete variable that probably takes a value around mu, but is fundamentally unbounded.

qnormal(mu, sigma, q)

Parameters

Name Description
mu
Required

The mean of the normal distribution.

sigma
Required

The standard deviation of the normal distribution.

q
Required
int

The smoothing factor.

Returns

Type Description

The stochastic expression.

quniform

Specify a uniform distribution of the form round(uniform(min_value, max_value) / q) * q.

This is suitable for a discrete value with respect to which the objective is still somewhat "smooth", but which should be bounded both above and below.

quniform(min_value, max_value, q)

Parameters

Name Description
min_value
Required

The minimum value in the range (inclusive).

max_value
Required

The maximum value in the range (inclusive).

q
Required
int

The smoothing factor.

Returns

Type Description

The stochastic expression.

randint

Specify a set of random integers in the range [0, upper).

The semantics of this distribution is that there is no more correlation in the loss function between nearby integer values, as compared with more distant integer values. This is an appropriate distribution for describing random seeds for example. If the loss function is probably more correlated for nearby integer values, then you should probably use one of the "quantized" continuous distributions, such as either quniform, qloguniform, qnormal or qlognormal.

randint(upper)

Parameters

Name Description
upper
Required
int

The exclusive upper bound for the range of integers.

Returns

Type Description

The stochastic expression.

uniform

Specify a uniform distribution from which samples are taken.

uniform(min_value, max_value)

Parameters

Name Description
min_value
Required

The minimum value in the range (inclusive).

max_value
Required

The maximum value in the range (inclusive).

Returns

Type Description

The stochastic expression.