> ## Documentation Index
> Fetch the complete documentation index at: https://wb-21fd5541-docs-2751.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Learn how to create configuration files for sweeps.

# Overview

A sweep combines a strategy for exploring hyperparameter values with the code that evaluates them. The strategy can be as simple as trying every option or as complex as Bayesian Optimization and Hyperband ([BOHB](https://arxiv.org/abs/1807.01774)).

This guide shows you how to define a sweep configuration that specifies which hyperparameters to search, which search strategy to use, and how to evaluate each run. Use it when you're setting up a new sweep or adapting an existing configuration to a different search method or parameter space.

Define a sweep configuration either in a [Python dictionary](https://docs.python.org/3/tutorial/datastructures.html#dictionaries) or a [YAML](https://yaml.org/) file. How you define your sweep configuration depends on how you want to manage your sweep.

<Note>
  Define your sweep configuration in a YAML file if you want to initialize a sweep and start a sweep agent from the command line. Define your sweep in a Python dictionary if you initialize a sweep and start a sweep entirely within a Python script or notebook.
</Note>

The following sections describe how to format your sweep configuration. See [Sweep configuration options](/models/sweeps/sweep-config-keys) for a comprehensive list of top-level sweep configuration keys.

## Basic structure

Sweep configurations are defined using either YAML or a Python dictionary, with key-value pairs and nested structures.

Use top-level keys within your sweep configuration to define qualities of your sweep search, such as the sweep [`name`](/models/sweeps/sweep-config-keys), the [`parameters`](/models/sweeps/sweep-config-keys#parameters) to search through, and the search [`method`](/models/sweeps/sweep-config-keys#method).

For example, the following code snippets show the same sweep configuration defined within a YAML file and within a Python dictionary. The sweep configuration specifies five top-level keys: `program`, `name`, `method`, `metric`, and `parameters`.

<Tabs>
  <Tab title="CLI">
    To manage sweeps interactively from the command line, define a sweep configuration in a YAML file.

    ```yaml title="config.yaml" theme={null}
    program: train.py
    name: sweepdemo
    method: bayes
    metric:
      goal: minimize
      name: validation_loss
    parameters:
      learning_rate:
        min: 0.0001
        max: 0.1
      batch_size:
        values: [16, 32, 64]
      epochs:
        values: [5, 10, 15]
      optimizer:
        values: ["adam", "sgd"]
    ```
  </Tab>

  <Tab title="Python script or notebook">
    Define a sweep in a Python dictionary data structure if you define your training algorithm in a Python script or notebook.

    The following code snippet stores a sweep configuration in a variable named `sweep_configuration`:

    ```python title="train.py" theme={null}
    sweep_configuration = {
        "name": "sweepdemo",
        "method": "bayes",
        "metric": {"goal": "minimize", "name": "validation_loss"},
        "parameters": {
            "learning_rate": {"min": 0.0001, "max": 0.1},
            "batch_size": {"values": [16, 32, 64]},
            "epochs": {"values": [5, 10, 15]},
            "optimizer": {"values": ["adam", "sgd"]},
        },
    }
    ```
  </Tab>
</Tabs>

The top-level `parameters` key nests the following keys: `learning_rate`, `batch_size`, `epochs`, and `optimizer`. For each nested key you specify, you can provide one or more values, a distribution, a probability, and more. For more information, see the [parameters](/models/sweeps/sweep-config-keys#parameters) section in [Sweep configuration options](/models/sweeps/sweep-config-keys).

## Double-nested parameters

Use nested parameters when you want to group related hyperparameters together or when your training code expects a nested configuration structure. To define a nested parameter, include an additional `parameters` key under the top-level parameter name.

The following example shows a sweep configuration with nested parameters `nested_category_1`, `nested_category_2`, and `nested_category_3`, each including the additional parameters `momentum` and `weight_decay`. The following code examples show the configuration in both a YAML file and a Python dictionary:

<Tabs>
  <Tab title="CLI">
    ```yaml theme={null}
    program: sweep_nest.py
    name: nested_sweep
    method: random
    metric:
      name: loss
      goal: minimize
    parameters:
      optimizer:
        values: ['adam', 'sgd']
      fc_layer_size:
        values: [128, 256, 512]
      dropout:
        values: [0.3, 0.4, 0.5]
      epochs:
        value: 1
      learning_rate:
        distribution: uniform
        min: 0
        max: 0.1
      batch_size:
        distribution: q_log_uniform_values
        q: 8
        min: 32
        max: 256
      nested_category_1:
        parameters:
          momentum:
            distribution: uniform
            min: 0.0
            max: 0.9
          weight_decay:
            values: [0.0001, 0.0005, 0.001]
      nested_category_2:
        parameters:
          momentum:
            distribution: uniform
            min: 0.0
            max: 0.9
          weight_decay:
            values: [0.1, 0.2, 0.3]
      nested_category_3:
        parameters:
          momentum:
            distribution: uniform
            min: 0.5
            max: 0.7
          weight_decay:
            values: [0.2, 0.3, 0.4]
    ```
  </Tab>

  <Tab title="Python script or notebook">
    ```python theme={null}
    {
      "program": "sweep_nest.py",
      "name": "nested_sweep",
      "method": "random",
      "metric": {
        "name": "loss",
        "goal": "minimize"
      },
      "parameters": {
        "optimizer": {
          "values": ["adam", "sgd"]
        },
        "fc_layer_size": {
          "values": [128, 256, 512]
        },
        "dropout": {
          "values": [0.3, 0.4, 0.5]
        },
        "epochs": {
          "value": 1
        },
        "learning_rate": {
          "distribution": "uniform",
          "min": 0,
          "max": 0.1
        },
        "batch_size": {
          "distribution": "q_log_uniform_values",
          "q": 8,
          "min": 32,
          "max": 256
        },
        "nested_category_1": {
          "parameters": {
            "momentum": {
              "distribution": "uniform",
              "min": 0.0,
              "max": 0.9
            },
            "weight_decay": {
              "values": [0.0001, 0.0005, 0.001]
            }
          }
        },
        "nested_category_2": {
          "parameters": {
            "momentum": {
              "distribution": "uniform",
              "min": 0.0,
              "max": 0.9
            },
            "weight_decay": {
              "values": [0.1, 0.2, 0.3]
            }
          }
        },
        "nested_category_3": {
          "parameters": {
            "momentum": {
              "distribution": "uniform",
              "min": 0.5,
              "max": 0.7
            },
            "weight_decay": {
              "values": [0.2, 0.3, 0.4]
            }
          }
        }
      }
    }
    ```
  </Tab>
</Tabs>

<Warning>
  Nested parameters defined in sweep configuration overwrite keys specified in a W\&B run configuration.

  As an example, suppose you have `train.py` script that initializes a run with a nested default:

  ```python theme={null}
  def main():
      with  wandb.init(config={"nested_param": {"manual_key": 1}}) as run:
          # Your training code here
  ```

  Your sweep configuration defines nested parameters under a top-level `"parameters"` key:

  ```python theme={null}
  sweep_configuration = {
      "method": "grid",
      "metric": {"name": "score", "goal": "minimize"},
      "parameters": {
          "top_level_param": {"value": 0},
          "nested_param": {
              "parameters": {
                  "learning_rate": {"value": 0.01},
                  "double_nested_param": {
                      "parameters": {"x": {"value": 0.9}, "y": {"value": 0.8}}
                  },
              }
          },
      },
  }

  sweep_id = wandb.sweep(sweep=sweep_configuration, project="[PROJECT]")
  wandb.agent(sweep_id, function=main, count=4)
  ```

  During a sweep run, `run.config["nested_param"]` reflects the subtree defined by the sweep configuration (`learning_rate` and `double_nested_param`). It doesn't include `manual_key`, which is defined in `wandb.init(config=...)`.
</Warning>

## Sweep configuration template

Use this template as a starting point for a new sweep configuration. It illustrates the most common parameter and early-termination patterns. Replace `hyperparameter_name` with the name of your hyperparameter and any values enclosed in brackets.

```yaml title="config.yaml" theme={null}
program: [INSERT]
method: [INSERT]
parameters:
  hyperparameter_name0:
    value: 0
  hyperparameter_name1:
    values: [0, 0, 0]
  hyperparameter_name:
    distribution: [INSERT]
    value: [INSERT]
  hyperparameter_name2:
    distribution: [INSERT]
    min: [INSERT]
    max: [INSERT]
    q: [INSERT]
  hyperparameter_name3:
    distribution: [INSERT]
    values:
      - [LIST-OF-VALUES]
      - [LIST-OF-VALUES]
      - [LIST-OF-VALUES]
early_terminate:
  type: hyperband
  s: [INSERT]
  eta: [INSERT]
  max_iter: [INSERT]
command:
- ${Command macro}
- ${Command macro}
- ${Command macro}
- ${Command macro}
```

To express a numeric value using scientific notation, add the YAML `!!float` operator, which casts the value to a floating-point number. For example, `min: !!float 1e-5`. For more information, see [Macro and custom command arguments example](#macro-and-custom-command-arguments-example).

## Sweep configuration examples

The following sweep configurations illustrate common scenarios. Use them as references when adapting a sweep to your own training script.

<Tabs>
  <Tab title="CLI">
    ```yaml title="config.yaml" theme={null}
    program: train.py
    method: random
    metric:
      goal: minimize
      name: loss
    parameters:
      batch_size:
        distribution: q_log_uniform_values
        max: 256
        min: 32
        q: 8
      dropout:
        values: [0.3, 0.4, 0.5]
      epochs:
        value: 1
      fc_layer_size:
        values: [128, 256, 512]
      learning_rate:
        distribution: uniform
        max: 0.1
        min: 0
      optimizer:
        values: ["adam", "sgd"]
    ```
  </Tab>

  <Tab title="Python script or notebook">
    ```python title="train.py" theme={null}
    sweep_config = {
        "method": "random",
        "metric": {"goal": "minimize", "name": "loss"},
        "parameters": {
            "batch_size": {
                "distribution": "q_log_uniform_values",
                "max": 256,
                "min": 32,
                "q": 8,
            },
            "dropout": {"values": [0.3, 0.4, 0.5]},
            "epochs": {"value": 1},
            "fc_layer_size": {"values": [128, 256, 512]},
            "learning_rate": {"distribution": "uniform", "max": 0.1, "min": 0},
            "optimizer": {"values": ["adam", "sgd"]},
        },
    }
    ```
  </Tab>
</Tabs>

### Bayes hyperband example

The following example combines Bayesian search with Hyperband early termination to stop underperforming runs early and preserve resources for more promising configurations.

```yaml theme={null}
program: train.py
method: bayes
metric:
  goal: minimize
  name: val_loss
parameters:
  dropout:
    values: [0.15, 0.2, 0.25, 0.3, 0.4]
  hidden_layer_size:
    values: [96, 128, 148]
  layer_1_size:
    values: [10, 12, 14, 16, 18, 20]
  layer_2_size:
    values: [24, 28, 32, 36, 40, 44]
  learn_rate:
    values: [0.001, 0.01, 0.003]
  decay:
    values: [1e-5, 1e-6, 1e-7]
  momentum:
    values: [0.8, 0.9, 0.95]
  epochs:
    value: 27
early_terminate:
  type: hyperband
  s: 2
  eta: 3
  max_iter: 27
```

The following tabs show how to specify either a minimum or maximum number of iterations for `early_terminate`:

<Tabs>
  <Tab title="Minimum number of iterations">
    The brackets for this example are `[3, 3*eta, 3*eta*eta, 3*eta*eta*eta]`, which equals `[3, 9, 27, 81]`.

    ```yaml theme={null}
    early_terminate:
      type: hyperband
      min_iter: 3
    ```
  </Tab>

  <Tab title="Maximum number of iterations">
    The brackets for this example are `[27/eta, 27/eta/eta]`, which equals `[9, 3]`.

    ```yaml theme={null}
    early_terminate:
      type: hyperband
      max_iter: 27
      s: 2
    ```
  </Tab>
</Tabs>

### Macro and custom command arguments example

This example shows how to construct the command that the sweep agent runs for each trial when you need finer control than the default invocation provides.

For more complex command-line arguments, you can use macros to pass environment variables, the Python interpreter, and additional arguments. [W\&B supports predefined macros](/models/sweeps/sweep-config-keys#command-macros) and custom command-line arguments that you can specify in your sweep configuration.

For example, the following sweep configuration (`sweep.yaml`) defines a command that runs a Python script (`run.py`) with the `${env}`, `${interpreter}`, and `${program}` macros replaced with the appropriate values when the sweep runs.

The `--batch_size=${batch_size}`, `--test=True`, and `--optimizer=${optimizer}` arguments use custom macros to pass the values of the `batch_size`, `test`, and `optimizer` parameters defined in the sweep configuration.

```yaml title="sweep.yaml" theme={null}
program: run.py
method: random
metric:
  name: validation_loss
parameters:
  learning_rate:
    min: 0.0001
    max: 0.1
command:
  - ${env}
  - ${interpreter}
  - ${program}
  - "--batch_size=${batch_size}"
  - "--optimizer=${optimizer}"
  - "--test=True"
```

The associated Python script `run.py` can then parse these command-line arguments using the `argparse` module:

```python title="run.py" theme={null}
# run.py
import wandb
import argparse


def str2bool(v: str) -> bool:
    """Convert a string such as "True" to a boolean, because argparse
    doesn't support boolean arguments by default.
    """
    if isinstance(v, bool):
        return v
    return v.lower() in ('yes', 'true', 't', '1')


parser = argparse.ArgumentParser()
parser.add_argument('--batch_size', type=int)
parser.add_argument('--optimizer', type=str, choices=['adam', 'sgd'], required=True)
parser.add_argument('--test', type=str2bool, default=False)
args = parser.parse_args()

# Initialize a W&B run
with wandb.init(project="test-project") as run:
    run.log({'validation_loss': 1})
```

See the [Command macros](/models/sweeps/sweep-config-keys#command-macros) section in [Sweep configuration options](/models/sweeps/sweep-config-keys) for a list of predefined macros you can use in your sweep configuration.

#### Boolean arguments

If your sweep passes boolean flags through command arguments, your training script needs extra handling because `argparse` doesn't interpret boolean strings by default.

The `argparse` module doesn't support boolean arguments by default. To define a boolean argument, use the [`action`](https://docs.python.org/3/library/argparse.html#action) parameter or use a custom function to convert the string representation of the boolean value to a boolean type.

For example, you can use the following code snippet to define a boolean argument. Pass `store_true` or `store_false` as an argument to `ArgumentParser`:

```python theme={null}
import wandb
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--test', action='store_true')
args = parser.parse_args()

args.test  # This will be True if --test is passed, otherwise False
```

You can also define a custom function to convert the string representation of the boolean value to a boolean type. For example, the following code snippet defines the `str2bool` function, which converts a string to a boolean value:

```python theme={null}
def str2bool(v: str) -> bool:
  """Convert a string to a boolean. This is required because
  argparse doesn't support boolean arguments by default.
  """
  if isinstance(v, bool):
      return v
  return v.lower() in ('yes', 'true', 't', '1')
```