Skip to content

Networks

A BOSH network is an IaaS-agnostic representation of the networking layer. The Director is responsible for configuring each instance group's networks with the help of the BOSH Agent and the IaaS. Networking configuration is usually assigned at the boot of the VM and/or when network configuration changes in the deployment manifest for already-running instance groups.

There are three types of networks that BOSH supports:

  • manual: The Director decides how to assign IPs to each instance based on the specified network subnets in the deployment manifest
  • dynamic: The Director defers IP selection to the IaaS
  • vip: The Director allows one-off IP assignments to specific instances to enable flexible IP routing (e.g. elastic IP)

Each type of network supports one or both IP reservation types:

  • static: IP is explicitly requested by the user in the deployment manifest
  • automatic: IP is selected automatically based on the network type
manual network dynamic network vip network
Static IP assignment Supported Not supported Supported
Automatic IP assignment Supported, default Supported Supported

General Structure

Networking configuration is usually done in three steps:

  • Configuring the IaaS: Outside of BOSH's responsibility
  • Example on AWS: User creates a VPC and subnets with routing tables.
  • Adding networks section to the deployment manifest to define networks used in this deployment
  • Example: User adds a manual network with a subnet and adds AWS subnet ID into the subnet's cloud properties.
  • Adding network associations for one or more networks to each instance group

All deployment manifests have a similar structure in terms of network definitions and associations:

# cloud-config.yml
---
networks:
- name: my-network
  ...

- name: my-other-network
  type: ...

  # IaaS specific attributes
  cloud_properties: { ... }

# deployment.yml
---
instance_groups:
- name: my-instance-group

  # Network associations for `my-instance-group`
  networks:
  - name: my-network
  ...

- name: my-multi-homed-instance-group
  networks:
  - name: my-network
  - name: my-other-network
  ...

- name: my-static-instance-group
  networks:
  - name: my-network
    # Static IP reservations for `my-instance-group`
    static_ips: [IP1]
  ...

See how to define each network type below.


Manual Networks

Manual networking allows you to specify one or more subnets and let the Director choose available IPs from one of the subnet ranges. A subnet definition specifies the CIDR range and, optionally, the gateway and DNS servers. In addition, certain IPs can be blacklisted (the Director will not use these IPs) via the reserved property.

Each manual network attached to an instance is typically represented as its own NIC in the IaaS layer.

Schema for manual network definition:

  • name [String, required]: Name used to reference this network configuration
  • type [String, required]: Value should be manual
  • subnets [Array, required]: Lists subnets in this network
    • range [String, required]: Subnet IP range that includes all IPs from this subnet
    • gateway [String, required]: Subnet gateway IP
    • dns [Array, optional]: DNS IP addresses for this subnet
    • reserved [Array, optional]: Array of reserved IPs and/or IP ranges. BOSH does not assign IPs from this range to any VM
    • static [Array, optional]: Array of static IPs and/or IP ranges. BOSH assigns IPs from this range to instances requesting static IPs. Only IPs specified here can be used for static IP reservations.
    • az [String, optional]: AZ associated with this subnet (should only be used when using first class AZs). Example: z1. Available in v241+.
    • azs [Array, optional]: List of AZs associated with this subnet (should only be used when using first class AZs). Example: [z1, z2]. Available in v241+.
    • cloud_properties [Hash, optional]: Describes any IaaS-specific properties for the subnet. Default is {} (empty Hash).

Example cloud config:

networks:
- name: my-network
  type: manual

  subnets:
  - range:    10.10.0.0/24
    gateway:  10.10.0.1
    dns:      [10.10.0.2]

    # IPs that will not be used for anything
    reserved: [10.10.0.2-10.10.0.10]

    cloud_properties: {subnet: subnet-9be6c3f7}

  - range:   10.10.1.0/24
    gateway: 10.10.1.1
    dns:     [10.10.1.2]

    # IPs that can only be used for static IP reservations within this subnet
    static: [10.10.1.11-10.10.1.20]

    cloud_properties: {subnet: subnet-9be6c6gh}

Manual networks use automatic IP reservation by default. They also support static IP reservation. To assign specific IPs to instances of the instance group, they must be specified in instance group's networks section, in the static_ips property for the associated network. That network's subnet definition must also specify them in its static property:

instance_groups:
- name: my-instance-group-with-static-ip
  instances: 2
  ...
  networks:
  - name: my-network

    # IPs associated with 2 instances of `my-instance-group-with-static-ip`
    static_ips: [10.10.1.11, 10.10.1.12]

Note

If an instance group uses static IP reservation, all instances must be given static IPs.

A common problem that you may run into is configuring multiple deployments to use overlapping IP ranges. The Director does not consider an IP to be "used" even if the Director used that IP in a different deployment. There are two possible solutions for this problem: reconfigure one of the deployments to use a different IP range, or use the same IP range but configure each deployment such that reserved IPs exclude the deployment from each other.

Note

While the Director usually selects the next available IP address, this behavior is not guaranteed.


Dynamic Networks

Dynamic networking defers IP selection to the IaaS. For example, AWS assigns a private IP to each instance in the VPC by default. By associating an instance group to a dynamic network, BOSH will pick up AWS-assigned private IP addresses.

Each dynamic network attached to an instance group is typically represented as its own NIC in the IaaS layer.

Dynamic networking only supports automatic IP reservations.

Schema for dynamic network definition:

  • name [String, required]: Name used to reference this network configuration
  • type [String, required]: Value should be dynamic
  • dns [Array, optional]: DNS IP addresses for this network
  • cloud_properties [Hash, optional]: Describes any IaaS-specific properties for the network. Default is {} (empty Hash).

Example cloud config:

networks:
- name: my-network
  type: dynamic
  dns:  [10.10.0.2]
  cloud_properties: {subnet: subnet-9be6c3f7}

Schema for dynamic network definition with multiple subnets (available in v241+):

  • name [String, required]: Name used to reference this network configuration
  • type [String, required]: Value should be dynamic
  • subnets [Array, required]: Lists subnets in this network.
    • dns [Array, optional]: DNS IP addresses for this subnet
    • az [String, optional]: AZ associated with this subnet (should only be used when using first class AZs). Example: z1.
    • azs [Array, optional]: List of AZs associated with this subnet (should only be used when using first class AZs). Example: [z1, z2].
    • cloud_properties [Hash, optional]: Describes any IaaS-specific properties for the subnet. Default is {} (empty Hash).

Example cloud config:

networks:
- name: my-network
  type: dynamic
  subnets:
  - {az: z1, cloud_properties: {subnet: subnet-9be6c3f7}}
  - {az: z2, cloud_properties: {subnet: subnet-9be6c384}}

Virtual IP (VIP) Networks

Virtual IP networking enables the association of an IP address that is not backed by any particular NIC. This flexibility enables users to remap a virtual IP to a different instance in cases of a failure. For IaaS specific implementation details, see the respective cloud provider docs.

VIP network static IPs can either be defined in the deployment manifest (static IP assignment) or in the cloud config (automatic IP assignment). The two assignment types cannot be combined for a given network.

Static IP Assignment

Schema for VIP network where static IPs are configured in the deployment manifest:

  • name [String, required]: Name used to reference this network configuration
  • type [String, required]: Value should be vip
  • cloud_properties [Hash, optional]: Describes any IaaS-specific properties for the network. Default is {} (empty Hash).

Sample cloud config and deployment manifest:

# cloud-config.yml
---
networks:
- name: my-vip-network
  type: vip

# deployment.yml
---
instance_groups:
- name: my-instance-group
  ...
  networks:
  - name: my-vip-network
    static_ips: [54.47.189.8]

Automatic IP Assignment

Note

Available as of BOSH Director version 269.0.0

Schema for VIP network where static IPs are configured in the cloud config for use across deployments:

  • name [String, required]: Name used to reference this network configuration
  • type [String, required]: Value should be vip
  • subnets [Array, optional]: Lists subnets in this network
    • az [String, optional]: AZ associated with this subnet (should only be used when using first class AZs). Example: z1.
    • azs [Array, optional]: List of AZs associated with this subnet (should only be used when using first class AZs). Example: [z1, z2].
    • static [Array, optional]: Array of static IPs and/or IP ranges. BOSH assigns IPs from this range to instances requesting static IPs. Only IPs specified here can be used for static IP reservations.
    • cloud_properties [Hash, optional]: Describes any IaaS-specific properties for the subnet. Default is {} (empty Hash).

Sample cloud config and deployment manifest:

# cloud-config.yml
---
networks:
- name: my-vip-network
  type: vip
  subnets:
  - azs: [z1, z2]
    static:
    - 203.0.113.10
    - 203.0.113.12

# deployment.yml
---
instance_groups:
- name: my-instance-group
  ...
  networks:
  - name: my-vip-network

Migrating from Static IP Assignment

To migrate from static IP assignment to automatic IP assignment:

  1. Add subnet definitions to the network definition in the cloud config. These will be used to associate sets of IPs to availability zones.
  2. Find all IPs currently listed in the static_ips property of each instance group's use of that network.
  3. Move those IPs into the static property of their respective subnet.
  4. Delete the static_ips property from the network configuration of the instance groups.
  5. Update the cloud config and redeploy with the new manifests.

Warning

You will need to perform these steps for each deployment using the VIP network. After enabling the VIP network for automatic IP assignment, deployments relying on it for static IP assignment will error on next deploy.

Sample manifests before migration:

# cloud-config.yml
---
networks:
- name: my-vip-network
  type: vip

# deployment.yml
---
instance_groups:
- name: my-instance-group
  azs: [z1, z2]
  ...
  networks:
  - name: my-vip-network
    static_ips:
    - 203.0.113.10
    - 203.0.113.12

Sample manifests after migration:

# cloud-config.yml
---
networks:
- name: my-vip-network
  type: vip
  subnets:
  - azs: [z1, z2]
    static:
    - 203.0.113.10
    - 203.0.113.12

# deployment.yml
---
instance_groups:
- name: my-instance-group
  ...
  networks:
  - name: my-vip-network

Multi-homed VMs

An instance group can be configured to have multiple IP addresses (multiple NICs) by being on multiple networks. Given that there are multiple network settings available for an instance group, the Agent needs to decide which network's DNS settings to use and which network's gateway should be the default gateway on the VM. Agent performs such selection based on the network's default property specified in the instance group.

Schema for default property:

  • default [Array, optional]: Configures this network to provide its settings for specific category as a default. Possible values are: dns, gateway and since bosh-release v258 addressable. All values can be specified together. addressable can be used to specify which IP address other instances see.

Example:

# cloud-config.yml
---
networks:
- name: my-network-1
  type: dynamic
  dns: [8.8.8.8]

- name: my-network-2
  type: dynamic
  dns: [4.4.4.4]

# deployment.yml
---
instance_groups:
- name: my-multi-homed-instance-group
  ...
  networks:
  - name: my-network-1
    default: [dns, gateway]
  - name: my-network-2

- name: my-other-multi-homed-instance-group
  ...
  networks:
  - name: my-network-1
    default: [dns]
  - name: my-network-2
    default: [gateway]

In the above example, VM allocated to my-multi-homed-instance-group instance group will have 8.8.8.8 as its primary DNS server and the default gateway will be set to my-network-1's gateway. VM allocated to my-other-multi-homed-instance-group instance group will also have 8.8.8.8 as its primary DNS server but the default gateway will be set to my-network-2's gateway.

Note

See CPI limitations to find which CPIs support this feature.

Note

See rakutentech/bosh-routing-release if you are looking for even more specific routing configuration.


CPI Limitations

The Director does not enforce how many networks can be assigned to each instance; however, each CPI might impose custom requirements either due to the IaaS limitations or simply because support was not yet implemented.

manual network dynamic network vip network
AWS Single per instance group Single per instance group Single, corresponds to an elastic IP
Azure Multiple per instance group Multiple per instance group Single, corresponds to a reserved IP
OpenStack Multiple per instance group Single per instance group Single, corresponds to a floating IP
vSphere Multiple per instance group Not supported Not supported
vCloud Multiple per instance group Not supported Not supported

CPI Specific cloud_properties