Skip to content

Agent Interactions

Here is an overview of the interactions between the CPI and the Agent on CloudStack as an example:

  • The CPI drives the IaaS and the Agent.
  • The Agent is a versatile process which configures the OS to leverage IaaS-provisioned resources (network interfaces, disks, etc.), and perform other BOSH tasks (job compilation, job instantiation, etc.)
  • The CPI asks the IaaS to instantiate VM template, VMs, volumes and possibly other constructs (floating IPs, security groups, connect LBs, etc.)
  • The director-to-agent communication happens through NATS-based messaging.
  • The initial configuration of the agent is done via the metadata server.

Note

If either of CPI, Director or stemcell does not support CPI API version 2, the BOSH registry is used to provide the information to configure the VM.

image

The following sequence diagram illustrates in more detail the CPI, agent, and registry interactions, in the case of the CloudStack CPI:

image

Agent settings

This section details the configuration and protocols supported by the Agent.

VM Configuration Locations provides a list of Agent configuration files and their roles.

agent.json file

/var/vcap/bosh/agent.json: Start up settings for the Agent that describe how to find bootstrap settings, and disable certain Agent functionality.

The loading agent.json file is described into config_test.go

The Platform part of the file is documented into LinuxOptions

The Infrastructure part of the file is documented into Options and in particular:

Sample agent.json which configures the agent to read from an HTTP metadata service at a custom URL:

{
  "Platform": {
    "Linux": {
      "CreatePartitionIfNoEphemeralDisk": true,
      "DevicePathResolutionType": "virtio"
    }
  },
  "Infrastructure": {
    "NetworkingType": "static",
    "Settings": {
      "Sources": [
        {
          "Type": "HTTP",
          "URI": "http://10.234.228.142"
        }
      ],
      "UseServerName": true,
      "UseRegistry": true
    }
  }
}

Sample agent.json which configures the agent to read from a config drive. See reference sample config-drive json. See config drive integration test:

{
  "Platform": {
    "Linux": {
      "UseDefaultTmpDir": true,
      "UsePreformattedPersistentDisk": true,
      "BindMountPersistentDisk": false,
      "DevicePathResolutionType": "virtio"
    }
  },
  "Infrastructure": {
    "Settings": {
      "Sources": [
        {
          "Type": "ConfigDrive",
          "DiskPaths": [
            "/dev/disk/by-label/CONFIG-2",
            "/dev/disk/by-label/config-2"
          ],
          "MetaDataPath": "ec2/latest/meta-data.json",
          "UserDataPath": "ec2/latest/user-data.json"
        }
      ],

      "UseServerName": true,
      "UseRegistry": true
    }
  }
}

Agent settings format

The current state of the JSON settings format supported by the Agent is documented in settings_test.go as well as in settings.go.

Some of the properties (for example blobstore or mbus URLs) can be specified in multiple places. In general the properties in the env section have precedence over the properties defined on root level.

Sample settings JSON:

{
  "agent_id": "agent-xxxxx",
  "blobstore": { },
  "disks": {
    "ephemeral": "/dev/sdb",
    "persistent": {},
    "raw_ephemeral": null,
    "system": "/dev/xvda"
  },
  "env": {
    "bosh": {
      "agent": {
        "settings": {
          "tmpfs": false
        }
      },
      "authorized_keys": [
        "--AUTHORIZED KEY--"
      ],
      "blobstores": [
        {
          "provider": "local",
          "options": {
            "endpoint": "http://xx.xx.xx.xx:25250",
            "password": "password",
            "blobstore_path": "/var/vcap/micro_bosh/data/cache",
            "user": "agent"
          }
        }
      ],
      "ipv6": {
        "enable": false
      },
      "keep_root_password": true,
      "mbus": {
        "cert": {
          "ca": "--MBUS CA--",
          "certificate": "--MBUS CERT--",
          "private_key": "--MBUS PRIVATE KEY--"
        },
        "urls": [
          "nats://nats:[email protected]:4222"
        ]
      },
      "ntp": [],
      "parallel": 1,
      "password": "*",
      "remove_dev_tools": false,
      "remove_static_libraries": false,
      "swap_size": 8192
    },
    "persistent_disk_fs": "",
    "persistent_disk_mount_options": null,
    "persistent_disk_partitioner": ""
  },
  "mbus": "nats://nats:[email protected]:4222",
  "networks": {
    "default": {
      "default": [
        "dns",
        "gateway"
      ],
      "dns": [
        "8.8.8.8"
      ],
      "gateway": "10.0.1.1",
      "ip": "10.0.1.6",
      "mac": "aa:aa:aa:aa:aa:aa",
      "netmask": "255.255.255.0",
      "preconfigured": false,
      "resolved": false,
      "type": "manual",
      "use_dhcp": true
    }
  },
  "ntp": [],
  "vm": {
    "name": "vm-yyyy"
  }
}

settings.json file

/var/vcap/bosh/settings.json: Local copy of the bootstrap settings used by the Agent to configure network and system properties for the VM. They are refreshed every time Agent is restarted.

The settings.json format is the same as the settings format initially transferred to the agent.

Agent settings sources

Regardless of the source of the agent settings, the JSON document is the same. The agent settings as described above are transferred to the agent using one of the following ways:

  • Metadata server is used if all involved components (director, CPI, agent) support CPI API V2.
  • If one of the components does not yet support CPI API V2, the agent reads its settings from the registry.

Tip

See CPI API V2 and CPI V2 Migration Guide for more information on how the CPI, Agent, and Director behave in a registry-less environment.

CPI API V1

This section describes the agent interactions prior to CPI API V2

Metadata Server

The metadata server initially serves the registry URL and DNS server list.

Following is a sample content of the user-data part of the HTTP metadata

{
  "server": {"name": "vm-384sd4-r7re9e"},
  "registry": {"endpoint": "http://192.168.0.255:8080/client/api"},
  "dns": {"nameserver": ["10.234.50.180"]}
}

The supported format of the metadata server by the bosh-agent is documented in UserDataContentsType and http_metadata_service_test.go, along with the expected behavior of the bosh agent when reading this config.

Registry

The registry provides bosh-side metadata to the bosh agent.

From the Warden CPI documentation:

  • The registry is used by the CPI to pass data to the Agent. The registry is started on a server specified by registry properties.
  • If SSH tunnel options are provided, a reverse ssh tunnel is created from the MicroBOSH VM to the registry, making the registry available to the agent on remote machine.
Registry HTTP protocol

The Agent expects to communicate with the bosh registry over a REST API documented in api_controller_spec.rb

Reference registry client and servers implementations are available in: