Cloud Foundry on AWS - Part 1: Deploying a MicroBOSH
2 Jul, 2013
The recommended deployment mechanism for Cloud Foundry uses BOSH, "an open source tool chain for release engineering, deployment, and lifecycle management of large-scale distributed services". Once it's up and running, Cloud Foundry actually has no dependencies on BOSH, and in fact, it's technically possible to deploy CF by other means. But BOSH makes it easier to manage an existing Cloud Foundry deployment, so we'll stick with that.
(A word of warning: the official documentation for "Deploying Cloud Foundry on AWS" currently describes a installation process involving the bootstrap-cf-plugin gem. I recommend avoiding that path, as it conflates installation of BOSH and Cloud Foundry in a confusing way. Let's stay focused on BOSH, for now.)
BOSH is itself a distributed system, and can be deployed across multiple nodes (e.g. EC2 instances) for better performance and resilience. For our purposes, though, it's sufficient to have all the BOSH components installed on a single node. Such a single-node deployment is called a "MicroBOSH".
The BOSH project distributes MicroBOSH as stemcells; where "stemcell" is their term for a raw machine image. They provide an AWS-specific stemcells, which can be used to build AMIs, as well as stemcells for other target infrastructures (like vSphere and OpenStack).
Before you get too excited, though, you'll need an "inception server". Despite the fancy name, this is nothing more than an EC2 instance in your target AWS region, that you can SSH into, to use as a staging point for MicroBOSH installation. It's required because:
- you'll be downloading some large BLOBs, and it's best to keep them inside the AWS network
- in order to turn the MicroBOSH stemcell into an AMI, you need to create a mount an EBS volume.
Any Linux instance running Ruby 1.9.x should do. If you don't have one handy, you can use one of:
Once you have an inception server running, SSH on in.
Option A: bosh-bootstrap
At this point, you're welcome to take a shortcut by way of Dr Nic's bosh-bootstrap project, which mostly automates the remainder of the process.
Option B: DIY
After some initial experiments with bosh-bootstrap, I wanted to understand more about what it was doing, so ended up building my MicroBOSH manually, using bosh-bootstrap as a guideline. If you're a sucker for punishment, like me, then read on ...
You'll need the BOSH command-line toolset. Create a Gemfile containing:
source "https://rubygems.org" source 'https://s3.amazonaws.com/bosh-jenkins-gems/' gem "bosh_cli", "~> 1.5.0.pre" gem "bosh_cli_plugin_micro", "~> 1.5.0.pre"
and run "bundle install" to install the bits you need.
Now, use the AWS console (or the API) to set some stuff up:
- create a keypair (e.g. "mybosh")
- create a security group (e.g. "bosh") that allows inbound connections on the following ports:
- 22 (for SSH)
- 4222 (for the "nats" pub/sub protocol)
- 6868 (for the BOSH agent)
- 25250 (for the BOSH blobstore)
- 25555 (for the BOSH director)
- 25777 (for the BOSH registry)
- 53 (UDP, for DNS)
- allocate an Elastic IP address (e.g. A.B.C.D)
Pick a name for your MicroBOSH, e.g. "mybosh", and create a configuration file for it:
$ mkdir -p ~/microbosh/deployments/mybosh $ vi ~/microbosh/deployments/mybosh/micro_bosh.yml
Here's an example micro_bosh.yml file:
name: mybosh logging: level: DEBUG network: type: dynamic vip: A.B.C.D resources: persistent_disk: 4096 cloud_properties: instance_type: m1.medium cloud: plugin: aws properties: aws: access_key_id: YOURKEY secret_access_key: YOURSECRET region: ap-southeast-2 ec2_endpoint: ec2.ap-southeast-2.amazonaws.com default_security_groups: - bosh default_key_name: mybosh ec2_private_key: /home/ubuntu/.ssh/mybosh.pem apply_spec: agent: blobstore: address: A.B.C.D nats: address: A.B.C.D properties: aws_registry: address: A.B.C.D
You'll need to :
- insert the appropriate AWS access/secret key
- change the region (unless "ap-southeast-2" is what you want)
- replace "A.B.C.D" with the Elastic IP address you allocated earlier
Now, you need a MicroBOSH image to install, so download the latest MicroBOSH stemcell:
$ cd ~/microbosh $ curl -O http://bosh-jenkins-artifacts.s3.amazonaws.com/micro-bosh-stemcell/aws/latest-micro-bosh-stemcell-aws.tgz
Now it's time to crank up the MicroBOSH!
Be patient; it takes a while. Actually, you should consider installing a terminal multiplexer like tmux on your inception server, and running this step within a tmux session, just in case you get disconnected while the deployment is in progress.
$ cd ~/microbosh/deployments $ bosh micro deployment mybosh/ Deployment set to '/home/ubuntu/microbosh/deployments/mybosh/micro_bosh.yml' $ bosh -n micro deploy ../latest-micro-bosh-stemcell-aws.tgz Verifying stemcell... File exists and readable OK Using cached manifest... Stemcell properties OK Stemcell info ------------- Name: micro-bosh-stemcell Version: 776 Deploy Micro BOSH unpacking stemcell (00:00:16) uploading stemcell (00:10:48) creating VM from ami-c51380ff (00:00:32) waiting for the agent (01:01:20) create disk (00:04:16) mount disk (00:00:06) stopping agent services (00:00:01) applying micro BOSH spec (00:00:19) starting agent services (00:00:00) waiting for the director (00:00:18) Done 11/11 01:18:06 WARNING! Your target has been changed to `https://126.96.36.199:25555'! Deployment set to '/home/ubuntu/microbosh/deployments/mybosh/micro_bosh.yml' Deployed `mybosh/micro_bosh.yml' to `https://mybosh:25555', took 01:18:06 to complete
Note the id of the AMI produced; you can use this for future MicroBOSH deployments in the same region, bypassing the stemcell download and conversion processes, e.g.
$ bosh -n micro deploy ami-c51380ff
Actually, if you're really lucky, Pivotal might have already baked a MicroBOSH AMI in your target region, in which case you can just use that, and save yourself a lot of time (and network traffic).
$ AWS_REGION=us-east-1 $ curl http://bosh-jenkins-artifacts.s3.amazonaws.com/last_successful_micro-bosh-stemcell-aws_ami_$AWS_REGION ami-427b092b
At time of writing, they're only baking AMIs for us-east-1, so for other regions you'll have to resort to downloading the stemcell, as described above.
Once your MicroBOSH is running, you should be able to connect to it:
$ bosh target https://188.8.131.52:25555 Target set to `mybosh' Your username: admin Enter password: ***** Logged in as `admin' $ bosh status Config /home/ubuntu/.bosh_config Director Name mybosh URL https://184.108.40.206:25555 Version 1.5.0.pre.776 (release:6191c586 bosh:6191c586) User admin UUID eac3cf02-845d-4817-aa55-7626a071304a CPI aws dns enabled (domain_name: microbosh) compiled_package_cache disabled snapshots disabled Deployment not set
Stay tuned for the next exciting episode!