Search code examples
amazon-s3cloud-foundryblobstorecephcf-bosh

Bosh CEPH Blobstore

My team wants to store the blobs for our BOSH release in a remote blobstore. However we have an internal CEPH / Rados store that we want to use. I know that Rados has S3 compatible interfaces so I was wondering how I could enable this as the final blobstore.

I know that typically I'd only need to give the access key, secrete key, and bucket to BOSH. But now I also have an IP host (and probably in the future a url) that specifies where the bucket exists. Is there currently a way to set this up?

Solution

  • As an example on how BOSH can interact with a CEPH / Rados store:

    Storing packages in an internal CEPH / Rados store

    This requires configuring the release to know about the intended blobstore. Keep in mind that when you run bosh upload blobs the command is parsed by the BOSH CLI rather than a BOSH or MicroBOSH VM if you have happened to target one. Assume that you have a store at address IP_ADDRESS:PORT with an existent bucket named BUCKET and keys ACCESS_KEY and SECRET_ACCESS_KEY. The config/final.yml file should look like:

    ---
blobstore:
  provider: s3
  options:
    bucket_name: BUCKET
    host: IP_ADDRESS
    s3_port: PORT

    But the config/private.yml file should look like:

    ---
blobstore:
  s3:
    access_key_id: ACCESS_KEY
    secret_access_key: SECRET_ACCESS_KEY

    Depending on how the CEPH store is set up, it may be necessary to turn off ssl verification as well which would include adding ssl_verify_peer: false under the s3_port option (i.e. the nesting would be blobstore -> s3 -> ssl_verify_peer. At this point calling bosh upload blobs will work as expected.

    Telling a BOSH VM about the CEPH / Rados store The deployment manifest file needs to have the items in the private.yml and the bucket name in the blobstore properties section.

    EDIT Aug 20, 2015 According to the CEPH website http://ceph.com/docs/master/radosgw/s3/#api there is currently no support for Bucket policies. Therefore the fact that a bucket is readable will not be inherited by a bucket's objects. This means that if a CEPH store is used in place of the official Amazon S3 for storage, the config/private.yml file is mandatory.