NAV
shell

More information on Migrating to BOSH Backup and Restore can be found here.

THE BASICS

Welcome to the CF OPS User Guide. CF OPS is an automation utility that is based on the Pivotal prescribed way to back up and restore an installation of Pivotal Cloud Foundry. You can view code examples in the dark area to the right.

This tool can be placed on a jumpbox that is able to communicate with the network that your Ops Manager virtual machine is on, or it can be placed directly onto the Ops Manager VM.

We recommend that you make redundant copies of your backups and that you store them in multiple locations in order to minimize the risk of losing your backups in the event of a disaster.

We also recommend that you attempt a test restore on every backup that you make in order to validate that the backup can be restored from.

PCF Components

The CF OPS utility backs up your Ops Manager and installed tile configuration, as well as the state of your Elastic Runtime. Pivotal Support can help you with any issues encountered while backing up or restoring these components of the platform.

Additionally, the Pivotal Field Solutions team has developed experimental plugins for backing up Pivotal services that can be used in conjunction with CF OPS. These plugins are unsupported and are intended to be used only in consultation with a Pivotal Solutions Engineer.

Compatibility

CF OPS is currently compatible with PCF 1.6, 1.7, 1.8, 1.9 and 1.10 installations on vSphere, Openstack, and Amazon Web Services.

Authentication

CF OPS works with Ops Manager installations with internal authentication as well as SAML-based authentication. Previous versions of CF OPS accepted an administrator username and password but this authentication mechanism will be deprecated in the next release. For all new installations of CF OPS, an admin client must be created with the uaac cli and the client credentials will need to be provided to the tool instead of administrator username and password.

Creating an admin client with the uaac cli

From a command line with Ruby installed, install the cf-uaac gem:

$ gem install cf-uaac

Target your Ops Manager IP:

$ uaac target https://YOUR_OPSMANAGER/uaa

Login as an administrator, and fetch your administrator token

$ uaac token sso get
Client ID:  opsman
Client secret:
Passcode (from https://YOUR_OPSMANAGER/uaa/passcode):  XXXXXX

Successfully fetched token via owner passcode grant.
Target: https://YOUR_OPSMANAGER/uaa
Context: admin, from client opsman

On all PCF installations (internal authentication as well as SAML-based authentication), a UAA client with Ops Manager administrative privileges must be created before CF OPS can be used to back up and restore a PCF installation.

Create a new client

$ uaac client add -i
Client ID:  NEW_CLIENT_NAME
New client secret:  DESIRED_PASSWORD
Verify new client secret:  DESIRED_PASSWORD
scope (list):  opsman.admin
authorized grant types (list):  client_credentials
authorities (list):  opsman.admin
access token validity (seconds):  43200
refresh token validity (seconds):  43200
redirect uri (list):
autoapprove (list):
signup redirect url (url):
  scope: opsman.admin
  client_id: NEW_CLIENT_NAME
  resource_ids: none
  authorized_grant_types: client_credentials
  autoapprove:
  access_token_validity: 43200
  refresh_token_validity: 43200
  action: none
  authorities: opsman.admin
  name: NEW_CLIENT_NAME
  signup_redirect_url:
  lastmodified: 1478530665397
  id: NEW_CLIENT_NAME

INSTALLATION

Downloading CF OPS

$ wget [URL TO DESIRED RELEASE]

The latest CFOps release can be downloaded from here.

The cfops version compatible with PCF 1.6 can be downloaded from here.

Installing on a Linux jump box

You can use the scp command to transfer the downloaded linux binary to your jumpbox

$ scp [ROUTE TO CFOPS]/CFOPS [jumpbox-user]@[ROUTE TO JUMPBOX]:

Alternatively, if your jumpbox has access to the internet, you can use the wget command after sshing into your jumpbox

$ ssh [jumpbox-user]@[ROUTE TO JUMPBOX] -i [YOUR_CERTIFICATE].pem

[jumpbox-user]@[ROUTE TO JUMPBOX]:~$ wget [URL TO DESIRED RELEASE]

You may need to change the permissions associated with the CFOPS binary in order to make it executable

$ chmod 755 cfops

CF OPS can be installed on a Linux jump box that has access to the Ops Manager Virtual Machine. Please ensure that the jump box has sufficient disk space to house the number of backups you intend to keep on it.

Installing on the Ops Manager VM

You can use the scp command to transfer the downloaded linux binary to your Ops Manager VM

$ scp [ROUTE TO CFOPS]/CFOPS [jumpbox-user]@[ROUTE TO OPS MANAGER]:

Alternatively, if your Ops Manager has access to the internet, you can use the wget command after sshing into your Ops Manager

$ ssh [jumpbox-user]@[ROUTE TO OPS MANAGER] -i [YOUR_CERTIFICATE].pem

[jumpbox-user]@[ROUTE TO OPS MANAGER]:~$ wget [URL TO DESIRED RELEASE]

You may need to change the permissions associated with the CFOPS binary in order to make it executable

$ chmod 755 cfops

CF OPS can also be installed directly on the Ops Manager Virtual Machine. Please ensure that your Ops Manager VM has sufficient disk space to house the number of backups you intend to keep on it.

BACKUP

Ops Manager Backup

$ ./cfops backup \
  -opsmanagerhost [Ops Manager VM hostname] \
  -clientid [Ops Manager admin client id] \
  -clientsecret [Ops Manager admin client secret] \
  -opsmanageruser ubuntu \
  -destination . \
  -tile ops-manager

This step can be also be run in debug mode by setting LOG_LEVEL=debug:

$ LOG_LEVEL=debug ./cfops backup

The following fields are required in order to perform an Ops Manager backup:

Full flag Compact flag Description Value
-opsmanagerhost -omh Ops Manager VM hostname
-clientid -cid Ops Manager admin client id
-clientsecret -cis Ops Manager admin client secret
-opsmanageruser -omu Ops Manager user ubuntu
-destination -d Backup destination path
-tile -t Component name ops-manager

Elastic Runtime Backup

$ ./cfops backup \
  -opsmanagerhost [Ops Manager VM hostname] \
  -clientid [Ops Manager admin client id] \
  -clientsecret [Ops Manager admin client secret] \
  -opsmanageruser ubuntu \
  -destination . \
  -tile elastic-runtime
  -nfs full

This step can be also be run in debug mode by setting LOG_LEVEL=debug:

$ LOG_LEVEL=debug ./cfops backup

The following fields are required in order to perform an Elastic Runtime backup:

Full flag Compact flag Description Value
-opsmanagerhost -omh Ops Manager VM hostname
-clientid -cid Ops Manager admin client id
-clientsecret -cis Ops Manager admin client secret
-opsmanageruser -omu Ops Manager user ubuntu
-destination -d Backup destination path
-tile -t Component name elastic-runtime
-nfs -nfs Type of nfs backup bp / lite / full (see below)
-nfs flag value Description
-nfs bp Backs up buildpacks only and necessitates re-pushing apps
-nfs lite Backs up buildpacks, app droplets, and source
-nfs full Backs up buildpacks, app droplets, source, and app cache

RESTORE

Ops Manager Restore

$ ./cfops restore \
  -opsmanagerhost [Ops Manager VM hostname] \
  -omr [Ops Manager passphrase]
  -clientid [Ops Manager admin client id] \
  -clientsecret [Ops Manager admin client secret] \
  -opsmanageruser ubuntu \
  -destination . \
  -tile ops-manager

This step can be also be run in debug mode by setting LOG_LEVEL=debug:

$ LOG_LEVEL=debug ./cfops restore

The following fields are required in order to perform an Ops Manager restore:

Full flag Compact flag Description Value
-opsmanagerhost -omh Ops Manager VM hostname
-omr -omr Ops Manager passphrase
-clientid -cid Ops Manager admin client id
-clientsecret -cis Ops Manager admin client secret
-opsmanageruser -omu Ops Manager user ubuntu
-destination -d Path to Ops Manager backup
-tile -t Component name ops-manager

After performing the Ops Manager restore, you will need to apply changes in order to create all missing VMs.

If the BOSH director VM is not present in the environment being restored to

$ ssh -i ops_mgr.pem ubuntu@OPS-MGR-IP

$ cd /var/tempest/workspaces/default/deployments/
$ sudo mv bosh-state.json bosh-state.json.old

Elastic Runtime Restore

$ ./cfops restore \
  -opsmanagerhost [Ops Manager VM hostname] \
  -clientid [Ops Manager admin client id] \
  -clientsecret [Ops Manager admin client secret] \
  -opsmanageruser ubuntu \
  -destination . \
  -tile elastic-runtime

This step can be also be run in debug mode by setting LOG_LEVEL=debug:

$ LOG_LEVEL=debug ./cfops restore

The Elastic Runtime restore can only be performed after the Ops Manager restore is complete, since the missing VMs will need to be created before their state can be restored.

The following fields are required in order to perform an Elastic Runtime restore:

Full flag Compact flag Description Value
-opsmanagerhost -omh Ops Manager VM hostname
-clientid -cid Ops Manager admin client id
-clientsecret -cis Ops Manager admin client secret
-opsmanageruser -omu Ops Manager user ubuntu
-destination -d Path to Elastic Runtime backup
-tile -t Component name elastic-runtime

TROUBLESHOOTING

Authentication/Connection Errors

Ops Manager

error in save http request {"error":"Your UAA access token has expired and Your UAA access token does not have either \"opsman.admin\" or \"opsman.user\" scope"}

Generic authentication problem - check that your credentials are correct and are being provided.

From a command line with Ruby installed, install the cf-uaac gem:

$ gem install cf-uaac

Target your Ops Manager IP:

$ uaac target https://YOUR_OPSMAN_IP/uaa

Log in to your Ops Manager with the Client name “opsman”:

$ uaac token owner get

Client name: opsman
Client secret:
User name: YOUR_USERNAME_HERE
Password: YOUR_PASSWORD_HERE

Retrieve your Ops Manager access token:

$ uaac context

You can use the uaac CLI to determine whether a token can be successfully retrieved with the provided credentials. See the Ops Manager API documentation at YOUR_OPSMANAGER/docs for more details on how to install and use the uaac CLI.

This can also be caused by connecting to OM using anything other than the IP address or FQDN with which Ops Manager was deployed. The command will not work if you deployed Ops Manager with an IP address and are trying to connect to it with a domain name, or vice versa.

Elastic Runtime

Error connecting to server %v dial tcp YOUR_IP:25555: getsockopt: connection timed out

OR

Error connecting to director using %s, UserId-[%s] and Password[%s]

OR

createCliCommand.go:49] there was an error: invalid director credentials running backup on elastic-runtime tile:tile

Usually this means your jump box cannot connect to your director VM or the internal MySQL server VM over the specified port. Try installing CF OPS directly on Ops Manager or change your network access controls so your jump box can access these VMs.

Tile Not Found Errors

tile '' not found

Indicates malformed command. Every parameter must be provided to the command, otherwise it will not run. This error can occur when the destination flag (-d) has not been set

Restore Errors

422 Error

422 Unprocessable Entity

Has a number of causes. Check /tmp/logs/production.log on the OM box

Causes include the following:

500 Error

500 Internal Server Error

This usually affects CF OPS installations that are running directly on the Ops Manager. Usually comes from NGINX complaining the OM box is full. The /tmp folder is cleared out only infrequently. Logs should confirm this. Check disk space and free up some if necessary

Filing A Support Ticket

Sample backup command with LOG_LEVEL=debug

$ LOG_LEVEL=debug ./cfops backup \
  -opsmanagerhost [Ops Manager VM hostname] \
  -clientid [Ops Manager admin client id] \
  -clientsecret [Ops Manager admin client secret] \
  -opsmanageruser ubuntu \
  -destination . \
  -tile ops-manager

Sample cURL command to retrieve installation_settings.json

$ curl "https://YOUR_OPS_MANAGER/api/installation_settings" \
  -X GET \
  -H "Authorization: Bearer UAA_ACCESS_TOKEN"

Instructions:

MIGRATING TO BBR

BOSH Backup and Restore (BBR) is the officially support product for backup and restore of Pivotal Cloud Foundry. BBR works with PCF1.11 and later versions. As end of General Availability is approaching for PCF1.10 (December 2017), customers using CFOps will need to migrate to BBR.

The overall workflow is similar for CFOps and BBR: they are both binaries run from a jumpbox, and when backup is run the generated artifact is stored on the jumpbox.

BBR, like CFOps, supports backup and restore of ERT configured with an internal MySQL and internal blobstore.

The main differences between BBR and CFOps are:

Workflow

The manual steps for backup of PCF using BBR are here and restore are here. Instructions for setting up a jumpbox for bbr are here. Pivotal recommends automating backups using Concourse. Pivotal has created pipeline tasks and an exemplar pipeline here. The Concourse worker running BBR needs access to the VMs in ERT. Details on required access can be found here.

Data Services

Support for BBR has not yet been added to Pivotal Data Services. Operators are recommended to use the backups configurable per data service tile in Ops Manager.