Install Chef Automate with Chef Habitat Builder On-prem

This guide details how to install Chef Automate and deploy Chef Habitat Builder on-prem together. Enterprise customers may wish to set up an on-premises Chef Habitat Builder depot to store Chef Habitat packages for use by their own Chef Habitat Studios and Supervisors.

This guide covers setting up Chef Automate and Chef Habitat Builder on-prem, and bootstrapping Chef Habitat Builder on-prem with curated core seed lists from the Chef Habitat public Builder.

The Chef installer includes everything necessary to get started with Chef Automate and Chef Habitat Builder on-prem. Bootstrapping Chef Habitat Builder requires:

An outward bound HTTPS connection
An existing Chef Habitat public Builder account.

System Requirements

This guide demonstrates setup of Chef Habitat Builder on-prem by installing the components with the Chef Automate deployment utility. Contact your Chef representative before using this implementation in production for more specific scale recommendations.

Hardware Requirements

The minimum with which Chef Automate and Chef Habitat Builder can be deployed on a single host is:

16 GB of RAM.
130 GB of disk space, available to /hab.
4 vCPUs.
Inbound network connectivity from LAN (HTTP/HTTPS) is required for internal clients to access the on-premises Chef Habitat Builder.

For deployments that are expected to see production-scale workload, we recommend:

64 GB of RAM.
200 GB of diskspace, available to /hab.
16 vCPUs.

Roughly 80 GB of the disk space is designated for Chef Automate; the rest is used for Chef Habitat Builder and the artifacts it stores. The current implementation uses MinIO for Chef Habitat artifact storage; we do not support using Artifactory for artifact storage.

Operating System

Chef Automate and Chef Habitat Builder require:

a Linux kernel of version 3.2 or greater
systemd as the init system
useradd
curl or wget
The shell that starts Chef Automate should have a max open files setting of at least 65535
Run the installation and bootstrapping procedures as the superuser or use sudo at the start of each command.

Supported Topologies

Chef Automate and Chef Habitat Builder on the same host
Chef Automate and Chef Habitat Builder on different hosts

Unsupported Topologies

high-availability/DR/multinode Builder

Get Started with Chef Automate and Chef Habitat On-prem

Download the Chef Automate Installer

Download and unzip the installer:

curl https://packages.chef.io/files/current/latest/chef-automate-cli/chef-automate_linux_amd64.zip | gunzip - > chef-automate && chmod +x chef-automate

Deploy Chef Automate and Chef Habitat Builder On-prem

Deploying Chef Habitat Builder with Chef Automate requires a Chef Automate license. If you already have a Chef Automate license, you may use it for the deployment. Otherwise, you can accept the 30-day trial license when you first sign in to Chef Automate.

If you are deploying Chef Habitat Builder with Chef Automate in an airgapped environment, follow the documentation on building an airgap bundle.

You can deploy Chef Habitat Builder either with a full Chef Automate installation or with the Chef Automate auth stack only.

When deploying Chef Habitat Builder on the same host as Chef Automate, authentication to Chef Automate will be configured as part of the deployment.

Deploy Chef Automate and Chef Habitat Builder

To deploy Chef Automate and Chef Habitat Builder on the same host, specify both the builder and automate products on the command line. For example:

 ./chef-automate deploy --product builder --product automate

Accept the license with y.

Deploy Chef Habitat Builder with Chef Automate Auth

To deploy Chef Habitat Builder on a different host than Chef Automate, specify only the builder product on the command line. Chef Automate UI and Auth services only will also be deployed to support authentication.

When deploying Chef Habitat Builder on a different host than Chef Automate the supported authentication options are LDAP or SAML. Follow the standard instructions for Chef Automate LDAP or Chef Automate SAML configuration.

For example:

 ./chef-automate deploy --product builder

Accept the license with y.

Please note that the Chef Automate UI will only support managing Users, Groups, and Authorization policies when deployed without the full Chef Automate stack. If you wish to enable all of Chef Automate at a later time, you can update the product configuration to include the complete Chef Automate stack. For example:

Create a new patch configuration toml as config.toml and update the products to include both builder and automate:
```
[deployment.v1.svc]
  products = ["builder", "automate"]
```

Patch the configuration to deploy the rest of the Chef Automate services:

./chef-automate config patch config.toml

You should see output similar to:

Updating deployment configuration

Applying deployment configuration
  Installed automate-opensearch
  Installed automate-es-gateway
  Installed event-service
  Installed es-sidecar-service
  Installed event-feed-service
  Installed secrets-service
  Installed applications-service
  Installed notifications-service
  Installed nodemanager-service
  Installed compliance-service
  Installed ingest-service
  Installed config-mgmt-service
  Installed data-feed-service
  Installed event-gateway
  Started automate-opensearch
  Started automate-es-gateway
  Started event-service
  Started es-sidecar-service
  Started event-feed-service
  Started secrets-service
  Started applications-service
  Started notifications-service
  Started nodemanager-service
  Started compliance-service
  Started ingest-service
  Started config-mgmt-service
  Started data-feed-service
  Started event-gateway
  Started automate-load-balancer
Success: Configuration patched

Add Chef Habitat Builder to a Chef Automate Installation

Patch an existing Chef Automate installation to add Chef Habitat Builder:

Create a patch.toml file to add builder to the list of products to deploy:
```
   [deployment.v1.svc]
   products=["automate", "builder"]
```

Apply the patch to the Chef Automate installation:

   sudo chef-automate config ./patch.toml

The command output shows the Chef Habitat Builder services being added:

   Updating deployment configuration

   Applying deployment configuration
     Installed automate-minio
     Installed automate-builder-memcached
     Installed automate-builder-api
     Installed automate-builder-api-proxy
     Started automate-minio
     Started automate-builder-memcached
     Started automate-builder-api
     Started automate-builder-api-proxy
     Started automate-load-balancer
   Success: Configuration patched

View your login credentials in the terminal with:

cat automate-credentials.toml

You should see output similar to:

url = "https://automate.example.com"
username = "admin"
password = "abcdefgh1234567890PASSWORDSTRING"

Navigate to https://automate.example.com in your browser. If you cannot access the site in Google Chrome, try Mozilla Firefox or Microsoft Edge.
Sign in to Chef Automate using the credentials provided during deployment.
Select Applications in the top navigation bar
Select Chef Habitat Builder in the left sidebar, which redirects the browser to the Chef Habitat Builder login site
Select Sign in with Chef Automate
Sign into Chef Habitat Builder using the same credentials used with Chef Automate

Generate a Chef Habitat on-prem Builder Personal Access Token

You need a Personal Access Token for Chef Habitat on-prem in order to bootstrap packages and to perform authenticated operations with the hab client.

Select your Gravatar icon on the top right corner of the Chef Habitat Builder web page, and then select Profile. This will take you to a page where you can generate your access token. Make sure to save it securely.

Create the Core Origin

Once you are signed in to the Chef Habitat Builder UI, select the New Origin button and enter core as the name for the origin.

Access Chef Habitat Builder On-prem With Chef Habitat Command-Line Tools

Use the https://automate.example.com/bldr/v1 URL when accessing your Chef Habitat Builder installation with the Chef Habitat command-line tools. The Chef Habitat command-line tools recognize the HAB_BLDR_URL environment variable, which you can set on the command line with:

export HAB_BLDR_URL=https://automate.example.com/bldr/v1/

Access the Chef Habitat Builder On-prem REST API

To access the REST API for your on-prem installation of Chef Habitat Builder, you must specify your Builder authentication token as a bearer token in your request’s Authorization header. For example:

curl -H "Authorization: Bearer <your-habitat-builder-auth-token>" https://automate.example.com/bldr/v1/...

Bootstrap Chef Habitat Builder with Core Packages (Optional)

Prerequisites:

HTTPS connection
GitHub account
Public Chef Habitat Builder account
Public Chef Habitat Builder personal access token

Use seed lists to populate your on-premises Chef Habitat Builder installation with the packages required by your builds. Sample seed lists exist for the following scenarios:

Full core: the full contents of the upstream core origin. The x86_64 Linux set expands to 12GB, the Linux kernel2 set to 1GB, and the Windows set to 3.5GB.
Core dependencies: a subset of core consisting of commonly-used buildtime dependencies.
Effortless: packages used to start with the Effortless pattern. A complete Effortless implementation requires the contents of both the stable and the unstable channel.

Clone the Chef Habitat Builder On-prem Repository

To access the curated seed lists for bootstrapping Chef Habitat Builder on-prem, you will need to clone the Chef Habitat Builder on-prem repository using https.

git clone https://github.com/habitat-sh/on-prem-builder.git

Once the repository is successfully cloned, move into the on-prem-builder repository:

cd on-prem-builder

The Chef Automate installer uses a self-signed certificate. Copy the SSL public key certificate chain from Chef Automate into /hab/cache/ssl with this command:

cp /hab/svc/automate-load-balancer/data/automate.example.com.cert /hab/cache/ssl/automate.example.com.cert

Download Seed List Packages from the Public Chef Habitat Builder

Your host must have access to the internet to download the curated seed list packages from the public Chef Habitat Builder. If you have not already done so, create a user account and personal access token on the public Chef Habitat Builder.

Use the hab pkg download command with a seed list </path/to/seed_list> to download packages for your desired architecture <arch> from a channel <channel> to a directory <artifact-dir>:

HAB_AUTH_TOKEN=<your_public_builder_personal_access_token> hab pkg download --target <arch> --channel <channel> --file </path/to/seed_list> --download-directory <artifact-dir>

For example, to use the Effortless seed list to download x86_64-linux packages from the stable channel to the builder_bootstrap directory:

HAB_AUTH_TOKEN=<your_public_builder_personal_access_token> hab pkg download --target x86_64-linux --channel stable --file package_seed_lists/effortless_x86_64-linux_stable --download-directory builder_bootstrap

Bulk-Upload Seed List Packages to Chef Habitat Builder on-prem

Run the bulkupload command to upload artifacts from <artifact-dir> to the <channel> channel in the on-premises Chef Habitat Builder using the Builder API endpoint:

HAB_AUTH_TOKEN=<your_on-prem_Builder_personal_access_token> hab pkg bulkupload --url https://automate.example.com/bldr/v1 --channel <channel> <artifact-dir> --auto-create-origins

For example,

HAB_AUTH_TOKEN=<your_on-prem_Builder_personal_access_token> hab pkg bulkupload --url https://automate.example.com/bldr/v1 --channel stable builder_bootstrap/ --auto-create-origins

Should produce the output:

Preparing to upload artifacts to the 'stable' channel on https://automate.example.com/bldr/v1
Using builder_bootstrap/artifacts for artifacts and builder_bootstrap/keys signing keys
Found 46 artifact(s) for upload.
Discovering origin names from local artifact cache
Missing origin 'chef'
Origin 'core' already exists
Missing origin 'effortless'
Creating origin chef.
Created origin chef.
Creating origin effortless.
Created origin effortless.
75 B / 75 B | [===========================================] 100.00 % 1.31 MB/s d
Uploading public origin key chef-20160614114050.pub
...

The --auto-create-origins flag creates each origin listed in the <artifact-dir>/artifacts directory. If you omit the --auto-create-origins flag, use the Chef Habitat Builder UI to create the necessary origins before running the bulkupload command.

To finish up, return to your Chef Habitat Builder on-prem installation and view the packages that you’ve added to your core origin at https://automate.example.com/bldr/#/origins/core/packages.

Using Chef Habitat Builder

Because you are using an on-prem installation of Chef Habitat Builder, you must specify the Builder API endpoint of your installation when following the Habitat Builder documentation. This documentation covers using origin keys, using origin secrets, and uploading and promoting packages.

Operating Chef Habitat Builder

Chef Habitat Builder uses the same mechanisms that Chef Automate does for backups, log management, and uninstalling.

Logging errors

To change the log level for Chef Habitat Builder only, create a TOML file that contains the partial configuration below. Uncomment and change settings as needed, and then run chef-automate config patch </path/to/your-file.toml> to deploy your change.

[builder_api.v1.sys.log]
level = "debug"
scoped_levels = ["tokio_core=error", "tokio_reactor=error", "zmq=error", "hyper=error" ]

Setting up Chef Automate as an OAuth Provider for Chef Habitat Builder (Deprecated)

Warning

These instructions have been deprecated in favor of using the Chef Automate installer to deploy Chef Habitat on-prem.

To configure Chef Automate as an OAuth Provider for Chef Habitat Builder, create a TOML file with the partial configuration below. Run chef-automate config patch </path/to/your-file.toml> to deploy your change.

bldr_client_id and bldr_client_secret simply need to match what you configured for the corresponding values in Chef Habitat Builder (see below). However, we strongly recommend those values follow best practices for client_id and client_secret in the Oauth2 standard.

[session.v1.sys.service]
bldr_signin_url = "<your Builder fqdn>" # for example, "http://builder.test/"

# This needs to match what you configured OAUTH_CLIENT_ID as when you configured Habitat Builder.
bldr_client_id = "<your Habitat Builder Oauth2 Client ID>"

# This needs to match what you configured OAUTH_CLIENT_SECRET as when you configured Habitat Builder.
bldr_client_secret = "<your Habitat Builder Oauth2 Client Secret>"

In addition, add Chef Automate’s TLS certificate to Builder’s list of accepted certificates. Locate Chef Automate’s default self-signed certificate by running cat /hab/svc/automate-load-balancer/data/automate.example.com.cert, copy this default certificate, and then add it to your Builder instance’s list of accepted certificates.

-----BEGIN CERTIFICATE-----
MIIDfDCCAmSgAcaSldKaf...
-----END CERTIFICATE-----

If you are using a certificate signed by a trusted certificate authority instead of the default certificate, you can provide Builder with the root certificate authority for the signed certificate.

For more information, check out this further explanation on how to configure Builder to authenticate via Chef Automate.