Evolution from standard Octopus usage to large scale On-Premise deployments

Background

About 3-years ago, we started our move from deployment using MSI to using Octopus Deploy. The MSI setup was hard to maintain and manual to deploy/install. Octopus seemed like a good fit for automated deployments. For more than 1 1/2 years, we have now been running all our deployments to both test and production environments using Octopus.

Confirmit Horizons is a quite large application/platform that consists of 60+ services, API’s and web applications. The deployment process in Octopus has about 80 steps.

At the same time we have also stared our journey from a monolith to microservices so the number of services and API’s are constantly growing.

On our main Octopus server, we have about 40 environments. Most are test and staging environments. Our three On-Demand environments in US, UK and Australia are also defined on this server. There are a total of about 300 machines/tentacles in these environments.

We also have an Octopus test server mainly used for development/testing of microservices. It has about 60 projects, 10 environments and 50 machines/tentacles.

So far, all is standard usage of Octopus.

Challenges

We have a set of challenges that are not solved by standard usage of Octopus.

Developers

Since our background was a monolith, developers was used to cloning one repository and building all the code locally on their development machine. The move to microservices changes that.

Most new development is now done on microservices and web applications, in separate repositories, with their own build. Since Confirmit Horizons is a platform, they still need the rest of the packages (services/api’s/web applications/databases) installed locally for everything to work on their development machine. Since all deployment is now done with Octopus a developer will also need to have a local installation of Octopus to test the deployment of new packages they create.

Goal: Every developer should have a local installation of Octopus where they can deploy all parts of Confirmit Horizons except the packages they are currently working on. They can then also use this Octopus server to test the deployment of packages they work on.

Shared development environments

The different scrum teams own different parts of the platform. To be effective the team needs an environment where they can test all the packages they maintain together with the production version of the rest of the Confirmit Horizons platform. To not clutter the main Octopus server it makes most sense to use a local Octopus server for this purpose.

Goal: Each scrum team should have a shared Octopus server where all parts of Confirmit Horizons, except the packages the team maintain, are automatically deployed. The team’s packages are then deployed by separate project(s) in Octopus.

On-Premise installations

We have many customers that have On-Premise installations of Confirmit Horizons. They vary in size from small to very large. They have traditionally installed Confirmit Horizons using MSI. Since we are not using MSI ourselves anymore we need to move them over to use Octopus as we do. Some On-Premise customers will need more than one Octopus server since they have different environments with no connectivity between them.

Goal: All On-Premise customers should install Confirmit Horizons with Octopus.

Goal: The On-Premise Octopus server(s), Confirmit Horizons project, deployment process, variables, releases, etc… all need to be installed and updated automatically. Initial installation should only need to specify some basic settings in addition to the Octopus variables and the environment machines. During day-to-day operation they should only need to click deploy on the releases they want to deploy to their environments.

On-Premise offline installations

Some of our On-Premise customers have Confirmit Horizons installations in environments with no Internet connectivity. Standard downloading of Octopus packages from one of our Octopus NuGet feeds will not be an option in these cases. Since we want to use Octopus for deployment in all other scenarios, we would also like to do so in these cases.

Goal: On-Premise customers with environments without Internet connectivity should have the option of installing Confirmit Horizons with Octopus from a zip file copied into a local server in that environment.

Continuous Delivery and Frozen versions

We practice continuous delivery and deploy new releases to our On-Demand environments 4 times a week. Every week we publish one of these releases to On-Premise customers that subscribe to the Continuous Delivery releases.

Some On-Premise customers prefers to stay on the same version for a longer period and only upgrade if there are any critical hotfixes. For this reason, we also publish and support the two previous major versions as Frozen versions of Confirmit Horizons. About every 9 months, we create a new major release. Currently we support v22 CD(Continuous Delivery), v21 Frozen and v20 Frozen.

Goal: An On-Premise customer can select between Confirmit Horizons CD and two Frozen versions. They should be able to install and deploy all of these to different environments using the same Octopus server.

Goal: All Confirmit Horizons projects on an On-Premise Octopus server need to be automatically installed and updated. Any changes to the deployment process, variables, etc. needs to be automatically transferred to an On-Premise Octopus server along with new release definitions.

Additional projects/applications

In addition to the CD and Frozen versions described above there are also several optional parts of Confirmit Horizons that are deployed using Octopus. Examples of this is CATI and Flex. These Octopus projects and releases also needs to be installed/updated on On-Premise installations using the same mechanism as the standard Confirmit Horizons projects.

Summary of challenges

The challenges described above boils down to a few things:

  • Developer and shared development environments can be treated as On-Premise installation
  • An Octopus project needs to be copied from one Octopus server to another with a few adjustments
  • An Octopus release definition needs to be copied from one Octopus server to another
  • The Octopus server version needs to be controlled so that we can avoid having scripts that needs to work with several versions of Octopus. Breaking changes in Octopus do occur and we want to avoid any issues with this by knowing what version we are targeting
  • Offline

Our solution

To solve these challenges we need something on top of Octopus to orchestrate:

  • Installation and configuration of Octopus
  • Import of a Confirmit Horizons releases with the relevant deployment process and variable changes

We have chosen to use Chocolatey for this. It is a good fit.

  • OctopusDeploy, OctopusDeploy.Tentacle and OctopusTools already exists as chocolatey packages
  • PowerShell can be used for scripts in both Octopus and Chocolatey
  • We can control the version of installed packages with Chocolatey
  • Both Octopus and Chocolatey use NuGet feeds and can work offline if the feeds are configured to e.g. a local folder
  • Chocolatey itself can also be installed using offline scripts. The version of Chocolatey can be controlled like other dependencies

Build and distribution

We use TeamCity to build and package all Octopus packages. Once the build is green, we deploy the packages to Artifactory and create a release in our main Octopus server. The release is deployed to some test environments and more tests are run. Once a day the last green release is deployed to the On-Demand environments connected to the main Octopus server.

Every time a Confirmit Horizons release is created in Octopus we also start off another build that will extract the deployment process and release definition from Octopus and package it into a Chocolatey package. We create one version for each release of the Octopus project. This package has enough information to import the release on another Octopus server.

Our developers and the shared development environments use the Chocolatey package to install and update their local On-Premise installation of Confirmit Horizons with Octopus.

Once every week we pick the latest stable release and promote that to an official On-Premise feed in Artifactory. This feed replicates across our On-Demand environments. On-Premise customers use the Octopus and Chocolatey feeds from the On-Demand environment closest to their location for their installation(s).

Conventions

Project groups

The Confirmit Horizons CD and frozen versions will have separate project groups in Octopus.

Projects

All optional Confirmit Horizons projects will be a project belonging to one of the project groups. E.g., there will be different CATI projects for CD and each frozen version for Confirmit Horizons. Each project will get an Octopus lifecycle named after the project.

Library Variablesets

As part of the bootstrapping of the installation, we specify a list of environments where Confirmit Horizons will be installed. Each project will get a library variableset for each of the environments.

<Project Name>.<Environment Name>.Variables

If new environments are added to the list at a later stage new variableset’s are generated and added to the projects.

External feed

All projects in the installation use the same external feed in Octopus. This external feed is added by the installation and configured automatically.

Installation process

Bootstrapping

Bootstrapping is only needed on the initial installation. Bootstrapping of a new On-Premise installation use two files.

  1. defaultSettings.yaml is a config file with some basic information about the installation like:
    • Url’s to the NuGet feeds to use for Chocolatey and Octopus packages
    • Credentials to use for these NuGet feeds
    • Octopus installation information like installation path, server url, database server and credentials
    • List of environments to configure for Confirmit Horizons
    • Optional proxy settings to use for Chocolatey and Octopus
  2. InstallWithGUI.ps1 is a PowerShell file that starts the installation using the settings in the yaml file. You need to specify the project to install e.g. Confirmit Horizons or Confirmit Horizons v21 Frozen. You can optionally also specify the version number of the release you want to install if you do not want the latest.
    There is also an InstallAutomatic.ps1 file available that use additional configuration files for Octopus variables and Tentacle configuration to fully automate installation. We typically use this in automated tests.

The InstallWithGUI.ps1 and defaultSettings.yaml files are only used the first time to bootstrap the installation and are deleted after use.

InstallWithGUI.ps1 do the following:

  1. Install Chocolatey if it’s not already installed or upgrade chocolatey if the version is older than what we require
  2. Configure Chocolatey by adding the choco-confirmit source and configuring proxy settings if applicable
  3. Install the confirmit.on-premise.install.module chocolatey package containing the ConfirmitInstall PowerShell module
  4. Encrypt and save the installer settings provided in defaultSettings.yaml using functions in the ConfirmitInstall PowerShell module. These encrypted settings are used for all later configuration and updates
  5. Open the basic installer GUI using a PowerShell function provided in the ConfirmitInstall PowerShell module

Scheduled task

After the initial installation, the Octopus server is configured with a scheduled task that by default will run every night. This task will update the Chocolatey packages that represents releases for the different Confirmit Horizons projects. The update of a release package can trigger updating of the Octopus version and reconfiguration of the Octopus server if we have chosen to bundle these changes with the release.

The scheduled task runs in the background and is fully automated. The only thing the On-Premise user will see is that new releases are available for deployment in Octopus.

Chocolatey packages

We have created a set of Chocolatey packages to orchestrate the installation and configuration of Octopus.

confirmit.on-premise*.release

The release Chocolatey packages are used to import an Octopus project release. If the project do not exist, it is created according to the conventions described above. If it exists, it is updated to make sure it still follows the same conventions or adjusted according to new conventions we have chosen to use.

The package contains:

  • Deployment process. A json file with a copy of the deployment process originally used to create the release
  • Release definition. A json file with an export of the Octopus release definition. This basically is a list package versions
  • Default variables. A config file with a list of all Octopus variables that will be used in the library variableset for this project. If we add a new variable in a release the variablesets will be updated before the release is imported
  • Release notes. A json file with the release notes for N number of releases. The release notes is adjusted to match the target system so that the imported release contains all release notes for changes since the last time a release was imported

The Chocolatey install script basically do this:

Confirmit.OctopusTools.exe update
Octo.exe create-release
Confirmit.OctopusTools.exe update do all the heavy lifting to create and update the project, deployment process, library variablesets, etc… The deployment process is adjusted so that all package use the external feed. The release definition file is updated with correct release notes.

Octo.exe create-release simply creates the release.

All release packages are based on the same template and are easy to set up. All that is needed is to create a new build file and specify:

  • Source Octopus server
  • Source Octopus project
  • Release number
  • PackageId of Chocolatey package to create
  • Package title and description
  • A file with default Octopus variables for this project
  • A chocolateyinstall.ps1 file with some standard code and optionally some custom code
build.ps1
param(
    [parameter(Mandatory=$true)] [String] $Release,
    [parameter(Mandatory=$true)] [String] $OctopusUrl,
    [parameter(Mandatory=$true)] [String] $OctopusApiKey,
    [parameter(Mandatory=$true)] [String] $Project
)
 
$desc = @"
Import Confirmit v21 frozen release definition to your local Octopus On-Premise installation
The package depends on the confirmit.on-premise package and will upgrade Octopus version if needed
 
Initial installation requires a defaultSettings.yaml file in the current directory
See description for confirmit.on-premise package for more details
"@
..\build\build.ps1 -Release $Release -OctopusUrl $OctopusUrl -OctopusApiKey $OctopusApiKey -Project $Project -PackageId "confirmit.on-premise.v21frozen.release" -PackageTitle "Import Confirmit Horizons v21 Frozen release to On-Premise Octopus server" -PackageDescription $desc -PackageDir (Get-Location)

 

chocolateyinstall.ps1
# stop on all errors
$ErrorActionPreference = 'Stop'
 
. (Join-Path $PSScriptRoot "chocolateyinstall_include.ps1")
 
Install-ConfirmitPackage -OctopusProjectName "Confirmit Horizons v21 Frozen"
 
Write-Host "Done"  -ForegroundColor Green

confirmit.on-premise

This package is used to control the version of Octopus and the tools used to configure and update Octopus. This ensures that all tools and scripts are compatible with the choosen version of Octopus. Every time we want to deploy a new version of Octopus to On-Premise we change the Octopus dependency version number in the nuspec file of this package. The scheduled task that handles automatic import of new releases is created by this package.

<dependencies>
  <dependency id="OctopusTools" version="[4.14.1]" />
  <dependency id="OctopusDeploy" version="[3.12.7]" />
  <dependency id="OctopusDeploy.Tentacle" version="[3.12.7]" />
  <dependency id="confirmit.on-premise.install.module" version="[2.2.3]" />
</dependencies>

confirmit.on-premise.install.module

A PowerShell module with a set of functions used during bootstrapping and updates done by the scheduled task.

E.g:

  • Edit installation setting used to re-run the bootstrapping
  • Edit all variables in the library variableset with documentation of variables
  • Edit tentacle configuration with documentation of all roles. This information can be used to automatically install Octopus tentacle on all machines and add them to the environment in Octopus with the specified roles

Confirmit.Horizons.OctopusTools

A tool similar to the official OctopusTools. It has additional commands that we need to create and deploy releases.

PS C:\Windows\system32> confirmit.octopustools
Confirmit Octopus Deploy Command Line Tool, version 6.1.4.0
 
Usage: Confirmit.OctopusTools <command> [<options>]
 
Where <command> is one of:
 
  addallroles
   Add all project roles to a machine. Create and add machine to environment if needed
  create-apikey
   Create a Octopus apikey. Replaces and existing apikey with the same purpose
  create-defaultusers
   Create default On-Premise admin and deploy users and a deploy team with correct permissions
  customcode-project
   Create or update Confirmit CustomCode project
  export-default-variables
   Export list of variables with default values to file
  export-deploymentprocess
   Export the deployment process for the Confirmit project
  export-env-variables
   Export library variable set for a Octopus/Confirmit environment to file. Sensitive variables will NOT be exported
  export-migration-variables
   Export configuration settings used in Confirmit Horizons Octopus variableset from Confirmit to file. This command HAS to be executed on a Confirmit server. The exported variables WILL contain passwords in clear text
  export-tentacleconfig
   Export current tentacle configuration for an environment. If environment is empty a template file will be created
  get-servercert
   Returns the Octopus server certificate
  help
   Prints this help text
  import-env-variables
   Import library variable set for a Octopus/Confirmit environment from file. All variable names that end with .Password will be configured as sensitive variables
  import-tentacleconfig
   Add or update tentacle(s) with the specified roles. Tentacles not specified in config file will be removed from the environment
  role-package-report
   Create a CSV file with a role/package report. Output file is role-package-report.csv
  update
   Update Confirmit project and all connected environment variable sets. Prepare Confirmit release import file. Will create environment and project if needed
 
Or use Confirmit.OctopusTools help <command> for more details.

 

confirmit.poweryaml

A PowerShell module with functions to work with YAML config files.

Offline

To support offline installations from a zip file we had to do a few more things:

  • The NuGet feeds for Chocolatey and Octopus are configured as local folders
  • All Chocolatey packages that download MSI’s as part of the installation needs to be re-packed so that they contain the MSI file. This applies to OctopusDeploy and OctopusDeploy.Tentacle
  • We need to include Chocolatey in the zip file so that we can handle offline installation of Chocolatey on servers that don’t have Chocolatey or where Chocolatey needs to be updated

With these additional steps the offline installation basically works as the online version with a few manual steps to import a new Confirmit Horizons release:

  • Copy the confirmit.on-premise.release.<version>.zip to the target Octopus server
  • Unzip the file
  • Run setup.ps1 and you are done

 

setup.ps1 will simply unzip the packages in the Chocolatey and Octopus feeds and copy them to the local feed folders and then run a choco update command.

 


 

This was basically a braindump of our usage of Octopus in On-Premise scenarios. Of course there are a lot of details missing, but I hope it gave you some insight into a possible way of supporting this. This works well for us and is currently used to deploy to hundreds of On-Premise environments (counting real On-Premise customers, test installations and developers)

 

Jan Lønsetteig, SVP – Horizons Architecture

Author: Jan Lønsetteig

Senior Vice President – Horizons Architecture.
#Architecture #Development #Deployment #DevOps #Automation