Cloud Orchestration

how to design automation processes
declarative (also often known as the model)
imperative (also often known as the workflow or procedural)

how they can be used together.

Declarative/Model-Based Automation
The fundamental of the declarative model is that of the desired state
The principle is that we declare, using a model, what a system should look like.
The model is “data driven,” which allows data, in the form of attributes or variables, to be injected into the model at run-time to bring it to a desired state.
A declarative process should not require the user to be aware of the current state
usually bring it to a required state using a concept known as idempotent

Idempotent
if you deploy version 10 of a component to a development environment and it is currently at version 8, changes 9 and 10 will be applied
If you deploy the same release to a test environment where version 5 is installed, changes 6 through 10 will be applied
if you deploy it to a production system where it has never been deployed, it will apply changes 1 through 10
Each of the deployment processes brings it to the same state regardless of where they were initially
The user, therefore, does not need to be aware of the current state

Maintaining State
as a model changes over time—and may have a current, past or future state.
how can a model be applied to a target system, how can it know what changes to make

State Management
three methods used to keep an environment or system and its desired state models in line:

Maintain an inventory of what has been deployed.
This where we maintain the state of what has been deployed, and what is in the desired state of a release, and only apply the difference
Validate/compare the desired state with what has been been deployed.
Just make it so. The most obvious example here is anything that is stateless, such as a container.you would just want to instantiate a new one that has the new configuration you require.


Imperative/Procedural/Workflow-Based Automation
a series of actions are executed in a specific order to achieve an outcome.
For an application deployment, this is where the process of “how the application needs to be deployed” is defined, and a series of steps in the workflow are executed to deploy the entire application

A standard example might include
Some pre-install/validation steps
Some install/update steps
Finally, some validation to verify what we have automated has worked as expected

The often-cited criticism of this approach is that we end up with lots of separate workflows that layer changes onto our infrastructure and applications—and the relationship between these procedures are not maintained. The user, therefore, needs to be aware of the current state of a target before they know which workflow to execute. That means the principle of the desired state/idempotent is hard to maintain—and each of the workflows are tightly coupled to the applications

Puppet is an example of what is seen as a Declarative automation tool while Chef is said to be Imperative.
Do they both support concepts of the desired state and are idempotent? The answer is, of course, Yes
Is it possible to use a workflow tool to design a tightly coupled release that is not idempotent? The answer is, again, yes


What are the Benefits of Workflows?
The benefit of using a workflow is that we are able to define relationship and dependencies for our units of automation—and orchestrate them together.
Procedural workflows also allow us to, for example, deploy component A & B on Server1, then deploy component C or Server2 and then continue to deploy D on Server1. This give us much greater control in orchestrating multi-component releases.

Workflow for Applications
a stateless loosely coupled micro service on a cloud native platform

Model for Components
The concept of something being Idempotent—I can deploy any version to any target system if it has never been deployed to before
if we want to introduce a new change to an already existing environment or even if we want to roll a unit of automation back

Imperative Orchestration & Declarative Automation
a combination of Declarative (or model-driven) units of automation, coordinated by Imperative (or workflow-based) Orchestration can be used to achieve this
An imperative/workflow-based orchestrator also will allow you to executive not only declarative automation but also autonomous imperative units of automation should you need to.

My recommendation is that an application workflow defines the order and dependencies of components being deployed.
The Declarative Model for a component determines what action needs to be taken to bring a target system into compliance with the model

https://devops.com/perfect-combination-imperative-orchestration-declarative-automation/#disqus_thread


The debate hearkens back to concepts of declarative, model-based programming (the most like Puppet's approach)
The declarative approach requires that users specify the end state of the infrastructure they want, and then Puppet's software makes it happen.
some proponents of the imperative/procedural approach to automation say the declarative model can break down when there are subtle variations in the environment, since declarative configuration files must be made for every edge case in the environment.
Puppet Labs gets into the software-defined infrastructure game with new modules that allow it to orchestrate infrastructure resources such as network and storage devices.


an imperative, or procedural, approach (the one generally taken by Puppet rival Chef).
The imperative/procedural approach takes action to configure systems in a series of actions.

https://searchitoperations.techtarget.com/news/2240187079/Declarative-vs-imperative-The-DevOps-automation-debate

• There're many ways to do this, from provisioning shell scripts to use boxes with Chef already installed. A clean, reliable,

and repeatable way is to use a Vagrant plugin to do just that—vagrant-omnibus. Omnibus is a packaged Chef


Terraform can also be used to manipulate Docker. The classical usage is against an already
running Docker server on the network, but it will work exactly the same locally with your own
Docker installation. Using Terraform for controlling Docker, we'll be able to trigger
Docker image updates dynamically, execute containers with every imaginable option, manipulate Docker
networks, and use Docker volumes.


Vagrant is a tool focused for managing development environments and Terraform is a tool for building infrastructure.
Terraform can describe complex sets of infrastructure that exists locally or remotely.

Vagrant provides a number of higher level features that Terraform doesn't. Synced folders, automatic networking, HTTP tunneling, and more are features provided by Vagrant to ease development environment usage. Because Terraform is focused on infrastructure management and not development environments, these features are out of scope for that project.
https://www.vagrantup.com/intro/vs/terraform.html

• we explained why we picked Terraform as our IAC tool of choice and not Chef, Puppet, Ansible, SaltStack, or CloudFormation.

https://blog.gruntwork.io/an-introduction-to-terraform-f17df9c6d180

• Terraform and OpenStack

Terraform has a number of modules that will allow you manage your OpenStack infrastructure. It does support some (but not all) of the components of OpenStack, namely:
    Block Storage
    Compute
    Networking
    Load Balancer
    Firewall
    Object Storage
https://www.stratoscale.com/blog/openstack/tutorial-how-to-use-terraform-to-deploy-openstack-workloads/

• Terraform is a tool from HashiCorp that can be used to deploy and manage the cloud infrastructure easily by defining configuration files. It is similar to OpenStack Heat. However, unlike Heat which is specific to OpenStack, Terraform is provider-agnostic and can work with multiple cloud platforms such as OpenStack, AWS and VMware.

Terraform supports OpenStack Compute resources such as Instance, Floating i/p, Key Pair, Security Group and Server Group. It supports OpenStack Network resources such as Network, Subnet, Router and Router interface, Floating i/p.  For Block Storage it supports Volume and for Object Storage it supports Containers.
https://platform9.com/blog/how-to-use-terraform-with-openstack/

• Terraform vs. CloudFormation, Heat, etc.

Tools like CloudFormation, Heat, etc. allow the details of an infrastructure to be codified into a configuration file.
Terraform similarly uses configuration files to detail the infrastructure setup, but it goes further by being both cloud-agnostic and enabling multiple providers and services to be combined and composed.
The configuration files allow the infrastructure to be elastically created, modified and destroyed. Terraform is inspired by the problems they solve.
For example, Terraform can be used to orchestrate an AWS and OpenStack cluster simultaneously, while enabling 3rd-party providers like Cloudflare and DNSimple to be integrated to provide CDN and DNS services.
https://www.terraform.io/intro/vs/cloudformation.html

• Terraform vs. Boto, Fog, etc.

Libraries like Boto, Fog, etc. are used to provide native access to cloud providers and services by using their APIs.
https://www.terraform.io/intro/vs/boto.html

• Terraform vs. Chef, Puppet, etc.

Configuration management tools install and manage software on a machine that already exists. Terraform is not a configuration management tool, and it allows existing tooling to focus on their strengths: bootstrapping and initializing resources
https://www.terraform.io/intro/vs/chef-puppet.html

• Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state and then executes it to build the described infrastructure
The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc.

Terraform is used to create, manage, and update infrastructure resources such as physical machines, VMs, network switches, containers, and more. Almost any infrastructure type can be represented as a resource in Terraform.

A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Terraform Enterprise, DNSimple, CloudFlare).
https://www.terraform.io/docs/providers/index.html

• Provisioners are used to execute scripts on a local or remote machine as part of resource creation or destruction. Provisioners can be used to bootstrap a resource, cleanup before destroy, run configuration management, etc.

https://www.terraform.io/docs/provisioners/index.html

• Step 2 — Setting Up a Virtual Environment

Virtual environments enable you to have an isolated space on your computer for Python projects, ensuring that each of your projects can have its own set of dependencies that won’t disrupt any of your other projects.
https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04

• Terraform is to easily deploy our infrastucture and orchestrate our Docker environment.

The base idea is to manage and provision your infrastructure by code (configuration files, scripts, etc).
Terraform is an example of IaC that converts your infrastructure to human friendly config files that are JSON compatible.
http://t0t0.github.io/internship%20week%209/2016/05/02/terraform-docker.html

• Top 3 Terraform Testing Strategies for Ultra-Reliable Infrastructure-as-Code

Aside from CloudFormation for AWS or OpenStack Heat, it's the single most useful open-source tool out there for deploying and provisioning infrastructure on any platform.

This post will also briefly cover deployment strategies for your infrastructure as they relate to testing. 

Software developers use unit testing to check that individual functions work as they should. 

They then use integration testing to test that their feature works well with the application as a whole.

Being able to use terraform plan to see what Terraform will do before it does it is one of Terraform's most stand-out features.

it was designed for practitioners to see what would happen before a run occurred, check that everything looks good before applying it and then running terraform apply to add the finishing touches.

Disadvantages

It's hard to spot mistakes this way 

wanted to deploy five replicas of your Kubernetes master instead of three.

You check the plan really quickly and apply it, thinking that all is good.

deployed three masters instead of five

The remediation is harmless in this case: change your variables.tf to deploy five Kubernetes masters and redeploy. 

However, the remediation would be much more painful if this were done in production and your feature teams had already deployed services onto it.

trudging through chains of repositories or directories to find out what a Terraform configuration is doing is annoying at best and damaging at worst

To avoid the consequences from the approach above, your team might decide to use something like serverspec, Goss and/or InSpec to execute your plans first into a sandbox, automatically confirm that everything looks good then tear down your sandbox and collect results. 

If this were a pipeline in Jenkins or Bamboo CI

Orange border indicates steps executed within sandbox.

Advantages

Removes the need for long-lived development environments and encourages immutable infrastructure

Well-written integration tests provide enough confidence to do away with this practice completely.

Every sandbox environment created by an integration test will be an exact replica of production because every sandbox environment will ultimately become production.

This provides a key building block towards infrastructure immutability whereby any changes to production become part of a hotfix or future feature release, and no changes to production are allowed or even needed.

Documents your infrastructure

You no longer have to wade through chains of modules to make sense of what your infrastructure is doing. If you have 100% test coverage of your Terraform code (a caveat that is explained in the following section), your tests tell the entire story and serve as a contract to which your infrastructure must adhere

Allows for version tagging and "releases" of your infrastructure

Because integration tests are meant to test your entire system cohesively, you can use them to tag your Terraform code with git tag or similar. This can be useful for rolling back to previous states (especially when combined with a blue/green deployment strategy) or enabling developers within your organization to test differences in their features between iterations of your infrastructure.

They can serve as a first-line-of-defense 

let's say that you created a pipeline in Jenkins or Bamboo that runs integration tests against your Terraform infrastructure twice daily and pages you if an integration test fails.

you receive an alert saying that an integration test failed. Upon checking the build log, you see an error from Chef saying that it failed to install IIS because the installer could not be found. After digging some more, you discover that the URL that was provided to the IIS installation cookbook has expired and needs an update

After cloning the repository within which this cookbook resides, updating the URL, re-running your integration tests locally and waiting for them to pass, you submit a pull request to the team that owns this repository asking them to integrate

You just saved yourself a work weekend by fixing your code proactively instead of waiting for it to surface come release time.

Disadvantages

It can get quite slow

Depending on the number of resources your Terraform configuration creates and the number of modules they reference, doing a Terraform run might be costly

It can also get quite costly

Performing a full integration test within a sandbox implies that you mirror your entire infrastructure (albeit at a smaller scale with smaller compute sizes and dependencies) for a short time

Obtaining code coverage is hard

Terraform doesn't yet have a framework for obtaining a percentage of configurations that have a matching integration test.

This means that teams that choose to embark on this journey need to be fastidious about maintaining a high bar for code coverage and will likely write tools themselves that can do this (such as scanning all module references and looking for a matching spec definition).

kitchen-terraform is the most popular integration testing framework for Terraform at the moment.

Goss is a simple validation/health-check framework that lets you define what a system should look like and either validates against that definition or provides an endpoint

integration testing enables you to test interactions between components in an entire system. 

Unit testing, on the other hand, enables you to test those individual components in isolation. 

Advantages

It enables test-driven development 

Test-driven development is a software development pattern whereby every method in a feature is written after writing a test describing what that feature is expected to do.

Faster than integration tests 

Disadvantages

unit tests complement integration tests. They do not replace them. 

https://www.contino.io/insights/top-3-terraform-testing-strategies-for-ultra-reliable-infrastructure-as-code

• Cloudify is an open source cloud orchestration platform, designed to automate the deployment, configuration and remediation of application and network services across hybrid cloud and stack environments.

Cloudify uses a declarative approach based on TOSCA, in which users focus on defining the desired state of the application through a simple DSL, and Cloudify takes care of reaching this state, all while continuously monitoring the application to ensure that it maintains the desired SLAs in the case of failure or capacity shortage.

https://cloudify.co/product/

The new Cloudify release now provides coverage for 90% of the workloads being used in large enterprises today - from hybrid cloud models through our plugin support of the top 5 clouds, alongside containerized and non-containerized workloads

http://getcloudify.org/

• Cloudify vs. Terraform; How they compare

There are many ways to break down the types of automation, whether it’s imperative versus declarative, or orchestration versus desired configuration state. 

Terraform Strengths

Diversity

Easy to Get Started

Fast Standup

Wide Adoption (and Open Source)

Cloudify Strengths 

Full-Scale Service Orchestration

Controlled Operations as Code or GUI

Terraform Use Cases
Terraform has become even easier to use to create entire environments that can be managed from a single state file.
Creating Terraform stacks for lab, POC, and testing environments is fantastic considering the ease and speed of deployment
Terraform really shines when it is managing stacks utilizing statelessness, such as auto-scaling groups, lambda functions, and network resources.
It’s better to avoid having Terraform manage individual stateful instances or volumes because it can easily destroy resources.

Cloudify Use Cases
Instead of having deployments such as “Web frontend” or “Mongo Cluster,” blueprints can better resemble “ERP system,” or “BI Solution.”
Cloudify is TOSCA compliant and is designed to provide end-to-end orchestration.
This cradle-to-grave construct means that users can deploy an environment and manage that environment through Cloudify until decommission. This provides enterprise customers change management, auditing, and provisioning capabilities to control deployments of interdependent resources through their entire lifecycle
Blueprints can be a single component or mixed ecosystems containing thousands of servers.

https://cloudify.co/2018/10/22/terraform-vs-cloudify/

• InfraKit is a toolkit for infrastructure orchestration. With an emphasis on immutable infrastructure, it breaks down infrastructure automation and management processes into small, pluggable components. These components work together to actively ensure the infrastructure state matches the user's specifications.

https://github.com/docker/infrakit

• Juju deploys everywhere: to public or private clouds.

https://jujucharms.com/
Getting started with Foreman
Today, there is a great number of available tools allowing quick OS deployment and configuration, status monitoring and maintenance of the desired configuration. Here, the absolute leader for Win is SCCM. While full-featured analogues for *nix have just started to take on momentum. Nowadays, an administrator has to cope with a variety of tools and each of them performs its own role. This is convenient for development, but greatly complicates the support, while the results are not quite obvious. Foreman project, to be more precise, The Foreman is, in fact, an add-on for some open source solutions, which provides system management throughout system lifecycles from deployment and configuration to monitoring (Provisioning, Configuration, Monitoring). With it you can easily automate any repetitive tasks, manage changes on thousands of servers located on bare hardware or in the cloud, monitoring their status. The concept of server groups “config group” allows giving commands to multiple systems regardless of their location.
For example, Foreman is used in RHOS to configure the nodes. It is written using Ruby and JavaScript. Foreman operates in two modes:
Foreman consists of several components that can be deployed either on a single server, or on multiple servers:
Smart Proxy — is an autonomous web component which is placed on the host and allows Foreman connection to TFTP, DHCP (ISC DHCP, MS DHCP), DNS (Bind, MS DNS), Chef Proxy, Realm (FreeIPA), Puppet and Puppet CA. One Smart Proxy can manage multiple services, but the autonomous installation is also possible;
WebGUI, CLI and API management interfaces;
Configuration Management — the complete solution for configuration management based on Puppet and Chef, including Puppet ENC (external node classifier) with integrated support for parameterized classes and parameter hierarchy;
DBMS (MySQL, PostgreSQL or SQLite)— storage of settings and metadata managed computers.
https://hackmag.com/devops/getting-started-with-foreman/

• Foreman is a complete lifecycle management tool for physical and virtual servers. We give system administrators the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage servers, on-premise or in the cloud.

https://www.theforeman.org/

How to get started with the Foreman sysadmin tool

Full Stack Automation with Katello & The Foreman

Life cycle management with Foreman and Puppet

Red Hat Satellite 6 comes with improved server and cloud management

• Cobbler is an install server; batteries are included

Cobbler is a Linux installation server that allows for rapid setup of network installation environments. It glues together and automates many associated Linux tasks so you do not have to hop between lots of various commands and applications when rolling out new systems, and, in some cases, changing existing ones.
https://fedorahosted.org/cobbler/

• CF BOSH is a cloud-agnostic open source tool for release engineering, deployment, and lifecycle management of complex distributed systems.

https://www.cloudfoundry.org/bosh/

• BOSH is a project that unifies release engineering, deployment, and lifecycle management of small and large-scale cloud software. BOSH can provision and deploy software over hundreds of VMs. It also performs monitoring, failure recovery, and software updates with zero-to-minimal downtime.

In addition, BOSH supports multiple Infrastructure as a Service (IaaS) providers like VMware vSphere, Google Cloud Platform, Amazon Web Services EC2, Microsoft Azure, and OpenStack. There is a Cloud Provider Interface (CPI) that enables users to extend BOSH to support additional IaaS providers such as Apache CloudStack and VirtualBox.
https://bosh.io/docs/

• Python library for interacting with many of the popular cloud service providers using a unified API.

Resource you can manage with Libcloud are divided in the following categories:
Cloud Servers and Block Storage - services such as Amazon EC2 and Rackspace CloudServers
Cloud Object Storage and CDN - services such as Amazon S3 and Rackspace CloudFiles
Load Balancers as a Service - services such as Amazon Elastic Load Balancer and GoGrid LoadBalancers
DNS as a Service - services such as Amazon Route 53 and Zerigo
https://libcloud.apache.org/