MOLECULE + TERRAFORM

Un paseo por la inmutabilidad

JORDI

PERALTA

© 2019 Binlogic.

¿Qué vamos a ver?

• CI/CD

• Universo infinito de herramientas

• DevOps/SRE: también nos gusta dormir

• Inmutabilidad al rescate

• Tooling para ser "inmutable"

• Packer

• Terraform

© 2019 Binlogic.

CI / CD

• Continuous Integration

• Continuous Delivery

• Continuous Deployment

• Continuous Improvement

• Continuous Testing

• Continuous Failing... (Fail Fast)

© 2019 Binlogic.

CI / CD more...

Pipeline perfecta:

git push DEV INT QA STG PROD

Magic Words: ( R3PA pattern )

• Reliability | Reproducibility | Repeatability

• Predictability | Availability

© 2019 Binlogic.

Universo infinito de herramientas

Objetivo: ser Netflix(+5000), Amazon(+5000), Airbnb(+500)...

© 2019 Binlogic.

… y más

En busca de la Pipeline perfecta...

© 2019 Binlogic.

para...

• Ser los primeros

• Crear impacto

• Crear necesidad

• Augmentar "revenue"

• …

© 2019 Binlogic.

In short,

GANAR MÁS PASTA!!!

DevOps/SRE

Brutal Realidad:

• Infra heterogénea

• Entornos distintos

• No Pipeline

• Los Deployments son ARTE!!! ( craftsmanship )

• No R3PA-Pattern at all!!!

© 2019 Binlogic.

...dormir???, para qué???

• Somos aguerridos, bravos,

valientes, locos...

• Vivimos para esto

• NO comemos, NO bebemos, NO

paramos...NO lo necesitamos

• La misión es lo primero

• Nos batimos el cobre por nuestro

objetivo.

© 2019 Binlogic.

Realidad... augmentada ;-)

• Nos gusta lo que hacemos

• Pero también nos gusta el tiempo libre

• Queremos reconocimiento, fama, dinero...

• Pero mola tener el finde

• Nos gusta ser imprescindibles

• Pero no todo el tiempo

• Queremos dejar huella

• Pero que no nos persiga para siempre

...

© 2019 Binlogic.

Somos humanos, sin más!!!

Approach...

Sumemos a lo anterior:

• El humano es 100% falible

• La herramientas usadas no siempre

están del todo maduras o no son las

correctas

• Mezclar "churras con merinas" porque

es "bleeding edge"

© 2019 Binlogic.

Inmutabilidad

• Nuevo paradigma

• Infra intocable tras deployment

• Intro cambio >>> nueva infra

• No SSH, No RDP, no nothing...

• No HUMANS allowed!!!

© 2019 Binlogic.

Infra mutable

• Despliegues en directo

• Modificaciones de ficheros de

entorno

• Adaptaciones de configuración

• Actividades de administración

clásica

• SSH, RDP, etc...

© 2019 Binlogic.

Inmutabilidad - Beneficios

• Confiabilidad

• Consistencia

• Reproducibilidad

• Previsibilidad

© 2019 Binlogic.

En muchas ocasiones:

• Simplifica el proceso de despliegue

• Evita el "fine tuning" de lo clásico

• Rollback diferente

Inmutabilidad - ¿Cómo se consigue?

Una posibilidad es:

• Molecule: como TDD approach a la

provisión de imágenes

• Packer: generación efectiva de la

imagen

• Terraform: generación efectiva de la

infraestructura

© 2019 Binlogic.

Molecule

TDD framework para infra

• Ansible

• Docker/AWS/GCP/Vagrant...

• TestInfra

• Flake8

• Ansible lint / YAML lint

• Ansible syntax

© 2019 Binlogic.

Molecule – Commands (skeleton)

Creamos un nuevo Ansible Role

Echemos un ojo

© 2019 Binlogic.

$ molecule init role -r dataops

--> Initializing new role dataops...

Initialized role in /var/tmp/dataops successfully.

$ cd dataops && ls

defaults handlers meta molecule README.md tasks vars

Molecule – Tests

© 2019 Binlogic.

$ vim molecule/default/tests/test_default.py

import os

import testinfra.utils.ansible_runner

testinfra_hosts = ...

def test_hosts_file(host):

f = host.file('/etc/hosts')

assert f.exists

assert f.user == 'root'

assert f.group == 'root'

Molecule

Demo

© 2019 Binlogic.

Really simple one

Molecule – TDD approach

• Ansible role

• La idea sería satisfacer los tests

• Echemos un ojo a tasks/main.yml

• Desarrollo de tareas

• Test, test, test and check!!!

© 2019 Binlogic.

Molecule – Commands (ansible)

© 2019 Binlogic.

$ molecule converge

--> Validating schema /var/tmp/dataops/molecule/default/molecule.yml.Validation completed successfully.--> Test matrix

└── default├── dependency

├── create├── prepare└── converge

--> Action: 'converge'

PLAY [Converge] ****************************************************************TASK [Gathering Facts] *********************************************************ok: [instance]

PLAY RECAP *********************************************************************instance : ok=1 changed=0 unreachable=0 failed=0 skipped=0

Molecule – Commands (testinfra)

© 2019 Binlogic.

$ molecule verify

--> Validating schema /var/tmp/dataops/molecule/default/molecule.yml.Validation completed successfully.--> Test matrix

└── default└── verify

--> Scenario: 'default'--> Action: 'verify'--> Executing Testinfra tests found in /var/tmp/dataops/molecule/default/tests/...

============ test session starts ============rootdir: /var/tmp/dataops/molecule/default

plugins: testinfra-1.19.0collected 1 item

tests/test_default.py . [100%]

============ 1 passed in 2.69 seconds ============Verifier completed successfully.

Molecule – Commands (whole picture)

© 2019 Binlogic.

Una vez desarrollado el Role

$ molecule list

$ molecule syntax && molecule lint

$ molecule idempotence

$ molecule verify

$ molecule destroy

Y como repaso final: $ molecule test

Packer

© 2019 Binlogic.

Generación de imágenes

• Builders:

AWS, GCP, Azure, DigitalOcean, QEMU,

VirtualBox...

• Provisioners:

Ansible, Chef, Puppet, Salt, Shell...

© 2019 Binlogic.

$ cat dataops.json

..."builders": [{

"type": "googlecompute","project_id": "{{user ̀ project`}}",

"machine_type": "{{user ̀ img_type`}}","source_image_family": "{{user ̀ source_img_family`}}","region": "{{user ̀ region`}}",

"zone": "{{user ̀ zone`}}","image_name": "{{user ̀ img_name`}}",

"image_family": "{{user ̀ img_family`}}","image_description": "{{user ̀ img_description`}}",

...

Packer – builders

© 2019 Binlogic.

$ cat dataops.json

..."disk_size": 10,"disk_type": "pd-ssd",

"ssh_timeout": "30s","ssh_username": "{{user ̀ ssh_username`}}",

"ssh_private_key_file": "{{user ̀ ssh_keyfile`}}","omit_external_ip": "{{user ̀ omit_external_ip`}}","use_internal_ip": "{{user ̀ use_internal_ip`}}",

"subnetwork": "{{user `subnetwork`}}","tags": ["packer","dataops","ssh","all","private"]

}],...

Packer – builders

© 2019 Binlogic.

$ cat dataops.json

..."provisioners": [{

"type": "shell","inline": [

"echo '------PROVISION STARTS------'","sudo apt-get update","sudo apt-get -y upgrade",

"sudo apt-get -y install python-dev python-pip","echo 'ansible installed'"

]},

...

Packer – provisioners

© 2019 Binlogic.

$ cat dataops.json

...{

"type": "ansible-local",

"playbook_dir": "{{user ̀ playbook_dir`}}","playbook_file": "{{user ̀ playbook_dir`}}/{{user ̀ playbook_file`}}",

"role_paths": ["{{user ̀ playbook_dir`}}/roles/dataops"

]

},...

Packer – provisioners

© 2019 Binlogic.

$ bash dataops.sh

Template validated successfully.googlecompute output will be in this color.

==> googlecompute: Checking image does not exist...==> googlecompute: Creating temporary SSH key for instance...

==> googlecompute: Using image: ubuntu-1804-bionic-v20190617==> googlecompute: Creating instance...

googlecompute: Loading zone: us-east1-b

googlecompute: Loading machine type: n1-standard-1googlecompute: Requesting instance creation...

googlecompute: Waiting for creation operation to complete...googlecompute: Instance has been created!

==> googlecompute: Waiting for the instance to become running...

googlecompute: IP: 35.243.232.182==> googlecompute: Using ssh communicator to connect: 35.243.232.182

==> googlecompute: Waiting for SSH to become available...

Packer – Demo

Terraform

© 2019 Binlogic.

• Infrastructure as Code ( IaC )

• Unique source of truth

• R3PA pattern

• Posibilidad de "soñar" en dormir

• Blue/Green deployments

• Canary Releases

• Rolling updates...

© 2019 Binlogic.

$ cat autoscaler.tf

resource "google_compute_autoscaler" "bg_dataops_gcautoscale" {name = "bg-dataops-gcautoscale"

project = "${var.project}"zone = "${var.zone}"

target = "${google_compute_instance_group_manager.dataops_gcigm.self_link}"autoscaling_policy = {

max_replicas = "${var.as_max_replicas}"

min_replicas = "${var.as_min_replicas}"cooldown_period = "${var.as_cooldown}"

cpu_utilization {target = "${var.as_cpu_usage}"

}

}}

Terraform

© 2019 Binlogic.

$ terraform apply -var-file=dataops.tfvars ../configurations/

Acquiring state lock. This may take a few moments...data.google_compute_image.bg_dataops_img: Refreshing state...

An execution plan has been generated and is shown below.Resource actions are indicated with the following symbols:

+ createTerraform will perform the following actions:+ google_compute_autoscaler.bg_dataops_gcautoscale

id: <computed>autoscaling_policy.#: "1"

autoscaling_policy.0.cooldown_period: "30"autoscaling_policy.0.cpu_utilization.#: "1"autoscaling_policy.0.cpu_utilization.0.target: "0.9"

autoscaling_policy.0.max_replicas: "10"autoscaling_policy.0.min_replicas: "3"

...

Terraform – Demo

© 2019 Binlogic.