Location : Bangalore
Date : 27th April, 2016
Name of the Speaker : Uchit Vyas
Company Name : Opex Software, Pune
IT Automation Summit 2016
http://www.unicomlearning.com/2016/IT_Automation_Summit/
Bangalore
27th April 2016
VCs on line 1
Scaling under pressure
with
Chef, Packer and Terraform
Who am I: Uchit Vyas
8+ years of exciting experience
Infrastructure automation domain
Leading Cloud Infrastructure
automation in Opex Software
Introduction
Single metric we measure: Speed with correctness
Mission: Automate IT. Create Time
Focus: SAAS applications
SAAS Startup Dream
Scaling with SPEED
How to make this happen, technically?
Images: 4actionmarketing.net, dreamstime.com, thenextweb.com
Pressure from investors
How to make this happen, technically?
Images: startupinitiative.com
SAAS is value: Step 1 of 3
Clear value is delivered
SAAS is value: Step 2 of 3
Clear value is delivered
Onboarding new customers rapidly
SAAS is value: Step 3 of 3
Clear value is delivered
Onboarding new customers rapidly
Upgrade fast, maintain lead,
make value irresistible
Mortality Rate
Images: jeepneymanilaph.files.wordpress.com
Execution Problems
All good
Execution Problems
All good
Harden the OS skill not available immediately
Automate app deploy/cfg after thought
Automate Testing fully after thought
Execution Problems
All good
Harden the OS skill not available immediately
Automate app deployment after thought
Automate app configuration after thought
Microservices independently upgradable?
Lead starting to lose speed, competition!
Images: avopress.com, animationmagazine.net, thenextweb.com
1412/05/15
DevOps + Strong Ops
Dev
Test
Ops
15
Transform test automation for DevOps
Test bed
Infrastructure
creation
Automation
Automated
Test Result
Analytics
Test
Automation
(Selenium,
QTP etc)
Strong Ops to scale
Helpdesk Monitoring
Configuration
Management
Auto-Self
healing
Synthetic
Monitoring
Log analytics
Good News
OS Hardening: Why is it imp?
protecting IP
unfair competition
cybersecurity
private data
direct or indirect
attacks via cloud
President Barack Obama delivers remarks at the Business Roundtable offices in Washington
September 16, 2015.
OS Hardening
Use Chef templates and Terraform
Include following in server configuration definition
Security Rules, Password policies,
Secure SSH, Compliance policy
Important agents (AV, monitoring)
Use “Chef-Vault” for storing secrets
Sample code: login definition
template '/etc/login.defs' do
source 'login.defs.erb'
mode '0444'
owner 'root'
group 'root'
variables(
additional_user_paths: node['env’] …
)
end
Sample code: erb file
'sample.erb'
<% if @port != 80 -%>
Listen <%= @port %>
<% end -%>
Two simple, but powerful concepts
a) Expression evaluation
b) Variable value replacement
Sample code: login definition
variables(
password_max_age: node['auth']['pw_max_age'],
password_min_age: node['auth']['pw_min_age'],
login_retries: node['auth']['retries'],
login_timeout: node['auth']['timeout'],
chfn_restrict: '', # "rwh"
allow_login_without_home:
node['auth']['allow_homeless'],
…
)
Quality gates: Serverspec
High speed lab creation
Terraform is parallelized
Auto-sequencing based on graphs
Terraform can integrate with any layer of the
stack
Lab setup on pre-existing servers, provisioning
servers from scratch - both are supported
Speed of provisioning
Why we chose terraform for high-speed scaling
Number of Machines Chef-metal Terraform
10 2.7 minutes 1.20 minutes
30 3.9 minutes 3.08 minutes
40 5.6 minutes 3.36 minutes
60 9.4 minutes 7.08 minutes
100 15.2 minutes 7.41 minutes
Distributing load elegantly
Next few slides explain the code.
Key highlighted portions are in a red rectangle
Use cases 1: Synthetic monitoring
Use case 2: Load balancing across all regions in a cloud
How to scale across AZs? How to scale across clouds?
.tf Provisioning (AWS multi AZs)
provider "aws" {
region = "us-west-2"
access_key = "XXXXXXX"
secret_key = "XXXXXXXXX"
}ia
variable "region"
{
default = "us-west-2"
}
variable "region_az" {
default = {
"us-east-1" = "us-east-1a,us-east-1c,us-east-1d,us-east-1e"
"us-west-1" = "us-west-1a,us-west-1b,us-west-1c"
"us-west-2" = "us-west-2a,us-west-2b,us-west-2c"
"eu-west-1" = "eu-west-1a,eu-west-1b,eu-west-1c"
"eu-central-1" = "eu-central-1a,eu-central-1b"
"ap-southeast-1" = "ap-southeast-1a,ap-southeast-1b"
"ap-northeast-1" = "ap-northeast-1a,ap-northeast-1b,ap-northeast-1c"
"ap-southeast-2" = "ap-southeast-2a,ap-southeast-2b"
"sa-east-1" = "sa-east-1a,sa-east-1b,sa-east-1c"
}
}
Variables for AZs and AMIs
Lets use this region to start
Region based AZ map
variable "region_az" {
default = {
"us-east-1" = "us-east-1a,us-east-1c,us-east-1d,us-east-1e"
"us-west-1" = "us-west-1a,us-west-1b,us-west-1c"
"us-west-2" = "us-west-2a,us-west-2b,us-west-2c"
"eu-west-1" = "eu-west-1a,eu-west-1b,eu-west-1c"
"eu-central-1" = "eu-central-1a,eu-central-1b"
"ap-southeast-1" = "ap-southeast-1a,ap-southeast-1b"
"ap-northeast-1" = "ap-northeast-1a,ap-northeast-1b,ap-northeast-1c"
"ap-southeast-2" = "ap-southeast-2a,ap-southeast-2b"
"sa-east-1" = "sa-east-1a,sa-east-1b,sa-east-1c"
}
}
variable "ami"
{
default = ...
used in lookup
variable "ami"
{
default =
{
"description" = "Ubuntu server 14.04 ami id"
"us-west-1" = "ami-df6a8b9b"
"us-west-2" = "ami-5189a661"
"us-east-1" = "ami-d05e75b8"
"eu-west-1" = "ami-47a23a30"
"eu-central-1" = "ami-accff2b1"
"ap-northeast-1" = "ami-936d9d93"
"ap-southeast-1" = "ami-96f1c1c4"
"ap-southeast-2" = "ami-69631053"
"sa-east-1" = "ami-4d883350"
}
}
Region based Ubuntu AMI map
resource "aws_instance" "web" {
ami = "${lookup(var.ami, var.region)}"
instance_type = "${var.instance_type}"
count = "${var.servers}"
availability_zone =
"${element(split(",",lookup(var.region_az,
var.region)),
count.index%length
(split(",",lookup(var.region_az,
var.region))
))}"
Resource declaration
resource "aws_instance" "web" {
ami = "${lookup(var.ami, var.region)}"
instance_type = "${var.instance_type}"
count = "${var.servers}"
availability_zone =
"${element(split(",",lookup(var.region_az,
var.region)),
count.index%length
(split(",",lookup(var.region_az,
var.region))
))}"
Scaling and iterating
Math library added from
terraform v0.4
Simulating iteration
Scaling to 100s of servers
Multi-cloud distribution
Step 1
Creating an image
for each cloud
Step 2
Using that image in
the code that we
just saw
Creating image using Packer
"provisioners": [{
"type": "shell",
"inline": [
"sleep 30",
"sudo apt-get update",
"sudo apt-get install -y redis-
server"
]
}] Provisioning Redis server
Strong Ops - Chef provisioner
"provisioners": [{
"type": "chef-client",
"server_url”: “https://mychefserver.com/”
}]
Cookbooks and Recipes:
Helpdesk, monitoring server, monitoring clients, antivirus
agents, auto-healing servers, analytics data couriers...
Using Chef client-server
App context: Data bags
Creating image using Packer
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "us-east-1",
"source_ami": "ami-de0d9eb7",
"instance_type": "t1.micro",
"ssh_username": "ubuntu",
"ami_name": "packer-example {{timestamp}}"
},
Build in Amazon Cloud
Same code for multi-cloud
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "us-east-1",
"source_ami": "ami-de0d9eb7",
"instance_type": "t1.micro",
"ssh_username": "ubuntu",
"ami_name": "packer-example {{timestamp}}"
},
{
"type": "digitalocean",
"api_token": "{{user `do_api_token`}}",
"image": "ubuntu-14-04-x64",
"region": "nyc3",
"size": "512mb"
}],
Build in Amazon Cloud Build in Digital Ocean Cloud
Beautiful multi-color output
amazon-ebs output will be in this color.
digitalocean output will be in this color.
==> digitalocean: Creating temporary ssh key for droplet...
==> amazon-ebs: Prevalidating AMI Name...
==> amazon-ebs: Inspecting the source AMI...
==> amazon-ebs: Creating temporary keypair: packer 55f6c5e5-2b50-c8c3-5e37-7d246b6f0bca
==> amazon-ebs: Creating temporary security group for this instance...
==> amazon-ebs: Authorizing access to port 22 the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
==> digitalocean: Creating droplet...
==> digitalocean: Waiting for droplet to become active...
Pilot of great idea - clear value delivered - all good
Onboarding new customers rapidly without sacrificing quality
Harden the OS - chef-templates, packer, Terraform
Automate app deployment - Terraform + Chef
Automate app configuration - Terraform + Chef + Data bags
Pivot fast, maintain the lead, make value irresistible
DevOps for building new features fast - Adopt DevOps early
Upgrade customers DevOps pipeline + Roles + Environments
Hope this helps!
The 4th question in daily agile scrums that was never asked
4th question: Are you blocking Ops?
Meaning:
a) Are there any new binaries created/deleted?
b) Are there any configuration files that changed?
c) Are there any configuration attributes that changed?
Why DevOps projects fail
Recap, takeaway
Easy to code infrastructure
Easy to replicate infrastructure
Easy to distribute infrastructure across AZs
Easy to balance infrastructure across regions
Easy to balance infrastructure across clouds, countries
Easy to integrate with other apps
The 4th scrum question
IT Automation Summit 2016
http://www.unicomlearning.com/2016/IT_Automation_Summit/
Speaker Name: Uchit Vyas
Email ID: uchit.vyas@opexsoftware.com
Organized by
UNICOM Trainings & Seminars Pvt. Ltd.
contact@unicomlearning.com
Opex Software Thanks You
AUTOMATE IT. CREATE TIME
Hybrid SAAS Apps
App is on AWS (DO, Azure, other clouds)
Appliance, Sync process is on-prem with the tenant
Upgrading customers: Env
knife environment create ClientSetOne
name "ClientSetOne"
description "The packages and configuration used by ClientSetOne"
cookbook_versions({
"nginx" => "<= 1.1.0",
"apt" => "= 0.0.1"
})
override_attributes ({
"nginx" => {
"listen" => [ "80", "443" ]
},
"mysql" => {
"root_pass" => "root"
}
})
Upgrading customers: Associate
Each node can be in exactly one environment
STEP1: knife node edit clientOneAppliance
STEP2: {
"name": "clientOneAppliance",
"chef_environment": "ClientSetOne",
"normal": {
"tags": [
]
},
"run_list": [
"role[web_server]"
]
}
Language Integration challenges
Source: http://sdtimes.com/poll-the-language-i-use-most-at-work-is/
Terraform JSON file which can be executed
by terraform
Java beans for the all available TF resources.
Opening TF to Java world
Terraform Resource
(Java)
Transpiler with
Notification Provisioner
Injection
Terraform Resource
(JSON)
Convert TF Java bean into TF JSON. Injects
echos using local_exec.
Terraform JSON
Resource
Terraform
Executor(Java)
Terraform Action
Handler
Notify Action Handlers for
1. Resource Created
2. Provisioner Execution Started
3. Provisioner Execution Done
Execute Terraform JSON File
Terraform Executor(Java)

Scaling with Automation

  • 1.
    Location : Bangalore Date: 27th April, 2016 Name of the Speaker : Uchit Vyas Company Name : Opex Software, Pune IT Automation Summit 2016 http://www.unicomlearning.com/2016/IT_Automation_Summit/ Bangalore 27th April 2016
  • 2.
    VCs on line1 Scaling under pressure with Chef, Packer and Terraform
  • 3.
    Who am I:Uchit Vyas 8+ years of exciting experience Infrastructure automation domain Leading Cloud Infrastructure automation in Opex Software
  • 4.
    Introduction Single metric wemeasure: Speed with correctness Mission: Automate IT. Create Time Focus: SAAS applications
  • 5.
    SAAS Startup Dream Scalingwith SPEED How to make this happen, technically? Images: 4actionmarketing.net, dreamstime.com, thenextweb.com
  • 6.
    Pressure from investors Howto make this happen, technically? Images: startupinitiative.com
  • 7.
    SAAS is value:Step 1 of 3 Clear value is delivered
  • 8.
    SAAS is value:Step 2 of 3 Clear value is delivered Onboarding new customers rapidly
  • 9.
    SAAS is value:Step 3 of 3 Clear value is delivered Onboarding new customers rapidly Upgrade fast, maintain lead, make value irresistible
  • 10.
  • 11.
  • 12.
    Execution Problems All good Hardenthe OS skill not available immediately Automate app deploy/cfg after thought Automate Testing fully after thought
  • 13.
    Execution Problems All good Hardenthe OS skill not available immediately Automate app deployment after thought Automate app configuration after thought Microservices independently upgradable? Lead starting to lose speed, competition! Images: avopress.com, animationmagazine.net, thenextweb.com
  • 14.
  • 15.
    15 Transform test automationfor DevOps Test bed Infrastructure creation Automation Automated Test Result Analytics Test Automation (Selenium, QTP etc)
  • 16.
    Strong Ops toscale Helpdesk Monitoring Configuration Management Auto-Self healing Synthetic Monitoring Log analytics
  • 17.
  • 18.
    OS Hardening: Whyis it imp? protecting IP unfair competition cybersecurity private data direct or indirect attacks via cloud President Barack Obama delivers remarks at the Business Roundtable offices in Washington September 16, 2015.
  • 19.
    OS Hardening Use Cheftemplates and Terraform Include following in server configuration definition Security Rules, Password policies, Secure SSH, Compliance policy Important agents (AV, monitoring) Use “Chef-Vault” for storing secrets
  • 20.
    Sample code: logindefinition template '/etc/login.defs' do source 'login.defs.erb' mode '0444' owner 'root' group 'root' variables( additional_user_paths: node['env’] … ) end
  • 21.
    Sample code: erbfile 'sample.erb' <% if @port != 80 -%> Listen <%= @port %> <% end -%> Two simple, but powerful concepts a) Expression evaluation b) Variable value replacement
  • 22.
    Sample code: logindefinition variables( password_max_age: node['auth']['pw_max_age'], password_min_age: node['auth']['pw_min_age'], login_retries: node['auth']['retries'], login_timeout: node['auth']['timeout'], chfn_restrict: '', # "rwh" allow_login_without_home: node['auth']['allow_homeless'], … )
  • 23.
  • 24.
    High speed labcreation Terraform is parallelized Auto-sequencing based on graphs Terraform can integrate with any layer of the stack Lab setup on pre-existing servers, provisioning servers from scratch - both are supported
  • 25.
    Speed of provisioning Whywe chose terraform for high-speed scaling Number of Machines Chef-metal Terraform 10 2.7 minutes 1.20 minutes 30 3.9 minutes 3.08 minutes 40 5.6 minutes 3.36 minutes 60 9.4 minutes 7.08 minutes 100 15.2 minutes 7.41 minutes
  • 30.
    Distributing load elegantly Nextfew slides explain the code. Key highlighted portions are in a red rectangle Use cases 1: Synthetic monitoring Use case 2: Load balancing across all regions in a cloud How to scale across AZs? How to scale across clouds?
  • 31.
    .tf Provisioning (AWSmulti AZs) provider "aws" { region = "us-west-2" access_key = "XXXXXXX" secret_key = "XXXXXXXXX" }ia
  • 32.
    variable "region" { default ="us-west-2" } variable "region_az" { default = { "us-east-1" = "us-east-1a,us-east-1c,us-east-1d,us-east-1e" "us-west-1" = "us-west-1a,us-west-1b,us-west-1c" "us-west-2" = "us-west-2a,us-west-2b,us-west-2c" "eu-west-1" = "eu-west-1a,eu-west-1b,eu-west-1c" "eu-central-1" = "eu-central-1a,eu-central-1b" "ap-southeast-1" = "ap-southeast-1a,ap-southeast-1b" "ap-northeast-1" = "ap-northeast-1a,ap-northeast-1b,ap-northeast-1c" "ap-southeast-2" = "ap-southeast-2a,ap-southeast-2b" "sa-east-1" = "sa-east-1a,sa-east-1b,sa-east-1c" } } Variables for AZs and AMIs Lets use this region to start
  • 33.
    Region based AZmap variable "region_az" { default = { "us-east-1" = "us-east-1a,us-east-1c,us-east-1d,us-east-1e" "us-west-1" = "us-west-1a,us-west-1b,us-west-1c" "us-west-2" = "us-west-2a,us-west-2b,us-west-2c" "eu-west-1" = "eu-west-1a,eu-west-1b,eu-west-1c" "eu-central-1" = "eu-central-1a,eu-central-1b" "ap-southeast-1" = "ap-southeast-1a,ap-southeast-1b" "ap-northeast-1" = "ap-northeast-1a,ap-northeast-1b,ap-northeast-1c" "ap-southeast-2" = "ap-southeast-2a,ap-southeast-2b" "sa-east-1" = "sa-east-1a,sa-east-1b,sa-east-1c" } } variable "ami" { default = ... used in lookup
  • 34.
    variable "ami" { default = { "description"= "Ubuntu server 14.04 ami id" "us-west-1" = "ami-df6a8b9b" "us-west-2" = "ami-5189a661" "us-east-1" = "ami-d05e75b8" "eu-west-1" = "ami-47a23a30" "eu-central-1" = "ami-accff2b1" "ap-northeast-1" = "ami-936d9d93" "ap-southeast-1" = "ami-96f1c1c4" "ap-southeast-2" = "ami-69631053" "sa-east-1" = "ami-4d883350" } } Region based Ubuntu AMI map
  • 35.
    resource "aws_instance" "web"{ ami = "${lookup(var.ami, var.region)}" instance_type = "${var.instance_type}" count = "${var.servers}" availability_zone = "${element(split(",",lookup(var.region_az, var.region)), count.index%length (split(",",lookup(var.region_az, var.region)) ))}" Resource declaration
  • 36.
    resource "aws_instance" "web"{ ami = "${lookup(var.ami, var.region)}" instance_type = "${var.instance_type}" count = "${var.servers}" availability_zone = "${element(split(",",lookup(var.region_az, var.region)), count.index%length (split(",",lookup(var.region_az, var.region)) ))}" Scaling and iterating Math library added from terraform v0.4 Simulating iteration Scaling to 100s of servers
  • 37.
    Multi-cloud distribution Step 1 Creatingan image for each cloud Step 2 Using that image in the code that we just saw
  • 38.
    Creating image usingPacker "provisioners": [{ "type": "shell", "inline": [ "sleep 30", "sudo apt-get update", "sudo apt-get install -y redis- server" ] }] Provisioning Redis server
  • 39.
    Strong Ops -Chef provisioner "provisioners": [{ "type": "chef-client", "server_url”: “https://mychefserver.com/” }] Cookbooks and Recipes: Helpdesk, monitoring server, monitoring clients, antivirus agents, auto-healing servers, analytics data couriers... Using Chef client-server
  • 40.
  • 41.
    Creating image usingPacker "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "ami-de0d9eb7", "instance_type": "t1.micro", "ssh_username": "ubuntu", "ami_name": "packer-example {{timestamp}}" }, Build in Amazon Cloud
  • 42.
    Same code formulti-cloud "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "ami-de0d9eb7", "instance_type": "t1.micro", "ssh_username": "ubuntu", "ami_name": "packer-example {{timestamp}}" }, { "type": "digitalocean", "api_token": "{{user `do_api_token`}}", "image": "ubuntu-14-04-x64", "region": "nyc3", "size": "512mb" }], Build in Amazon Cloud Build in Digital Ocean Cloud
  • 43.
    Beautiful multi-color output amazon-ebsoutput will be in this color. digitalocean output will be in this color. ==> digitalocean: Creating temporary ssh key for droplet... ==> amazon-ebs: Prevalidating AMI Name... ==> amazon-ebs: Inspecting the source AMI... ==> amazon-ebs: Creating temporary keypair: packer 55f6c5e5-2b50-c8c3-5e37-7d246b6f0bca ==> amazon-ebs: Creating temporary security group for this instance... ==> amazon-ebs: Authorizing access to port 22 the temporary security group... ==> amazon-ebs: Launching a source AWS instance... ==> digitalocean: Creating droplet... ==> digitalocean: Waiting for droplet to become active...
  • 44.
    Pilot of greatidea - clear value delivered - all good Onboarding new customers rapidly without sacrificing quality Harden the OS - chef-templates, packer, Terraform Automate app deployment - Terraform + Chef Automate app configuration - Terraform + Chef + Data bags Pivot fast, maintain the lead, make value irresistible DevOps for building new features fast - Adopt DevOps early Upgrade customers DevOps pipeline + Roles + Environments Hope this helps!
  • 45.
    The 4th questionin daily agile scrums that was never asked 4th question: Are you blocking Ops? Meaning: a) Are there any new binaries created/deleted? b) Are there any configuration files that changed? c) Are there any configuration attributes that changed? Why DevOps projects fail
  • 46.
    Recap, takeaway Easy tocode infrastructure Easy to replicate infrastructure Easy to distribute infrastructure across AZs Easy to balance infrastructure across regions Easy to balance infrastructure across clouds, countries Easy to integrate with other apps The 4th scrum question
  • 47.
    IT Automation Summit2016 http://www.unicomlearning.com/2016/IT_Automation_Summit/ Speaker Name: Uchit Vyas Email ID: uchit.vyas@opexsoftware.com Organized by UNICOM Trainings & Seminars Pvt. Ltd. contact@unicomlearning.com
  • 48.
    Opex Software ThanksYou AUTOMATE IT. CREATE TIME
  • 49.
    Hybrid SAAS Apps Appis on AWS (DO, Azure, other clouds) Appliance, Sync process is on-prem with the tenant
  • 50.
    Upgrading customers: Env knifeenvironment create ClientSetOne name "ClientSetOne" description "The packages and configuration used by ClientSetOne" cookbook_versions({ "nginx" => "<= 1.1.0", "apt" => "= 0.0.1" }) override_attributes ({ "nginx" => { "listen" => [ "80", "443" ] }, "mysql" => { "root_pass" => "root" } })
  • 51.
    Upgrading customers: Associate Eachnode can be in exactly one environment STEP1: knife node edit clientOneAppliance STEP2: { "name": "clientOneAppliance", "chef_environment": "ClientSetOne", "normal": { "tags": [ ] }, "run_list": [ "role[web_server]" ] }
  • 52.
    Language Integration challenges Source:http://sdtimes.com/poll-the-language-i-use-most-at-work-is/
  • 53.
    Terraform JSON filewhich can be executed by terraform Java beans for the all available TF resources. Opening TF to Java world Terraform Resource (Java) Transpiler with Notification Provisioner Injection Terraform Resource (JSON) Convert TF Java bean into TF JSON. Injects echos using local_exec.
  • 54.
    Terraform JSON Resource Terraform Executor(Java) Terraform Action Handler NotifyAction Handlers for 1. Resource Created 2. Provisioner Execution Started 3. Provisioner Execution Done Execute Terraform JSON File Terraform Executor(Java)