// Feed your brain               GR8Conf US 2012
        gr8.technologies.each{                July 30-31
          yourBrain << it            Minneapolis, MN, US
        }




The Grails App that Deploys Netflix to the Cloud
Extended Edition Slides

Joe Sondow, Netflix
   @joesondow #gr8conf
Slides online
http://slideshare.net/joesondow




   @joesondow #gr8conf
Who am I?




 @joesondow #gr8conf
Who am I?
Still time to sneak out to another session




   @joesondow #gr8conf
Who am I?




 @joesondow #gr8conf
Who am I?
Joe Sondow




 @joesondow #gr8conf
Who am I?
Joe Sondow
Netflix since 2010




 @joesondow #gr8conf
Who am I?
Joe Sondow
Netflix since 2010
Asgard lead




 @joesondow #gr8conf
Who am I?
Joe Sondow
Netflix since 2010
Asgard lead
Grails




 @joesondow #gr8conf
Who am I?
Joe Sondow
Netflix since 2010
Asgard lead
Grails
jQuery




 @joesondow #gr8conf
Why am I here?




 @joesondow #gr8conf
Why am I here?
Why am I here?
Sell you something
Why am I here?
Sell you something
Discuss business plans
Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Give back to community
Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Give back to community
Steal your engineers
Asgard
Asgard
Asgard
Screen shots
Asgard
Application list
Asgard
Auto Scaling Group list
Asgard
Cluster deployment, ready for fast rollback
Asgard
Asgard
Application deployment
Asgard
Application deployment
Cloud management
Asgard
Application deployment
Cloud management
Started 2010
Asgard
Application deployment
Cloud management
Started 2010
Open source June 2012
Asgard
Application deployment
Cloud management
Started 2010
Open source June 2012
http://netflix.github.com
Asgard
Application deployment
Cloud management
Started 2010
Open source June 2012
http://netflix.github.com
100’s of Jira tickets
Asgard
Application deployment
Cloud management
Started 2010
Open source June 2012
http://netflix.github.com
100’s of Jira tickets
Actively developed
User forum
https://groups.google.com/group/AsgardUsers
Source code and download
https://github.com/Netflix/asgard
Joe Sondow   Clay McCoy   Jason Gritman
  @joesondow   @claymccoy   @jgritman




Asgard Team
July 2012
With more than 25 million streaming members in the
United States, Canada, Latin America, the United
Kingdom and Ireland, Netflix, Inc. (NASDAQ: NFLX) is
the world's leading internet subscription service for
enjoying movies and TV programs.
(from http://ir.netflix.com)
Freedom and Responsibility
Freedom and Responsibility
Corporate culture and the Cloud
Freedom and Responsibility
Freedom and Responsibility
Cloud SOA
Freedom and Responsibility
Cloud SOA
100’s of services
Freedom and Responsibility
Cloud SOA
100’s of services
Small teams
Freedom and Responsibility
Cloud SOA
100’s of services
Small teams
Independent releases
Freedom and Responsibility
Cloud SOA
100’s of services
Small teams
Independent releases
Controlled chaos
Regional URLs
Regional URLs
Let’s see some code
Regional URLs
Cloud object URLs were unique, conventional, intuitive




Default Grails URL mapping is excellent
Regional URLs
Later, Netflix needed to use multiple Amazon regions
Regional URLs
Region is another URL dimension




Add URL mapping, but don’t break existing links
Regional URLs




RegionService.groovy
Regional URLs




RegionService.groovy
Regional URLs




RegionService.groovy

Grails may soon provide a better hook
Regional URLs
Some controllers should never be regional



RegionService detects RegionAgnostic annotation
Cloud deployment model
Cloud deployment model
Applications and Clusters
Cloud deployment model
Cloud deployment model

   Auto Scaling
      Group
Cloud deployment model

    Auto Scaling
       Group




    Launch
  Configuration
Cloud deployment model

                   Elastic Load
                     Balancer
    Auto Scaling
       Group




    Launch
  Configuration
Cloud deployment model

                    Elastic Load
                      Balancer
    Auto Scaling
       Group




    Launch
  Configuration

                   Amazon Machine
                       Image
Cloud deployment model

                    Elastic Load
                      Balancer
    Auto Scaling
       Group




                   Security
                    Group


    Launch
  Configuration

                   Amazon Machine
                       Image
Cloud deployment model

                    Elastic Load
                      Balancer
    Auto Scaling
       Group




                   Security         Instances
                    Group


    Launch
  Configuration

                   Amazon Machine
                       Image
Cloud deployment model

                    Elastic Load
                      Balancer
    Auto Scaling
       Group




                   Security         Instances
                    Group


    Launch
  Configuration

                   Amazon Machine
                       Image
Cloud deployment model

                    Elastic Load
                      Balancer
    Auto Scaling
       Group




                   Security         Instances
                    Group


    Launch
  Configuration

                   Amazon Machine
                       Image
Cloud deployment model

                    Elastic Load
                      Balancer
    Auto Scaling
       Group




                   Security         Instances
                    Group


    Launch
  Configuration

                   Amazon Machine
                       Image
Cloud deployment model
Cloud deployment model




                    Search
Cloud deployment model




            API
                    Search
Cloud deployment model
                     Ratings




            API
                    Search
Cloud deployment model
Streaming Starts
                         Ratings




                   API
                         Search
Cloud deployment model
 Streaming Starts
                          Ratings




Autocomplete        API
                          Search
Cloud deployment model
                    Sign Up
 Streaming Starts
                              Ratings




Autocomplete           API
                              Search
Cloud deployment model
                      Sign Up
 Streaming Starts
                                       Ratings

                      Application
        Application

                                       Application



Autocomplete              API
                                      Search


   Application          Application
                                      Application
Inventing the Application
Inventing the Application
 Problem:
 Application is not an Amazon concept
Inventing the Application
 Problem:
 Application is not an Amazon concept


 Solution:
 Create an Application domain in SimpleDB
 Enforce naming conventions on Amazon objects
Fast Rollback
Fast Rollback
Optimism causes outages
Fast Rollback
Optimism causes outages
Production traffic is unique
Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Monitor results
Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Monitor results
Revert traffic quickly
Fast Rollback
Fast Rollback


                     api-frontend




   api-usprod-v007
Fast Rollback


                     api-frontend




   api-usprod-v007                  api-usprod-v008
Fast Rollback


                     api-frontend




   api-usprod-v007                  api-usprod-v008
Fast Rollback


                     api-frontend




   api-usprod-v007                  api-usprod-v008
Fast Rollback


                     api-frontend




   api-usprod-v007                  api-usprod-v008
Fast Rollback


                     api-frontend




   api-usprod-v007
Inventing the Cluster
Inventing the Cluster
 Problem:
 Two ASGs with one function but different names
Inventing the Cluster
 Problem:
 Two ASGs with one function but different names


 Solution:
 Append version number in reserved format
 Parse ASG name to determine long-term “cluster”
Inventing the Cluster
Instead of keeping a database in sync, use naming
conventions to store the source in truth in Amazon’s API

api                  Application
api-usprod           Cluster
api-usprod-v007      Auto Scaling Group
api-usprod-v008      Auto Scaling Group
Database Aversion
Database Aversion
Storing metadata on cloud objects
Database Aversion
Database Aversion
 Naming conventions
Database Aversion
 Naming conventions
 Tagging conventions
Database Aversion
 Naming conventions
 Tagging conventions
 No GORM domain objects
Database Aversion
 Naming conventions
 Tagging conventions
 No GORM domain objects
 AWS Java SDK
Database Aversion
 Naming conventions
 Tagging conventions
 No GORM domain objects
 AWS Java SDK
 Less to go out of sync
Database Aversion
 Naming conventions
 Tagging conventions
 No GORM domain objects
 AWS Java SDK
 Less to go out of sync
 Shared source of truth
Open Source, Closed Config
Open Source, Closed Config
Pull company-specific details out of Asgard
Open Source, Closed Config
Open Source, Closed Config
Open Source, Closed Config
                   Asgard for
                   Netflix is
                   configured to
                   use company-
                   specific
                   extension
                   points such
                   as standard
                   utility links
                   for instances
Open Source, Closed Config
    Out-of-the-box Asgard installation
    has no instance utility links
Open Source, Closed Config
           Netflix specific $ASGARD_HOME/Config.groovy
link {

      // Avoid GStrings here because these Strings are stored dynamic templates for arbitrary server names.
      String logUrlStart = 'http://${server}:7777'
      String configUrlStart = 'http://${server}:9999/AdminConfig'
      instanceLinkGroupingsToLinkTemplateLists = [
              'Logs': [
                      new TextLinkTemplate(logUrlStart + '/Admin/list?view=tomcat/catalina.out', 'catalina.out'),
                      new TextLinkTemplate(logUrlStart + '/Admin/list', 'Log File Archive'),
                      new TextLinkTemplate(logUrlStart + '/Admin/threaddumps', 'Thread Dumps'),
                      new TextLinkTemplate(logUrlStart + '/AdminProxy', 'Admin Proxy Info'),
                      new TextLinkTemplate(logUrlStart + '/AdminStatus', 'Admin Proxy Status'),
                      new TextLinkTemplate(logUrlStart + '/GC/index', 'GC Visualization')
              ],
              'Netflix Configuration': [
                      new TextLinkTemplate(configUrlStart + '/prop.html', 'NetflixConfiguration Properties Console'),
                      new TextLinkTemplate(configUrlStart + '/libs.html', 'Libraries Console'),
                      new TextLinkTemplate(configUrlStart + '/machineProps', 'Machine Readable Properties'),
                      new TextLinkTemplate(configUrlStart + '/webapp/META-INF/MANIFEST.MF', 'Manifest'),
              ]
      ]
}
Open Source, Closed Config
 grails-app/conf/Config.groovy
 references external configuration file
 ~/.asgard/Config.groovy
 https://github.com/Netflix/asgard/blob/master/grails-app/conf/Config.groovy


asgardHome = System.getenv('ASGARD_HOME') ?: System.getProperty('ASGARD_HOME') ?:
        "${System.getProperty('user.home')}/.asgard"

// Locations to search for config files that get merged into the main config.
// Config files can either be Java properties files or ConfigSlurper scripts.
grails.config.locations = [
        "file:${asgardHome}/Config.groovy",
        'classpath:sourceVersion.properties'
]
Open Source, Closed Config
 External Config.groovy also hold the AWS account
 credentials, or references for finding them.
grails {
    awsAccounts=["178123456789"]
    awsAccountNames=["178123456789":"prod"]
}
secret {
    accessId="AKIAILAND0E3TMSJA8BA"
    secretKey="Od0AMY/dEC3P0/wZQjpmRtm2MVZc9330epDk+pwm"
}
cloud {
    accountName="prod"
    publicResourceAccounts=["amazon"]
}
Caching the Cloud
Caching the Cloud
Responsive, massive, multi-region metadata
Caching the Cloud
Caching the Cloud
 Large counts
Caching the Cloud
 Large counts
 Many types
Caching the Cloud
 Large counts
 Many types
 Complex relationships
Caching the Cloud
 Large counts
 Many types
 Complex relationships
 Multiple regions
Caching the Cloud
 Large counts
 Many types
 Complex relationships
 Multiple regions
 Consistent single objects
Caching the Cloud
 Large counts
 Many types
 Complex relationships
 Multiple regions
 Consistent single objects
 Eventually consistent lists
Caching the Cloud
class Caches {
    final CachedMap<AppRegistration> allApplications
    final CachedMap<ApplicationMetrics> allApplicationMetrics
    final CachedMap<HardwareProfile> allHardwareProfiles
    final MultiRegionCachedMap<MetricAlarm> allAlarms
    final MultiRegionCachedMap<ApplicationInstance> allApplicationInstances
    final MultiRegionCachedMap<AutoScalingGroup> allAutoScalingGroups
    final MultiRegionCachedMap<AvailabilityZone> allAvailabilityZones
    final MultiRegionCachedMap<Cluster> allClusters
    final MultiRegionCachedMap<DBInstance> allDBInstances
    final MultiRegionCachedMap<DBSecurityGroup> allDBSecurityGroups
    final MultiRegionCachedMap<DBSnapshot> allDBSnapshots
    final MultiRegionCachedMap<String> allDomains
    final MultiRegionCachedMap<FastProperty> allFastProperties
    final MultiRegionCachedMap<Image> allImages
    final MultiRegionCachedMap<Instance> allInstances
    final MultiRegionCachedMap<InstanceTypeData> allInstanceTypes
    final MultiRegionCachedMap<KeyPairInfo> allKeyPairs
    final MultiRegionCachedMap<LaunchConfiguration> allLaunchConfigurations
    // etc.
    // etc.
    // etc.
}
Caching the Cloud
class Caches {
    final CachedMap<AppRegistration> allApplications
    final CachedMap<ApplicationMetrics> allApplicationMetrics
    final CachedMap<HardwareProfile> allHardwareProfiles
    final MultiRegionCachedMap<MetricAlarm> allAlarms
    final MultiRegionCachedMap<ApplicationInstance> allApplicationInstances
    final MultiRegionCachedMap<AutoScalingGroup> allAutoScalingGroups
    final MultiRegionCachedMap<AvailabilityZone> allAvailabilityZones
    final MultiRegionCachedMap<Cluster> allClusters
    final MultiRegionCachedMap<DBInstance> allDBInstances
    final MultiRegionCachedMap<DBSecurityGroup> allDBSecurityGroups
    final MultiRegionCachedMap<DBSnapshot> allDBSnapshots
    final MultiRegionCachedMap<String> allDomains
    final MultiRegionCachedMap<FastProperty> allFastProperties
    final MultiRegionCachedMap<Image> allImages
    final MultiRegionCachedMap<Instance> allInstances
    final MultiRegionCachedMap<InstanceTypeData> allInstanceTypes
    final MultiRegionCachedMap<KeyPairInfo> allKeyPairs
    final MultiRegionCachedMap<LaunchConfiguration> allLaunchConfigurations
    // etc.
    // etc.
    // etc.
}
Caching the Cloud
class AwsRdsService implements CacheInitializer, InitializingBean {

    MultiRegionAwsClient<AmazonRDS> awsClient
    Caches caches

    void initializeCaches() {
        caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })
    }

    private List<DBInstance> retrieveDBInstances(Region region) {
        awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()
    }

    Collection<DBInstance> getDBInstances(UserContext userContext) {
        caches.allDBInstances.by(userContext.region).list()
    }
}
Caching the Cloud
class AwsRdsService implements CacheInitializer, InitializingBean {

    MultiRegionAwsClient<AmazonRDS> awsClient
    Caches caches

    void initializeCaches() {
        caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })
    }

    private List<DBInstance> retrieveDBInstances(Region region) {
        awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()
    }

    Collection<DBInstance> getDBInstances(UserContext userContext) {
        caches.allDBInstances.by(userContext.region).list()
    }
}
Caching the Cloud
class AwsRdsService implements CacheInitializer, InitializingBean {

    MultiRegionAwsClient<AmazonRDS> awsClient
    Caches caches

    void initializeCaches() {
        caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })
    }

    private List<DBInstance> retrieveDBInstances(Region region) {
        awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()
    }

    Collection<DBInstance> getDBInstances(UserContext userContext) {
        caches.allDBInstances.by(userContext.region).list()
    }
}
Caching the Cloud
class AwsRdsService implements CacheInitializer, InitializingBean {

    MultiRegionAwsClient<AmazonRDS> awsClient
    Caches caches

    void initializeCaches() {
        caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })
    }

    private List<DBInstance> retrieveDBInstances(Region region) {
        awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()
    }

    Collection<DBInstance> getDBInstances(UserContext userContext) {
        caches.allDBInstances.by(userContext.region).list()
    }
}
Caching the Cloud
Visual Language for the Cloud
Visual Language for the Cloud
Tango open source icons
Visual Language for the Cloud
Visual Language for the Cloud
 AWS is intimidating
Visual Language for the Cloud
 AWS is intimidating
 Many object types
Visual Language for the Cloud
 AWS is intimidating
 Many object types
 Help newbie users
Visual Language for the Cloud
 AWS is intimidating
 Many object types
 Help newbie users
 Reduce cognitive load
Visual Language for the Cloud
 AWS is intimidating
 Many object types
 Help newbie users
 Reduce cognitive load
 Make it easy
Visual Language for the Cloud
 AWS is intimidating
 Many object types
 Help newbie users
 Reduce cognitive load
 Make it easy
 Avoid surprises
Visual Language for the Cloud
Visual Language for the Cloud
At a glance, these nav bar items look alike.
Visual Language for the Cloud
At a glance, these nav bar items look alike.
Visual Language for the Cloud
Visual Language for the Cloud



Some screens have multiple action
buttons that look too similar.
Visual Language for the Cloud



Some screens have multiple action
buttons that look too similar.
Visual Language for the Cloud
Visual Language for the Cloud




  Because of naming conventions,
  these links look alike.
Visual Language for the Cloud




  Because of naming conventions,
  these links look alike.
Visual Language for the Cloud
Visual Language for the Cloud
           The indicators for the
           current AWS region are
           too easy to miss.
Visual Language for the Cloud
           The indicators for the
           current AWS region are
           too easy to miss.
Visual Language for the Cloud



These availability zones
are important to
recognize at a glance
but their names look
similar, and they appear
on many screens.
Visual Language for the Cloud



These availability zones
are important to
recognize at a glance
but their names look
similar, and they appear
on many screens.
Visual Language for the Cloud
Tango Icons
Tango Icons
http://tango.freedesktop.org/
Tango Icons
http://tango.freedesktop.org/
http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines
Tango Icons
http://tango.freedesktop.org/
http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines

http://commons.wikimedia.org/wiki/Tango_icons
Tango Icons
http://tango.freedesktop.org/
http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines

http://commons.wikimedia.org/wiki/Tango_icons
Used by Firefox, Jenkins, GIMP, OpenOffice, VMWare
REST API in Grails
REST API in Grails
Enable external mashups with cloud data
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
ApplicationController.groovy
Offline Development
Offline Development
Makes on a plane
Offline Development
Mock data
Mock behavior
System property switch offline=true
Mock Data
http://asgardprod/us-east-1/autoScaling/list.json
Mock Data
Parse JSON
Mock Behavior
Override Amazon Java client methods
System Property
grails run-app -Doffline=true
Why not the AWS console?
Why not the AWS console?
No, not to keep me busy
Why not the AWS console?
Why not the AWS console?
Why not the AWS console?
Hide keys
Why not the AWS console?
Hide keys
Customize model
Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Automate workflow
Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Automate workflow
Log changes
Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Automate workflow
Log changes
Integrate systems
Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Automate workflow
Log changes
Integrate systems
Create REST API
@NetflixOSS
@NetflixOSS
http://techblog.netflix.com
@NetflixOSS
http://techblog.netflix.com
http://netflix.github.com
http://github.com/Netflix/asgard

Thank you
http://github.com/Netflix/asgard

Thank you
Questions?


  github.com/joesondow
  @joesondow
  slideshare.net/joesondow
  linkedin.com/in/joesondow

Asgard: Using Grails to Deploy Netflix to AWS (Extended Slides)

Editor's Notes