© 2018 Citrix | Confidential
Learnings from 8x8’s Experience
JUNE 18, 2020
Business Continuity with
Microservices-Based Apps and DevOps
Pankaj Gupta, Citrix PankajOnCloud
Lance Johnson, 8x8
Michael Laws, 8x8
2 © 2018 Citrix | Confidential
Micro-services Based Application for Business Continuity
Portability
Modular App
Architecture
Break application in small
services, which can be
developed, deployed and
auto-scaled independently.
Faster Releases
Very frequent application
updates (even many times a
day)
Continuous Delivery to
automate deployment
Self contained runtime
Application portability
across Clouds
Read “A Pandemic Plan for Application Architecture” at https://tinyurl.com/K8sApp
3 © 2018 Citrix | Confidential3 © 2019 Citrix | Confidential
Diverse Stakeholders Have Unique Needs for Business
Continuity
Application & infrastructure security
Container security & API gateways
Automation
DevSecOps
SRE
Application availability
Observability
Incident response
Postmortems
DevOps
Faster release &
deployment cycles
CI/CD & automation
Canary & progressive
rollout
User experience
Troubleshooting
Microservice discovery &
routing
Developers
Network policy & compliance
Manage, control & monitor network
Resources & capacity planning
(visibility)
NetOps
Platform Team
Platform governance
Operational
efficiency
Developer agility
4 © 2018 Citrix | Confidential4 © 2019 Citrix | Confidential
Citrix Micro Services App Delivery Principles
Integrated With Your
Platform & Tools
Get apps to production fast with vast K8s
platform & opensource tools support
Performance
& Scale
Support large clusters &
very dynamic microservices
Consistent App &
API Security
Break the silos
of monoliths & microservices
Architecture
Flexibility
Move at speed of your IT skill set
Balance benefits & complexity
Complete
Observability Stack
Gain visibility & troubleshoot
problems faster, Break the silos
Production-Grade Solution at the Speed of Business
Read “7 Key Considerations for Microservices-Based Application Delivery” at https://tinyurl.com/7Consideration
5 © 2018 Citrix | Confidential
For Your Every Step for Transition to Hybrid Multi Cloud
Single Software Code Base
across Portfolio for
Features, APIs & Operational Consistency, and Innovation Velocity
Hardware
Appliances
DC & Private Cloud, MPX
Virtual Appliances
DC & Private Cloud, VPX
In Public Cloud
VPX for AWS, Azure, GCP
Containers
Public & Private Cloud, CPX
Bare Metal
Private & Public Cloud, BLX
Multi-Tenant
MSP & Private Cloud,
SDX
Citrix ADC Portfolio: Built for Multi-Cloud
Most Comprehensive, Feature Rich & Software-Centric
6 © 2018 Citrix | Confidential6 © 2019 Citrix | Confidential
Complexity
Benefits
Unified Ingress
2-Tier Ingress
Service Mesh Lite
Service Mesh
Low High
High
Balance the tradeoff
between benefits and complexity
Simplest and quickest way
to production
Simple for network-savvy
platform teams
Best observability &
security but complex
Service mesh-like
benefits but simpler
Microservices App Delivery Architecture Choices
Read “The Best Way to Select a Proxy Architecture for Microservices Application Delivery” https://tinyurl.com/ProxyArch
7 | © 8x8, Inc. All Rights Reserved.
DevOps “Three Body Problem”
8 | © 8x8, Inc. All Rights Reserved.
Introductions
Lance Johnson, Director of Engineering, Cloud R&D
https://twitter.com/johnsonrl
https://github.com/rljohnsn
https://www.linkedin.com/in/lancejohnsn/
Michael Laws, Sr. Site Reliability Engineer @8x8
https://www.linkedin.com/in/mikelaws/
9 | © 8x8, Inc. All Rights Reserved.
Metrics
10 | © 8x8, Inc. All Rights Reserved.
Unified Communications
11 | © 8x8, Inc. All Rights Reserved.
At a Glance
12 | © 8x8, Inc. All Rights Reserved.
Unified Communications
Meetings
Chat
Voice
Call Center
13 | © 8x8, Inc. All Rights Reserved.
The Three Body Problem
Unsolvable mathematical problem
14 | © 8x8, Inc. All Rights Reserved.
DevOps Three Body Problem
Deployments
• Ssh
• Jenkins Pipelines
• HyperLoop
• Ansible
• Chef
• Terraform
• Cloudformation
Customers
• DNS
• LTM
• GTM
• Route53
• Geo
• Proxy
• CloudFlare
• Ambassador/Envoy
Dependencies
• Data Centers
• Clouds
• Files
• Databases
• Random VPC
• Public
• Private
How did 8x8 approach the problem?
15 | © 8x8, Inc. All Rights Reserved.
“What is your Quest?”
16 | © 8x8, Inc. All Rights Reserved.
DevOps Fundamentals
• Safety
• Provide a way for changes to be validated and tested.
• Automation
• Deliver the changes the same way for any environment
• Knowledge Sharing
• Learn from each other with open collaboration
• Autonomy
• Immediate visibility that changes deployed are safe
• Continuously verified for every change
17 | © 8x8, Inc. All Rights Reserved.
Adoption of Microservices: 100s of Them!
● Which
○ Foundational Framework services (Cloud8)
○ Shared Services
○ Stateless Application Services
● Why
○ Divide and conquer
○ Technology refresh
○ Team Autonomy
18 | © 8x8, Inc. All Rights Reserved.
Design Goals
● Uniform deployment regardless of environments
● Service Discovery
● High availability and automatic recovery
● Container Packaging
● Native Kubernetes
● Public Charts
● Scaffolding Charts
19 | © 8x8, Inc. All Rights Reserved.
Deployment Environments - Cloud
20 | © 8x8, Inc. All Rights Reserved.
Deployment Environments - Hybrid
21 | © 8x8, Inc. All Rights Reserved.
Deployment Environment - On Premise
* Mainline K8s (1.17+)
* L3 Leaf-Spine
* 3 Zones/DC
* Zone Storage
(& CSI Topology)
* BGP (Calico + Bird)
- Citrix ADC
- Route Reflectors
- BGP to Host
22 | © 8x8, Inc. All Rights Reserved.
Deployment Environment - Anatomy of a Zone
Requirements:
* Horizontal Scale
* Zone isolation
topology.kubernetes.io/region=us-west
topology.kubernetes.io/zone=sjc01a
* Specialized network
Use-cases (DSR, ECMP, Anycast)
* Service IPs (VIPs) via BGP
* K8s-native APIs/resources
kind: Service
type: LoadBalancer
23 | © 8x8, Inc. All Rights Reserved.
Load Balancing Tier - Citrix ADC
Key Decision Factors:
Supports Kubernetes Services and Ingresses (and Ingress Classes)
Supports Complex, Zone-Aware Infrastructures
Full-Feature BGP Support (e.g. Route Reflectors, Auth, RHI, ECMP)
Horizontal/Linear Scale (using VPX in VMware)
Rich Feature Support via CRD/Annotation (Cert Mgmt, Rules, Policy, IPAM, etc.)
Simple, Hands-Off, Reliable Deployment and Configuration
Flexible Licensing Options & Editions (MPX, VPX, CPX, etc.)
24 | © 8x8, Inc. All Rights Reserved.
Recommendations
Reserve System Resources05
● System level, guarantee the OS a percentage
● Namespace LimitResource defaults
● Right size your workers
Cloud First04
● Started in the cloud for a uniform experience
● 100% of the resources were automatable
Self Service03
● Both engineering and operations could react
independently
Alignment02
● Strong collaboration between operations and
development
● Shared goals between teams
Choose your battles01
● Only stateless microservices initially
● New framework from the ground up
25 | © 8x8, Inc. All Rights Reserved.
Recommendations
Centralize Metrics and Logs10
● Prometheus has vast library of existing dashboards
● Collect and ship using Daemonsets
● Newrelic with Pagerduty
Common Logging Format09
● Simply it for the teams, send to standard out
● Build into your framework or endorse a specific format
Ingress vs Services08
● Use an ingress strategy
● Saves money
● Provides Telemetry
Security07
● Limit kubectl usage
● Helm safer
● Cluster Role for a power user
Embrace A Service Mesh06
● Telemetry
● Better Security
● Connect desperate clusters
26 | © 8x8, Inc. All Rights Reserved.

Business Continuity with Microservices-Based Apps and DevOps: Learnings from 8x8’s Experience

  • 1.
    © 2018 Citrix| Confidential Learnings from 8x8’s Experience JUNE 18, 2020 Business Continuity with Microservices-Based Apps and DevOps Pankaj Gupta, Citrix PankajOnCloud Lance Johnson, 8x8 Michael Laws, 8x8
  • 2.
    2 © 2018Citrix | Confidential Micro-services Based Application for Business Continuity Portability Modular App Architecture Break application in small services, which can be developed, deployed and auto-scaled independently. Faster Releases Very frequent application updates (even many times a day) Continuous Delivery to automate deployment Self contained runtime Application portability across Clouds Read “A Pandemic Plan for Application Architecture” at https://tinyurl.com/K8sApp
  • 3.
    3 © 2018Citrix | Confidential3 © 2019 Citrix | Confidential Diverse Stakeholders Have Unique Needs for Business Continuity Application & infrastructure security Container security & API gateways Automation DevSecOps SRE Application availability Observability Incident response Postmortems DevOps Faster release & deployment cycles CI/CD & automation Canary & progressive rollout User experience Troubleshooting Microservice discovery & routing Developers Network policy & compliance Manage, control & monitor network Resources & capacity planning (visibility) NetOps Platform Team Platform governance Operational efficiency Developer agility
  • 4.
    4 © 2018Citrix | Confidential4 © 2019 Citrix | Confidential Citrix Micro Services App Delivery Principles Integrated With Your Platform & Tools Get apps to production fast with vast K8s platform & opensource tools support Performance & Scale Support large clusters & very dynamic microservices Consistent App & API Security Break the silos of monoliths & microservices Architecture Flexibility Move at speed of your IT skill set Balance benefits & complexity Complete Observability Stack Gain visibility & troubleshoot problems faster, Break the silos Production-Grade Solution at the Speed of Business Read “7 Key Considerations for Microservices-Based Application Delivery” at https://tinyurl.com/7Consideration
  • 5.
    5 © 2018Citrix | Confidential For Your Every Step for Transition to Hybrid Multi Cloud Single Software Code Base across Portfolio for Features, APIs & Operational Consistency, and Innovation Velocity Hardware Appliances DC & Private Cloud, MPX Virtual Appliances DC & Private Cloud, VPX In Public Cloud VPX for AWS, Azure, GCP Containers Public & Private Cloud, CPX Bare Metal Private & Public Cloud, BLX Multi-Tenant MSP & Private Cloud, SDX Citrix ADC Portfolio: Built for Multi-Cloud Most Comprehensive, Feature Rich & Software-Centric
  • 6.
    6 © 2018Citrix | Confidential6 © 2019 Citrix | Confidential Complexity Benefits Unified Ingress 2-Tier Ingress Service Mesh Lite Service Mesh Low High High Balance the tradeoff between benefits and complexity Simplest and quickest way to production Simple for network-savvy platform teams Best observability & security but complex Service mesh-like benefits but simpler Microservices App Delivery Architecture Choices Read “The Best Way to Select a Proxy Architecture for Microservices Application Delivery” https://tinyurl.com/ProxyArch
  • 7.
    7 | ©8x8, Inc. All Rights Reserved. DevOps “Three Body Problem”
  • 8.
    8 | ©8x8, Inc. All Rights Reserved. Introductions Lance Johnson, Director of Engineering, Cloud R&D https://twitter.com/johnsonrl https://github.com/rljohnsn https://www.linkedin.com/in/lancejohnsn/ Michael Laws, Sr. Site Reliability Engineer @8x8 https://www.linkedin.com/in/mikelaws/
  • 9.
    9 | ©8x8, Inc. All Rights Reserved. Metrics
  • 10.
    10 | ©8x8, Inc. All Rights Reserved. Unified Communications
  • 11.
    11 | ©8x8, Inc. All Rights Reserved. At a Glance
  • 12.
    12 | ©8x8, Inc. All Rights Reserved. Unified Communications Meetings Chat Voice Call Center
  • 13.
    13 | ©8x8, Inc. All Rights Reserved. The Three Body Problem Unsolvable mathematical problem
  • 14.
    14 | ©8x8, Inc. All Rights Reserved. DevOps Three Body Problem Deployments • Ssh • Jenkins Pipelines • HyperLoop • Ansible • Chef • Terraform • Cloudformation Customers • DNS • LTM • GTM • Route53 • Geo • Proxy • CloudFlare • Ambassador/Envoy Dependencies • Data Centers • Clouds • Files • Databases • Random VPC • Public • Private How did 8x8 approach the problem?
  • 15.
    15 | ©8x8, Inc. All Rights Reserved. “What is your Quest?”
  • 16.
    16 | ©8x8, Inc. All Rights Reserved. DevOps Fundamentals • Safety • Provide a way for changes to be validated and tested. • Automation • Deliver the changes the same way for any environment • Knowledge Sharing • Learn from each other with open collaboration • Autonomy • Immediate visibility that changes deployed are safe • Continuously verified for every change
  • 17.
    17 | ©8x8, Inc. All Rights Reserved. Adoption of Microservices: 100s of Them! ● Which ○ Foundational Framework services (Cloud8) ○ Shared Services ○ Stateless Application Services ● Why ○ Divide and conquer ○ Technology refresh ○ Team Autonomy
  • 18.
    18 | ©8x8, Inc. All Rights Reserved. Design Goals ● Uniform deployment regardless of environments ● Service Discovery ● High availability and automatic recovery ● Container Packaging ● Native Kubernetes ● Public Charts ● Scaffolding Charts
  • 19.
    19 | ©8x8, Inc. All Rights Reserved. Deployment Environments - Cloud
  • 20.
    20 | ©8x8, Inc. All Rights Reserved. Deployment Environments - Hybrid
  • 21.
    21 | ©8x8, Inc. All Rights Reserved. Deployment Environment - On Premise * Mainline K8s (1.17+) * L3 Leaf-Spine * 3 Zones/DC * Zone Storage (& CSI Topology) * BGP (Calico + Bird) - Citrix ADC - Route Reflectors - BGP to Host
  • 22.
    22 | ©8x8, Inc. All Rights Reserved. Deployment Environment - Anatomy of a Zone Requirements: * Horizontal Scale * Zone isolation topology.kubernetes.io/region=us-west topology.kubernetes.io/zone=sjc01a * Specialized network Use-cases (DSR, ECMP, Anycast) * Service IPs (VIPs) via BGP * K8s-native APIs/resources kind: Service type: LoadBalancer
  • 23.
    23 | ©8x8, Inc. All Rights Reserved. Load Balancing Tier - Citrix ADC Key Decision Factors: Supports Kubernetes Services and Ingresses (and Ingress Classes) Supports Complex, Zone-Aware Infrastructures Full-Feature BGP Support (e.g. Route Reflectors, Auth, RHI, ECMP) Horizontal/Linear Scale (using VPX in VMware) Rich Feature Support via CRD/Annotation (Cert Mgmt, Rules, Policy, IPAM, etc.) Simple, Hands-Off, Reliable Deployment and Configuration Flexible Licensing Options & Editions (MPX, VPX, CPX, etc.)
  • 24.
    24 | ©8x8, Inc. All Rights Reserved. Recommendations Reserve System Resources05 ● System level, guarantee the OS a percentage ● Namespace LimitResource defaults ● Right size your workers Cloud First04 ● Started in the cloud for a uniform experience ● 100% of the resources were automatable Self Service03 ● Both engineering and operations could react independently Alignment02 ● Strong collaboration between operations and development ● Shared goals between teams Choose your battles01 ● Only stateless microservices initially ● New framework from the ground up
  • 25.
    25 | ©8x8, Inc. All Rights Reserved. Recommendations Centralize Metrics and Logs10 ● Prometheus has vast library of existing dashboards ● Collect and ship using Daemonsets ● Newrelic with Pagerduty Common Logging Format09 ● Simply it for the teams, send to standard out ● Build into your framework or endorse a specific format Ingress vs Services08 ● Use an ingress strategy ● Saves money ● Provides Telemetry Security07 ● Limit kubectl usage ● Helm safer ● Cluster Role for a power user Embrace A Service Mesh06 ● Telemetry ● Better Security ● Connect desperate clusters
  • 26.
    26 | ©8x8, Inc. All Rights Reserved.