Client Deployment of IBM Cloud Private (IBM #Think2019 #5964)

IBM Confidential
Client Deployment of IBM Cloud Private
#5964A
—
Michael Elder
IBM Distinguished Engineer – IBM
Multicloud Platform
@mdelder
Yong Feng
IBM Senior Technical Staff Member – IBM
Cloud Private
@luckyfengyong
Think 2019 / 5964A / Feb 15, 2019 / © 2019 IBM Corporation

Please note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice and at IBM’s sole discretion.
Information regarding potential future productsis intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future productsis not a commitment,
promise, or legal obligation to deliver any material, code or functionality. Information about
potential future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for
our productsremains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks
in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an
individual user will achieve results similar to those stated here.

What’s
included in
IBM Cloud
Private? 3Think 2019 / 5964A / Feb 15, 2019 / © 2019 IBM Corporation

Reference Architecture
4
http://ibm.biz/icpreferencearch

Topology Architecture
5
http://ibm.biz/icptopologyarch

Available Resources
6
http://bit.ly/icp-planning
Operator guides were produced as a joint
effort between engineering, support, and
teams in the field
Designed to provide real world guidance
Always under improvement – give us your
feedback!

How should
you plan
your specific
architecture?Think 2019 / 5964A / Feb 15, 2019 / © 2019 IBM Corporation

8
Critical ArchitectureDecisions
HA Workload Feature
• How many failed
nodes can be
tolerated?
• Do you need
availability zones?
• Upgrade with zero
downtime?
• What characteristics
define your workload:
cpu-intensive, memory-
intensive or others?
• What phase delivery
lifecycle: dev, test, UAT
or production?
• What is your required
throughput from your
consumers?
• Monitoring?
• Logging?
• Metering?
• Vulnerability Scan?
Security
• Do you need stringent
isolation for multiple
cluster consumers?
• Is certificate
management required?
• Is full PCI compliance
required?
• SELinux and firewall?

How should
you design
a cluster?

Provision
infrastructure
Prepare external
service
Ready for installation
1
# of
cluster
How many
clusters?
2
Host
group
What kind of host
group, how many
hosts in the host
groups and
what’s the size of
hosts?
3
Network
Network
topology, ingress
of management
control plan and
user workload
4
Storage
Storage for
management
services and
user workload
Five Key Principles Define IBM’s Approach…
5
Infra
Infrastructure
utilities leveraged
from infrastructure
manager
Design cluster with six key factors
6
Config
Configuration of OS
of host,
configuration of
management
services,
configuration of
external services
Installation
configuration

# of Clusters
11
The overall approach within the enterprise
Cost:
Aligned with the organisational units
Network latency:
Aligned with the geography
Scalability:
Aligned with the size of the managed nodes
Environment Requirement:
Aligned with the number of the distinct environment such as test, UAT and
production (whether namespaces isolation achieves the desired goals?)

Host Groups
12
Determine optional host group
• etcd: Enable for large scale cluster
• management: Enable when loads of management services are high
• va: Enable when vulnerability advisor and mutation advisor are enabled
• proxy: Enable when throughput of accessing to services from outside
cluster is high
Determine resource isolation
• Dedicate proxy node for a namespace
• Dedicate worker node for a namespace

Number and Size of Host
13Think 2019 / 5964A / Feb 15, 2019 / © 2019 IBM Corporation
Machine Role Number vCPU (>= 2.4 GHz) Memory Disk Space Comment
Master 3 16 32GB 500GB 3 for HA
Management 2 16 32GB 500GB 2 for HA
Proxy 2 4 16GB 400GB 2 for HA
Vulnerability Advisor 1 8 32GB 500GB Optional (none-HA)
WorkerNodes 2-50 8 32GB 400GB
A typical production environment
http://ibm.biz/icpcapacityplan

Management Node considerations
Separate management node from master node
– CPU, Memory and Disk intensive services running in management nodes
Increate the number of management node for large cluster
– Adding more management nodes not only increase the high availability but
balance the load of management services

Proxy Node considerations
Proxy nodes scale better
vertically vs horizontally as
shown in the figure. Notice
that 1 Proxy Node of 8 vCPU
supports nearly the same
workload as 3 Proxy Nodes
of 4 vCPU
– Rather than adding more
nodes, it is better to increase
the size of the node

Network - Resources
External Load Balancer
– ELB for master node
– ELB for proxy node
VIP
– Recommend ELB for
Production environment
Container network
– Network policy
Host network
External Network Controller

Network - Firewall Rule
Protocol within cluster
– ipip (94) of IPV4
Port number
– Externally access to master and proxy nodes
– Internally access between master, proxy, management, va, etcd and worker
nodes
http://ibm.biz/icpportnumber

Network - DNS
DNS resolving of services
– <service>.<namespace>.sv
c.<cluster_domain>
Join upstream DNS chain
– Pick up upstream DNS
configuration from host
automatically
– Specify upstream DNS
configuration explicitly

Storage – Management Service
February 19, 2019 ICP Solutioning Guide 101 | IBM Confidential | IBM Cloud Solutioning Centers
Shared storage
– Image Registry: Large capacity which depends on the number of images
– License Audit Log: Small capacity
Local storage
– Docker: https://docs.docker.com/storage/storagedriver/select-storage-driver/
– etcd: High IOPS, SSD is preferred
– MongoDB: SATA is OK, but SSD is better.
– Elasticsearch: Large capacity

Storage – User Application
Storage options hosted on IBM Cloud Provider cluster
– GlusterFS
– Ceph block storage by using Rock
– Minio
Storage options hosted outside IBM Cloud Provider cluster
– vSphere storage provider
– Network file system
– IBM Spectrum Scale
Storage options allowed by Kubernetes
– https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes

Storage - Backup
Kubernetes cluster state
– etcd: http://ibm.biz/icpbackup
Persistent volumes
– Traditional base backup tools can be used for backing up nodes and file system.

Storage – Backup with VM Solution
http://ibm.biz/icpbackupwithvmware

Infrastructure Provider
Infrastructure Metadata
– Host topology such as available zone
– Labels
Network
– NSX-T by vSphere
– ALB/ELB by AWS
– F5
Storage
– Datestore by vSphere
– EBS by AWS

Infrastructure Provider (Cont’d)
AWS
– AWS Cloud Provider
https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#aws
vSphere
– vSphere Cloud Provider
https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/overview.html
F5
– F5 Network Solution
https://clouddocs.f5.com/containers/v2/kubernetes/

Configuration - OS
Security
– Enable Selinux to get better security protection on host resource
– Enable local firewall to get better security protection on network
Kernel Parameter
– Network related parameters
http://ibm.biz/icposkernelparam
– Virtual memory related parameter for elasticsearch
vm.max_map_count

Configuration – External Service
LDAP/AD
– Prepare LDAP/AD for user authentication
http://ibm.biz/icpldap
Key Management Service
– Prepare KMS for secret encryption
http://ibm.biz/icpkmssecret
Vault
– Prepare HashiCorp Vault for certificate manager
http://ibm.biz/icpvault

Configuration – Management Service
Docker
– Follow CIS security benchmark
– Storage driver
Kubernetes
– Scheduler policy
– Configuration for large cluster: http://ibm.biz/icplargecluster
etcd
https://coreos.com/etcd/docs/latest/tuning.html
ELK
http://ibm.biz/icpelktuning

Examples

IBM Cloud Private in AWS
Leverage available zone
– Master/mgmt/va across available
zone
– User application across available
zone
AWS ALB/NLB
– Load balancer for management
control plane
– Load balancer for user application
– Security group to control network
access
EBS as persistent storage
http://ibm.biz/icponaws

Large Scale Cluster (1000 nodes)
Size of host
– etcd/Master/Management/Proxy/VA: 36 CPU, 60 GM memory and 10 GB networ
OS kernel parameter
– Network and virtual memory: net.core.somaxconn, net.ipv4.neigh.default.gc_thresh, fs.file-max …
Calico
– Enable router reflector
etcd
– --heartbeat-interval=500, --election-timeout=2500, --snapshot-count=5000
Kubernetes
– memory cache, communication timeout, API throttle, parallelism of ops

Multiple Tenants with Isolation
Proxy in DMZ can only
access service from
tenant A
Proxy in intranet can only
access service from
tenant B
Services from tenant A
and Service from tenant B
are running in different
workers and cannot
access between each
other

Air-gapped Environment
Proxy configuration for Docker
## Docker environment setup
docker_env:
- HTTP_PROXY=http://1.2.3.4:3128
- HTTPS_PROXY=http://1.2.3.4:3128
- NO_PROXY=localhost,127.0.0.1,{{ cluster_CA_domain }}
Proxy configuration for helm-api
tiller_http_proxy: http://1.2.3.4:3128
tiller_https_proxy: http://1.2.3.4:3128
http://ibm.biz/icpairgapped

Notices and disclaimers
© 2019 International BusinessMachinesCorporation. No part of this
document maybe reproducedor transmittedin any form without
written permission from IBM.
U.S. Government Users Restricted Rights — use,duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations(including information relating to
products that have not yet been announcedby IBM) hasbeen reviewed
for accuracyasof the date of initial publication andcouldinclude
unintentional technical or typographical errors. IBM shall have no
responsibility to update this information. This document is distributed
“as is” without any warranty, either express or implied.In no event,
shall IBM be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity.IBM products and
servicesare warrantedper the termsand conditionsof the agreements
under which they are provided.
IBM productsare manufacturedfrom newpartsor new and used parts.
In some cases, a product may not be new and mayhave been previously
installed. Regardless, our warrantytermsapply.”
Any statements regarding IBM's future direction,intent or product
plans are subject to change orwithdrawal without notice.
Performance data containedherein wasgenerallyobtainedin a
controlled, isolated environments. Customer examplesare presented as
illustrationsof how those
customershave used IBM productsand the results they mayhave
achieved. Actual performance, cost, savingsor other results in other
operating environmentsmay vary.
Referencesin thisdocument to IBM products, programs, or servicesdoes
not implythat IBM intendsto make such products, programsor services
available in all countriesin which IBM operates or does business.
Workshops, sessions and associatedmaterialsmayhave been prepared
by independent session speakers, and do not necessarily reflect the
views of IBM. All materialsanddiscussions are provided for
informational purposesonly, andare neither intendedto, nor shall
constitute legal or other guidance or advice to any individual participant
or their specific situation.
It isthe customer’sresponsibility to insure itsown compliance with legal
requirementsand to obtain advice of competent legal counsel asto
the identification andinterpretation of anyrelevant laws and regulatory
requirementsthat mayaffect the customer’sbusiness andany actions
the customer mayneed to take to complywith such laws. IBM does not
provide legal advice or represent or warrant that its servicesor products
will ensure that the customer followsany law.

Notices and disclaimers
continued
Information concerning non-IBM products was obtainedfrom the
suppliers of those products, their published announcementsor other
publicly available sources. IBM has not tested those products about this
publication andcannot confirm the accuracyof performance,
compatibilityor anyother claimsrelatedto non-IBM
products. Questionson the capabilities of non-IBM products should be
addressed to the suppliers of those products. IBM does not warrant the
quality of any third-party products, or the abilityof any such third-party
products to interoperate with IBM’s products. IBM expressly disclaims
all warranties, expressed orimplied, including but not limited to, the
implied warranties of merchantability and fitness fora purpose.
The provision of the information containedherein isnot intendedto, and
does not, grant anyright or license under any IBM patents, copyrights,
trademarksor other intellectual propertyright.
IBM, the IBM logo, ibm.com and[names of other referencedIBM
products andservices used in the presentation]are trademarksof
International Business MachinesCorporation, registeredin many
jurisdictionsworldwide. Other product and service namesmight
be trademarksof IBM or other companies. A current list of IBM
trademarksisavailable on the Web at “Copyright and trademark
information” at: www.ibm.com/legal/copytrade.shtml.

IBM Confidential
Thank you

®
https://www.ibm.com/legal/us/en/c opytrade.s html

Client Deployment of IBM Cloud Private (IBM #Think2019 #5964)

More Related Content

What's hot

Similar to Client Deployment of IBM Cloud Private (IBM #Think2019 #5964)

More from Michael Elder

Recently uploaded

Client Deployment of IBM Cloud Private (IBM #Think2019 #5964)