Downloaded 54 times
![Starting
containers#cloud-config
coreos:
units:
- name: docker.service
command: start
- name: es.service
command: start
content: |
[Unit]
After=docker.service
Requires=docker.service
Description=starts ElasticSearch container
[Service]
TimeoutStartSec=0
ExecStartPre=/usr/bin/docker pull dockerfile/elasticsearch
ExecStart=/usr/bin/docker run -d -p 9200:9200 -p 9300:9300
dockerfile/elasticsearch](https://image.slidesharecdn.com/goasguen-cernvm-2015-150305112450-conversion-gate01/75/On-Docker-and-its-use-for-LHC-at-CERN-29-2048.jpg)
![?
[
{
"image": "wordpress",
"name": "wordpress",
"cpu": 10,
"memory": 200,
"essential": true,
"links": [
"mysql"
],
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"environment": [
{
"name": "WORDPRESS_DB_NAME",
"value": "wordpress"
},
…](https://image.slidesharecdn.com/goasguen-cernvm-2015-150305112450-conversion-gate01/75/On-Docker-and-its-use-for-LHC-at-CERN-49-2048.jpg)
![Starting
containers#cloud-config
coreos:
units:
- name: docker.service
command: start
- name: es.service
command: start
content: |
[Unit]
After=docker.service
Requires=docker.service
Description=starts ElasticSearch container
[Service]
TimeoutStartSec=0
ExecStartPre=/usr/bin/docker pull dockerfile/elasticsearch
ExecStart=/usr/bin/docker run -d -p 9200:9200 -p 9300:9300
dockerfile/elasticsearch](https://crownmelresort.com/image.slidesharecdn.com/goasguen-cernvm-2015-150305112450-conversion-gate01/75/On-Docker-and-its-use-for-LHC-at-CERN-29-2048.jpg)
![?
[
{
"image": "wordpress",
"name": "wordpress",
"cpu": 10,
"memory": 200,
"essential": true,
"links": [
"mysql"
],
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"environment": [
{
"name": "WORDPRESS_DB_NAME",
"value": "wordpress"
},
…](https://crownmelresort.com/image.slidesharecdn.com/goasguen-cernvm-2015-150305112450-conversion-gate01/75/On-Docker-and-its-use-for-LHC-at-CERN-49-2048.jpg)
Uploaded bySebastien Goasguen
On Docker and its use for LHC at CERN
This document discusses Docker containers and CoreOS. It summarizes Sebastien Goasguen's background working with high performance computing, cloud computing, and various open source projects. It then discusses how Docker simplifies application deployment and portability using containers and image sharing. CoreOS is introduced as a Linux distribution optimized for Docker with tools like etcd and Fleet for managing distributed applications across containers. Kubernetes is presented as a system for orchestrating Docker containers across multiple hosts and providing services like replication and high availability.
More Related Content
![[OpenInfra Days Korea 2018] Day 2 - E4 - 딥다이브: immutable Kubernetes architecture](https://cdn.slidesharecdn.com/ss_thumbnails/e41530linuxkit-k8s-180704110158-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to On Docker and its use for LHC at CERN
On Docker and its use for LHC at CERN
- 1.
- 2. Background • Joined CitrixOSS team in July 2012 • Associate professor at Clemson University prior • High Performance Computing, Grid computing (OSG, TG) • At CERN summer 2009/2010, help build LXCLOUD based on opennebula • http://sebgoa.blogspot.com @sebgoa
- 3. What do Ido ? • Apache CloudStack and licloud committer + PMC member • Looking at techs and how they work together • Half dev, half community manager, + half event planner
- 4.
- 5.
- 6. VMWare 1998 Xen 2003 HW assistedVirt 2005 EC2 2006 Opennebula Eucalyptus 2008 CloudStack 2010 Openstack 2010 GCE 2012
- 7. Goals • Utility computing •Elasticity of the infrastructure • On-demand • Pay as you go • Multi-tenant • Programmable access
- 8. So what… Let’s assumethis is solved. What is not solved: - Application deployment - Application scalability - Application portability - Application composability
- 9.
- 10. Docker • Linux container(LXC +) • Application deployment • PaaS • Portability • Image sharing via DockerHub • Ease of packaging applications
- 11. Building docker images Fairuse from http://blog.octo.com/en/docker-registry-first-steps/
- 12.
- 13. Installation $ sudo curl-sSL https://get.docker.com/ubuntu/ | sudo sh $ sudo yum install docker
- 14. Use $ docker runbusybox echo foobar Foobar $ docker run –ti ubuntu:14.04 /bin/bash root@0156ad334ca4:/#
- 15. The App store $docker push runseb/application $ docker pull runseb/application $ docker run –d runseb/application
- 16.
- 17. Networking Bridge in thehost Port mapping to expose services on the host Chain DOCKER (1 references) target prot opt source destination ACCEPT tcp -- anywhere 172.17.0.4 tcp dpt:www
- 18.
- 19. Other gotchas • Noinit system in the container • Foreground processes • Root • Data volumes • Data persistence • How small does an image get for real applications ?
- 20.
- 21.
- 22.
- 23.
- 24. CoreOS • Linux distribution •Rolling upgrades • Minimal OS • Docker support • etcd and fleet tools to manage distributed applications based on containers. • Cloud-init support • Systemd units
- 25.
- 26.
- 27.
- 28.
- 29. Starting containers#cloud-config coreos: units: - name: docker.service command:start - name: es.service command: start content: | [Unit] After=docker.service Requires=docker.service Description=starts ElasticSearch container [Service] TimeoutStartSec=0 ExecStartPre=/usr/bin/docker pull dockerfile/elasticsearch ExecStart=/usr/bin/docker run -d -p 9200:9200 -p 9300:9300 dockerfile/elasticsearch
- 30. Opportunity CERN cloud tooffer templates for: • Coreos • Snappy • Atomic Create a coreOS OEM upstream with cern specific contextualization
- 31.
- 32. CoreOS clustering etcd HAkey value store • Raft election algorithm • Writes when majority in cluster has committed update • e.g 5 nodes, tolerates 2 nodes failure fleet distributed init system (schedules systemd units in a cluster) • Submits systemd units cluster wide • Affinity, anti-affinity, global “scheduling”
- 33.
- 34. “Where are yougoing to run coreOS ?” “Where are you going to run Docker ?“
- 35. - Bare metalcluster - Public Clouds - Private Clouds
- 36. “How are yougoing to manage containers running on multiple Docker Hosts ?”
- 37. Docker schedulers • DockerSwarm • Citadel • CoreOS Fleet • Lattice from CF incubator • Clocker (via blueprints) • … • Kubernetes
- 38. Opportunity Experiment with adedicated cluster for container based applications. Or use a public cloud one:
- 39.
- 40. Kubernetes • Docker application orchestration •Google GCE, rackspace, Azure providers • Deployable on CoreOS • Container replication • HA services
- 42. Cloud (e.g CloudStackbased = exoscale, openstack based = cern cloud) coreOS coreOS coreOS K* K* K* Docker container Docker container Docker container API calls to Kubernetes API
- 43.
- 44. { "id": "redis-master-2", "kind": "Pod", "apiVersion":"v1beta1", "desiredState": { "manifest": { "version": "v1beta1", "id": "redis-master-2", "containers": [{ "name": "master", "image": "dockerfile/redis", "ports": [{ "containerPort": 6379, "hostPort": 6379 … "labels": { "name": "redis-master" } } Kubernetes Pod
- 45. Standardizing on pod Lookat the differences between: - k8s pod - AWS ECS task - Ansible Docker playbook - Fig file
- 46. ? - hosts: wordpress tasks: -name: Run mysql container docker: name=mysql image=mysql detach=true env="MYSQL_ROOT_PASSWORD=wordpressdocker,MYSQL_DATABASE=wordpress, MYSQL_USER=wordpress,MYSQL_PASSWORD=wordpresspwd" - name: Run wordpress container docker: image=wordpress env="WORDPRESS_DB_NAME=wordpress,WORDPRESS_DB_USER=wordpress, WORDPRESS_DB_PASSWORD=wordpresspwd" ports="80:80" detach=true links="mysql:mysql"
- 47. ? wordpress: image: wordpress links: - mysql ports: -"80:80" environment: - WORDPRESS_DB_NAME=wordpress - WORDPRESS_DB_USER=wordpress - WORDPRESS_DB_PASSWORD=wordpresspwd mysql: image: mysql volumes: - /home/docker/mysql:/var/lib/mysql environment: - MYSQL_ROOT_PASSWORD=wordpressdocker - MYSQL_DATABASE=wordpress - MYSQL_USER=wordpress - MYSQL_PASSWORD=wordpresspwd
- 48. ? apiVersion: v1beta1 id: wordpress desiredState: manifest: version:v1beta1 id: wordpress containers: - name: wordpress image: wordpress ports: - containerPort: 80 volumeMounts: # name must match the volume name below - name: wordpress-persistent-storage # mount path within the container mountPath: /var/www/html env: - name: WORDPRESS_DB_PASSWORD # change this - must match mysql.yaml password value: yourpassword volumes: - name: wordpress-persistent-storage source: # emptyDir: {} persistentDisk: # This GCE PD must already exist. pdName: wordpress-disk fsType: ext4 labels: name: wpfrontend kind: Pod
- 49. ? [ { "image": "wordpress", "name": "wordpress", "cpu":10, "memory": 200, "essential": true, "links": [ "mysql" ], "portMappings": [ { "containerPort": 80, "hostPort": 80 } ], "environment": [ { "name": "WORDPRESS_DB_NAME", "value": "wordpress" }, …
- 50. Opportunity What type ofLHC applications could take advantage of such a model ? • Highly distributed (in the sense of many isolated functions, not X jobs) • Long running services • Scalable layers
- 51.
- 52. Clouds and BigData •Object store + compute IaaS to build EC2+S3 clone • BigData solutions as storage backends for image catalogue and large scale instance storage. • BigData solutions as workloads to clouds.
- 53. EC2, S3 clone •An open source IaaS with an EC2 wrapper e.g Opennebula, CloudStack • Deploy a S3 compatible object store – separately- e.g riakCS • Two independent distributed systems deployed Cloud = EC2 + S3
- 54. Big Data as IaaSbackend “Big Data” solutions can be used as image catalogue .
- 55. Even use BareMetal
- 56. A note onScheduling • Core problem of computer science • knapsack is NP complete • Central scheduling has been used for a long time in HPC • Optimizing the cluster utilization requires multi-level scheduling (e.g backfill, preemption etc..) • Google Omega paper 2013 • Mesos 2009/2011, ASF Dec 2011
- 57.
- 58. Food for thought IfMesos is the answer… Mesos Framework for managing VM ? Workload sharing in your data-center: • Big Data • VM • Services • Containers Cloud and BigData
- 59. Conclusions • Docker isa technology to watch to create distributed applications • Not a replacement for VMs • Packaging experiments applications could be challenging • Supporting the docker networking model in the CERN environment will be difficult. • Could Mesos be used to fill up the clusters and collocate batch and interactive services ?
- 60.
- 61.