Taming AEM deployments

APACHE SLING & FRIENDS TECH MEETUP
BERLIN, 25-27 SEPTEMBER 2017
Taming AEM deployments
Jakub Wądołowski, Cognifide

The road to efficient AEM delivery
2https://flic.kr/p/z8g25R
▪ Starts on your computer
▪ Ends in production
▪ Includes everything in between

Local environment
4https://flic.kr/p/b93YGx
▪ Everyone runs the same
▪ It contains the entire technology stack
▪ Start over any time you need it

What’s in the box?
6https://flic.kr/p/cMBbTu
▪ AEM Author & AEM Publish
▪ Preinstalled packages
▪ Dispatcher
▪ Solr Master & Slave
▪ knot.x

Vagrant in practice (1/3)
7
# Install prerequisites:
# * Vagrant
# * VirtualBox
# * ChefDK
$ git clone git@example.org:xyz/vagrant.git
$ cd project-vagrant
$ rake init
$ rake up

8
▪ http://localhost:4502 (AEM Author)
▪ http://localhost:4503 (AEM Publish)
▪ https://example.vagrant (Dispatcher)
▪ http://localhost:8983 (Solr)
▪ etc

9
$ rake provision:publish
...
Chef Client finished, 8/196 resources updated in 02
minutes 02 seconds
$ rake provision:httpd
...
Chef Client finished, 4/41 resources updated in 19
seconds

Vagrant talk
10
https://goo.gl/hu216e

Continuous integration
12https://flic.kr/p/nCGank
▪ Code integration
▪ Tests
▪ Release

Use case (1/4)
14
▪ 17 Chef cookbook repos
▪ 2 CI related repos
▪ 3 config repos
▪ 4 app repos
▪ 2 infrastructure repos
https://flic.kr/p/bF3Nwm

Use case (2/4)
15
▪ Each repository has its own lifecycle
▪ Simple Git flow
https://flic.kr/p/peZfvA

Use case (3/4)
16
$ tree -L 1
.
├── aem
├── domain
├── knotx-autocomplete
├── knotx-download
├── knotx-filters
├── knotx-index
├── knotx-search
├── pom.xml
└── solr
8 directories, 1 file
https://flic.kr/p/fnPQuz

Use case (3/4)
17
$ tree -L 1
.
├── aem
├── domain
├── pom.xml
└── solr

Use case (3/4)
18
$ tree -L 1
.
├── aem
├── domain
├── pom.xml
└── solr

Use case (3/4)
19
$ tree -L 1
.
├── aem
├── domain
├── solr
└── pom.xml

Use case (3/4)
20
$ tree -L 1
.
├── aem
├── domain
├── solr
└── pom.xml

Use case (4/4)
21
▪ Compatibility matrix is no longer needed
▪ End-to-end testing
▪ One release contains multiple artifacts

Pipelines
22https://flic.kr/p/qG26gA
▪ Variety of choices
▪ Jenkins
▪ GoCD
▪ Bamboo
▪ etc
▪ Pipeline as code (Jenkinsfile)

23
node('master') {
stage('Checkout') {
dir('app') {
git url: 'git://example.com/aem-project.git', branch: 'master'
}
dir('tests') {
git url: 'git://example.com/tests.git', branch: 'master'
}
}
stage('Build') {
node('aem') {
dir('app/aem') {
sh 'mvn clean install'
}
}
}
stage('Test') {
dir('tests') {
sh './gradlew clean contractTests'
}
}
}

24
node('master') {
stage('Checkout') {
dir('app') {
}
dir('tests') {
}
}
stage('Build') {
node('aem') {
dir('app/aem') {
}
}
}
stage('Test') {
dir('tests') {
}
}
}

25
node('master') {
stage('Checkout') {
dir('app') {
}
dir('tests') {
}
}
stage('Build') {
node('aem') {
dir('app/aem') {
}
}
}
stage('Test') {
dir('tests') {
}
}
}

26
node('master') {
stage('Checkout') {
dir('app') {
}
dir('tests') {
}
}
stage('Build') {
node('aem') {
dir('app/aem') {
}
}
}
stage('Test') {
dir('tests') {
}
}
}

Pipeline improvements
40
▪ Dynamically defined stages
▪ Local & global configuration files
▪ Custom shared library
https://flic.kr/p/juzu1W

41
#!groovy
@Library('cognifide-library@master')
import com.cognifide.jenkinslib.Pipeline
def pipeline = new Pipeline(this)
node('master') {
withMaven(maven: 'Maven 3') {
stage('Quality Gate: Sonarqube') {
sonarqube pipeline.CFG.sonarqube
}
stage('Build') {
// ...
}
stage('Clear dispatcher cache') {
knifeExec query: "chef_environment:${pipeline.CFG.chef.environmentName}
AND tags:dispatcher"
}
}
}

42
#!groovy
node('master') {
}
stage('Build') {
// ...
}
}
}
}

43
#!groovy
node('master') {
}
stage('Build') {
// ...
}
}
}
}

44
#!groovy
node('master') {
}
stage('Build') {
// ...
}
}
}
}

45
#!groovy
node('master') {
}
stage('Build') {
// ...
}
}
}
}

46
#!groovy
node('master') {
}
stage('Build') {
// ...
}
}
}
}

47
notifications:
enabled: true
recipients:
- 'dev-team@example.com'
deploy:
profiles:
aem-author: 'dev-author'
aem-publish1: 'dev-publish1'
aem-publish2: 'dev-publish2'
tests:
bobcat:
profiles:
- 'dev'
- 'grid'
mvnParams: '-Dfork.count=4'
chef:
environmentName: 'dev'

Testing
48
▪ Unit tests
▪ Bobcat
▪ github.com/Cognifide/bobcat
▪ Zalenium & Docker
▪ github.com/zalando/zalenium
▪ AET
▪ github.com/Cognifide/aet
https://flic.kr/p/juzgT9

Release
49https://flic.kr/p/EQbNYd
▪ Release != deployment
▪ Unnoticeable process
▪ Artifact store is a must

Infrastructure setup
52
▪ Multiple AWS environments
▪ Terraform modules
▪ Bootstrap process
https://flic.kr/p/qCk82v

Service discovery
53
▪ Interconnectivity
▪ Hardcoded IPs are not an option
▪ Adapts to changes by itself
https://flic.kr/p/qr4XXW

Consul features
55
▪ DNS-based service discovery
▪ Load balancing
▪ Health checks

Consul talk
56
https://goo.gl/aZjKsR

Chef
59
▪ Configuration management
▪ Each service has its own cookbook
▪ Deployment steps written in DSL
https://flic.kr/p/W25Lmv

Chef example
60
▪ Install a package
▪ Restart AEM
▪ Remove default flush agent
https://flic.kr/p/9qsZDC

61
cq_package 'Author: Core app' do
username 'admin'
password 'admin'
instance 'http://localhost:4502'
source 'https://artifacts.example.com/xyz/1.0/xyz-1.0.3-full.zip'
http_user 'basic_auth_user'
http_pass 'basic_auth_password'
action :deploy
end

62
username 'admin'
password 'admin'
action :deploy
end

63
username 'admin'
password 'admin'
action :deploy
end

64
username 'admin'
password 'admin'
action :deploy
end

65
username node['cq']['author']['credentials']['login']
password node['cq']['author']['credentials']['password']
instance "http://localhost:#{node['cq']['author']['port']}"
source "https://artifacts.example.com/xyz/#{ver_dir}/"
"xyz-#{ver}-full.zip"
http_user node['xyz-webapp']['nexus']['user']
http_pass node['xyz-webapp']['nexus']['password']
action :deploy
end

66
username 'admin'
password 'admin'
action :deploy
notifies :restart, 'service[cq62-author]', :immediately
end
service 'cq62-author' do
action :nothing
end

67
username 'admin'
password 'admin'
action :deploy
end
action :nothing
end

68
username 'admin'
password 'admin'
action :deploy
end
action :nothing
end

69
username 'admin'
password 'admin'
action :deploy
end
action :nothing
end

70
cq_jcr 'Author: /etc/replication/agents.author/flush' do
path '/etc/replication/agents.author/flush'
username 'admin'
password 'admin'
action :delete
end

71
username 'admin'
password 'admin'
action :delete
end

72
username 'admin'
password 'admin'
action :delete
end

73
username 'admin'
password 'admin'
action :delete
end

74
$ chef-client
# ...
Chef Client finished, 39/652 resources updated in 09 minutes 46 seconds
$ chef-client
# ...

75
$ chef-client
# ...
$ chef-client
# ...

76
$ grep cq_package recipes/author*.rb | wc -l
70
$ grep cq_jcr recipes/author*.rb | wc -l
20
$ grep cq_package recipes/publish*.rb | wc -l
67
$ grep cq_jcr recipes/publish*.rb | wc -l
11

What’s currently available?
77
▪ cq_package
▪ cq_osgi_config
▪ cq_osgi_bundle
▪ cq_osgi_component
▪ cq_user
▪ cq_jcr
▪ groovy_script
▪ apm_script
▪ …
https://flic.kr/p/qhRjAF

AEM cookbook
78
github.com/jwadolowski/cookbook-cq

Orchestration
80
▪ Deployment pipelines
▪ Driven by “knife ssh” command
https://flic.kr/p/eYhy4y

82
$ knife ssh
'chef_environment:xyz-prod AND tags:aem AND tags:author'
'sudo -u apache /bin/touch /path/to/maintenance/maintenance.lock'

83
$ knife ssh

84
$ knife ssh

86
$ knife ssh
'chef_environment:xyz-prod AND tags:dispatcher AND tags:us-east-1a'
'sudo -u apache /bin/touch /path/to/maintenance/deployment.lock'

87
$ knife ssh

88
$ knife ssh

90
$ knife ssh
'chef_environment:xyz-prod AND (
(tags:dispatcher AND tags:us-east-1a) OR
(tags:aem AND tags:publish AND tags:us-east-1a)
)'
'sudo /usr/bin/chef-client'

91
$ knife ssh
)'

92
$ knife ssh
)'

93
$ knife ssh
'sudo -u apache /usr/bin/rm -f /path/to/maintenance/deployment.lock'

Lessons learned
98
▪ Fail fast
▪ Enforce correct exit codes
▪ How to resume failed pipeline?
https://flic.kr/p/jzAyNF

Taming AEM deployments

More Related Content

What's hot

Similar to Taming AEM deployments

More from Jakub Wadolowski

Recently uploaded

Taming AEM deployments