Download as PDF, PPTX
![ECS Task Definition
{
"containerDefinitions": [
{
"command": [
"bin/worker"
],
"workingDirectory": "/app",
"essential": true,
"image": $imageURI,
"logConfiguration": {
"logDriver": "awslogs"
},
"name": $containerName
}
],
"cpu": "256",
"memory": "512",
"executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
"family": "my-consumer-fargate",
"networkMode": "awsvpc",
"requiresCompatibilities": [
"FARGATE"
]
}](https://image.slidesharecdn.com/microserviceswithrabbitmq-180603111650/75/Ruby-Microservices-with-RabbitMQ-43-2048.jpg)
![ECS Task
export taskDefinition=my-consumer-fargate
aws ecs register-task-definition
--cli-input-json file://my-consumer-fargate-task-definition.json
aws ecs run-task --launch-type FARGATE
--cluster $clusterName
--task-definition $taskDefinition
--network-configuration
"awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"](https://image.slidesharecdn.com/microserviceswithrabbitmq-180603111650/75/Ruby-Microservices-with-RabbitMQ-44-2048.jpg)
![ECS Service
• runs and maintains the required number of tasks
associated with the elastic load balancer
aws ecs create-service --cluster $clusterName
--service-name my-consumer-service
--task-definition $taskDefinition
--desired-count 2
--launch-type FARGATE
--network-configuration
"awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"](https://image.slidesharecdn.com/microserviceswithrabbitmq-180603111650/75/Ruby-Microservices-with-RabbitMQ-45-2048.jpg)
![ECS Task Definition
{
"containerDefinitions": [
{
"command": [
"bin/worker"
],
"workingDirectory": "/app",
"essential": true,
"image": $imageURI,
"logConfiguration": {
"logDriver": "awslogs"
},
"name": $containerName
}
],
"cpu": "256",
"memory": "512",
"executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
"family": "my-consumer-fargate",
"networkMode": "awsvpc",
"requiresCompatibilities": [
"FARGATE"
]
}](https://crownmelresort.com/image.slidesharecdn.com/microserviceswithrabbitmq-180603111650/75/Ruby-Microservices-with-RabbitMQ-43-2048.jpg)
![ECS Task
export taskDefinition=my-consumer-fargate
aws ecs register-task-definition
--cli-input-json file://my-consumer-fargate-task-definition.json
aws ecs run-task --launch-type FARGATE
--cluster $clusterName
--task-definition $taskDefinition
--network-configuration
"awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"](https://crownmelresort.com/image.slidesharecdn.com/microserviceswithrabbitmq-180603111650/75/Ruby-Microservices-with-RabbitMQ-44-2048.jpg)
![ECS Service
• runs and maintains the required number of tasks
associated with the elastic load balancer
aws ecs create-service --cluster $clusterName
--service-name my-consumer-service
--task-definition $taskDefinition
--desired-count 2
--launch-type FARGATE
--network-configuration
"awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"](https://crownmelresort.com/image.slidesharecdn.com/microserviceswithrabbitmq-180603111650/75/Ruby-Microservices-with-RabbitMQ-45-2048.jpg)
More Related Content
Similar to Ruby Microservices with RabbitMQ
Ruby Microservices with RabbitMQ
- 1.
- 2. Content • Intro from monolithdecoupling to microservices integration • The Main Part microservices messaging with RabbitMQ/AMQP • Ruby Code Refactoring from ActiveRecord Callback to Consumer as a Microservice • A Simple Microservice Deployment in docker container to AWS ECS
- 3. Let's start witha bad example
- 4. A better example ...or can we just publish the event and let subscribed consumers do the work? a complete to-do list for this event
- 5. Decoupling Publisher / Producer Subscriber/ Consumer Subscriber / Consumer Subscriber / Consumer
- 6. Monolith vs Microservices vs Complexityin Interactions Code Complexity
- 7. Microservice • a smallprogram that handle one task • independently deployable • work together by communicating so that can accomplish the larger task • integrated system on the whole provides value https://martinfowler.com/ articles/microservices.html https://youtu.be/wgdBVIX9ifA
- 8. App Integration Options •File Transfer • Shared Database • Remote Procedure Invocation • Messaging While all four approaches solve essentially the same problem, each style has its distinct advantages and disadvantages. http://www.enterpriseintegrationpatterns.com/patterns/messaging/IntegrationStylesIntro.html
- 9. File Transfer Consumer A ConsumerB Consumer C DATA FILE Producer Export Import
- 10. Shared Database Consumer A ConsumerB Consumer C Producer DB
- 11. Remote Procedure Invocation Consumer A ConsumerB Consumer C Producer Call Result
- 12. Messaging Exchanges Bindings Queues ConsumerA Consumer B Consumer C Producer AMQP 0-9-1 Message Broker
- 13. • a binary,networking protocol with minimal overhead • originated in 2003 by John O'Hara at JPMorgan Chase • version 0.9.1 was specified in 2008 • in 2011 OASIS standards group change it to a significantly different beast
- 14. AMQP 0-9-1 • virtualhost is a logical partition within the server (a multi-tenant system similar to virtual hosts in Apache/Nginx) • each of virtual hosts can have many connections, channels, queues, exchanges and some other things • bindings binds an exchange to a queue or many queues • exchange can be: direct, fanout, with topic or headers • queue is a buffer that holds messages on behalf of a consumer (or consumers)
- 15. The Producer • externalapplication which creates messages and decides: • how the attributes should be configured for routing • from which exchange the messages should start from • what is the actual payload that is being sent
- 16. The Message • anatomic unit of processing • consists of: • content (body, bare message or payload) • attributes - metadata about the message like content type, encoding, routing key, whether the message will be persistent or not, and whether it has a priority level, etc. • is created (published) by the producer • may go through more than one exchange before landing in the right queue
- 17. The Exchange • Directexchange route messages based on an exact match with the specified routing key • Fanout exchange automatically route the message to all the queues known to them (ignores the routing key) • Topic exchange pattern match on the routing key (topic) to route the messages • Header exchange use the message header attributes for matching the queue • Default (anonymous) exchange is a direct exchange (created automatically) which routes messages with empty exchange name - it compares routing key with the queue name
- 18. Multiple Queues and Consumers ConsumerA Queue 1 Queue 2 Queue 3 Consumer B Consumer C
- 19. The Consumer • externalapplication which is subscribed to one or more queues • alerted whenever a message shows up in the subscribed queue • can poll the queue at regular intervals to see which messages were added in the queue since the last time it made the request • can send acknowledgments back to RabbitMQ that the message has been: • received (known as ack), or • rejected (nack for negative acknowledgments)
- 20. Bindings • Routing Key •Publisher: exchange.publish(payload, routing_key: 'foo') • Queue Binding: queue.bind(exchange, routing_key: 'foo') • Headers • Publisher: exchange.publish(payload, headers: { ... }) • Queue Binding: queue.bind(exchange, arguments: { ... })
- 21. • an messagebroker implemented with Erlang/OTP • implements all the AMQP 0.9.1 concepts: messages, queues, exchanges, bindings, virtual hosts ... • ... and with plugins for other messaging protocols such as STOMP, XMPP, and more • has fantastic client support in a variety of popular languages: Ruby, Python, Java, PHP, C#, JavaScript, Go, Elixir, Objective-C, Swift, ...
- 22. Gems • Bunny -AMPQ client for Ruby (MRI) • March Hare - AMPQ client for JRuby • Sneakers - a background consumer/worker • Hutch - asynchronous inter-service communication • RabbitMQ HTTP API client (various management features) • Ruby RabbitMQ Clients Blog: http:// blog.rubyrabbitmq.info/
- 23. Features • No messageloss • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues • Message Ordering • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation ... and many more
- 24. Back to Refactoring ...or can we just publish the event and let subscribed consumers do the work?
- 25.
- 26. Publish Message withTopic
- 27.
- 28.
- 29.
- 30.
- 31.
- 32. Messaging Patterns (65) CreativeCommons Attribution 4.0 license.http://www.enterpriseintegrationpatterns.com/patterns/messaging/index.html
- 33. Microservice Messaging Benefits • language-agnostic •amplifies loose-coupling • makes scaling and rearchitecting simpler • better reliability and availability - messages will be waiting until the consumer is ready to process them • multiple communication patterns: events, message/reply, notification, async request-response, pub/sub, etc.
- 34. Microservice Messaging Drawbacks • introducingnew complexity into the system (the message broker) • a failure in the message broker can cause severe effects on the system (highly-available message broker) • big messages can cause network congestion • "eventual data consistency" (instead of immediate consistency across different microservices)
- 35. Message Consumer Microservice Deployment withDocker to AWS ECS
- 36. Installing Docker CEon Workstation • macOS: https://store.docker.com/editions/community/docker-ce- desktop-mac • Linux Ubuntu: https://store.docker.com/editions/community/docker-ce- server-ubuntu • or any other Linux distro: https://store.docker.com/search? offering=community&operating_system=linux&type=edition
- 37. My Ruby ProjectStructure
- 38.
- 39.
- 40. Elastic Container Registry exportrepositoryName=my_consumer export repositoryURI=123456789012.dkr.ecr.us-east-1.amazonaws.com export containerName=my_consumer export ver=1.0.0 export imageURI=$repositoryURI/$repositoryName:$ver aws ecr create-repository --repository-name $repositoryName eval $(aws ecr get-login --no-include-email) docker tag $containerName:$ver $imageURI docker push $imageURI
- 41. Elastic Container Service (ECS) •logical way to group resources (Tasks and Services) • currently provides two launch types: • EC2 • Fargate (abstract away EC2 instances) collection of ECS resources: https://github.com/nathanpeck/awesome-ecs export clusterName=staging aws ecs create-cluster --cluster-name $clusterName
- 42. ECS Cluster Building Blocks TaskDefinition • specify the resources for a Docker container or group of containers, such as: docker image (registry), ports, CPU/RAM, logs, any volumes, environment variables, etc. Task • running containers according to the Task Definition (one-off or long-running) • TaskRole allow access to S3, DynamoDB, etc. Service • manage long-running tasks (defines desired count and replace failed containers) • integrates with Elastic Load Balancer
- 43. ECS Task Definition { "containerDefinitions":[ { "command": [ "bin/worker" ], "workingDirectory": "/app", "essential": true, "image": $imageURI, "logConfiguration": { "logDriver": "awslogs" }, "name": $containerName } ], "cpu": "256", "memory": "512", "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", "family": "my-consumer-fargate", "networkMode": "awsvpc", "requiresCompatibilities": [ "FARGATE" ] }
- 44. ECS Task export taskDefinition=my-consumer-fargate awsecs register-task-definition --cli-input-json file://my-consumer-fargate-task-definition.json aws ecs run-task --launch-type FARGATE --cluster $clusterName --task-definition $taskDefinition --network-configuration "awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"
- 45. ECS Service • runsand maintains the required number of tasks associated with the elastic load balancer aws ecs create-service --cluster $clusterName --service-name my-consumer-service --task-definition $taskDefinition --desired-count 2 --launch-type FARGATE --network-configuration "awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"
- 46. Deploying Updates 1. Builda new image and push it to the repository 2. Create a new revision of the Task Definition (revision numbers increment automatically) 3. Update Service to use new Task Definition revision or simply use: https://github.com/silinternational/ecs-deploy ecs-deploy -c $clusterName -n my-consumer-service -i $imageURI
- 47. Advanced Example ofAWS ECS Deployment with Terraform VPC with 2 subnets (1 public and 1 private) in each Availability Zone https://github.com/duduribeiro/terraform_ecs_fargate_example https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f
- 48. AWS CodeBuild AWS CodePipeline screenshotsfrom: https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f
- 49. AWS ECS vs Kubernetes 1.AWS ECS can not be run on-premise 2. AWS ECS lacks advanced features These two differences can either be seen as weakness or as strengths.
- 50.