Download to read offline
More Related Content
Similar to Event-driven architecture patterns in highly scalable image storage solution- AWS Community Day Oceanic 2025
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Event-driven architecture patterns in highly scalable image storage solution- AWS Community Day Oceanic 2025
- 1. Firdaws Aboulaye &Vadym Kazulkin ip.labs Event-driven architecture patterns in highly scalable image storage solution Vadym Kazulkin
- 2. Firdaws Aboulaye &Vadym Kazulkin ip.labs Vadym Kazulkin ip.labs GmbH Bonn, Germany Co-Organizer of the Java User Group Bonn v.kazulkin@gmail.com @VKazulkin https://dev.to/vkazulkin https://github.com/Vadym79/ https://de.slideshare.net/VadymKazulkin/ https://www.linkedin.com/in/vadymkazulkin https://www.iplabs.de/ Contact
- 3. Firdaws Aboulaye &Vadym Kazulkin ip.labs Firdaws Aboulaye ip.labs GmbH Bonn, Germany Software Engineer aboufred@gmail.com https://www.linkedin.com/in/faboulaye https://www.iplabs.de/ Contact https://github.com/faboulaye
- 4. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea About ip.labs
- 5. Firdaws Aboulaye &Vadym Kazulkin ip.labs Agenda Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Motivation behind the reimplementation of our image storage solution to one based on AWS Serverless services Current architecture and APIs of our image storage solution Event-driven patterns in our image storage solution Challenges and lessons learned
- 6. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Motivation behind the reimplementation of our image storage solution Product/Strategical challenges remained: • Our main application also contained completely self-written eCommerce system • Shop frontend, shop backend, user management, payment and ordering workflow, statistics/BI • Our application contained external APIs to enable User SSO and External Cart integrations • Integration and maintenance costs were high for our customers
- 7. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Motivation behind the reimplementation of our image storage solution Product/Strategical challenges remained: • Business expressed the need to support direct integrations with the popular eCommerce solutions • Magento/Adobe Cloud, Shopify • Image handling/workflow was a part of this monolithical front and backend • Allthough it provided generic functionallity it was tightly coupled to the self-written eCommerce solution
- 8. Firdaws Aboulaye &Vadym Kazulkin ip.labs Why we chose AWS Serverless for image the storage solution? • We already experimented with AWS Serverless services for the new development on a smaller scale since our early days in AWS in 2018 • We saw a lot of benefits to fully utilize the power of the AWS cloud • further increasing the speed of delivery • focus on our core capabilities by relying on the AWS managed services
- 9. Firdaws Aboulaye &Vadym Kazulkin ip.labs Why we chose AWS Serverless for image the storage solution? • It became also a cultural thing • Developers started to learn, build and share knowledge in AWS Serverless
- 10. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image Storage Solution image storage solution=consumer file storage = CFS
- 11. Firdaws Aboulaye &Vadym Kazulkin ip.labs Current architecture our image storage solution File storage modules: • File API • Project API • Ordering API • Config API • Support API • Scheduler Jobs • Reporting API • Migration API • …
- 12. Firdaws Aboulaye &Vadym Kazulkin ip.labs AWS Services involved in Event Driven Architectures • SQS • SNS • EventBridge • Kinesis Data Streams /Firehose • DynamoDB Streams • CloudWatch Event /Event Bridge Scheduler
- 13. Firdaws Aboulaye &Vadym Kazulkin ip.labs Current architecture our image storage solution File storage modules: • File API • Project API • Ordering API • Config API • Customer Support API • Scheduler Jobs • Reporting API • Migration API • …
- 14. Firdaws Aboulaye &Vadym Kazulkin ip.labs Different workflows and life cycles of images • (Mobile) upload • Mobile upload + long term storage • Save and load projects • Saved projects sharing (in different modes) • Saved (internal and external) cart items • Purchased items • Manual (by the end user) or automated (saved project expirations, user log outs) image deletions and clean ups
- 15. Firdaws Aboulaye &Vadym Kazulkin ip.labs Queues and DLQs everywhere Use SQS for: • Decoupling image thumbnail(s) creation with “S3 object created event” • “Post Upload Lambda” takes care of it in batches
- 16. Firdaws Aboulaye &Vadym Kazulkin ip.labs Queues and DLQs everywhere Use SQS for: • “Files Usage Handling” Queue for events that saved projects or files/images were deleted (actively by the user or expired) • “Process File Usage Handling Lambda “ checks in batches whether “deleted” files/images are still referenced in any kinds of projects. • If not, Lambda marks them as “not_in_use” in DynamoDB “File Table”. • Files/Images themselves will be cleaned up later asynchronously (see scheduled events section)
- 17. Firdaws Aboulaye &Vadym Kazulkin ip.labs User State Observer with SQS (FIFO), EventBridge and Lambda • “User State FIFO SQS Queue” to preserve the order of the user state events (user log in, user log out) • Use “External Events Observer EventBridge” for multiple targets (in our case 2 Lambda functions Project and File User State Observer) • In case of “user log out “ Lambdas clean up not assigned mobile upload files (file level) to any project and not ordered cart items (project level)
- 18. Firdaws Aboulaye &Vadym Kazulkin ip.labs Change Log Pattern with DynamoDB Streams and Lambda • Requirement to implement “change log” for certain operations in the Config and Support APIs : 1. Config API captures all configuration account changes (i.e. how long saved projects are stored) 2. Support API captures all actions performed by the internal support team (i.e. saved project access for repair, or temporarily owner change)
- 19. Firdaws Aboulaye &Vadym Kazulkin ip.labs Change log Pattern with DynamoDB Streams and Lambda • Use DynamoDB Stream in combination with filters (via structures called FilterCriteria) to control what events to send to what Lambda function • Control the batch size when processing your DynamoDB Stream • Store all “change log” data in the single DynamoDB table
- 20. Firdaws Aboulaye &Vadym Kazulkin ip.labs Scheduled Events with CloudWatch Event Rules and different event-driven patterns with EventBridge and SNS 1. Finding expired saved projects, cart items, mobile upload long term images, handling their deletion and handling expiration notification (via SNS Topic, see “handling expiration notification”) 2. Triggering storage calculation for accounting purpose on per account base via EventBridge with 2 Lambda function targets: for the file and project level
- 21. Firdaws Aboulaye &Vadym Kazulkin ip.labs Handling Expiration Notifications with SNS to multiple SQS queues fan-out pattern • Our central systems are handling email delivery for their customer • SNS-SQS fan-out pattern with SNS message filtering rule based on the customer group number • Each central system subscribes to its individual SQS queue containing expiration notification messages for its customer group
- 22. Firdaws Aboulaye &Vadym Kazulkin ip.labs Migration API Migrate existing projects from the legacy system to the new file storage. Type of migration: • Batch mode • On demand
- 23. Firdaws Aboulaye &Vadym Kazulkin ip.labs Different event-driven patterns put together in Migration API Solution: • Export projects metadata to S3 (storage- first pattern) • Number of projects to be migrated per run is configurable • Sent notification on S3 “object created” event to “Migration SQS queue” • Implemented Lambda message batching from SQS queue • Implemented partial batch responses • Implemented the DynamoDB BatchWriteItem operation A single call to BatchWriteItem can transmit up to 16MB of data over the network, consisting of up to 25 item put or delete operations
- 24. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Challenges and lessons learned Architect with AWS Serverless quotas and technical concepts in mind General architectural decisions SQS vs SNS vs Kinesis vs EventBridge Aurora (Serverless) vs DynamoDB vs Aurora DSQL Challenging Serverless observability
- 25. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Service Quotas
- 26. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Service Quotas
- 27. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea account and current region limits Service Quotas
- 28. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Service Quotas
- 29. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Service Quotas Request History
- 30. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Service Quotas
- 31. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Serverless Application
- 32. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html Concurrency is the number of in-flight requests your AWS Lambda function is handling at the same time Lambda Concurrency
- 33. Firdaws Aboulaye &Vadym Kazulkin ip.labs Quota Description Value Adju stab le Mitigation Concurrent executions/ Concurrency limit The maximum number of events that functions can process simultaneously in the current region 1.000 Rearchitect Burst Concurrency Limit After the initial burst, concurrency scales by 1000 executions every 10 seconds up to your account concurrency limit. Each function within an account now scales independently from each other • US West (Oregon), US East (N. Virginia), Europe (Ireland)=3.000 • Asia Pacific (Tokyo), Europe (Frankfurt), US East (Ohio)=1000 • All other Regions=500 Use provisioned concurrency Lambda Important Service Quotas New
- 34. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea https://aws.amazon.com/de/blogs/aws/aws-lambda-functions-now-scale-12-times-faster-when-handling-high-volume-requests/ Lambda Concurrency and throttling
- 35. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Serverless Application
- 36. Firdaws Aboulaye &Vadym Kazulkin ip.labs Quota Description Value Adjustable Throughput per Standard Queue Standard queues support a nearly unlimited number of transactions per second (TPS) per API action. Nearly unlimited SQS (Standard) Important Service Quotas
- 37. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea https://aws.amazon.com/about-aws/whats-new/2023/11/aws-lambda-polling-scale-rate-sqs-event-source/?nc1=h_ls https://aws.amazon.com/blogs/compute/introducing-faster-polling-scale-up-for-aws-lambda-functions-configured-with-amazon-sqs/ • When a Lambda function subscribes to an SQS queue, Lambda polls the queue as it waits for messages to arrive. It consumes messages in batches, starting with 5 functions at a time • If there are more messages in the queue, Lambda adds up to 300 functions/concurrent executions per minute, up to 1250 concurrent executions, to consume those messages from the SQS queue • This scaling behavior is managed by AWS and cannot be modified • To process more messages, you can optimize your Lambda configuration for higher throughput Lambda scaling with SQS standard queues
- 38. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea https://aws.amazon.com/de/blogs/compute/understanding-how-aws-lambda-scales-when-subscribed-to-amazon-sqs-queues/ https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting • Increase the allocated memory for your Lambda function • Optimize batching behavior: • by default, Lambda batches up to 10 messages in a queue to process them during a single Lambda execution. You can increase this number up to 10,000 messages, or up to 6 MB of messages in a single batch for standard SQS queues • If each payload size is 1024KB (the maximum message size for SQS), Lambda can only take 6 messages per batch, regardless of the batch size setting • Implement partial batch responses Lambda scaling with SQS standard queues
- 39. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea • Understand the concepts of distributed systems • Token Bucket Algorithm • How asynchronous invocation patterns work • polling from the queue and synchronously invoke Lambda function • Retries with (exponential) backoff pattern and jitter, load- shedding and how AWS services and AWS SDKs support them • Understand individual service specific terms • Concurrency, transactions per second (TPS) • Throttle/Concurrency limit, burst limit • Event Source Mapping (ESM) Understand Technical Concepts
- 40. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea • Know, understand and observe the service quotas • Architect with service quotas in mind • AWS adjusts them from time to time • In case I’d like to request the quota increase, provide a valid justification for the new desired value • Service quotas are valid per AWS account (per region) • Use different AWS accounts for development and testing • Use different AWS accounts for independent (micro-)services • Separate AWS accounts on the team level • Use AWS Organizations General best practices for Service Quotas
- 41. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Challenges and lessons learned Architect with AWS Serverless quotas and technical concepts in mind General architectural decisions SQS vs SNS vs Kinesis vs EventBridge Aurora (Serverless) vs DynamoDB vs Aurora DSQL Challenging Serverless observability
- 42. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea When to use SQS, SNS, Kinesis and EventBridge https://www.serverlessguru.com/tips/sqs-vs-sns-vs-kinesis-vs-eventbridge
- 43. Firdaws Aboulaye &Vadym Kazulkin ip.labs Database Choice DynamoDB Aurora Aurora Serverless vs vs vs Aurora DSQL
- 44. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea Challenges and lessons learned Architect with AWS Serverless quotas and technical concepts in mind General architectural decisions SQS vs SNS vs Kinesis vs EventBridge Aurora (Serverless) vs DynamoDB vs Aurora DSQL Challenging Serverless observability
- 45. Firdaws Aboulaye &Vadym Kazulkin ip.labs Current architecture our image storage solution You need • Observability (Logging, Monitoring, Tracing) • Alerting • Incident Management solution (PagerDuty) https://aws.amazon.com/blogs/compute/introducing-new-asynchronous-invocation-metrics-for-aws-lambda/
- 46. Firdaws Aboulaye &Vadym Kazulkin ip.labs Thank you Developing highly scalable image storage with AWS Serverless at ip.labs
- 47. Firdaws Aboulaye &Vadym Kazulkin ip.labs Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea