Download to read offline
Uploaded byAWS Chicago
Craig Johnson When VPCs Attack: Real-Life Cloud Networking Fails (and Fixes)
AWS Community Day Midwest 2025 Craig Johnson When VPCs Attack: Real-Life Cloud Networking Fails (and Fixes) Ever been haunted by an overlapping CIDR block? Or lost hours chasing a security group issue, only to discover a rogue NACL? You’re not alone. In this session, we’ll walk through a series of real-world AWS networking mishaps—anonymized but true—and break down how they were uncovered and resolved using native AWS tools like Reachability Analyzer, Flow Logs, and Route Tables. From broken PrivateLink endpoints to peering chaos and sneaky subnet misalignments, you’ll learn to spot the signals, structure your debugging flow, and build your own “uh-oh detection kit.” This is a fast-paced, lightly humorous talk designed for both newcomers and experienced cloud builders. Come for the fails—leave with the tools to prevent them.
More Related Content
Similar to Craig Johnson When VPCs Attack: Real-Life Cloud Networking Fails (and Fixes)
Craig Johnson When VPCs Attack: Real-Life Cloud Networking Fails (and Fixes)
- 1. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. When VPCs Attack: Real-Life Cloud Networking Fails (and Fixes) And How I Survived as a Traditional Network Engineer Craig Johnson Principal Solutions Architect Forward Networks
- 2. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. It’s-a me! Principal Solutions Architect Forward Networks Craig Johnson • On prem network engineer for way too long🌐 • Operations, architecture, post sales, pre-sales, etc • 8 years at the mothership in post sales • Traditional CLI junkie who thought ‘the cloud’ meant someone else’s router • AWS Community Builder • 3x CCIE (Expired lol) • TX(NUG) Group leader • Cloud Native for the past 4 years @captainpacket
- 3. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Native troubleshooting for the Network Engineer 01 Are my regular tools good enough? 05 Within a VPC 02 Building the infra – AWS Network Manager 06 What about flow logs? 03 On Prem connectivity too? 07 How this can help you 04 Inter-region
- 4. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. My tools have changed There’s no router to login to anymore! Lots of places to check routes Understanding traffic flow is more critical than ever! Routing protocols don’t help me Ping and traceroute don’t work very well Flow logs only work if traffic is active
- 5. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Network engineer goodies!
- 6. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. In this house we use the CLI!
- 7. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Creating your Global Network map aws networkmanager create-global-network <<<<<< Initial creation aws networkmanager create-site <<<<<<< Create your on-prem sites –-global-network-id x --location Latitude=string,Longitude=string #Optional aws networkmanager create-device --global-network-id x --site-id x --location Latitude=string,Longitude=string #Optional aws networkmanager create-link –-global-network-id x <<<< Create link associated with on-prem site –-site-id x --bandwidth UploadSpeed=x, Downloadspeed=x • Multi-account also possible with IAMRoleForAWSNetworkManagerCrossAccountResourceAccess permission from mgmt account • Routers and Sites can have lat/long locations which makes for pretty diagrams, but not required • aws networkmanager update-* commands can update any tags, physical addresses, etc
- 8. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Associating everything together aws networkmanager register-transit-gateway --global-network-id x --transit-gateway-arn x ^^^^^ Register TGWs to Global Network aws networkmanager associate-link --global-network-id --device-id x --link-id x aws networkmanager associate-customer-gateway <<<<<<< Create your on-prem routers --global-network-id x --customer-gateway—arn --device-id x –-link-id x • Getting the TGW and CGW is easier with the GUI, but can be iterated on each region with aws ec2 describe-transit-gateways • Routers and Sites can have lat/long locations which makes for pretty diagrams
- 9. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Global Network created!
- 10. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Global Network created!
- 11. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Inter-region routing – Route Analyzer aws networkmanager start-route-analysis <<<<<< Start the Analysis --global-network-id x --source TransitGatewayAttachmentArn=string,IpAddress=string --destination TransitGatewayAttachmentArn=string,IpAddress=string --include-return-path <<<< Always include the return path!!! --use-middleboxes #Optional – include if there are 3rd party devices in the path aws networkmanager start-route-analysis <<<<<< Start the Analysis --global-network-id x --route-analysis-id x <<<< ID from start-route-analysis • This can be run as often as needed • ARN construction: arn:aws:ec2:<region>:<account>:transit-gateway-attachment/tgw-attach-xxxxx
- 12. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Route Analyzer results: { "RouteAnalysis": { "GlobalNetworkId": ”<id>", "OwnerAccountId": ”<account>", "RouteAnalysisId": "4b75d541-c65c-46ca-baa1-b12da36209a0", "StartTimestamp": "2025-06-03T16:50:01-05:00", "Status": "SUCCEEDED", "Source": { "TransitGatewayAttachmentArn": "arn:aws:ec2:us-east-1:<account>:transit-gateway-attachment/tgw-attach-x", "TransitGatewayArn": "arn:aws:ec2:us-east-1:535928845597:transit-gateway/tgw-x", "IpAddress": "192.168.102.5" }, "Destination": { "TransitGatewayAttachmentArn": "arn:aws:ec2:us-west-2:535928845597:transit-gateway-attachment/tgw-attach-x", "TransitGatewayArn": "arn:aws:ec2:us-west-2:535928845597:transit-gateway/tgw-x", "IpAddress": "192.168.101.5” "ForwardPath": { "CompletionStatus": { "ResultCode": "CONNECTED" "ReturnPath": { "CompletionStatus": { "ResultCode": "CONNECTED"
- 13. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Route Analyzer results (continued): "Path": [ { "Sequence": 0, "Resource": { "IsMiddlebox": false } }, { "Sequence": 1, "Resource": { "IsMiddlebox": false } }, { "Sequence": 2, "Resource": {
- 14. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Route Analyzer (visual)
- 15. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Troubleshooting deeper • What about connectivity within a region? • What about security checking • Network ACLs? • Security Groups • I could use VPC flow logs to see if the traffic is working? • That doesn’t help me if: • Traffic isn’t running right now • Or actually help me solve the problem • VPC Reachability analyzer to the rescue!
- 16. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Setting up a path in a region aws ec2 create-network-insights-path --source <resource> <<< Lots to choose from --destination <resource> --protocol <TCP or UDP> aws ec2 start-network-insights-analysis --network-insights-path-id <resource> <<< Now to analyze the path aws ec2 describe-network-insights-analysss --network-insights-analysis-ids <resource> <<< Now to analyze the path • Analysis is a point in time – can be run multiple times • Predefined paths that can be saved opens up possibilities….
- 17. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. VPC Reachability analyzer results: { "NetworkInsightsAnalyses": [ { "NetworkInsightsAnalysisId": "nia-0cd72c90598500aac", "NetworkInsightsAnalysisArn": "arn:aws:ec2:us-east-2:<account>:network-insights-analysis/nia-0cd72c90598500aac", "NetworkInsightsPathId": "nip-0d67aa595cb9d008b", "StartDate": "2025-06-03T11:22:42.347000+00:00", "Status": "succeeded", "NetworkPathFound": false, "ForwardPathComponents": [ "Explanations": [ { "Direction": "ingress", "ExplanationCode": "ENI_SG_RULES_MISMATCH", "NetworkInterface": { "Id": "eni-070d49116b014c8f2", "Arn": "arn:aws:ec2:us-east-2:<account>:network-interface/eni-070d49116b014c8f2" }, "SecurityGroups": [ { "Id": "sg-0e2eccecb25fe3954", "Arn": "arn:aws:ec2:us-east-2:<account>:security-group/sg-0e2eccecb25fe3954"
- 18. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. VPC Reachability Analyzer (visual)
- 19. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. One more time! { "NetworkInsightsAnalyses": [ { "NetworkInsightsAnalysisId": "nia-060dfac0572f26fdc", "NetworkInsightsAnalysisArn": "arn:aws:ec2:us-east-2:535928845597:network-insights-analysis/nia- 060dfac0572f26fdc", "NetworkInsightsPathId": "nip-0d67aa595cb9d008b", "StartDate": "2024-06-09T11:31:26.118000+00:00", "Status": "succeeded", "NetworkPathFound": true, "ForwardPathComponents": [
- 20. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Not just for troubleshooting! • Every analysis I run is a saved entry at a point in time • Could I make this proactive? • Ideas: • Use within a Lambda to check pre/post any network changes • Integrate into CI/CD pipeline for ticketing to guarantee network uptime • Define critical application flows that can be cross verified • Mean Time to Innocence! • The Network is NOT the problem
- 21. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Wrapping up • Path tools ARE available to help you when connectivity fails • Even to your on prem and other middleboxes! • Take the time to build your map in Network Manager • Don’t rely on flow logs • Think about building proactive checks and notifications • Enables self service to your users • Great opportunity to use AI/ML from your flow logs to automate this
- 22. © 2024, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Thank you!