Peter Sankauskas | @pas256 | June 2025
he/him
Access Denied:
Understanding & Debugging
AWS IAM
Typical AWS usage
2
An error occurred (AccessDenied) when calling the
ListObjectsV2 operation: Access Denied
Replace this talk with Amazon Q
3
🤮
User: arn:aws:sts::12345678901:sux/to-be-you
is not authorized to perform
aws:SimpleAction
on resource
arn:aws:ec2:us-tirefire-1:12345678901:service/abcd1234
because
I haven't had my coffee yet.
Standard IAM error message format
4
• How is IAM is designed?
• How does it evaluate policies?
• What are the di
ff
erent types of policies?
• When is each one useful?
• What is this error trying to tell me?
• What techniques can I use to debug permission errors?
Agenda
5
Peter Sankauskas
• Fractional CTO @ PAS Ventures
• Help startups hire,
fi
nd PMF and raise funding
• AWS Community Hero since 2014
• Answers for AWS community survey
• Advanced AWS Meetup in SF
Who am I?
6
Inspiration
7
a.k.a great artist steel
1) Speci
fi
cation
When you define access policies, you specify which IAM principals are
allowed to perform which actions on speci
fi
c AWS resources and under
which conditions.
2) Enforcement
IAM enforces this access by evaluating the AWS request with the policies
you de
fi
ned and returns either allow or deny.
What are IAM Policies?
8
{
"Statement":[{
"Sid": "unique",
"Effect": "effect",
"Principal": "principal",
"Action": "action",
"Resource": "resource",
"Condition": {
"operator": { "key": "value" }
}
}]
}
IAM Policy Language - PARC model
9
Principal - The entity that is allowed or denied access
arn:aws:iam::123456789012:user/you
Action - Type of access that is allowed or denied
s3:PutObject
Resource - The AWS resource(s) the action will act on
arn:aws:s3:::my-bucket
Condition - But only under these conditions
aws:MultiFactorAuthPresent = true
IAM Policy Evaluation
10
Decision
starts at Deny
Evaluate all
applicable
policies
Is there
an explicit
Deny?
Final Decision =
Deny
Yes
Final Decision =
Allow
Yes
Is there an
Allow?
No Final Decision =
Deny
No
(default)
(explicit)
What's not matching?
11
Context of
your request
Your de
fi
ned
policies
Matching
Allow
or
Deny
Identity-Based Policies
Trust Policies
VPC Endpoint Policies
Resource Control Policies
Service Control Policies
Resource-Based Policies
Permission Boundaries
Session Policies
Policy types
12
• Identity-Based Policies
• Attach to a Principal
• e.g. User, Group, Role, Session
• Resource-Based Policies
• Attach to a Resource
• e.g. S3 Bucket, VPC Endpoint
•
Policy types
Resource-Based Policy
Identity-Based Policy
Effective Permissions
= UNION
13
*within an AWS Account
• Service Control Policies (SCP)
• Getting more common
• Set at the Organization level
• Limit permissions granted to Identities
• Resource Control Policies (RCPs)
• New - released Dec 2024
• Set at the Organization level
• Limit permissions granted on Resources
Organization policy types
14
SCP or RCP
Identity-Based Policy
Effective Permissions
= INTERSECTION
Policy type matrix
15
Policy type Act on Set at Used to Example
Identity-based Principals Account level
Grant
permissions
Grant granular permissions for
Users, Groups and Roles.
Resource-based Resources Account level
Grant
permissions
Grant cross-account access to
a Resource. Control access
from a Resource.
Service Control
(SCP)
Principals Organization level
Deny
permissions
Disable access to services by
Users, Groups or Roles.
Resource Control
(RCP)
Resources Organization level
Deny
permissions
Disable access to Resources.
SCP default policy
16
IAM Policy Evaluation
17
Decision
starts at Deny
Evaluate all
applicable
policies
Is there
an explicit
Deny?
Final Decision =
Deny
Yes
Final Decision =
Allow
Yes
Is there an
Allow?
No Final Decision =
Deny
No
(default)
(explicit)
Example SCP policy - Guardrail
18
{
"Statement":[{
"Effect": "Deny",
"Action": [
"config:DeleteConfigRule",
"config:DeleteConfigurationAggregator",
"config:DeleteEvaluationResults",
"config:PutConfigRule",
"config:PutConfigurationAggregator"
],
"Resource": "*",
"Condition": {
"ArnNotLike": { "aws:PrincipalARN": "arn:aws:iam::*:role/AWSControlTowerExecution" }
}
}]
}
RCP default policy
19
Organization Policies
20
• Permission Boundaries
• Powerful way to limit privilege escalation,
but not widely adopted outside of larger
enterprises
• Scoped-down policies
• Assume a role (via STS) with a minimum
set of permissions
•
Some additional policy types
21
Permission Boundary
Identity-Based Policy
Effective Permissions
= INTERSECTION
• Allows you to de
fi
ne which Principals you trust to assume a role.
• Not to be confused with what the role can do
IAM Role Trust Policy
22
Within an AWS account
• Identity UNION Resource
Across multiple AWS accounts
• Identity INTERSECT Resource
• Need explicit Allows on both sides
• Create a "trust hug"
Permission across accounts
23
S3 Bucket
Resource
User
Principal
Final Decision =
Allow
Yes
Do
both
accounts allow
the request?
Final Decision =
Deny
No
Account A Account B
Pop Quiz
How policies work together within an account
25
IAM User: pas256 S3 Bucket: my-bucket
{
"Statement":[{
"Effect": "Allow",
"Action": "s3:PutObject"
"Resource": "*"
}]
}
{
no policy defined
}
aws s3 cp my-file.txt s3://my-bucket/ Allow
Account A
OR
How policies work together within an account
26
IAM User: pas256 S3 Bucket: my-bucket
{
no policy defined
}
{
"Statement":[{
"Effect": "Allow",
"Principal": {"AWS": "pas256"}
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::my-bucket/*"
}]
}
aws s3 cp my-file.txt s3://my-bucket/ Allow
Account A
OR
How policies work together across accounts
27
IAM User: pas256 S3 Bucket: your-bucket
{
no policy defined
}
{
"Statement":[{
"Effect": "Allow",
"Principal": {"AWS": "pas256"}
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::your-bucket/*"
}]
}
aws s3 cp my-file.txt s3://your-bucket/
AND
Account A Account B
Deny
How policies work together across accounts
28
IAM User: pas256 S3 Bucket: your-bucket
{
"Statement":[{
"Effect": "Allow",
"Action": "s3:PutObject"
"Resource": "*"
}]
}
{
"Statement":[{
"Effect": "Allow",
"Principal": {"AWS": "pas256"}
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::your-bucket/*"
}]
}
aws s3 cp my-file.txt s3://your-bucket/
AND
Account A Account B
Allow
How policies work together across accounts
29
IAM User: pas256 S3 Bucket: your-bucket
{
no policy defined
}
{
no policy defined
}
aws s3 cp my-file.txt s3://your-bucket/
Account A Account B
SCP
{
"Statement":[{
"Sid": "FullAWSAccess",
"Effect": "Allow",
"Action": "*"
"Resource": "*"
}]
}
Deny
AND AND
How policies work together across accounts
30
IAM User: pas256 S3 Bucket: your-bucket
{
no policy defined
}
{
"Statement":[{
"Effect": "Allow",
"Principal": {"AWS": "pas256"}
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::your-bucket/*"
}]
}
aws s3 cp my-file.txt s3://your-bucket/
Account A Account B
SCP
{
"Statement":[{
"Sid": "FullAWSAccess",
"Effect": "Allow",
"Action": "*"
"Resource": "*"
}]
}
Deny
AND AND
How policies work together across accounts
31
IAM User: pas256 S3 Bucket: your-bucket
{
"Statement":[{
"Effect": "Allow",
"Action": "s3:PutObject"
"Resource": "*"
}]
}
{
"Statement":[{
"Effect": "Allow",
"Principal": {"AWS": "pas256"}
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::your-bucket/*"
}]
}
aws s3 cp my-file.txt s3://your-bucket/
Account A Account B
Allow
SCP
{
"Statement":[{
"Sid": "FullAWSAccess",
"Effect": "Allow",
"Action": "*"
"Resource": "*"
}]
}
AND AND
Troubleshooting
First, make sure the Principal you think
you're using is indeed being used!
• Activity appears within 4 hours
• Tracks 400 days of history
• Tracks attempts, not just success
• Tracks Users and Roles
Verify Last Accessed
33
Documentation
Inspect using CloudTrail
34
35
Inspect using CloudTrail
• View last 90 days in AWS Web Console
• Must be enabled
• Create one, and ONLY ONE, trail (otherwise $$$!!!)
• Not 100% coverage by default
• e.g. No s3:PutObject coverage
• Can enable logging data events to get s3:PutObject
• Careful. This can cost $$$ with high volume projects
Inspect using CloudTrail
36
aws sts decode-authorization-message
--encoded-message 'rGACT....GDcV' |
jq -r '.DecodedMessage' |
jq
Decode the error
37
{
"allowed": false,
"explicitDeny": false,
"matchedStatements": {
"items": []
},
"failures": {
"items": []
},
"context": {
"principal": {
"id": "AROA2UC3D7TR6HATBG7IO:pas256",
"arn": "arn:aws:sts::730335542499:assumed-role/SSO/pas256"
},
"action": "CreateSecurityGroup",
"resource": "arn:aws:ec2:us-west-2:730335542499:vpc/vpc-079df256ef81d3c40",
"conditions": {
"items": [
{
"key": "ec2:ResourceTag/ManagedBy",
"values": {
"items": [
{
"value": "Terraform"
}
]
}
},
{
"key": "aws:Region",
"values": {
"items": [
{
"value": "us-west-2"
}
]
}
},
{
"key": "aws:Resource",
"values": {
"items": [
{
"value": "vpc/vpc-079df256ef81d3c40"
}
]
}
},
{
"key": "ec2:VpcID",
"values": {
"items": [
{
"value": "vpc-079df256ef81d3c40"
}
]
}
},
Decode the error
38
IAM Policy Simulator
39
https://policysim.aws.amazon.com/
AWS IAM Policy Visualizer
40
https://iam-liart.vercel.app/
• Experiment
• Get hands-on
• The IAM docs are actually really good
• Watch Brigid's talk
• https://www.youtube.com/watch?
v=YQsK4MtsELU
This is the tip of the iceberg
41
Thank you
🎉
🙏
👏
🥳

Peter Sankauskas Access Denied: Understanding & Debugging AWS IAM

  • 1.
    Peter Sankauskas |@pas256 | June 2025 he/him Access Denied: Understanding & Debugging AWS IAM
  • 2.
    Typical AWS usage 2 Anerror occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
  • 3.
    Replace this talkwith Amazon Q 3 🤮
  • 4.
    User: arn:aws:sts::12345678901:sux/to-be-you is notauthorized to perform aws:SimpleAction on resource arn:aws:ec2:us-tirefire-1:12345678901:service/abcd1234 because I haven't had my coffee yet. Standard IAM error message format 4
  • 5.
    • How isIAM is designed? • How does it evaluate policies? • What are the di ff erent types of policies? • When is each one useful? • What is this error trying to tell me? • What techniques can I use to debug permission errors? Agenda 5
  • 6.
    Peter Sankauskas • FractionalCTO @ PAS Ventures • Help startups hire, fi nd PMF and raise funding • AWS Community Hero since 2014 • Answers for AWS community survey • Advanced AWS Meetup in SF Who am I? 6
  • 7.
  • 8.
    1) Speci fi cation When youdefine access policies, you specify which IAM principals are allowed to perform which actions on speci fi c AWS resources and under which conditions. 2) Enforcement IAM enforces this access by evaluating the AWS request with the policies you de fi ned and returns either allow or deny. What are IAM Policies? 8
  • 9.
    { "Statement":[{ "Sid": "unique", "Effect": "effect", "Principal":"principal", "Action": "action", "Resource": "resource", "Condition": { "operator": { "key": "value" } } }] } IAM Policy Language - PARC model 9 Principal - The entity that is allowed or denied access arn:aws:iam::123456789012:user/you Action - Type of access that is allowed or denied s3:PutObject Resource - The AWS resource(s) the action will act on arn:aws:s3:::my-bucket Condition - But only under these conditions aws:MultiFactorAuthPresent = true
  • 10.
    IAM Policy Evaluation 10 Decision startsat Deny Evaluate all applicable policies Is there an explicit Deny? Final Decision = Deny Yes Final Decision = Allow Yes Is there an Allow? No Final Decision = Deny No (default) (explicit)
  • 11.
    What's not matching? 11 Contextof your request Your de fi ned policies Matching Allow or Deny
  • 12.
    Identity-Based Policies Trust Policies VPCEndpoint Policies Resource Control Policies Service Control Policies Resource-Based Policies Permission Boundaries Session Policies Policy types 12
  • 13.
    • Identity-Based Policies •Attach to a Principal • e.g. User, Group, Role, Session • Resource-Based Policies • Attach to a Resource • e.g. S3 Bucket, VPC Endpoint • Policy types Resource-Based Policy Identity-Based Policy Effective Permissions = UNION 13 *within an AWS Account
  • 14.
    • Service ControlPolicies (SCP) • Getting more common • Set at the Organization level • Limit permissions granted to Identities • Resource Control Policies (RCPs) • New - released Dec 2024 • Set at the Organization level • Limit permissions granted on Resources Organization policy types 14 SCP or RCP Identity-Based Policy Effective Permissions = INTERSECTION
  • 15.
    Policy type matrix 15 Policytype Act on Set at Used to Example Identity-based Principals Account level Grant permissions Grant granular permissions for Users, Groups and Roles. Resource-based Resources Account level Grant permissions Grant cross-account access to a Resource. Control access from a Resource. Service Control (SCP) Principals Organization level Deny permissions Disable access to services by Users, Groups or Roles. Resource Control (RCP) Resources Organization level Deny permissions Disable access to Resources.
  • 16.
  • 17.
    IAM Policy Evaluation 17 Decision startsat Deny Evaluate all applicable policies Is there an explicit Deny? Final Decision = Deny Yes Final Decision = Allow Yes Is there an Allow? No Final Decision = Deny No (default) (explicit)
  • 18.
    Example SCP policy- Guardrail 18 { "Statement":[{ "Effect": "Deny", "Action": [ "config:DeleteConfigRule", "config:DeleteConfigurationAggregator", "config:DeleteEvaluationResults", "config:PutConfigRule", "config:PutConfigurationAggregator" ], "Resource": "*", "Condition": { "ArnNotLike": { "aws:PrincipalARN": "arn:aws:iam::*:role/AWSControlTowerExecution" } } }] }
  • 19.
  • 20.
  • 21.
    • Permission Boundaries •Powerful way to limit privilege escalation, but not widely adopted outside of larger enterprises • Scoped-down policies • Assume a role (via STS) with a minimum set of permissions • Some additional policy types 21 Permission Boundary Identity-Based Policy Effective Permissions = INTERSECTION
  • 22.
    • Allows youto de fi ne which Principals you trust to assume a role. • Not to be confused with what the role can do IAM Role Trust Policy 22
  • 23.
    Within an AWSaccount • Identity UNION Resource Across multiple AWS accounts • Identity INTERSECT Resource • Need explicit Allows on both sides • Create a "trust hug" Permission across accounts 23 S3 Bucket Resource User Principal Final Decision = Allow Yes Do both accounts allow the request? Final Decision = Deny No Account A Account B
  • 24.
  • 25.
    How policies worktogether within an account 25 IAM User: pas256 S3 Bucket: my-bucket { "Statement":[{ "Effect": "Allow", "Action": "s3:PutObject" "Resource": "*" }] } { no policy defined } aws s3 cp my-file.txt s3://my-bucket/ Allow Account A OR
  • 26.
    How policies worktogether within an account 26 IAM User: pas256 S3 Bucket: my-bucket { no policy defined } { "Statement":[{ "Effect": "Allow", "Principal": {"AWS": "pas256"} "Action": "s3:PutObject" "Resource": "arn:aws:s3:::my-bucket/*" }] } aws s3 cp my-file.txt s3://my-bucket/ Allow Account A OR
  • 27.
    How policies worktogether across accounts 27 IAM User: pas256 S3 Bucket: your-bucket { no policy defined } { "Statement":[{ "Effect": "Allow", "Principal": {"AWS": "pas256"} "Action": "s3:PutObject" "Resource": "arn:aws:s3:::your-bucket/*" }] } aws s3 cp my-file.txt s3://your-bucket/ AND Account A Account B Deny
  • 28.
    How policies worktogether across accounts 28 IAM User: pas256 S3 Bucket: your-bucket { "Statement":[{ "Effect": "Allow", "Action": "s3:PutObject" "Resource": "*" }] } { "Statement":[{ "Effect": "Allow", "Principal": {"AWS": "pas256"} "Action": "s3:PutObject" "Resource": "arn:aws:s3:::your-bucket/*" }] } aws s3 cp my-file.txt s3://your-bucket/ AND Account A Account B Allow
  • 29.
    How policies worktogether across accounts 29 IAM User: pas256 S3 Bucket: your-bucket { no policy defined } { no policy defined } aws s3 cp my-file.txt s3://your-bucket/ Account A Account B SCP { "Statement":[{ "Sid": "FullAWSAccess", "Effect": "Allow", "Action": "*" "Resource": "*" }] } Deny AND AND
  • 30.
    How policies worktogether across accounts 30 IAM User: pas256 S3 Bucket: your-bucket { no policy defined } { "Statement":[{ "Effect": "Allow", "Principal": {"AWS": "pas256"} "Action": "s3:PutObject" "Resource": "arn:aws:s3:::your-bucket/*" }] } aws s3 cp my-file.txt s3://your-bucket/ Account A Account B SCP { "Statement":[{ "Sid": "FullAWSAccess", "Effect": "Allow", "Action": "*" "Resource": "*" }] } Deny AND AND
  • 31.
    How policies worktogether across accounts 31 IAM User: pas256 S3 Bucket: your-bucket { "Statement":[{ "Effect": "Allow", "Action": "s3:PutObject" "Resource": "*" }] } { "Statement":[{ "Effect": "Allow", "Principal": {"AWS": "pas256"} "Action": "s3:PutObject" "Resource": "arn:aws:s3:::your-bucket/*" }] } aws s3 cp my-file.txt s3://your-bucket/ Account A Account B Allow SCP { "Statement":[{ "Sid": "FullAWSAccess", "Effect": "Allow", "Action": "*" "Resource": "*" }] } AND AND
  • 32.
  • 33.
    First, make surethe Principal you think you're using is indeed being used! • Activity appears within 4 hours • Tracks 400 days of history • Tracks attempts, not just success • Tracks Users and Roles Verify Last Accessed 33 Documentation
  • 34.
  • 35.
  • 36.
    • View last90 days in AWS Web Console • Must be enabled • Create one, and ONLY ONE, trail (otherwise $$$!!!) • Not 100% coverage by default • e.g. No s3:PutObject coverage • Can enable logging data events to get s3:PutObject • Careful. This can cost $$$ with high volume projects Inspect using CloudTrail 36
  • 37.
    aws sts decode-authorization-message --encoded-message'rGACT....GDcV' | jq -r '.DecodedMessage' | jq Decode the error 37
  • 38.
    { "allowed": false, "explicitDeny": false, "matchedStatements":{ "items": [] }, "failures": { "items": [] }, "context": { "principal": { "id": "AROA2UC3D7TR6HATBG7IO:pas256", "arn": "arn:aws:sts::730335542499:assumed-role/SSO/pas256" }, "action": "CreateSecurityGroup", "resource": "arn:aws:ec2:us-west-2:730335542499:vpc/vpc-079df256ef81d3c40", "conditions": { "items": [ { "key": "ec2:ResourceTag/ManagedBy", "values": { "items": [ { "value": "Terraform" } ] } }, { "key": "aws:Region", "values": { "items": [ { "value": "us-west-2" } ] } }, { "key": "aws:Resource", "values": { "items": [ { "value": "vpc/vpc-079df256ef81d3c40" } ] } }, { "key": "ec2:VpcID", "values": { "items": [ { "value": "vpc-079df256ef81d3c40" } ] } }, Decode the error 38
  • 39.
  • 40.
    AWS IAM PolicyVisualizer 40 https://iam-liart.vercel.app/
  • 41.
    • Experiment • Gethands-on • The IAM docs are actually really good • Watch Brigid's talk • https://www.youtube.com/watch? v=YQsK4MtsELU This is the tip of the iceberg 41
  • 43.