A security toolkit for Amazon S3
Another day, another leaky Amazon S3 bucketDon’t be the... next... big... data... leak
— The Register, 12 Jul 2017
Battle-tested at Instacart
Installation
Run:
pip install s3tkpip install awscli
aws configureCommands
Scan
Scan your buckets for:
- ACL open to public
- policy open to public
- logging enabled
- versioning enabled
- default encryption enabled
s3tk scans3tk scan my-bucket my-bucket-2s3tk scan "my-bucket*"s3tk scan --log-bucket my-s3-logs --log-bucket other-region-logs --log-prefix "{bucket}/"s3tk scan --skip-logging --skip-versioning --skip-default-encryptions3tk scan --sns-topic arn:aws:sns:...List Policy
List bucket policies
s3tk list-policys3tk list-policy my-bucket my-bucket-2s3tk list-policy --namedSet Policy
Note: This replaces the previous policy
Only private uploads
s3tk set-policy my-bucket --no-object-aclDelete Policy
Delete policy
s3tk delete-policy my-bucketEnable Logging
Enable logging on all buckets
s3tk enable-logging --log-bucket my-s3-logss3tk enable-logging my-bucket my-bucket-2 --log-bucket my-s3-logs{bucket}/ by default)s3tk enable-logging --log-bucket my-s3-logs --log-prefix "logs/{bucket}/"--dry-run flag to testA few notes about logging:
- buckets with logging already enabled are not updated at all
- the log bucket must in the same region as the source bucket - run this command multiple times for different regions
- it can take over an hour for logs to show up
Enable Versioning
Enable versioning on all buckets
s3tk enable-versionings3tk enable-versioning my-bucket my-bucket-2--dry-run flag to testEnable Default Encryption
Enable default encryption on all buckets
s3tk enable-default-encryptions3tk enable-default-encryption my-bucket my-bucket-2encrypt command for thisUse the
--dry-run flag to testScan Object ACL
Scan ACL on all objects in a bucket
s3tk scan-object-acl my-buckets3tk scan-object-acl my-bucket --only "*.pdf"s3tk scan-object-acl my-bucket --except "*.jpg"Reset Object ACL
Reset ACL on all objects in a bucket
s3tk reset-object-acl my-bucketUse the
--dry-run flag to testSpecify certain objects the same way as scan-object-acl
Encrypt
Encrypt all objects in a bucket with server-side encryption
s3tk encrypt my-buckets3tk encrypt my-bucket --kms-key-id arn:aws:kms:...s3tk encrypt my-bucket --customer-key secret-key--dry-run flag to testSpecify certain objects the same way as scan-object-acl
Note: Objects will lose any custom ACL
Delete Unencrypted Versions
Delete all unencrypted versions of objects in a bucket
s3tk delete-unencrypted-versions my-bucketUse the
--dry-run flag to testSpecify certain objects the same way as scan-object-acl
Scan DNS
Scan Route 53 for buckets to make sure you own them
s3tk scan-dnsCredentials
Credentials can be specified in
~/.aws/credentials or with environment variables. See this guide for an explanation of environment variables.You can specify a profile to use with:
AWS_PROFILE=your-profile s3tkIAM Policies
Here are the permissions needed for each command. Only include statements you need.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Scan",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:GetBucketAcl",
"s3:GetBucketPolicy",
"s3:GetBucketLogging",
"s3:GetBucketVersioning",
"s3:GetEncryptionConfiguration"
],
"Resource": "*"
},
{
"Sid": "ScanDNS",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"route53:ListHostedZones",
"route53:ListResourceRecordSets"
],
"Resource": "*"
},
{
"Sid": "ListPolicy",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:GetBucketPolicy"
],
"Resource": "*"
},
{
"Sid": "SetPolicy",
"Effect": "Allow",
"Action": [
"s3:PutBucketPolicy"
],
"Resource": "*"
},
{
"Sid": "DeletePolicy",
"Effect": "Allow",
"Action": [
"s3:DeleteBucketPolicy"
],
"Resource": "*"
},
{
"Sid": "EnableLogging",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:PutBucketLogging"
],
"Resource": "*"
},
{
"Sid": "EnableVersioning",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
" s3:PutBucketVersioning"
],
"Resource": "*"
},
{
"Sid": "EnableDefaultEncryption",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:PutEncryptionConfiguration"
],
"Resource": "*"
},
{
"Sid": "ResetObjectAcl",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObjectAcl",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
},
{
"Sid": "Encrypt",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutO bject"
],
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
},
{
"Sid": "DeleteUnencryptedVersions",
"Effect": "Allow",
"Action": [
"s3:ListBucketVersions",
"s3:GetObjectVersion",
"s3:DeleteObjectVersion"
],
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
}
]
}Access Logs
Amazon Athena is great for querying S3 logs. Create a table (thanks to this post for the table structure) with:
CREATE EXTERNAL TABLE my_bucket (
bucket_owner string,
bucket string,
time string,
remote_ip string,
requester string,
request_id string,
operation string,
key string,
request_verb string,
request_url string,
request_proto string,
status_code string,
error_code string,
bytes_sent string,
object_size string,
total_time string,
turn_around_time string,
referrer string,
user_agent string,
version_id string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1',
'input.regex' = '([^ ]*) ([^ ]*) \\[(.*?)\\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) \\\"([^ ]*) ([^ ]*) (- |[^ ]*)\\\" (-|[0-9]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) (\"[^\"]*\\") ([^ ]*)$'
) LOCATION 's3://my-s3-logs/my-bucket/';SELECT
date_parse(time, '%d/%b/%Y:%H:%i:%S +0000') AS time,
request_url,
remote_ip,
user_agent
FROM
my_bucket
WHERE
requester = '-'
AND status_code LIKE '2%'
AND request_url LIKE '/some-keys%'
ORDER BY 1CloudTrail Logs
Amazon Athena is also great for querying CloudTrail logs. Create a table (thanks to this post for the table structure) with:
CREATE EXTERNAL TABLE cloudtrail_logs (
eventversion STRING,
userIdentity STRUCT<
type:STRING,
principalid:STRING,
arn:STRING,
accountid:STRING,
invokedby:STRING,
accesskeyid:STRING,
userName:String,
sessioncontext:STRUCT<
attributes:STRUCT<
mfaauthenticated:STRING,
creationdate:STRING>,
sessionIssuer:STRUCT<
type:STRING,
principalId:STRING,
arn:STRING,
accountId:STRING,
userName:STRING>>>,
eventTime STRING,
eventSource STRING,
eventName STRING,
awsRegion STRING,
sourceIpAddress STRING,
userAgent STRING,
errorCode STRING,
errorMessage STRING,
requestId STRING,
eventId STRING,
r esources ARRAY<STRUCT<
ARN:STRING,
accountId:STRING,
type:STRING>>,
eventType STRING,
apiVersion STRING,
readOnly BOOLEAN,
recipientAccountId STRING,
sharedEventID STRING,
vpcEndpointId STRING,
requestParameters STRING,
responseElements STRING,
additionalEventData STRING,
serviceEventDetails STRING
)
ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://my-cloudtrail-logs/'SELECT
eventTime,
eventName,
userIdentity.userName,
requestParameters
FROM
cloudtrail_logs
WHERE
eventName LIKE '%Bucket%'
ORDER BY 1Best Practices
Keep things simple and follow the principle of least privilege to reduce the chance of mistakes.
- Strictly limit who can perform bucket-related operations
- Avoid mixing objects with different permissions in the same bucket (use a bucket policy to enforce this)
- Don’t specify public read permissions on a bucket level (no
GetObjectin bucket policy) - Monitor configuration frequently for changes
Bucket Policies
Only private uploads
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObjectAcl",
"Resource": "arn:aws:s3:::my-bucket/*"
}
]
}Notes
The
set-policy, enable-logging, enable-versioning, and enable-default-encryption commands are provided for convenience. We recommend Terraform for managing your buckets.resource "aws_s3_bucket" "my_bucket" {
bucket = "my-bucket"
acl = "private"
logging {
target_bucket = "my-s3-logs"
target_prefix = "my-bucket/"
}
versioning {
enabled = true
}
}Upgrading
Run:
pip install s3tk --upgradepip install git+https://github.com/ankane/s3tk.git --upgradeDocker
Run:
docker run -it ankane/s3tk aws configuredocker commit $(docker ps -l -q) my-s3tkdocker run -it my-s3tk s3tk scanHistory
View the changelog