AWS CloudTrail is an AWS service that helps you enable operational and risk auditing, governance, and compliance of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs. [ Source : AWS Documentation ]
Cloudtrail can store logs in S3 bucket, and name of every log file is unique.
Cloudtrail Log file name format. Log files are in json format and gz compressed and extension of file name is .json.gz. Read
CloudTrail log file examples - AWS Documentation
to check example logs of different AWS services.
AccountID_CloudTrail_RegionName_YYYYMMDDTHHmmZ_UniqueString.FileNameFormat
Cloudtrail Logs directory structure looks like:
AWSLogs
|--- AWS Account ID
|--- Cloudtrail Name
|--- AWS Each Regions
|--- Year
|--- Month
|--- Date
|--- Logs File in `.json.gz`
Analysing each .gz.` file is not best approach as it will take lot of time and creating timeline is also not easy. Therefore best approach is to unzip all files merge into single file and then use SIEM or any preferred Log Analysis tool.
0xdf already have a video on how to combine cloudtrail .json files into single log file.
But in some cases 0xdf commands not working like if files are not zipped then also extra steps are required, use below script.
#!/bin/bash
echo "[+] Provide cloudtrail Log directory complete path :"
read cloudtrail_logs_file_path
echo "[+] Checking If All Required Tools are installed ......"
if ! [ -x "$(command -v jq)" ]; then
echo "[-] jq is not installed"
exit 0
elif ! [ -x "$(command -v gunzip)" ]; then
echo "[-] gunzip is not installed"
exit 0
elif ! [ -x "$(command -v find)" ]; then
echo "[-] find is not installed"
exit 0
else
echo "[+] All Required Tools are installed"
fi
if [ -d $cloudtrail_logs_file_path ]; then
echo "[+] Checking if gz Cloudtrail Log Files are present in $cloudtrail_logs_file_path"
find $cloudtrail_logs_file_path -name *.json.gz > /tmp/cloudtrail_log_files_name_gz
file_size=$(wc -c /tmp/cloudtrail_log_files_name_gz | cut -d " " -f1 )
if [ $file_size -ge 1 ]; then
echo "[+] Unzipping gz Cloudtrail Log Files"
find $cloudtrail_logs_file_path -name *.json.gz -exec gunzip -d {} \;
else
echo "[-] No gz Cloudtrail Log Files Found"
fi
echo "[+] Collecting All Cloudtrail JSON Log File Names"
find $cloudtrail_logs_file_path -name '*.json' > /tmp/cloudtrail_log_files_name
file_size=$(wc -c /tmp/cloudtrail_log_files_name | cut -d " " -f1 )
if [ $file_size -ge 1 ]; then
echo "[+] Merging All Cloudtrail Log Files"
while IFS= read -r line; do cat $line | jq -c '.Records[]' >> /tmp/cloudtrail-merged-all-logs.json ; done < /tmp/cloudtrail_log_files_name
else
echo "[-] No JSON Cloudtrail Log Files Found"
exit 0
fi
if ! [ -d "$cloudtrail_logs_file_path/../AWS_CloudTrail_Merged_Logs" ]; then
mkdir $cloudtrail_logs_file_path/../AWS_CloudTrail_Merged_Logs
fi
AWS_CloudTrail_Merged_Logs_directory=$(realpath $cloudtrail_logs_file_path/../AWS_CloudTrail_Merged_Logs)
cp /tmp/cloudtrail-merged-all-logs.json $AWS_CloudTrail_Merged_Logs_directory/cloudtrail-merged-all-logs.json
echo "[+] CloudTrail Logs Merged Logs Stored At: $AWS_CloudTrail_Merged_Logs_directory/cloudtrail-merged-all-logs.json"
rm /tmp/cloudtrail_log_files_name /tmp/cloudtrail_log_files_name_gz /tmp/cloudtrail-merged-all-logs.json
else
echo "[-] Check If $cloudtrail_logs_file_path exists on system"
fi
Server access logging provides detailed records for the requests that are made to an Amazon S3 bucket. Server access logs are useful for many applications. For example, access log information can be useful in security and access audits. This information can also help you learn about your customer base and understand your Amazon S3 bill. [ Source : AWS Documentation ].
S3 Access Logs are not updated real-time and their log delivery to destination bucket can take mins or hours.
Server access log records are delivered on a best-effort basis. Most requests for a bucket that is properly configured for logging result in a delivered log record. Most log records are delivered within a few hours of the time that they are recorded, but they can be delivered more frequently. [ Source : AWS S3 Access Logs : Best-effort server log delivery ]
AWS S3 access logs file are not in json format these are long strings. Therefore if using any SIEM or Tool for log analysis field extraction needs to be done.
AWS S3 Access Logs Format, Check AWS S3 Access Logs LogsFormat Documentation for more details.
bucket_owner bucket_name request_time remote_ip requester request_id operation key request_uri http_status http_code bytes_sent object_size total_time turn_around_time referer user_agent version_id host_id sig_version cipher_suite auth_type host_header tls_version access_point_arn acl_required source_region
If using Splunk for log analysis regex is shared on Splunk Community Forum
| rex field=_raw "^\s*(?P<bucket_owner>\S+)(\s+(?P<bucket_name>\S+))(\s+\[(?P<request_time>[\w\/\s:+]+)\])(\s+(?P<remote_ip>\S+))(\s+(?P<requester>\S+))(\s+(?P<request_id>\S+))(\s+(?P<operation>\S+))(\s+(?P<key>\S+))(\s+(?:\"?)(?<request_uri>[-]|([^\"]+))(?:\"?))(\s+(?P<http_status>\S+))(\s+(?P<error_code>\S+))(\s+(?P<bytes_sent>\S+))(\s+(?P<object_size>\S+))(\s+(?P<total_time>\S+))(\s+(?P<turn_around_time>\S+))(\s+(?:\"?)(?<referrer>[-]|([^\"]+))(?:\"?))(\s+(?:\"?)(?<user_agent>[-]|([^\"]+))(?:\"?))(\s+(?P<version_id>\S+))(\s+(?P<host_id>\S+))?(\s+(?P<signature_version>\S+))?(\s+(?P<cipher_suite>\S+))?(\s+(?P<authentication_type>\S+))?(\s+(?P<host_header>\S+))?(\s+(?P<tls_version>\S+))?(\s+(?P<access_point_arn>\S+))?(\s+(?P<acl_required>\S+))?"