🔑 Key & Secrets Management

Level: Core Solves: Secure storage và rotation của secrets, encryption keys, và sensitive configuration

🎯 Mục tiêu (Outcomes)

Sau khi áp dụng kiến thức trong trang này, bạn sẽ có khả năng:

Chọn đúng Secret Storage Service dựa trên use case (KMS, Secrets Manager, Parameter Store)
Thiết kế KMS Key Hierarchy với CMKs và data keys
Triển khai Automatic Rotation cho database credentials và API keys
Cấu hình Cross-Account Sharing an toàn
Implement Secret Versioning cho zero-downtime rotation
Audit Secret Access với CloudTrail và alerting

✅ Khi nào dùng

Service	Use Case	Lý do
KMS CMK	Encryption keys	Key management, rotation, audit
Secrets Manager	Database credentials	Auto-rotation, RDS integration
Secrets Manager	API keys, tokens	Versioning, cross-account
Parameter Store Standard	Non-sensitive config	Free, simple
Parameter Store SecureString	Sensitive config không cần rotation	Cheaper than Secrets Manager

❌ Khi nào KHÔNG dùng

Pattern	Vấn đề	Thay thế
Hardcode secrets trong code	Security risk, khó rotate	Secrets Manager
Environment variables cho production	Visible trong console, logs	Secrets Manager
Parameter Store cho DB passwords	Không auto-rotation	Secrets Manager
Secrets Manager cho static config	Unnecessary cost	Parameter Store
AWS-managed keys cho compliance	Không control key policy	Customer-managed CMK

⚠️ Cảnh báo từ Raizo

"Một developer commit AWS access key vào GitHub. Trong vòng 10 phút, crypto miners đã spin up $50,000 EC2 instances. Secret scanning và rotation policy là critical. Assume secrets SẼ bị leak và plan accordingly."

Service Selection

Decision Matrix

┌─────────────────────────────────────────────────────────────────┐
│              SECRETS MANAGEMENT DECISION                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  What are you storing?                                          │
│                                                                 │
│  ├── Encryption keys → KMS                                      │
│  │   • Customer master keys (CMKs)                              │
│  │   • Data encryption keys                                     │
│  │   • Asymmetric keys for signing                              │
│  │                                                              │
│  ├── Database credentials → Secrets Manager                     │
│  │   • Automatic rotation                                       │
│  │   • RDS integration                                          │
│  │   • Cross-account sharing                                    │
│  │                                                              │
│  ├── API keys, tokens → Secrets Manager                         │
│  │   • Third-party service credentials                          │
│  │   • OAuth tokens                                             │
│  │   • Webhook secrets                                          │
│  │                                                              │
│  └── Configuration values → Parameter Store                     │
│      • Feature flags                                            │
│      • Environment-specific config                              │
│      • Non-sensitive settings                                   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Service Comparison

Feature	KMS	Secrets Manager	Parameter Store
Purpose	Key management	Secrets storage	Config storage
Rotation	Manual	Automatic	Manual
Pricing	$1/key/month	$0.40/secret/month	Free (Standard)
Size Limit	4KB	64KB	8KB (Advanced)
Versioning	No	Yes	Yes
Cross-Account	Yes	Yes	Limited

AWS KMS

Key Hierarchy

┌─────────────────────────────────────────────────────────────────┐
│                    KMS KEY HIERARCHY                            │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              Customer Master Key (CMK)                   │    │
│  │              • Never leaves KMS                          │    │
│  │              • Used to encrypt Data Keys                 │    │
│  └────────────────────────┬────────────────────────────────┘    │
│                           │                                     │
│                           ▼                                     │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              Data Encryption Key (DEK)                   │    │
│  │              • Generated by KMS                          │    │
│  │              • Used to encrypt actual data               │    │
│  │              • Stored encrypted alongside data           │    │
│  └────────────────────────┬────────────────────────────────┘    │
│                           │                                     │
│                           ▼                                     │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    Your Data                             │    │
│  │              • Encrypted with DEK                        │    │
│  │              • Stored in S3, EBS, etc.                   │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                 │
│  ENVELOPE ENCRYPTION:                                           │
│  Data → Encrypted with DEK → DEK encrypted with CMK             │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Key Types

Type	Management	Use Case
AWS Managed	AWS	Default encryption (S3, EBS)
Customer Managed	You	Custom policies, rotation
Customer Owned	You (CloudHSM)	Regulatory requirements

Key Policy Example

json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Enable IAM policies",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Allow key administrators",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/KeyAdminRole"
      },
      "Action": [
        "kms:Create*",
        "kms:Describe*",
        "kms:Enable*",
        "kms:List*",
        "kms:Put*",
        "kms:Update*",
        "kms:Revoke*",
        "kms:Disable*",
        "kms:Get*",
        "kms:Delete*",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow key usage",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/ApplicationRole"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey*"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "s3.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

AWS Secrets Manager

Secret Lifecycle

Secret Structure

json

{
  "username": "app_user",
  "password": "super-secret-password",
  "engine": "mysql",
  "host": "mydb.cluster-xxx.us-east-1.rds.amazonaws.com",
  "port": 3306,
  "dbname": "myapp"
}

Rotation Configuration

python

import boto3
import json

def create_secret_with_rotation():
    client = boto3.client('secretsmanager')
    
    # Create secret
    response = client.create_secret(
        Name='prod/myapp/database',
        Description='Production database credentials',
        SecretString=json.dumps({
            'username': 'app_user',
            'password': 'initial-password',
            'engine': 'mysql',
            'host': 'mydb.xxx.us-east-1.rds.amazonaws.com',
            'port': 3306,
            'dbname': 'myapp'
        }),
        Tags=[
            {'Key': 'Environment', 'Value': 'production'},
            {'Key': 'Application', 'Value': 'myapp'}
        ]
    )
    
    # Enable rotation
    client.rotate_secret(
        SecretId='prod/myapp/database',
        RotationLambdaARN='arn:aws:lambda:us-east-1:123456789012:function:SecretsRotation',
        RotationRules={
            'AutomaticallyAfterDays': 30,
            'ScheduleExpression': 'rate(30 days)'
        }
    )

Application Integration

python

import boto3
import json
from botocore.exceptions import ClientError

def get_secret(secret_name: str, region: str = 'us-east-1') -> dict:
    """Retrieve secret from Secrets Manager with caching"""
    
    client = boto3.client('secretsmanager', region_name=region)
    
    try:
        response = client.get_secret_value(SecretId=secret_name)
        
        if 'SecretString' in response:
            return json.loads(response['SecretString'])
        else:
            # Binary secret
            return response['SecretBinary']
            
    except ClientError as e:
        if e.response['Error']['Code'] == 'ResourceNotFoundException':
            raise ValueError(f"Secret {secret_name} not found")
        elif e.response['Error']['Code'] == 'InvalidRequestException':
            raise ValueError(f"Invalid request for secret {secret_name}")
        else:
            raise

# Usage
db_creds = get_secret('prod/myapp/database')
connection = mysql.connector.connect(
    host=db_creds['host'],
    user=db_creds['username'],
    password=db_creds['password'],
    database=db_creds['dbname']
)

Parameter Store

Parameter Hierarchy

┌─────────────────────────────────────────────────────────────────┐
│              PARAMETER STORE HIERARCHY                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  /myapp/                                                        │
│  ├── /myapp/production/                                         │
│  │   ├── /myapp/production/database/host                        │
│  │   ├── /myapp/production/database/port                        │
│  │   ├── /myapp/production/api/endpoint                         │
│  │   └── /myapp/production/feature/new-ui-enabled               │
│  │                                                              │
│  ├── /myapp/staging/                                            │
│  │   ├── /myapp/staging/database/host                           │
│  │   └── /myapp/staging/api/endpoint                            │
│  │                                                              │
│  └── /myapp/shared/                                             │
│      ├── /myapp/shared/log-level                                │
│      └── /myapp/shared/timeout                                  │
│                                                                 │
│  Benefits:                                                      │
│  • IAM policies can scope to path prefix                        │
│  • GetParametersByPath for bulk retrieval                       │
│  • Clear organization by environment                            │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Parameter Types

Type	Use Case	Encryption
String	Plain text config	No
StringList	Comma-separated values	No
SecureString	Sensitive values	KMS

Bulk Retrieval

python

import boto3

def get_parameters_by_path(path: str) -> dict:
    """Get all parameters under a path"""
    
    ssm = boto3.client('ssm')
    parameters = {}
    
    paginator = ssm.get_paginator('get_parameters_by_path')
    
    for page in paginator.paginate(
        Path=path,
        Recursive=True,
        WithDecryption=True
    ):
        for param in page['Parameters']:
            # Convert path to key
            key = param['Name'].replace(path, '').lstrip('/')
            parameters[key] = param['Value']
    
    return parameters

# Usage
config = get_parameters_by_path('/myapp/production/')
# Returns: {'database/host': 'xxx', 'database/port': '3306', ...}

Architecture

┌─────────────────────────────────────────────────────────────────┐
│           CROSS-ACCOUNT SECRETS SHARING                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Shared Services Account              Workload Account          │
│  ┌─────────────────────┐             ┌─────────────────────┐    │
│  │                     │             │                     │    │
│  │ ┌─────────────────┐ │             │ ┌─────────────────┐ │    │
│  │ │ Secrets Manager │ │             │ │   Application   │ │    │
│  │ │                 │◄┼─────────────┼─┤                 │ │    │
│  │ │ • DB creds      │ │  AssumeRole │ │ • Reads secrets │ │    │
│  │ │ • API keys      │ │             │ │                 │ │    │
│  │ └─────────────────┘ │             │ └─────────────────┘ │    │
│  │                     │             │                     │    │
│  │ ┌─────────────────┐ │             │                     │    │
│  │ │      KMS        │ │             │                     │    │
│  │ │ (Encryption key)│ │             │                     │    │
│  │ └─────────────────┘ │             │                     │    │
│  │                     │             │                     │    │
│  └─────────────────────┘             └─────────────────────┘    │
│                                                                 │
│  Requirements:                                                  │
│  1. Secret resource policy allows cross-account access          │
│  2. KMS key policy allows cross-account decrypt                 │
│  3. Workload account has IAM role with permissions              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Resource Policy

json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::222222222222:role/WorkloadAppRole"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:VersionStage": "AWSCURRENT"
        }
      }
    }
  ]
}

Best Practices

Secret Naming Convention

{environment}/{application}/{secret-type}/{name}

Examples:
- prod/payment-service/database/primary
- staging/api-gateway/credentials/stripe
- shared/infrastructure/ssh/bastion-key

Security Checklist

[ ] Use Secrets Manager for credentials that need rotation
[ ] Enable automatic rotation for database credentials
[ ] Use customer-managed KMS keys for sensitive secrets
[ ] Implement least privilege access to secrets
[ ] Enable CloudTrail logging for secret access
[ ] Use resource policies for cross-account sharing
[ ] Never hardcode secrets in application code
[ ] Rotate secrets on suspected compromise

⚖️ Trade-offs

Trade-off 1: Secrets Manager vs Parameter Store

Khía cạnh	Secrets Manager	Parameter Store SecureString
Cost	$0.40/secret/tháng	Free (Standard)
Rotation	Automatic	Manual
Cross-account	Native support	Cần Lambda
Versioning	Built-in	Limited
RDS integration	Native	Manual

Khuyến nghị:

Secrets Manager: Database credentials, API keys cần rotation
Parameter Store: Config values, flags, secrets ít thay đổi

Trade-off 2: AWS-managed vs Customer-managed KMS Keys

Khía cạnh	AWS-managed	Customer-managed
Cost	Free	$1/key/tháng
Key policy	AWS controls	You control
Rotation	Automatic (yearly)	You configure
Cross-account	Không	Có
Compliance	Limited proof	Full control

Trade-off 3: Rotation Frequency vs Operational Overhead

Rotation Period	Security	Overhead	Use Case
Daily	Rất cao	Cao	Ultra-sensitive
Weekly	Cao	Trung bình	Production credentials
30 days	Trung bình	Thấp	Standard
90 days	Thấp	Rất thấp	Low-risk

🚨 Failure Modes

Failure Mode 1: Rotation Breaks Application

🔥 Incident thực tế

Auto-rotation chạy nhưng application vẫn cache old password. Database connections fail. Production down 2 giờ. Team không biết về secret versioning.

Cách phát hiện	Cách phòng tránh
Application errors post-rotation	Test rotation trong staging
Database connection failures	Use `AWSCURRENT` và `AWSPREVIOUS` labels
Rotation Lambda errors	Monitor rotation Lambda metrics

Failure Mode 2: Secret Exposure

Cách phát hiỆn	Cách phòng tránh
CloudTrail unusual access	Restrict secret access với IAM
GitHub secret scanning alerts	Pre-commit hooks, git-secrets
GuardDuty findings	Resource policies cho secrets
Credential usage từ unknown locations	VPC endpoints, IP restrictions

Failure Mode 3: KMS Key Deletion

Cách phát hiện	Cách phòng tránh
ScheduleKeyDeletion CloudTrail event	SCPs block key deletion
Decryption failures	7-30 day deletion window
Data inaccessible	Key policy restrictions

🔐 Security Baseline

Secrets Security Requirements

Requirement	Implementation	Verification
No hardcoded secrets	Code scanning	Pre-commit hooks
Encryption at rest	KMS encryption	Default enabled
Access logging	CloudTrail data events	Enabled organization-wide
Rotation policy	Auto-rotation enabled	Compliance check
Least privilege	IAM policies	Access Analyzer

Secret Access Pattern

json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:*:*:secret:prod/*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:VersionStage": "AWSCURRENT"
        },
        "IpAddress": {
          "aws:VpcSourceIp": "10.0.0.0/8"
        }
      }
    }
  ]
}

📊 Ops Readiness

Metrics cần Monitoring

Metric	Source	Alert Threshold
Secret access from unknown principal	CloudTrail	Any
Rotation Lambda errors	CloudWatch	> 0
KMS key scheduled for deletion	CloudTrail	Any
Secret access denied	CloudTrail	Spike
Decryption failures	CloudWatch	> 0

Runbook Entry Points

Tình huống	Runbook
Suspected secret leak	`runbook/secret-rotation-emergency.md`
Rotation failure	`runbook/rotation-troubleshooting.md`
KMS key issues	`runbook/kms-key-recovery.md`
Cross-account access denied	`runbook/secrets-cross-account.md`
Application can't access secret	`runbook/secret-access-debug.md`

✅ Design Review Checklist

Secret Storage

[ ] Đúng service cho từng loại secret
[ ] Naming convention consistent
[ ] KMS key selection appropriate
[ ] Cross-account access configured nếu cần

Rotation

[ ] Auto-rotation enabled cho credentials
[ ] Rotation đã test trong non-prod
[ ] Application hỗ trợ secret versioning
[ ] Rotation Lambda monitored

Security

[ ] Least privilege IAM
[ ] CloudTrail logging enabled
[ ] No hardcoded secrets trong code
[ ] Secret scanning trong CI/CD

Operations

[ ] Access monitoring configured
[ ] Alerting cho anomalies
[ ] Runbooks documented
[ ] Emergency rotation procedure

📎 Liên kết

📎 GCP Secret Manager - So sánh với GCP's secrets management
📎 IAM Fundamentals - Access control cho secrets
📎 Security Posture - Secrets trong overall security strategy
📎 Terraform Security - Managing secrets in IaC
📎 Storage & Data Protection - Encryption patterns

🔑 Key & Secrets Management ​

🎯 Mục tiêu (Outcomes) ​

✅ Khi nào dùng ​

❌ Khi nào KHÔNG dùng ​

Service Selection ​

Decision Matrix ​

Service Comparison ​

AWS KMS ​

Key Hierarchy ​

Key Types ​

Key Policy Example ​

AWS Secrets Manager ​

Secret Lifecycle ​

Secret Structure ​

Rotation Configuration ​

Application Integration ​

Parameter Store ​

Parameter Hierarchy ​

Parameter Types ​

Bulk Retrieval ​

Cross-Account Secrets Sharing ​

Architecture ​

Resource Policy ​

Best Practices ​

Secret Naming Convention ​

Security Checklist ​

⚖️ Trade-offs ​

Trade-off 1: Secrets Manager vs Parameter Store ​

Trade-off 2: AWS-managed vs Customer-managed KMS Keys ​

Trade-off 3: Rotation Frequency vs Operational Overhead ​

🚨 Failure Modes ​

Failure Mode 1: Rotation Breaks Application ​

Failure Mode 2: Secret Exposure ​

Failure Mode 3: KMS Key Deletion ​

🔐 Security Baseline ​

Secrets Security Requirements ​

Secret Access Pattern ​

📊 Ops Readiness ​

Metrics cần Monitoring ​

Runbook Entry Points ​

✅ Design Review Checklist ​

Secret Storage ​

Rotation ​

Security ​

Operations ​

📎 Liên kết ​

🔑 Key & Secrets Management

🎯 Mục tiêu (Outcomes)

✅ Khi nào dùng

❌ Khi nào KHÔNG dùng

Service Selection

Decision Matrix

Service Comparison

AWS KMS

Key Hierarchy

Key Types

Key Policy Example

AWS Secrets Manager

Secret Lifecycle

Secret Structure

Rotation Configuration

Application Integration

Parameter Store

Parameter Hierarchy

Parameter Types

Bulk Retrieval

Cross-Account Secrets Sharing

Architecture

Resource Policy

Best Practices

Secret Naming Convention

Security Checklist

⚖️ Trade-offs

Trade-off 1: Secrets Manager vs Parameter Store

Trade-off 2: AWS-managed vs Customer-managed KMS Keys

Trade-off 3: Rotation Frequency vs Operational Overhead

🚨 Failure Modes

Failure Mode 1: Rotation Breaks Application

Failure Mode 2: Secret Exposure

Failure Mode 3: KMS Key Deletion

🔐 Security Baseline

Secrets Security Requirements

Secret Access Pattern

📊 Ops Readiness

Metrics cần Monitoring

Runbook Entry Points

✅ Design Review Checklist

Secret Storage

Rotation

Security

Operations

📎 Liên kết