Leaked Secret Response
Immediate Actions (Within 1 Hour)
1. Assume Compromised (Do Not Delay)
The secret is compromised the moment it's pushed to Git, even if the repo is private.
2. Rotate the Secret Immediately
Stop the bleeding first, investigate later.
# Example: Database password leaked
# 1. Change password at source
psql -U postgres -c "ALTER USER appuser PASSWORD 'new_secure_password_123!';"
# 2. Update Ansible Vault
ansible-vault edit inventory/production/group_vars/all/vault.yml
# Change vault_db_password to new password
# 3. Update GitHub Secrets (if used in CI/CD)
# Go to repo Settings → Secrets → Update DB_PASSWORD
# 4. Redeploy affected services with new secret
ansible-playbook playbooks/deploy.yml -i inventory/production
Timeline: Complete rotation within 1 hour of discovery.
3. Alert Security Team
# In Rocket.Chat #security channel
⚠️ Secret leaked in Git
Repo: company/api-server
Commit: abc123def
Secret type: Database password
Action taken: Rotated immediately
Status: Investigating
4. Identify What Was Exposed
Check:
- What secret? (password, API key, private key?)
- Where? (which repo, which commit?)
- When? (commit timestamp)
- Who had access? (public repo? private? team members?)
- Was it pushed to GitHub? (yes = assume public)
Secret-Specific Rotation
Database Password
# 1. Change at database
ALTER USER appuser PASSWORD 'new_password';
# 2. Update in Ansible Vault
ansible-vault edit inventory/production/group_vars/databases/vault.yml
# 3. Update running services
ansible-playbook playbooks/update-db-config.yml -i inventory/production
# 4. Verify connectivity
ansible all -i inventory/production -m shell -a "psql -U appuser -c 'SELECT 1;'"
API Key (Third-Party Service)
# 1. Revoke old key in provider dashboard (e.g., Stripe, SendGrid)
# 2. Generate new API key
# 3. Update in Ansible Vault + GitHub Secrets
ansible-vault edit inventory/production/group_vars/all/vault.yml
# Update GitHub repo Settings → Secrets
# 4. Redeploy services
ansible-playbook playbooks/deploy.yml -i inventory/production
SSH Private Key
# 1. Remove public key from all servers
ansible-playbook playbooks/remove-ssh-key.yml -e "ssh_key_fingerprint=XX:YY:ZZ"
# 2. Generate new key pair
ssh-keygen -t ed25519 -C "deploy@company.com"
# 3. Add new public key to authorized_keys via Ansible
# Update inventory/production/group_vars/all/users.yml
ansible-playbook playbooks/sync-ssh-keys.yml -i inventory/production
# 4. Update key in GitHub Secrets (for CI/CD)
SSL/TLS Private Key
# 1. Revoke certificate immediately
# Contact certificate authority to revoke
# 2. Generate new CSR and private key
openssl req -new -newkey rsa:2048 -nodes -keyout new.key -out new.csr
# 3. Obtain new certificate
# 4. Deploy via Ansible
ansible-playbook playbooks/update-ssl-certs.yml -i inventory/production
OAuth Token / JWT Secret
# 1. Rotate signing key
# Generate new secret key
# 2. Update in Ansible Vault + GitHub Secrets
ansible-vault edit inventory/production/group_vars/all/vault.yml
# 3. Deploy new config
ansible-playbook playbooks/deploy.yml -i inventory/production
# 4. Invalidate all existing tokens (force re-login)
# Depends on your auth system
Clean Git History
After rotating, clean the secret from Git history:
# Option 1: Using git-filter-repo (recommended)
pip install git-filter-repo
git filter-repo --path-glob '**/secrets.yml' --invert-paths
# Option 2: Using BFG Repo-Cleaner
# Download BFG: https://rtyley.github.io/bfg-repo-cleaner/
java -jar bfg.jar --delete-files secrets.yml
git reflog expire --expire=now --all
git gc --prune=now --aggressive
# Force push (⚠️ Notify team first!)
git push --force --all
git push --force --tags
⚠️ Warning: Force push rewrites history. Coordinate with team.
Incident Report
Document in post-incident review:
- What was leaked: Type of secret, service it belongs to
- How it was leaked: Accidental commit, wrong .gitignore, etc.
- When detected: Timestamp
- Exposure window: Time between leak and rotation
- Rotation completed: Timestamp
- Evidence of use: Any logs showing unauthorized access?
- Prevention: What we'll do differently
Template:
## Secret Leak Incident - YYYY-MM-DD
**Secret**: Production database password
**Leaked in**: Commit abc123 on 2025-11-10 14:23
**Detected**: 2025-11-10 14:35 (12 minutes exposure)
**Rotated**: 2025-11-10 14:47 (24 minutes from leak)
**Evidence of compromise**: None found in access logs
**Cleaned from Git**: Yes, force push at 14:55
**Root cause**: Developer committed .env file (should be in .gitignore)
**Prevention**:
- Added .env to global .gitignore
- Enabled pre-commit hook for secret scanning
- Added to onboarding checklist
- Scheduled training on secret management
Prevention Measures
After handling the incident, implement these to prevent recurrence:
1. Pre-Commit Hooks (Blocks commits)
# Install gitleaks pre-commit hook
# .pre-commit-config.yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.18.0
hooks:
- id: gitleaks
# Install for all developers
pre-commit install
2. GitHub Secret Scanning (Already Active)
Already enabled on GitHub. Ensure Rocket.Chat webhook configured for alerts.
3. Update .gitignore
# Add to .gitignore
.env
.env.*
*.secret
*.key
*.pem
vault-password.txt
credentials.json
4. Code Review Checklist
Add to PR template:
- No secrets in code
- .env files in .gitignore
- Secrets stored in Ansible Vault or GitHub Secrets
- Pre-commit hooks passing
5. Developer Training
- Add to onboarding: "Secret Management 101"
- Regular reminders in #engineering channel
- Share incident reports as learning
Monitoring for Unauthorized Use
After rotation, watch for:
# Check access logs for unauthorized access
# Example: Database connection logs
grep "authentication failed" /var/log/postgresql/postgresql.log
# Check API logs for old API key usage
grep "401 Unauthorized" /var/log/app/api.log | grep "old_api_key"
# Monitor for unusual activity
# - Login attempts from unknown IPs
# - Access to sensitive data
# - Unusual query patterns
Alert #security if suspicious activity detected.