当例行备份耗尽磁盘：修复 Kubernetes 中的 100% 磁盘使用率问题
在 DevOps 的世界里，一些最具挑战性的事件往往始于看似简单的问题。某天，在处理我们的 Kubernetes 基础设施时，我调查了一台主节点的根文件系统已达到 100% 磁盘使用率的情况。起初，这看起来像是一个典型的磁盘空间问题。也许是日志文件增长过大，容器镜像占用了过多存储空间，或者某个应用程序生成了意外的数据。然而，根本原因却更为隐蔽：卡住的 etcd 备份。以下是我如何诊断并解决该问题的过程。
初步症状
问题的第一个迹象是一个监控告警，表明我们 Kubernetes 主节点之一的根文件系统已满。鉴于 Kubernetes 主节点对集群运行至关重要，这是一个严重的问题。我立即使用标准的 df -h 命令检查了该节点的磁盘使用情况：
df -h
输出确认了问题：根分区已满 100%，没有可用空间。接下来，我查看该节点的日志，看看是否有明显的错误：
journalctl -xe
虽然日志显示了一些 Kubernetes 的常规活动，但没有出现“磁盘已满紧急情况”的明显提示。我还使用 du -sh * 和 find / -type f -size +1G -exec ls -lh {} \; 检查了大型文件或目录，但没有发现明显的罪魁祸首。
深入调查 etcd
鉴于 Kubernetes 严重依赖 etcd 进行状态管理，调查 etcd 的磁盘使用情况是合理的。我使用 etcdctl 命令检查了 etcd 数据库的大小：
etcdctl ls --recursive --prefix=/ | wc -l
输出显示了一组健康的键数量，但磁盘使用情况似乎仍然不对劲。然后我检查了 etcd 备份目录，该目录配置为在 /var/lib/etcd-backups 存储备份：
ls -lh /var/lib/etcd-backups
这就是问题变得明显的地方。备份目录非常大，占用了几乎所有的可用磁盘空间。鉴于备份通常是例行且自动的，手动备份失败的可能性不大。我怀疑可能是某个自动备份卡住了，原因可能是网络问题、配置错误或资源限制。
确认卡住的备份
为了验证我的怀疑，我检查了节点上运行的备份进程：
ps aux | grep etcd-backup
输出显示了多个 etcd-backup 进程，其中一些似乎处于卡住状态。我还检查了系统的 cron 任务，看看是否有与 etcd 备份相关的任务：
crontab -l
虽然没有明显的 etcd 备份 cron 任务，但该节点确实有一个自定义脚本通过 etcdctl 触发备份。我检查了这个脚本，看看是否有错误处理或重试机制：
cat /path/to/etcd-backup-script.sh
脚本看起来合理，但我注意到它没有优雅地处理失败。如果备份过程因任何原因失败，脚本会直接退出，不会尝试重试。
解决问题
立即的解决方案是删除卡住的备份以释放磁盘空间。但这并不是一个长期解决方案。为了防止类似问题再次发生，我需要改进备份过程。
-
限制备份重试次数：我将备份脚本修改为限制重试次数。如果备份在尝试一定次数后失败，脚本会通过 Slack 向团队发送警报：
```bash
!/bin/bash
MAX_RETRIES=3
RETRIES=0
while [ $RETRIES -lt $MAX_RETRIES ]; do
etcdctl snapshot save /var/lib/etcd-backups/backup-$$(date +%Y%m%d%H%M%S).db
if [ $? -eq 0 ]; then
echo "Backup succeeded."
exit 0
else
echo "Backup failed. Retrying..."
RETRIES=$((RETRIES+1))
sleep 10
fi
done
echo "Backup failed after $MAX_RETRIES retries. Alerting team."
curl -X POST -H 'Content-type: application/json' --data '{"text":"Etcd backup failed on $(hostname)"}' https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
exit 1
```
-
监控磁盘使用情况：我使用 Prometheus 和 Grafana 为 etcd 备份目录添加了监控告警。这将允许我们在磁盘使用问题变得严重之前就发现它们。
```yaml
示例 Prometheus 告警，用于 etcd 备份目录大小
- alert: HighEtcdBackupUsage
expr: vector(sum(kubelet_volume_stats_used_bytes{job="kubelet", volume_name="etcd-backups"})) / vector(sum(kubelet_volume_stats_capacity_bytes{job="kubelet", volume_name="etcd-backups"})) > 0.9
for: 5m
labels:
severity: critical
annotations:
summary: "High disk usage in etcd backup directory"
description: "The etcd backup directory on {{ $__meta_kubelet_node_name }} is using more than 90% of its capacity."
```
-
自动化清理：为了防止备份无限增长，我添加了一个 cron 任务来清理旧备份：
bash
0 0 * * * find /var/lib/etcd-backups -type f -name 'backup-*.db' -mtime +7 -delete
经验教训
这次事件让我学到了几个关于管理 Kubernetes 基础设施的重要教训：
-
自动备份需要健壮的错误处理：备份脚本应包含重试机制和警报，以确保失败能被及时发现和处理。
-
监控至关重要：主动监控有助于在问题变得严重之前发现它们。在这个案例中，监控 etcd 备份目录的大小将为我们节省数小时的排错时间。
-
定期清理是必要的：如果管理不当，即使是自动备份也可能失控。定期清理旧备份可以防止磁盘空间问题。
总结
如果未快速解决，由卡住的 etcd 备份引起的 100% 磁盘使用率问题可能会使 Kubernetes 集群瘫痪。通过改进备份脚本、实施监控和自动化清理，您可以防止此类问题再次发生。始终确保您的自动进程具有健壮的错误处理，并且您对其运行情况有可见性。
When a Routine Backup Fills the Disk: Fixing a 100% Disk Usage Issue in Kubernetes
In the world of DevOps, some of the most challenging incidents begin with something that looks simple. One day, while working on our Kubernetes infrastructure, I investigated a master node whose root filesystem had reached 100% disk usage. At first, it looked like a typical disk-space problem. Maybe logs had grown too large, container images were consuming storage, or some application was generating unexpected data. However, the root cause turned out to be something far more insidious: stuck etcd backups. Here’s how I diagnosed and resolved the issue.
The Initial Symptoms
The first sign of trouble was a monitoring alert indicating that the root filesystem on one of our Kubernetes master nodes was full. Given that Kubernetes masters are critical to the cluster's operation, this was a serious concern. I immediately checked the node’s disk usage with the standard df -h command:
df -h
The output confirmed the issue: the root partition was 100% full, with no available space. Next, I looked at the node’s logs to see if any obvious errors were present:
journalctl -xe
While the logs showed some general Kubernetes activity, nothing screamed "disk full emergency." I also checked for large files or directories using du -sh * and find / -type f -size +1G -exec ls -lh {} \;, but nothing stood out as a clear culprit.
Digging Deeper into etcd
Given that Kubernetes relies heavily on etcd for state management, it made sense to investigate etcd’s disk usage. I used the etcdctl command to check the size of etcd’s databases:
etcdctl ls --recursive --prefix=/ | wc -l
The output showed a healthy number of keys, but the disk usage still seemed off. I then checked the etcd backup directory, which was configured to store backups in /var/lib/etcd-backups:
ls -lh /var/lib/etcd-backups
This is where the problem became apparent. The backup directory was massive, consuming nearly all the available disk space. Given that backups are typically routine and automated, it seemed unlikely that a manual backup had failed. I suspected that an automated backup might have gotten stuck, either due to a network issue, a misconfiguration, or resource constraints.
Identifying the Stuck Backup
To confirm my suspicion, I examined the backup processes running on the node:
ps aux | grep etcd-backup
The output showed multiple etcd-backup processes, some of which appeared to be in a stuck state. I also checked the system’s cron jobs to see if any were related to etcd backups:
crontab -l
While there were no obvious cron jobs for etcd backups, the node did have a custom script that triggered backups via etcdctl. I examined this script to see if there were any error handling or retry mechanisms:
cat /path/to/etcd-backup-script.sh
The script looked reasonable, but I noticed that it didn’t handle failures gracefully. If the backup process failed for any reason, the script would simply exit without attempting a retry.
Resolving the Issue
The immediate solution was to free up disk space by deleting the stuck backups. However, this wasn’t a long-term fix. To prevent the same issue from recurring, I needed to improve the backup process.
-
Limit Backup Retries: I modified the backup script to limit the number of retries. If the backup failed after a certain number of attempts, the script would alert the team via Slack:
```bash
!/bin/bash
MAX_RETRIES=3
RETRIES=0
while [ $RETRIES -lt $MAX_RETRIES ]; do
etcdctl snapshot save /var/lib/etcd-backups/backup-$$(date +%Y%m%d%H%M%S).db
if [ $? -eq 0 ]; then
echo "Backup succeeded."
exit 0
else
echo "Backup failed. Retrying..."
RETRIES=$((RETRIES+1))
sleep 10
fi
done
echo "Backup failed after $MAX_RETRIES retries. Alerting team."
curl -X POST -H 'Content-type: application/json' --data '{"text":"Etcd backup failed on $(hostname)"}' https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
exit 1
```
-
Monitor Disk Usage: I added a monitoring alert for the etcd backup directory using Prometheus and Grafana. This would allow us to catch disk usage issues before they became critical.
```yaml
Example Prometheus alert for etcd backup directory size
- alert: HighEtcdBackupUsage
expr: vector(sum(kubelet_volume_stats_used_bytes{job="kubelet", volume_name="etcd-backups"})) / vector(sum(kubelet_volume_stats_capacity_bytes{job="kubelet", volume_name="etcd-backups"})) > 0.9
for: 5m
labels:
severity: critical
annotations:
summary: "High disk usage in etcd backup directory"
description: "The etcd backup directory on {{ $__meta_kubelet_node_name }} is using more than 90% of its capacity."
```
-
Automate Cleanup: To prevent backups from growing indefinitely, I added a cron job to clean up old backups:
bash
0 0 * * * find /var/lib/etcd-backups -type f -name 'backup-*.db' -mtime +7 -delete
Lessons Learned
This incident taught me a few important lessons about managing Kubernetes infrastructure:
-
Automated Backups Require Robust Error Handling: Backup scripts should include retry mechanisms and alerts to ensure that failures are caught and addressed promptly.
-
Monitoring is Crucial: Proactive monitoring can help detect issues before they become critical. In this case, monitoring the etcd backup directory size would have saved us hours of troubleshooting.
-
Regular Cleanup is Necessary: Even automated backups can grow out of control if not managed properly. Regular cleanup of old backups can prevent disk space issues.
Takeaway
A 100% disk usage issue caused by stuck etcd backups can bring a Kubernetes cluster to its knees if not addressed quickly. By improving backup scripts, implementing monitoring, and automating cleanup, you can prevent such issues from recurring. Always ensure that your automated processes have robust error handling and that you have visibility into their operation.
