ES故障处理实践

磁盘资源不足

异常现象

fibebeat日志写入失败
kibana登录失败

排查

查看kibana日志，发现TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block的报错，截图如下：

确定是磁盘问题导致ES将索引锁定（将read-only-allow-delete参数置为true）

解决方案

删除索引，释放磁盘空间

curl请求、postman请求、或者在kibana上删除索引

建议使用kibana的开发工具操作
1
2
3
4
# 获取索引
GET _cat/indices
# 删除索引
DELETE .ds-filebeat-8.1.0-2022.04.01-000008
扩容磁盘空间

将read-only-allow-delete参数置null

使用elastic账号put请求修改参数会报错

{
    "error": {
        "root_cause": [
            {
                "type": "security_exception",
                "reason": "action [indices:admin/settings/update] is unauthorized for user [elastic] with roles [superuser] on indices [.async-search], this action is granted by the index privileges [manage,all]"
            }
        ],
        "type": "security_exception",
        "reason": "action [indices:admin/settings/update] is unauthorized for user [elastic] with roles [superuser] on indices [.async-search], this action is granted by the index privileges [manage,all]"
    },
    "status": 403
}

这时需要替换为 kibana用户去操作。

优化方案

调整索引的生命周期策略，定期删除过期索引数据（比如3天后变更为温数据，7天后变为冷数据，20天后删除）