排障集锦：九九八十一难之第二十七难！文件已经rm掉 df查看空间仍未释放

不吃小白菜

发布于 2021-04-01 14:44:26

3520

发布于 2021-04-01 14:44:26

文章被收录于专栏：用户7721898的专栏

问题描述：

警告文件磁盘已经满了，通过：df -h， du -sh * 等指令配合使用，发现当前系统任然存在大量可以使用的空间。大量剩余的磁盘空间不清楚怎么丢失了…

问题重现：

通过指令：lsof | grep deleted 指令，查看当前系统句柄未释放情况

[root@bigdata-hadoop2:/proc/21893/fd]# lsof | grep deleted
zabbix_ag  1791        zabbix    3wW     REG              253,1         4     132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag  1806        zabbix    3w      REG              253,1         4     132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag  1807        zabbix    3w      REG              253,1         4     132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag  1808        zabbix    3w      REG              253,1         4     132502 /var/tmp/zabbix_agentd.pid (deleted)
python     3946          root   14w      REG              253,1         0     393228 /tmp/cmflistener-stderr---agent-3916-1532518455-HYSskK.log (deleted)
catalogd  21893        impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893        impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893        impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893        impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893        impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893        impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893        impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893        impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893        impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893        impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  2502  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  2502  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  2502  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  2502  impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893  2502  impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893  2502  impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893  2502  impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893  2502  impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893  2502  impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893  2502  impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  5443  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  5443  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  5443  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  5443  impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893  5443  impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893  5443  impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893  5443  impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893  5443  impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893  5443  impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893  5443  impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  5834  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  5834  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  5834  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  5834  impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893  5834  impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893  5834  impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893  5834  impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893  5834  impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893  5834  impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893  5834  impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  6688  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  6688  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  6688  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  6688  impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893  6688  impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893  6688  impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893  6688  impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893  6688  impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893  6688  impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893  6688  impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  6689  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  6689  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  6689  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  6689  impala  259r      REG              253,1  72198042

通过结果可以发现，当前系统存在大量文件已经被删除（deleted），可是文件句柄并没有被释放的情况。

问题已经有答案了，因为系统中存在大量文件句柄没有释放，导致当前占用的磁盘空间无法释放。文件句柄释放不掉的根本原因，是因为当前有线程对句柄进行引用。

找到占用磁盘的pid 手动kill掉解决

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2021/03/29 ，如有侵权请联系 cloudcommunity@tencent.com 删除

grep