Linux下数据文件删除文件系统空间不释放的问题

Linux下数据文件删除文件系统空间不释放的问题

首先数据文件删除文件系统空间不释放的问题不只出现在Linux平台，所有平台都可能有这样的问题。这里只是在Linux平台做一些测试，其他平台类似；其次只有将数据文件存放在文件系统中才会有此类问题，如果数据库存放在ASM中也不会有类似的问题，这篇文章的目的是对相关问题进行总结，熟悉一些工具和方法，这是比较重要的。

写这篇文章是因为客户有一套数据库，AIX 6.1，数据文件存放在文件系统中，在使用DROP TABLESPACE UNDO INCLUDING CONTENTS AND DATAFILES删除了UNDO表空间之后，文件系统空间没有得到释放。如何解决问题就成了写这篇文章的初衷。

空间没有释放我们可能是通过df命令看确认的，当我们用du去扫描目录的大小可能会发现df和du两个命令看到的空间使用情况是不同的，可能差别很大，下面这篇MOS文章说明了原因：

'du' and 'df' tools report different space utilization (文档 ID 457444.1)

In this Document

 

Symptoms

Cause

Solution

 

Applies to:

Linux OS - Version Oracle Linux 4.4 and later

Linux x86-64

Linux x86

Linux Kernel - Version: 4.4 to 5.3

 

 

Symptoms

The 'du' (/usr/bin/du) and 'df' (/bin/df) command output displays conflicting space utilisation values, for example:

# df -k /

Filesystem    1k-blocks    Used  Available Use% Mounted on

/dev/sda6      9288792  8672768    144120 99% /

 

# du -xsh /

2.1G /

 

In the example above, 'df' reports 8.6 Gb to have been used on the root (/) filesystem, whereas 'du' reports only 2.1 Gb to have been used.

 

Cause

The 'df' command reports how many disk blocks are used, whilst 'du' traverses the filesystem and reports the actual number of blocks used (directory by directory), including any relating to files in use by processes.

 

In most cases, space utilisation values returned from 'df' and 'du' will be consistent. However, the potential exists for a running process to delete a large file, say. In this instance, according to 'du', the large file no longer exists, so the blocks associated with the deleted file are not reported. With the process still running, and with an open file descriptor still held against the deleted file, 'df' continues to track and report all disk blocks used, including those associated with the deleted (phantom) file. In this situation, the disk space associated with the deleted file will only be relinquished back to the system when the process completely releases the deleted file's descriptor or the process terminates (either gracefully or killed).

Solution

The solution is to identify and stop (or kill) the process that continues to hold a file descriptor open for the deleted file. To do so, run the lsof command (/usr/sbin/lsof | grep deleted) as root to identify the holding process, for example:

 

# lsof | grep deleted

COMMAND     PID   USER    FD  TYPE  DEVICE      SIZE       NODE NAME

cannaserv  3825  canna    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

vmware     4295   root    6u   REG   253,0      6770   13074503 /tmp/vmware-root/ui-4295.log (deleted)

vmware-re  4316   root    6u   REG   253,0      6770   13074503 /tmp/vmware-root/ui-4295.log (deleted)

vmnet-nat  4448   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

vmware-se  4454   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

gdm-binar  4506   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

gconfd-2   5392   root   12wW  REG   253,0       609   13090818 /tmp/gconfd-root/lock/0t1188207163ut519551u0p5392r346479926k3219784492 (deleted)

vmware-vm  5822   root   57u   REG   253,0   6520832   13074477 /tmp/vmware-root/ram0 (deleted)

vmware-vm 16487   root   57u   REG   253,0  11153408   13074520 /tmp/vmware-root/ram0 (deleted)

kdeinit   17991   root   17u   REG   253,0     26712   13074524 /tmp/kde-root/khtmlcacheM7jXYb.tmp (deleted)

kdeinit   17991   root   18u   REG   253,0      5631   13074501 /tmp/kde-root/khtmlcacheZlJmda.tmp (deleted)

kdeinit   17991   root   21u   REG   253,0     44718   13074514 /tmp/kde-root/khtmlcacheH5m4lc.tmp (deleted)

 

In the example above, the 7th column in the output denotes the size of deleted files (in bytes). The 9th column denotes which file remains held open. The 1st and second columns denotes the process and pid that still holds the open file descriptor.

    之所以df和du命令看到的空间使用会有差别，原因在于du不统计已经删除的文件，df会统计已经删除的文件，但该文件依然被进程持有，只有等进程释放了该文件，df才不进行统计。通过lsof | grep deleted命令可以找出被删除的文件依然被进程持有的情况。

    通过上面这篇文章我们得知，在文件系统中删除某个大文件空间没有释放是因为依然有进程在持有它，如果找到相关进程就需要用到lsof命令和fuser命令，熟悉这两个命令非常的重要。

>lsof(list open files)是一个列出当前系统打开文件的工具。
>fuser用于标识访问文件或socket的进程信息。

   fuser也是我们平时常用的找进程的工具，但它不能列出详细的进程信息，更多的只是一个进程号，无法很好的过滤，在这个场景中lsof工具更加的合适。

另外，大家都知道，在Linux平台文件系统中，即使某个文件被删除，但是它如果任然有进程持有它，这个文件是可以被恢复的，这就是为什么数据库所有数据文件被删除，在实例没有停机的情况下可以完整恢复数据文件的原因，下面这篇MOS文章就讨论了在此情况下如何对数据文件进行恢复：

How To Recover Deleted Files on ext3/ext4 Filesystem (文档 ID 2056343.1)

 

In this Document

 

Goal

Solution

References

 

Applies to:

Linux OS - Version Oracle Linux 5.0 and later

Linux x86-64

Linux x86

Goal

How to recover deleted file on ext3/ext4 filesystem, Which still has file descriptor opened.

Solution

A file in Linux is a pointer to an inode which contains the file data (permissions, owner and where its actual content resides on the disk).

Deleting the file removes the link, but not the inode itself.

If any other process has this file open then inode is not released for writing until the process releases it.

So if a process still has the file open, the data are there somewhere, even though the directory listing shows no files.

# ll

total 4

-rw-r--r-- 1 root root 27 Sep 16 05:19 test

# rm test

# lsof /opt/test/test

COMMAND    PID  USER  FD  TYPE DEVICE SIZE/OFF    NODE NAME

less 21353 root 4r REG 252,2 27 260360 /opt/test/test (deleted)     <<<<<<<<<<<<<

Understanding output of "lsof" command:

COMMAND: Command using the file.

PID: PID of the file

USER: Owner of the file

FD: File descriptor. Different flags of File descriptor are as below:

#:      The number in front of flag(s) is the file descriptor number of used by the process to associated with the file

u:      File open with Read and Write permission

r:       File open with Read permission

w:      File open with Write permission

W:      File open with Write permission and with Write Lock on entire file

mem: Memory mapped file, usually for share library

TYPE: File type. Different flags of File type are as below:

REG - Regular file

DIR - Directory

DEVICE: major, minor number of the device where file resides.

SIZE/OFF: File size

NODE: inode number

NAME: File name

Now we know that process 21353 still has the file open, and the file descriptor is 4.

Now we can look into /proc and there will be a reference to the inode, from which the deleted file can be copied.

Following steps will help to recover the deleted files:

# ls -l /proc/21353/fd/4

  lr-x------ 1 root root 64 Sep 16 05:28 /proc/21353/fd/4 -> /opt/test/test (deleted)

# cp /proc/21353/fd/4 /opt/test/test.bkp

Now verify the content of the restored file.

Note: Don't use the -a flag with cp, as this will copy the (broken) symbolic link, rather than the actual file contents.

另外，找到是某个进程持有的文件，通过下面的方法可以看到这个进程相关的环境信息：

Checking the environment variables of ASM pmon process: It shows ORACLE_HOME is set to /oracle_grid/product/11.2.0.3/grid/ ( with 'slash' at the end )

# ps -ef | grep pmon
oracle 27232 1 0 May30 ? 00:07:05 asm_pmon_+ASM1

# cat /proc/27232/environ
__CLSAGFW_TYPE_NAME=ora.asm.typeORA_CRS_HOME=/oracle_grid/product/11.2.0.3/grid/HOSTNAME=aude3od015naboi.basdev.aurdev.national.com.auTERM=xtermSHELL=/bin/bash__CR......

   总结：对于此类问题，我们首先要明白为什么df和du在空间计算上有所差别，其次要熟悉lsof和fuser两个命令，找出继续持有文件的进程号，通过该进程号可以在/proc目录下恢复文件，查看进程的环境信息，甚至杀掉进程来释放空间。

最后通过一个简单的例子来结束这篇文章：

1.首先确保lsof工具已经安装到操作系统。
[root@rac01 Server]# rpm -ivh lsof-4.78-6.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:lsof                   ########################################### [100%]
[root@rac01 Server]# which lsof
/usr/sbin/lsof

2.在其中一个会话通过tail -f install2.log命令使tail进程持有该文件，在另一个会话通过rm -rf install2.log命令删除该文件。

3.使用lsof执行如下的操作：

[root@rac01 ~]# lsof | grep deleted
tail      6006      root    3r      REG                8,3     29544    4587629 /root/install2.log (deleted)
[root@rac01 ~]# cd /proc/6006/
[root@rac01 6006]# ls
attr        cmdline          cwd      fdinfo   loginuid   mounts      numa_maps      pagemap      schedstat  stat     task
auxv        comm             environ  io       maps       mountstats  oom_adj        personality  sessionid  statm    wchan
cgroup      coredump_filter  exe      latency  mem        net         oom_score      root         smaps      status
clear_refs  cpuset           fd       limits   mountinfo  ns          oom_score_adj  sched        stack      syscall
[root@rac01 6006]# cd fd
[root@rac01 fd]# ll
total 0
lrwx------ 1 root root 64 Dec  3 19:07 0 -> /dev/pts/0
lrwx------ 1 root root 64 Dec  3 19:07 1 -> /dev/pts/0
lrwx------ 1 root root 64 Dec  3 19:07 2 -> /dev/pts/0
lr-x------ 1 root root 64 Dec  3 19:07 3 -> /root/install2.log (deleted)
[root@rac01 fd]# cd ..
[root@rac01 6006]# cat environ 
HOSTNAME=rac01TERM=vt100SHELL=/bin/bashHISTSIZE=1000SSH_CLIENT=172.168.4.123 56823 22OLDPWD=/mnt/ServerSSH_TTY=/dev/pts/0USER=rootLS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:MAIL=/var/spool/mail/rootPATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/binINPUTRC=/etc/inputrcPWD=/rootLANG=en_US.UTF-8SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpassSHLVL=1HOME=/rootLOGNAME=rootSSH_CONNECTION=172.168.4.123 56823 172.168.4.200 22LESSOPEN=|/usr/bin/lesspipe.sh %sG_BROKEN_FILENAMES=1_=/usr/bin/tail
[root@rac01 6006]# lsof /root/    <<<< 找出持有/root目录下文件的进程
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
bash    4934 root  cwd    DIR    8,3     4096 4587521 /root/
tail    6006 root  cwd    DIR    8,3     4096 4587521 /root/
[root@rac01 6006]# lsof -c tail    <<<< 找出tail进程持有的文件
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
tail    6006 root  cwd    DIR    8,3     4096 4587521 /root
tail    6006 root  rtd    DIR    8,3     4096       2 /
tail    6006 root  txt    REG    8,3    37704 1448826 /usr/bin/tail
tail    6006 root  mem    REG    8,3 56479136 1446088 /usr/lib/locale/locale-archive
tail    6006 root  mem    REG    8,3  1720736 5242891 /lib64/libc-2.5.so
tail    6006 root  mem    REG    8,3   142488 5242884 /lib64/ld-2.5.so
tail    6006 root    0u   CHR  136,0      0t0       3 /dev/pts/0
tail    6006 root    1u   CHR  136,0      0t0       3 /dev/pts/0
tail    6006 root    2u   CHR  136,0      0t0       3 /dev/pts/0
tail    6006 root    3r   REG    8,3    29544 4587629 /root/install2.log (deleted)
[root@rac01 6006]# lsof +d /root/    <<<< 显示访问/root目录下文件的进程。
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
bash    4934 root  cwd    DIR    8,3     4096 4587521 /root/
tail    6006 root  cwd    DIR    8,3     4096 4587521 /root/
[root@rac01 6006]# lsof +D /root/    <<<< 显示访问/root目录及子目录下文件的进程。
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
bash    4934 root  cwd    DIR    8,3     4096 4587521 /root/
tail    6006 root  cwd    DIR    8,3     4096 4587521 /root/ 
[root@rac01 6006]# lsof -d 3 | grep -v grep | grep deleted    <<<< 显示持有文件FD为3的进程文件
tail      6006      root    3r   REG                8,3    29544 4587629 /root/install2.log (deleted)
[root@rac01 6006]# lsof -p 6006    <<<< 显示6006进程持有的文件
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
tail    6006 root  cwd    DIR    8,3     4096 4587521 /root
tail    6006 root  rtd    DIR    8,3     4096       2 /
tail    6006 root  txt    REG    8,3    37704 1448826 /usr/bin/tail
tail    6006 root  mem    REG    8,3 56479136 1446088 /usr/lib/locale/locale-archive
tail    6006 root  mem    REG    8,3  1720736 5242891 /lib64/libc-2.5.so
tail    6006 root  mem    REG    8,3   142488 5242884 /lib64/ld-2.5.so
tail    6006 root    0u   CHR  136,0      0t0       3 /dev/pts/0
tail    6006 root    1u   CHR  136,0      0t0       3 /dev/pts/0
tail    6006 root    2u   CHR  136,0      0t0       3 /dev/pts/0
tail    6006 root    3r   REG    8,3    29544 4587629 /root/install2.log (deleted)
[root@rac01 6006]# lsof -u root | grep deleted    <<<< 显示以root用户持有的进程文件
tail      6006 root    3r      REG                8,3     29544    4587629 /root/install2.log (deleted)
[root@rac01 6006]# cd /proc/6006/fd/
[root@rac01 fd]# cp 3 /root/install2.log    <<<< 恢复删除的install2.log文件
[root@rac01 fd]# cd /proc/6006/fd
[root@rac01 fd]# ll
total 0
lrwx------ 1 root root 64 Dec  3 19:07 0 -> /dev/pts/0
lrwx------ 1 root root 64 Dec  3 19:07 1 -> /dev/pts/0
lrwx------ 1 root root 64 Dec  3 19:07 2 -> /dev/pts/0
lr-x------ 1 root root 64 Dec  3 19:07 3 -> /root/install2.log (deleted)   <<<< 文件恢复后deleted状态未发生变化。
[root@rac01 fd]# cd /root/
[root@rac01 ~]# ls
anaconda-ks.cfg  core.11067  core.526  core.6027  core.7965  Desktop  install2.log  install.log  install.log.syslog

--end--