Stay hungry,Stay foolish

0%

Ext4文件系统inode限制

问题

盯日志,发现如下信息

1
2
2021-12-20T06:29:06.112829+08:00 test-server kernel: [17763482.361803] EXT4-fs warning (device dm-1): ext4_dx_add_entry:2190: Directory (ino: 17956865) index full, reach max htree level :2
2021-12-20T06:29:06.112851+08:00 test-server kernel: [17763482.361807] EXT4-fs warning (device dm-1): ext4_dx_add_entry:2194: Large directory feature is not enabled on this filesystem

Directory (ino: 17956865) index full第一反应哪个分区的inode满, 实际查看并没有

1
2
3
4
5
6
7
8
9
10
root@test-server[test]:~# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 4.0M 1.7K 4.0M 1% /dev
tmpfs 4.0M 3.0K 4.0M 1% /run
/dev/sda1 2.0M 202K 1.9M 10% /
tmpfs 4.0M 2 4.0M 1% /dev/shm
tmpfs 4.0M 5 4.0M 1% /run/lock
tmpfs 4.0M 18 4.0M 1% /sys/fs/cgroup
/dev/mapper/vg_test-lv_home 68M 11M 58M 16% /home
tmpfs 4.0M 10 4.0M 1% /run/user/0

查看device dm-1具体指哪个设备udevadm info -q all --name='/dev/dm-1'并根据相关信息确认其挂载点

关键信息如下: LVM分区/dev/vg_test/lv_home, 挂载点/home

1
2
3
4
5
6
P: /devices/virtual/block/dm-1
N: dm-1
E: DEVLINKS=/dev/vg_test/lv_home
E: DEVNAME=/dev/dm-1

... ...

找出存在大量文件的目录ino: 17956865

1
2
root@test-server[test]:~# find /home/ -inum 17956865
/home/test/deloy

或直接暴力统计

1
for i in `ls`; do echo $/home/$i; find /home/$i/ -type f | wc -l ; done

发现/home/test/deloy目录下存在1000多万文件

结合日志输出Large directory feature is not enabled on this filesystem, 翻man ext4,man tune2fs,一些关键特性

  • large_dir
  • large_file
  • dir_index
  • dir_nlink
  • … …

结合tune2fs -l /dev/mapper/vg_test-lv_home关键输出

1
2
3
4
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Free blocks: 208427635
Free inodes: 70840725
... ...

描述看以下方式可以解决问题(不建议,需最终确认这种场景是不是正常)

1
tune2fs -O large_dir /dev/mapper/vg_test-lv_home

疑问:inode既然没满,为什么会有告警?

继续翻和搜https://en.wikipedia.org/wiki/Ext4,发现关键信息: 大概意思就是

  • ext4自身没有做相关方面的限制
  • Linux 2.6.23以后引入的HTree索引做的,2-level HTree索引限定了约10-12 million(百万)
  • Linux 4.12后largedir特性开启的话3-level限定了约6 billion(十亿)
1
2
Unlimited number of subdirectories
ext4 does not limit the number of subdirectories in a single directory, except by the inherent size limit of the directory itself. (In ext3 a directory can have at most 32,000 subdirectories.)[16] To allow for larger directories and continued performance, ext4 in Linux 2.6.23 and later turns on HTree indices (a specialized version of a B-tree) by default, which allows directories up to approximately 1012 million entries to be stored in the 2-level HTree index and 2 GB directory size limit for 4 KiB block size, depending on the filename length. In Linux 4.12 and later the largedir feature enabled a 3-level HTree and directory sizes over 2 GB, allowing approximately 6 billion entries in a single directory.

有兴趣的可继续翻翻具体怎么做的限定… …