东八区盾山

0%

Ext4文件系统inode限制

发表于 更新于 分类于

问题

盯日志,发现如下信息

1
2
2021-12-20T06:29:06.112829+08:00 test-server kernel: [17763482.361803] EXT4-fs warning (device dm-1): ext4_dx_add_entry:2190: Directory (ino: 17956865) index full, reach max htree level :2
2021-12-20T06:29:06.112851+08:00 test-server kernel: [17763482.361807] EXT4-fs warning (device dm-1): ext4_dx_add_entry:2194: Large directory feature is not enabled on this filesystem

Directory (ino: 17956865) index full第一反应哪个分区的inode满, 实际查看并没有

1
2
3
4
5
6
7
8
9
10
root@test-server[test]:~# df -ih
Filesystem                  Inodes IUsed IFree IUse% Mounted on
udev                          4.0M  1.7K  4.0M    1% /dev
tmpfs                         4.0M  3.0K  4.0M    1% /run
/dev/sda1                     2.0M  202K  1.9M   10% /
tmpfs                         4.0M     2  4.0M    1% /dev/shm
tmpfs                         4.0M     5  4.0M    1% /run/lock
tmpfs                         4.0M    18  4.0M    1% /sys/fs/cgroup
/dev/mapper/vg_test-lv_home    68M   11M   58M   16% /home
tmpfs                         4.0M    10  4.0M    1% /run/user/0

查看device dm-1具体指哪个设备udevadm info -q all --name='/dev/dm-1'并根据相关信息确认其挂载点

关键信息如下: LVM分区/dev/vg_test/lv_home, 挂载点/home

1
2
3
4
5
6
P: /devices/virtual/block/dm-1
N: dm-1
E: DEVLINKS=/dev/vg_test/lv_home
E: DEVNAME=/dev/dm-1

... ...

找出存在大量文件的目录ino: 17956865

1
2
root@test-server[test]:~# find /home/ -inum 17956865
/home/test/deloy

或直接暴力统计

1
for i in `ls`; do echo $/home/$i; find /home/$i/ -type f | wc -l ; done

发现/home/test/deloy目录下存在1000多万文件

结合日志输出Large directory feature is not enabled on this filesystem, 翻man ext4,man tune2fs，一些关键特性

  • large_dir
  • large_file
  • dir_index
  • dir_nlink
  • … …

结合tune2fs -l /dev/mapper/vg_test-lv_home关键输出

1
2
3
4
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Free blocks:              208427635
Free inodes:              70840725
... ...

描述看以下方式可以解决问题（不建议，需最终确认这种场景是不是正常）

1
tune2fs -O large_dir /dev/mapper/vg_test-lv_home

疑问：inode既然没满,为什么会有告警？

继续翻和搜https://en.wikipedia.org/wiki/Ext4，发现关键信息: 大概意思就是

  • ext4自身没有做相关方面的限制
  • Linux 2.6.23以后引入的HTree索引做的，2-level HTree索引限定了约10-12 million（百万）
  • Linux 4.12后largedir特性开启的话3-level限定了约6 billion（十亿）
1
2
Unlimited number of subdirectories
ext4 does not limit the number of subdirectories in a single directory, except by the inherent size limit of the directory itself. (In ext3 a directory can have at most 32,000 subdirectories.)[16] To allow for larger directories and continued performance, ext4 in Linux 2.6.23 and later turns on HTree indices (a specialized version of a B-tree) by default, which allows directories up to approximately 10–12 million entries to be stored in the 2-level HTree index and 2 GB directory size limit for 4 KiB block size, depending on the filename length. In Linux 4.12 and later the largedir feature enabled a 3-level HTree and directory sizes over 2 GB, allowing approximately 6 billion entries in a single directory.

有兴趣的可继续翻翻具体怎么做的限定… …