fuzzix.org

Cross-platform File Systems

On a previous blog o' mine I wrote up a short analysis on filesystem cross-platform compatibility and large file support ("cross-platform" in this instance meaning Linux, Windows and MacOS (sorry schrodinger)).

The conclusion of this piece was "UDF FTW", apart from the trifling issue of the Linux kernel truncating the bitmap on large UDF volumes. You can see the table from the original post below - (-ish) means supported via a FUSE module included with the Linux kernel, so works out-of-the-box but is not performant.

	Large Files	Linux	MacOS	Windows XP	Windows 7+
FAT	No	Native	Native	Native	Native
NTFS	Yes	Native(-ish)	Third-party	Native	Native
EXT 2/3/4	Yes	Native	Third-party	Third-party	Third-party
UDF	Yes	Native	Native	Native, read-only	Native

Since that post, the UDF filesystem module in the Linux kernel has been patched : udf: Fix bitmap overflow on large filesystems with small block size. Time to try this again on the ol' AMD fan rattler:

fuzzix@fatboy:~$ uname -a
Linux fatboy 3.9.7 #2 SMP Sat Jun 22 02:43:26 CDT 2013 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 6000+ AuthenticAMD GNU/Linux

The last time I tried this it was on a 160G disk. I only have a 1TB volume to hand right now - it will have to suffice. So, we've created a type 06 (FAT16) partition on the disk and formatted it with:

root@fatboy:~# mkudffs --media-type=hd --blocksize=512 /dev/sdb1
root@fatboy:~# df /dev/sdb1
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sdb1      976762287   238470 976523818   1% /mnt/usb0

Few hundred MB used for the UDF bitmap - not the most efficient FS, but not bad at this scale. Once mounted, the fun begins... Let's create 15 50G files...

root@fatboy:~# for i in `seq 1 15` ; do dd if=/dev/zero of=/mnt/usb0/tmp${i} bs=104857600 count=500 ; done

Hours later (fallocate not supported)...

root@fatboy:~# ls -lh /mnt/usb0/tmp* | awk '{print $5, $9}'
49G /mnt/usb0/tmp1
49G /mnt/usb0/tmp10
49G /mnt/usb0/tmp11
49G /mnt/usb0/tmp12
49G /mnt/usb0/tmp13
49G /mnt/usb0/tmp14
49G /mnt/usb0/tmp15
49G /mnt/usb0/tmp2
49G /mnt/usb0/tmp3
49G /mnt/usb0/tmp4
49G /mnt/usb0/tmp5
49G /mnt/usb0/tmp6
49G /mnt/usb0/tmp7
49G /mnt/usb0/tmp8
49G /mnt/usb0/tmp9

Erm, 49G files - close enough! This is much further than we got last time - no longer able to write anything to the disk after 22G had been filled. We can see now that we've managed to fill a decent chunk of a 1TB volume:

# df -h /dev/sdb1 
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       932G  733G  199G  79% /mnt/usb0

Success! So in conclusion, I refer you to my original conclusion, "UDF FTW". UDF supports large files, works out of the box on (recent) Linux, MacOS X and Windows Vista+.

by fuzzix on Fri, 26 Jul 2013 09:59.

Comments

fuzzix
Fri, 26 Jul 2013 11:03

Should point out, the small block size used in creating the UDF fs was an element of the "stress test" here.

Comment on this post

Text only, no HTML, * denotes a required field.

Jaws was never my scene and I don't like Star Wars

Cross-platform File Systems

Comments

Comment on this post