Will btrfs automatically compress existing files when compression is enabled?

Question

I chose btrfs as the format of my / filesystem in the Ubuntu 12.04 LTS installer. After the installation completed, I added compress=lzo to the mount options in /etc/fstab and rebooted.

Will the existing files be automatically compressed now, or must I explicitly do something to cause that to happen?

You don't have to reboot, you could just sudo mount -o remount / and the new fstab options will take effect. — sep332, Jan 16 '14 at 19:13
NB that you may need to use the compress-force option or some files will still remain uncompressed no matter what you do... — rogerdpack, Mar 12 '14 at 19:40
A note regarding compress-force. Normal compress has a feature in that if it detects that compression has little to no benefit, it doesn't bother continuing the compression for that particular file (thereby saving wasted CPU). By using compress-force, it attempts useless compression anyway.
Good examples where this applies are for files that are already compressed (multimedia, zip files, etc).

In other words, compress-force is generally a bad idea. ;) — zaTricky, Oct 15 '16 at 08:10

score 34 · Accepted Answer · edited Aug 20 '12 at 02:20

34

You will have to run btrfs fi defragment to force recompression of existing data. Otherwise, only new data will be compressed.

From the FAQ:

...consider remounting with -o compress, and either rewrite particular files in-place, or run btrfs fi defragment to recompress everything. This may take a while.

edited Aug 20 '12 at 02:20

ændrük

78,726

answered May 01 '12 at 17:28

Norbert Fabritius

364

1

btrfs fi defragment -r -clzo /path/to/fs – David Goodwin Jul 30 '15 at 14:25
3

What about the autodefrag mount option? Does that also make it compress previously uncompressed files when it auto-defrags? – Geremia Sep 20 '16 at 14:02
4

Make sure to use "-r" the recursive flag for compressing within directories. – Salami Oct 14 '16 at 17:30

score 10 · Answer 2 · edited Mar 08 '24 at 11:38

10

I've made what Norbert Fabritius said, but I didn't notice any compression in the existing files - df -h / before btrfs fi defragment = 658MB | df -h / after btrfs fi defragment = 658MB. New files are ok. Searching a little bring me this quote:

Running this:

# btrfs filesystem defragment ~/stuff

does not defragment the contents of the directory.

This is by design. btrfs fi defrag operates on the single filesystem object passed to it. This means that the command defragments just the metadata held by the directory object, and not the contents of the directory. If you want to defragment the contents of the directory, something like this would be more useful:

# find -xdev -type f -exec btrfs fi defrag '{}' \;

After this, my / it's occupping 656MB - nothing huge, but certainly there is compression.

Source: https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#Defragmenting_a_directory_doesn.27t_work

edited Mar 08 '24 at 11:38

Hi-Angel

4,915

answered May 17 '13 at 16:37

J. Neto

109

1
When using btrfs, do not use or trust df. 2. For a better compression ratio, use zlib (however, it's slower)

ignis

Nov 04 '13 at 12:00

4

Shouldn't your command line include defrag -clzo or defrag -czlib instead of just defrag, if you want to actually compress things? – jbo5112 Nov 17 '13 at 05:14

12

There's a "-r" recursive flag so you can just do this now: btrfs fi defrag -r -czlib ~/stuff – Salami Oct 14 '16 at 17:28

1

Just for reference, I got a 9.17 -> 5.89 GiB result with zstd:3 (default) on a fresh Ubuntu 22.04 install. Lovely. – martixy May 03 '23 at 07:29

Geoff · Answer 3 · 2013-10-03T13:56:37.970

According to Oracle's documentation, you can compress existing files on an existing, online filesystem by defragmenting each file in it with the -c, -clzo, or -czlib options. LZO is recommended for speed.

find / -xdev \( -type f -o -type d \) -exec btrfs filesystem defragment -v -clzo -- {} +

This uses the find command to run the btrfs defragmenter on every file in the root filesystem (given by the slash right after the "find" command at the beginning). If you have other subvolumes, you can use it again with the path of a subvolume (I have one at /home, for example) instead of the single slash.

You'll need root privileges for this, so add sudo to the front if you need to.

See:

Rostislav Stribrny · Answer 4 · 2013-03-16T20:42:12.960

According to https://btrfs.wiki.kernel.org/index.php/Mount_options#List_of_options following compression algorithms are available:


compress,compress-force
Enable compression. Starting with kernel 2.6.38 you can choose the algorithm for compression:

 - compress=zlib - Better compression ratio. It's the default and safe for olders kernels.
 - compress=lzo - Faster compression.
 - compress=no - Disables compression (starting with kernel 3.6). 

compress-force= - Enable compression even for files that don't compress well, like videos and dd images of disks. The options compress-force=zlib and compress-force=lzo works for kernels >2.6.38.

Note that old (before 2012) btrfs-progs versions will probably fail some operations (e.g. fsck) on filesystems with LZO compression.

Note: This post is answer for the additional question what type of compression algorithms are available which was asked below...so please don't blame me.

score 3 · Answer 5 · answered Mar 12 '15 at 02:58

According to ArchWiki:

Tip: Compression can also be enabled per-file without using the compress mount option; simply apply chattr +c to the file. When applied to directories, it will cause new files to be automatically compressed as they come.

Very nice! God bless BTRFS!

Also, from the BTRFS wiki:

Can I force compression on a file without using the compress mount option?

Yes. The utility chattr supports setting file attribute c that marks the inode to compress newly written data.

So the chattr method is for individual files while the btrfs filesystem defragment is for whole folders of files? — endolith, Jan 23 '20 at 22:17

Will btrfs automatically compress existing files when compression is enabled?

5 Answers5