How (and specifically from where) to prune Timeshift backups of large unwanted items (e.g., 400GB Steam folders)?

[MariusContusor]

As Timeshift (which is the only reason I can still use Arch with my limited expertise) doesn't allow you to specify folders or subvolumes to exclude, I run into the problem of an 80GB Steam download effectively becoming 160GB or so once it's backed up. I can always re-download a game, but this double-whammy of disk usage is becoming untenable. I am using BTRFS Timeshift.

Therefore, I would like to go into the (on my system) /run/timeshift/backup/@home/<user>/.local/share/Steam folder and remove some things.

So, here are my questions:

....A) If I were to go into said timeshifts backup folder (i.e., not the individual snapshots folder) and delete the several hundred GB of games in there, what would be the effect?
........A.1) Were I to restore, would my backups still restore the rest of my system, excluding Steam (that is, leaving it alone, OR: )?
........A.2) Would it cause some sort of journaling crisis and result in the deletion of my active Steam folder upon a restore?
........A.3) Is this even the right location, or should it be from the individual snapshots (e.g., timeshift/backup/timeshift-btrfs/snapshots/2021-07-18_18-00-01/@home/<user>/.local/share/Steam?
....B) Following up on A.3, can anyone explain the process Timeshift uses on BTRFS, such that I can feel comfortable in pruning these backups?

Doing a 'du' shows that both locations (i.e., timeshift/backup/@home vs timeshift/backup/timeshift-btrfs/snapshots/<timestamp>/@home...) contain the same files, of the same size. I'm not sure what kind of black-magic directory linking is going on, considering all the duplication, but then again, I'm far from competent in knowledge vis-a-vis BTRFS and Timeshift, so here we are.

I would be very appreciative for someone to answer the above questions and clear up for me the process of pruning things that should not be backed up. Otherwise, 1TB sort of becomes 500GB with Timeshift-BTRFS. Thank you!

 

[f33dm3bits]

The better option would be to set excludes and then create a new snapshot and remove the old ones with you Steam folder. For example Steamn folder I have under /mnt/games/Steam and this is what my excludes look like.

/mnt/**
/var/lib/libvirt/images/**

Just open the Timeshift interface and go to the "Filters" tab and then you can create a new exclude filter for the location of your Steam directory. I'm guessing you still have the default location set so you could do it like this, I'll give two examples and then you can use what you want. Also Timeshift is not a backup utility but a restore to a point in time tool for when your system breaks after an upgrade so it's not actually recommended by the developers to backup anything in your home directory or other user data. It should by default exclude your home directory so the only way that it's backing up your home directory is if you included it yourself. If you have configure Time shift to include your user's data you can configure excludes.

/home/tux/.steam/**
/home/tux/.steam/root/steamapps/common/**

Getting back to the two examples I was talking about, the First one is the default location of the default root Steam directory and the second one is the location of the default directory where all the Steam games are downloaded. Hard-linking is used so this is also the reason why I wouldn't manually deleted any of the directory structure in Timeshift because it won't remove the file or directory, I won't explain that since that's not the topic. After you have created the new exclude remove your old snapshots where the Steam folder is still included to so that you have more space and then create a new snapshot and your Steam directory will be excluded.

Lastly, I'm not using btrfs but since you are it could be that Timeshift uses the btrfs snapshot functionality so I don't know in what way btrfs creates snapshots and how Timeshift uses that. See the Timeshift topic @wizardfromoz from created with more details and help when it comes to using Timeshift.

 

[Condobloke]

G'day MariusContusor, Welcome to Linux.org

I am assuming from your questions that Timeshift is being stored on your main HD (or SSD)....is that correct?

 

[MariusContusor]

G'day MariusContusor, Welcome to Linux.org

I am assuming from your questions that Timeshift is being stored on your main HD (or SSD)....is that correct?

That is correct. Technically, it's a laptop on which I've got Garuda/Arch installed on an external USB3.1 NVME, with my internal NVME/HDD mounted in fstab to be able to transfer files from the windows installation. But yes, as far as Linux is concerned, the root folder and boot partition/etc are booted from the external, as is the /run/timeshift/ folder. They coexist on the same BTRFS partition, and I'm using BTRFS Timeshift, not RSYNC.

Filesystem     1K-blocks      Used Available Use% Mounted on
dev             16338368         0  16338368   0% /dev
run             16374612      2224  16372388   1% /run
/dev/sdb2      940435452 341141908 598548860  37% /
tmpfs           16374612    308524  16066088   2% /dev/shm
/dev/sdb2      940435452 341141908 598548860  37% /srv
/dev/sdb2      940435452 341141908 598548860  37% /home
/dev/sdb2      940435452 341141908 598548860  37% /root
/dev/sdb2      940435452 341141908 598548860  37% /var/tmp
/dev/sdb2      940435452 341141908 598548860  37% /var/log
/dev/sdb2      940435452 341141908 598548860  37% /var/cache
/dev/nvme0n1p3 499117140 391685232 107431908  79% /winvme
tmpfs           16374612       568  16374044   1% /home/guest
tmpfs           16374612     12172  16362440   1% /tmp
/dev/sdb1         262126       562    261564   1% /boot/efi
/dev/sda1      976760828 905154936  71605892  93% /winstorage
/dev/loop0        453376    453376         0 100% /var/lib/anbox/rootfs
tmpfs            3274920     23600   3251320   1% /run/user/1000
/dev/sdb2      940435452 341141908 598548860  37% /run/timeshift/backup

Everything I'm concerned about here is on /dev/sdb2/, from which the system is booting.

Question: Even if I look up how to put Steam in a different subvolume, Timeshift-BTRFS won't allow me to exclude said subvolume, would it? That is, in response to the post from f33dm3bits, would subvoluming even be useful when using the BTRFS variant of Timeshift? And again, if not, I need to know how to best remove unwanted backed up data from A) at least the existing backups, and B) if possible, from future backups. If I weren't using crash-prone KDE Plasma on Garuda of all things, perhaps the @User subvolume and even Timeshift itself wouldn't be as important, but one update without a mirrorlist refresh is all it takes to brick the install.

But yeah, my options in Timeshift for BTRFS only includes options for including the @home subvolume (where my user-files-I-want-to-save as well as my Steam-folder-I-want-to-omit are located), and BTRFS qgroups (which I don't currently understand).

Thank y'all for your assistance!

 

[f33dm3bits]

Question: Even if I look up how to put Steam in a different subvolume, Timeshift-BTRFS won't allow me to exclude said subvolume, would it? That is, in response to the post from f33dm3bits, would subvoluming even be useful when using the BTRFS variant of Timeshift? And again, if not, I need to know how to best remove unwanted backed up data from A) at least the existing backups, and B) if possible, from future backups. If I weren't using crash-prone KDE Plasma on Garuda of all things, perhaps the @User subvolume and even Timeshift itself wouldn't be as important, but one update without a mirrorlist refresh is all it takes to brick the install.

But yeah, my options in Timeshift for BTRFS only includes options for including the @home subvolume (where my user-files-I-want-to-save as well as my Steam-folder-I-want-to-omit are located), and BTRFS qgroups (which I don't currently understand).

Thank y'all for your assistance!

I have not used btrfs so I can't tell, what I do know is that no matter what filesystem you do use you should be able to include or exclude locations of which you want in your snapshot. As mentioned before Timeshift is NOT a backup a utility but restore to a point in time tool, you take a snapshot and if your system breaks after an update you restore your system to a snapshot.

Timeshift is similar to applications like rsnapshot, BackInTime and TimeVault but with different goals. It is designed to protect only system files and settings. User files such as documents, pictures and music are excluded. This ensures that your files remains unchanged when you restore your system to an earlier date. If you need a tool to backup your documents and files please take a look at the excellent BackInTime application which is more configurable and provides options for saving user files.

See the Timeshift documentation, also AFAIK btrfs already has a snapshot functionality so technically you don't need Timeshift to make snapshots when using btrfs. I was wrong about the part when it comes to Timeshift using hard-links, that is only when used in rsync mode.

In RSYNC mode, snapshots are taken using rsync and hard-links. Common files are shared between snapshots which saves disk space. Each snapshot is a full system backup that can be browsed with a file manager.

So maybe there is a way to remove directories from current snapshots when using btrfs, coming back to that Timeshift is a restore to a point back in time tool to restore a broken system to a working system. If your system is currently working it won't be a problem to remove your previous snapshots, you can create an exclude directory and then create a new snapshot so that you have a fresh snapshot without the unwanted files to which you can restore your system to. See the attached screenshot so you can see how I excluded specific directories from being included in my snapshots.

 

[wizardfromoz]

As Timeshift (which is the only reason I can still use Arch with my limited expertise) doesn't allow you to specify folders or subvolumes to exclude,

It does actually, but only with the EXT4-RSYNC version, not BTRFS version.

I will have more information in my Timeshift thread, on my tomorrow (DownUnder).

https://www.linux.org/threads/timeshift-similar-solutions-safeguard-recover-your-linux.15241/

G'day @MariusContusor and welcome to linux.org

Nice work @f33dm3bits

Other Helpers take note that usual solutions for Timeshift will not work in the OP's situation.

More tomorrow

Chris Turner
wizardfromoz

 

[Tolkem]

Why not snapper? It does exclude home by default and configuration is much more flexible, it is just a txt.conf file. IMHO, on BTRFS, snapper is the tool you want to use, and timeshift for EXT4. https://wiki.archlinux.org/title/Snapper