• We did not send an email asking for donations - please read this post.

Sparse File Unpack turns into Infinite Data Suck

LinuxBirdFly

New Member
Joined
Sep 6, 2022
Messages
14
Reaction score
0
Credits
116
Sparse files are files that have holes or empty gaps in the data. They are often used for virtual machine images. For example, Sparse files could enable the storage of a 20G virtual machine disk in like 10G.



I’m trying to install software which involves a sparse image file. However, when I unzip/unpack it as per the documentation, it just continues to expand forever. So it will continue to be a larger and larger file, beyond 100G, and never actually finish unpacking. Here’s the command from the documentation:



Code:
tar -xvf File1*.libvirt.xz



When I asked for help from the software’s support, they told me it’s possible that I do not have proper support for sparse files on my Linux machine. However, when I test creating a sparse file with:



Code:
truncate -s 1g Test



It appears to work properly. Has anyone dealt with infinite Sparse files before?
 


dos2unix

Well-Known Member
Joined
May 3, 2019
Messages
1,571
Reaction score
1,191
Credits
10,043
tar -xvf File1*.libvirt.xz

This has little to do with the fact it's a sparse file.
This is also a compressed file. Like a zip file.

Compressed files can easily grow to 10x or more of their compressed size.
 
OP
LinuxBirdFly

LinuxBirdFly

New Member
Joined
Sep 6, 2022
Messages
14
Reaction score
0
Credits
116
This has little to do with the fact it's a sparse file.
This is also a compressed file. Like a zip file.

Compressed files can easily grow to 10x or more of their compressed size.
Is there any way to know the actual eventual real size? Because it just keeps growing infinitely

The creator of the file even acknowledges it should not be growing beyond 100GB
 
OP
LinuxBirdFly

LinuxBirdFly

New Member
Joined
Sep 6, 2022
Messages
14
Reaction score
0
Credits
116
This has little to do with the fact it's a sparse file.
This is also a compressed file. Like a zip file.

Compressed files can easily grow to 10x or more of their compressed size.
Do you still stand by your original statement if the file creator says it shouldn't grow beyond 100GB?
 

dos2unix

Well-Known Member
Joined
May 3, 2019
Messages
1,571
Reaction score
1,191
Credits
10,043
Do you still stand by your original statement if the file creator says it shouldn't grow beyond 100GB?

I don't recall stating that. It is possible that it could be larger, although unusual.
 

JasKinasis

Well-Known Member
Joined
Apr 25, 2017
Messages
1,570
Reaction score
2,232
Credits
11,611
Sounds to me like it might be a bug in whatever software you’re using to unpack the sparse files!
What software is this?

Otherwise, it might be related to the file-system you’re using.
Sparse files are only supported on certain file-systems. In Linux, sparse files are supported on ext4, btrfs, XFS and ZFS/openZFS.
If you’re using Windows file-systems like FAT, VFAT, FAT32, exFAT, NTFS - they’re unsupported.

Even in Windows, sparse files are only supported under NTFS, or ReFS. And even then, only under special circumstances! I think the file-system has to have specific properties/environment variables set in it.

So if your file is on an external drive that is formatted in one of the FAT variants, or NTFS. That would potentially be problematic.

Most Linux distros use ext4 by default, with some starting to use btrfs as their default fs. So if your file is stored on a drive that uses one of those file-systems - and you’re seeing this infinite growth, I would strongly suggest that there is a bug in the program you’re using to unpack the sparse files.
 
OP
LinuxBirdFly

LinuxBirdFly

New Member
Joined
Sep 6, 2022
Messages
14
Reaction score
0
Credits
116
Sounds to me like it might be a bug in whatever software you’re using to unpack the sparse files!
What software is this?

Otherwise, it might be related to the file-system you’re using.
Sparse files are only supported on certain file-systems. In Linux, sparse files are supported on ext4, btrfs, XFS and ZFS/openZFS.
If you’re using Windows file-systems like FAT, VFAT, FAT32, exFAT, NTFS - they’re unsupported.

Even in Windows, sparse files are only supported under NTFS, or ReFS. And even then, only under special circumstances! I think the file-system has to have specific properties/environment variables set in it.

So if your file is on an external drive that is formatted in one of the FAT variants, or NTFS. That would potentially be problematic.

Most Linux distros use ext4 by default, with some starting to use btrfs as their default fs. So if your file is stored on a drive that uses one of those file-systems - and you’re seeing this infinite growth, I would strongly suggest that there is a bug in the program you’re using to unpack the sparse files.
The software is Whonix for KVM

I tried it on my Linux Mint harddrive, and then an external drive. Isn't Mint ext4?

It's being unzipped via tar command which is the default archive manager right? so I should reinstall that?
 

JasKinasis

Well-Known Member
Joined
Apr 25, 2017
Messages
1,570
Reaction score
2,232
Credits
11,611
Looking at this 2015 thread on the whonix website's forum:

It seems that this is nothing new. It's not a bug, it's normal.

It seems that during extraction, tar will report that it is extracting huge amounts of data. So the "apparent" size of the extracted images is over 100G. But if you check the actual, physical size of the file - it's only 2.5G. Or at least, that was the case back then. The actual sizes of the disk images for the current version might be completely differrent now.

But it seems that when extracting these sparse files, tar and the file-manager will report the sizes of the files as many times larger than they actually are.

So I'd advise you to just let the extraction finish and don't worry about it! The physical files are nowhere near as big as the system is reporting. What you are seeing is the "apparent" size of the sparse file, NOT its actual size!

This wiki page tells you how to check the "real" size of the disk image and its "apparent" size:
 
$100 Digital Ocean Credit
Get a free VM to test out Linux!

Linux.org Hosting Donations
Consider making a donation

Members online

No members online now.

Top