Samsung 8x0 EVO and Queued TRIM. Is it a big deal?

[Atlantean]

I have not one but two 860 EVO drives, and I guess many of you use drives of the 8x0 line too. I came across this piece of news:

Samsung 860/870 SSDs Continue Causing Problems For Linux Users - Phoronix

www.phoronix.com

Now the questions are: What is queued trim exactly? And is this a big deal or a nothingburger?

When I run a search for queued trim all the relevant results I get are related to this particular issue.

Edit: Also found this, though it's too technical for me:

201693 – Samsung 860 EVO NCQ Issue with AMD SATA Controller

bugzilla.kernel.org

 

[KGIII]

Queued TRIM is TRIM that's set to run at certain intervals or on certain events. There's continuous TRIM which is very wasteful of resources as it runs TRIM constantly, like on every single file creation or every single delete action performed on the drive. There are

So, obviously queued TRIM is better.

However, with modern SSDs you don't need to run TRIM all that often. I probably reboot every 15 days and see the TRIM in my boot log for most of those.

Now, as I understand it, Samsung tries to incorporate TRIM into the software (yes, there's software in your SSD - assuming you didn't know) to do wear-leveling and other drive maintenance operations like blocking bad sectors. If I'm understanding correctly, some versions of this software - with specific models, is incompatible with Linux. It leads to the drive not handling requests properly and thus I/O bottlenecks or even errors.

If I had to guess, my guess would be that Samsung's not supporting the various ISO standards for this - but I couldn't name those standards offhand. (We could probably look them up, but we're just laypersons.) Linux, presumably, adheres to those standards.

Other Samsung drives are just fine. They're not trying to do their own TRIM functions.

So, it's a case of a feature that's poorly implemented and harmful on Linux. Thus the kernel will now blacklist those particular drives. You can still use them, but you can expect poor performance - probably in the form of degredation over time. I'm not sure how you can resolve this - perhaps check if you can manually run TRIM and, if so, schedule it with CRON? I really don't know and that's way above my pay grade.

Maybe try running 'fstrim' if you notice slowdown and hope for the best?

Heck, I may not even be understanding it correctly - but I'm pretty sure I have a handle on it.

If asked for general advice concerning this, I'd say consider not using Samsung SSDs until they get their crap squared away. This isn't the first time they've had issues. It was a rather rough start for SSDs in general, but the kernel devs have a lot of test results at their disposal and can kinda customize based on the drive (that's the blacklisting thing they do for some SSDs, where certain features aren't available).

I am not affiliated - but I've had AMAZING results with a fairly new company that's also obscenely inexpensive. That company is TeamGroup. I've now used a number of their products and have been nothing but impressed by them.

 

[Atlantean]

Maybe try running 'fstrim' if you notice slowdown and hope for the best?

What I think I gathered from the little further research I did:

Queued TRIM is indeed the one that takes place constantly, it is so called because the TRIM instructions are "queued" between other read/write instructions given to the drive during normal use. The alternative to that is one-off or scheduled TRIM, which requires all read/write activity to be temporarily suspended to take place.

I did run sudo fstrim --verbose --all in my machine and was flabbergasted to see a whopping 700 GB being "trimmed" off my disk.

Later I learned the following: in encrypted disks all TRIM seems disabled by default. Trimming the disk might even decrease the security provided by encryption by revealing all blank sectors to an eventual attacker. Without knowing this, I rebooted the system and ran trhe fstrim command again, and the same 700GB were trimmed. I don't know if by "decrypting" the disk again at reboot all the blank space is again marked as non-blank encrypted data?

As to the issue itself: it affects primarily machines with older ATI/AMD SATA controllers. For those machines the only way to mitigate the issue is to disable native command queuing (NCQ) for all disk activity (not only TRIM), and that has a significant impact on performance (that much is in the article itself). Other machines with SATA controllers by other manufacturers are also affected, but for those it suffices to disable NCQ trimming, with minimal impact on performance. With NCQ trimming disabled the alternative seems really to be to schedule TRIM with CRON once a week or so. For me, however, that may be moot, as I am running an encrypted disk and might not be able to benefit from TRIM anyway.

I believe there are other vendors offering quality storage, but overall I find this issue to be a pity cause otherwise Samsung SSDs are often described as incredibly performant and durable.

 

[KGIII]

I believe there are other vendors offering quality storage, but overall I find this issue to be a pity cause otherwise Samsung SSDs are often described as incredibly performant and durable.

If there do turn out to be issues, it'd be cool if you'd add to this in the future so that others can learn from it.

Also, I did not know that about encrypted drives and TRIM. That's entirely new to me.

flabbergasted to see a whopping 700 GB being "trimmed" off my disk.

That'd probably make me pee my pants a little. Well, not too much - 'cause I have backups! But, still...

 

[Atlantean]

If there do turn out to be issues, it'd be cool if you'd add to this in the future so that others can learn from it.

I'll continue to use my disks normally. If I run into any issue and learn more while troubleshooting I'll make sure to post it here.

That'd probably make me pee my pants a little. Well, not too much - 'cause I have backups! But, still...

What really made me almost pee my pants were the 500Mb or so trimmed from boot/efi and another 400Mb or so from /boot, as I thought it had deleted the bootloader! I have backups too, but was not to thrilled about the prospect of doing a full reinstall or attempting a bootloader repair (still have to learn how to perform those)

 

[KGIII]

What really made me almost pee my pants were the 500Mb or so trimmed from boot/efi and another 400Mb or so from /boot, as I thought it had deleted the bootloader!

I've never seen quite anything like that.

I did watch the results of an employee who mistakenly de-duped the production database (about a terabyte in size in like the year 2000) instead of doing it to staging and pushing the results to production as scheduled.

At least a dozen of us watched that one...

As memory serves, said employee lost the credentials to production.

 

[wizardfromoz]

What really made me almost pee my pants were the 500Mb or so trimmed from boot/efi...

I promise not to pee my pants, but @Atlantean do you remember what size the ESP was to start with?

Cheers

Wiz

 

[Atlantean]

I've never seen quite anything like that.

I promise not to pee my pants, but @Atlantean do you remember what size the ESP was to start with?

sda1 was (and is) a FAT partition 537Mb in size (and only 1.2% full). That's on boot/efi and it's a EFI system partition.

sda2 is a Ext4 partition 768Mb in size with 491Mb free. That's in /boot and I guess that's where grub is.

sda3 is the main partition with nearly 800Gb free. This one is encrypted with LUKS.

Why sda1 and sda2 seem much bigger than they need to be I don't know. That's the Ubuntu installer's doing.

When I ran the fstrim command the first time I got these results:

/boot/efi: 505.8 MiB (530321408 bytes) trimmed on /dev/sda1
/boot: 466 MiB (488570880 bytes) trimmed on /dev/sda2
/: 744.9 GiB (799815979008 bytes) trimmed on /dev/mapper/vgubuntu-root

I reckoned it was just "trimming" a bunch of empty sectors in every partition. When I ran the command a second time it trimmed zero. So I thought "job done" despite the oddness. But when I ran the command a third time after a reboot and entering my encryption password once again I got the exact same results as above, to the dot. So I figure it has something to do with my disk's encryption, but if you ask me what exactly the command does when I run it I would be at a loss to explain it.

 

[KGIII]

I did some research last night and then some reading again today. I'm not sure where to send you, but I'd advise you to search for the following phrase:

rd.luks.options=discard

That's an option/kernel boot parameter available for those who want to use TRIM on an encrypted SSD.

Then again, the author of LUKS is pretty sure you shouldn't use TRIM on an encrypted drive at all. Though that article is more than a decade old and this above boot parameter is much newer.

TRIM & dm-crypt ... problems?

Some SSDs (Solid State Drive) implement TRIM command which is used to inform the disk (block device) that sectors are no longer used. If you...

asalor.blogspot.com

I like it when I see a post that like your post.

 

[Atlantean]

After running a search for that parameter and reading the article you posted it seems that I did no greater harm to my disk by running the fstrim command than reducing the security of the encryption by a little. If that!

That parameter is added to grub in order to allow discard operations such as TRIM in a LUKS encrypted drive. I checked the grub file and the line that should carry that parameter says only:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

There is no reference to a "cryptdevice" as in some examples I've seen.

Then there is the fact that in my system discard seems to be allowed despite the absence of that parameter. fstrim, for instance, did not return an error message, but seemed to work as intended. Seemed. This article I found by searching the parameter you mentioned also includes another method of verifying it:

https://confluence.jaytaala.com/display/TKB/Enable+periodic+TRIM+-+including+on+a+LUKS+partition

This is by running

lsblk --discard

I ran that and got this very weird result:

According to the artlicle non-zero entries in the DISC-GRAN and DISC-MAX columns mean that discard is already allowed. Now I also have those 13 "loops", and I have no idea what they are. And the mystery remains as to why I have so much to "trim" after a reboot. I wonder if the command trims anything or if ends up being just a "dry run".

I thought of updating the kernel to see if it makes any difference with the changes implemented for those Samsung drives, namely the blacklisting for NCQ trimming, but I guess I'll wait for the release of the 22.04 LTS in April with a new kernel. Perhaps I should follow the old adage, "if it ain't broken..."

 

[KGIII]

I'd agree with your choice - waiting for the next LTS and kernel.

Thanks for the information - this is mostly new stuff to me. I know what TRIM is and what it does, but never considered the ramifications of running it on an encrypted drive - nor of an SSD vendor that was still getting blacklisted. (As mentioned in a previous post, SSDs were pretty touch and go in Linux at first. I was a pretty early adopter.)

 

[wizardfromoz]

Now I also have those 13 "loops", and I have no idea what they are.

Loop devices are typical to, but not confined to, Ubuntu, and if you Google

ubuntu loop devices

you can find some reading. They are often, nowadays, related to the installation of Snap packages.

Wiz