The problem here is to do with USB read/write speeds. The read/write speeds on USB devices vary wildly from device to device. But in all cases, their write speeds are much slower than their read speeds. Especially with USB1 and USB2 devices. I think the write speeds are a bit better with USB3 devices, but only when plugged into USB3 ports.
Without persistence, data is only ever read from the USB drive, so it is blazingly fast.
As soon as you add persistence and the OS starts writing to USB, that's when things slow down. If you encrypt the persistent volume, it will slow it down even more (thanks to the performance hit you'll take from the on-the-fly encryption/decryption of data.)
If you want to use a persistent USB install, this is something you have to live with. When the system is undergoing heavy disk usage and there are a lot of simultaneous reads and writes going on, things will inevitably slow down. Shockingly so at times. You just have to live with it. Basically, under heavy disk use, programs start hanging while they wait for their data to be written to disk. Or while they are waiting to read from the disk. When this happens, it all seems like it's the CPU that's being thrashed, or that you are running out of memory, but in fact it's just programs waiting to read from/write to disk.
The only real form of mitigation would be to try to find a USB device with faster read/write speeds to install Linux onto - assuming that your USB ports are able to use the device at those speeds too!
Ultimately, the speed of the USB device and the supported speeds that your USB ports can use are the biggest factors.
You could also try using a distro with a less resource hungry wm/desktop environment, that might help. In the past, I've found that Crunchbang worked really well on USB. I used to use that via a 'live with persistence' style install and also with a more traditional 'HD style' install on my USB thumbdrives back when the HD on my old laptop died.
But sadly Crunchbang is no more. Since Phil/Corenomial announced the end of the Crunchbang project, some of the Crunchbang community have started working on forks/alternatives. But I just haven't had time to check any of them out yet. So I can't really comment on them. I'm sure the others here will be able to point you towards some slightly lighter distros that work well on USB thumbdrives.
I did also have Arch installed on another USB drive with dwm as the wm. That worked really well too. But Arch is more of an advanced distro. You might want to take a look at Arch when you have a bit more experience using Linux. Installing Arch is an education in itself and will teach you a lot about Linux.
In all cases, regardless of the distro: When using a USB install - If it writes to disk, whether it be 'Live with persistence' or a traditional HD style install - it will be slower than a pure Live installation with no persistence. It's just a limitation of using USB!