If your hardware is fully compatible with the default kernel, then I see no reason at all why you should compile your own kernel. The performance benefits would probably be negligible, and it would be quite a hassle - you could even end up potentially breaking your installation. So yes, it would be quite a waste of time, and it wouldn't even be a good way to speed things up.
Reasons to compile your own kernel include: 1) fun, 2) learning experience, 3) sense of accomplishment.
But beware. I have a friend who won't use Linux. When I ask why, he won't answer. But I've assembled the details I've gleaned and I think he spent so much time compiling and messing with his Linux kernel that his wife put the law down.
With so many distros out there that already attempt to optimize the kernel for a handful of characteristics, it really doesn't make sense to build your own kernel if you are simply wanting to use Linux for a purpose. Just spend the time to find a well supported distro that has your goals in mind.
That said, it is invaluable for understanding the OS and there many situations in which this is useful, especially when you are new to OSs and system programming. Understanding different build parameters and devices is en excellent learning tool. If you are wanting to write device drivers, it is also very helpful.
Overall, just keep in mind the purpose you have for building the kernel and if it is mostly related to use of the system then definitely do not bother fighting with kernel builds. It does quickly eat up hours of your life.
I have only compiled my own kernel one time, and the reason was for space alone. Often the default kernel has tons of drivers and other software patches that bloat it, as well as slowing of loading time. If you do not plan on changing your hardware configuration for some time to come, and you have a pretty stable software set to boot, compiling your own kernel can allow you to save spave, increase loading time, as well as optimize settings so that it runs faster in general. If you are really wanting to look under the hood, then I would suggest going even further and building your own linux kernel from the ground up, using linux from scratch or similar process. Not only will you learn alot, but you will discover that there were often things that you did not even know you disliked in the default kernels or distros offered and this will allow you to tweak to your hearts content. While it might seem like one of the geekiest things a person can do, almost anyone that can follow instructions to a tee will be able to accomplish this is you take the time to read up on it an follow the instructions step by step.
Most often there is a reason for you to compile your own kernel. First, you might have some hardware not supported in your current kernel, forcing you to compile your own.
Second, you might want to compile your own kernel because the one you have has too much hardware support compiled in. As an example, if you have a system with only (E)IDE harddisks, you don't your kernel to support several SCSI controllers. The same goes for odd hardware like CDROM:s connected to soundblaster sound cards and other similar things.
Recompiling your kernel will make it smaller, taking upp less memory. On todays computers that's often not a problem, since they have enough memory anyway.
Security is another aspect. Sometimes some security related bug is found in the kernel code, and a fix is released as a patch or in the next kernel release. When you recompile your kernel your computer won't be vulnerable to that security problem any more.