I went the exercise of trying optimize all the drivers once, it took a very long time.
While it has been a long time (almost a plural, decades) since I've tinkered with optimizing a kernel for my own hardware, the sauce has not been worth the squeeze on modern hardware for a long time.
Now, I might modify something I compile to use all the cores I have available...