windows no idea, but others in the forum might be able to answer that.
To check an ubuntu server for malware running on said ubuntu server: rkhunter, chkrootkit
This topic isn't that popular in Linux because its compliance, not security. We (Linux nerds) generally prefer to setup systems securely so they don't get hacked in the first place. Once an attacker is inside your system, it is extremely easy to avoid detection by any kind of scanner - its not about the scanners quality, its just not possible.
What we do is IDS - intrusion detection systems. That means, for example, analyzing the network traffic of a specific server to see if somebody is sending data to a weird destination, to make one simple example. Those run agents on the host as well, however the first thing you would do after becoming root on a host would be to disable that.
That topic is a bit complex.
If you are unfamiliar with Linux in general, and you want to "install something to check if your box got hacked" and be done with it, that doesn't exist (it doesn't exist for windows either - that whole concept is called "compliance instead of security").
You can become a linux expert and install + configure + maintain a SIEM like splunk, wazuh, surricata or similar. But thats actual work
PS: kali is for pentesting, as in trying to attack your own servers to see if they can be exploited - which you can automate as well of course.
PPS: If you are not a bank just configure your ubuntu server reasonably secure and you will be ok in 99% of the cases. We are
currently working on a thread that describes how to do that.