I'm an experienced UNIX/Linux engineer with no real modern DB experience; other than they run on my servers.
The environment I'm running is a mixture of Ubuntu and RHEL. I was hoping for a more generalized discussion and solution.
I'm running into a growing number of issues where jobs are being submitted to app or DB servers, where the jobs for similar functions of differing sizes. The larger jobs (time and resources to process) are monopolizing resources and causing smaller, quicker jobs to back up in the runnable queue.
I'm exploring load balancer algorithms to better manage this. There are cases where LBs are not feasible, at least for the current architectures. An example of the problem is that you have web/app servers submitting jobs. Transaction A is processing a small number of records in a DB. Transaction B is a more complex and much larger task (creating a report) using the same DB.
I'm seeing situations where the larger job is getting more resources, causing the smaller jobs to stall, the run queue grows, eventually causing the server to thrash.
The applications are typically commercial products, so changing the application may be difficult. I'm looking for a way to control this at the OS layer? These are jobs submitted by web or app servers, so using nice(1) on larger jobs seems too simple and would need something to predict that the job is going to be a resource hog. Would using Cgroups help limit the resources used by any one job? Any suggestions for other methods to explore?
Thanks in advance
The environment I'm running is a mixture of Ubuntu and RHEL. I was hoping for a more generalized discussion and solution.
I'm running into a growing number of issues where jobs are being submitted to app or DB servers, where the jobs for similar functions of differing sizes. The larger jobs (time and resources to process) are monopolizing resources and causing smaller, quicker jobs to back up in the runnable queue.
I'm exploring load balancer algorithms to better manage this. There are cases where LBs are not feasible, at least for the current architectures. An example of the problem is that you have web/app servers submitting jobs. Transaction A is processing a small number of records in a DB. Transaction B is a more complex and much larger task (creating a report) using the same DB.
I'm seeing situations where the larger job is getting more resources, causing the smaller jobs to stall, the run queue grows, eventually causing the server to thrash.
The applications are typically commercial products, so changing the application may be difficult. I'm looking for a way to control this at the OS layer? These are jobs submitted by web or app servers, so using nice(1) on larger jobs seems too simple and would need something to predict that the job is going to be a resource hog. Would using Cgroups help limit the resources used by any one job? Any suggestions for other methods to explore?
Thanks in advance
