We are running a heavy service on Debian. Our service generates lots of file i/o and its performance relies on how well Linux kernel caches files.
Our system has 64GB memory. Total file size is 500GB (lots of small files). These files are served via reiserfs on raid. The kernel is 3.2.9 now. (this is not a new issue. We have this issue for several years.)
Usually, most of file content are cached and our system runs smoothly. free(1) command shows 14G memory cache, 10G memory buffer, and 8G memory are free.
However, in our peak hour, I found linux kernel released most of cache/buffer and kept them in free state. In my opinion, it is a huge waste to keeps memory in free state and I guess it is the reason that our service overloads in busy hours. free(1) command show something like 2G cache, 2G buffer, and 28G free.
My question is, in what situation linux kernel want to release memory instead of caching/buffering. And how to avoid this behavior? Is there any configuration we can tune?
(Let me know if you need more information about our hardware/system/service)
Is it possible to put anything in a ramdisk? Explicit cache if you will. Not sure if what you're doing suits this, and you have probably thought aboout it, but i'll throw it out there ;)
Have a look here: http://www.linuxinsight.com/proc_sys_vm_hierarchy.html
there are a few bits you can tune easily, just by "cat"ing values in. vfs_cache_pressure & zone_reclaim_mode jumps out, caveat: only one i have ever tuned is swappiness.
There are two keys reasons this would happen.
One, that I hope you can rule out, is some person or script commanding it to do so. For example, someone sending a command to
The other is if that memory is required for some other purpose. If an application requires a lot of memory, the kernel will give the application that memory, reducing the cache as needed. When the application is done with that memory, it will become free. It will only fill back up with cached data over time.
There may be a few things you can do if this is the case. For example, if the memory usage is happening because you're trying to process four requests at once, you may wish to reconfigure to do two requests at once maximum. If this lets you keep more data in cache, it may be a performance win overall.
free memory is probably caused by one or more short lived processes that require lots of memory. Linux kernel will use free memory for caches but those caches will be dropped instead of using swap in case some process acquires more memory even for a short period. From the numbers you provided it seems that the system will see an extra 20 GB RAM load during your peak hours and that is too much for your system because caches get too small.
I would try logging running processes near the time you're seeing your caches dropping (for example, log
date; free -m; ps auxw every 5 seconds to a file until you see the problem).