search by tags

for the user

adventures into the land of the command line

open file limits for nginx and rabbitmq scale

OS

There are separate limits for different users, so make sure to run this as the user your process is using if you want to check.

$ ulimit -n
1024

There is a soft limit and a hard limit.

The soft limit is the actual limit your processes have to obey.

$ ulimit -Sn
1024

The hard limit is the maximum number the soft limit can be set to.

$ ulimit -Hn
1024

To set the open file limit for the current active shell session (lost on logout)

$ ulimit -n 4096
$ ulimit -n
4096

To set the open file limit permanently

$ vim /etc/security/limits.conf

*               soft    nofile         4096
*               hard    nofile         4096

-- or for a specific user --

root            soft    nofile         4096
root            hard    nofile         4096

You need to then reboot the machine for it to take effect.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15720
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096        <------------------- we changed this one
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15720
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

NGINX

There are three options you can focus on here.

worker_processes = should be equal to the number of cpu cores available on the machine
worker_connections = important for the number of concurrent active connections, default is 1024
worker_rlimit_nofile = represents the number of file handles for each connection, x1 for a file server, x2 if the connection is being proxied, ie, one in and one out eg, if you expect to proxy 100 connections, this number would need to be 200

These options should be set in the main nginx config file.

$ vim /etc/nginx/nginx.conf

.
.
worker_processes auto;
.
.

events {
    worker_connections 4096;
    .
    .
}

worker_rlimit_nofile 4096;

http {
    .
    .
    .
}

Check the nginx process’s limits to see that they reflect the changes on the os.

$ ps -ef | grep nginx
root       1556      1  0 Jan04 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data   1557   1556  0 Jan04 ?        00:00:08 nginx: worker process                           
www-data   1558   1556  0 Jan04 ?        00:00:09 nginx: worker process                           
www-data   1560   1556  0 Jan04 ?        00:00:00 nginx: cache manager process                    
root      61143  60933  0 15:15 pts/0    00:00:00 grep --color=auto nginx

The worker processes are the important ones to check as…

$ cat /proc/1557/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             15720                15720                processes
Max open files            4096                 4096                 files     <--------------------
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       15720                15720                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us  

…the master process is not the one opening and closing file handles, but managing the worker processes

$ cat /proc/1556/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             15720                15720                processes
Max open files            1024                 4096                 files     <--------------------
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       15720                15720                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us  

RABBITMQ

In addition to the os nolimit value, we also need to add this:

$ vim /etc/default/rabbitmq-server

ulimit -n 512000

Check the rabbitmq process limits as well.

$ ps -ef | grep rabbit
rabbitmq   2002      1  0 Jan04 ?        00:00:00 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq   2037      1  0 Jan04 ?        00:00:00 /bin/sh -e /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq   2358   2037  1 Jan04 ?        00:25:56 /usr/lib/erlang/erts-5.10.4/bin/beam -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl ...
rabbitmq   2742   2358  0 Jan04 ?        00:00:01 inet_gethost 4
rabbitmq   2743   2742  0 Jan04 ?        00:00:01 inet_gethost 4
rabbitmq   5070   2742  0 Jan04 ?        00:00:00 inet_gethost 4
root      38633  38610  0 15:18 pts/0    00:00:00 grep --color=auto rabbit

$ cat /proc/2002/limits
$ cat /proc/2037/limits
$ cat /proc/2358/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             7656                 7656                 processes
Max open files            512000               512000               files     <--------------------
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       7656                 7656                 signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us