Wilson Mar bio photo

Wilson Mar

Hello. Join me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

Set this higher to prevent errors during high load


Overview

Operating systems (Linux and macOS included) have settings which limit the number of files and processes that are allowed to be open. This limit protects the system from being overrun. But its default is usually set too low, when machines had way less power. Thus a “gotcha” that is only apparent when “too many files open” crashes appear only under load (as in during a stress test or production spike).

man bash, the Bash manual says the ulimit command (common among Linux flavors) provides “control over the resources available to the shell and to processes started by it”.

  1. Obtain the current limit of file descriptors

    ulimit -n

    An example output: “256” or “10032”.

    PROTIP: On MacOS, the maximum number that can be specified is 12288.

  2. Obtain the current limit of processes

    ulimit -u

    An example output: “1418”.

On macOS

  1. Obtain the current limit:

    launchctl limit maxfiles

    The response output should have numbers like this:

    maxfiles    65536          200000

    The first number is the “soft” limit and the second number is the “hard” limit.

    Configuration changes are necessary if lower numbers are displayed, such as:

    maxfiles    256            unlimited
  2. If the soft limit is too low (such as 256), set the current session to:

    sudo launchctl limit maxfiles 65536 200000

    Some set it to 1048576 (over a million).

    Since sudo is needed, you are prompted for a password.

    PROTIP: Because this would go back to defaults on reboot, add this command in your ~/.bash_profile

    Alternately, install Facebook’s Watchman utility which watches and adjusts automatically.

    PROTIP: Take both a full/complete backup to ensure fall-back. Also run a benchmark performance measurement before and after changing the configuration to detect issues.

  3. Over several versions, Apple has been changing the way system-wide open file limits can be set upon restart.

    Yosemite and older

    NOTE: On older MacOSX Yosemite, to make the settings permanent, increase the limits, edit (or create) file /etc/launchd.conf to contain:

    limit maxfiles 65536 200000

    Sierra and newer versions

    Newer versions of macOS no not reference the file due to security considerations.

    On newer macOS Sierra and High Sierra, Dejan Kitic and this found that two plist files need to be added.

  4. Copy in folder /Library/LaunchDaemons/ plist files from a GitHub repository:

    • https://github.com/wilsonmar/mac-setup/blob/master/configs/limit.maxfiles.plist at 524288
    • https://github.com/wilsonmar/mac-setup/blob/master/configs/limit.maxproc.plist at 2048

sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist
   sudo launchctl load -w /Library/LaunchDaemons/limit.maxproc.plist

The files’ ownership needs to be changed to “root:wheel”.

Their permissions need to be “-rw-r–r–”, set by sudo chmod 644.

  1. So how do you turn csrutil off? Google says sudo csrutil disable… Ummm, not so easy, can only be done in Recovery Mode. So, reboot, hold command + R to enter Recovery Mode, once there open terminal and do csrutil disable… Finally a breakthrough…disabled.

  2. Now you can adjust the process limit on Mac OS X Yosemite and El Capitan versions:

    sudo ulimit -n 65536 200000

    Since sudo is needed, you are prompted for a password.

  3. To increase the inotify watchers max limit, edit (or create) file /etc/sysctl.conf (inherited from BSD) to contain:

    kern.maxfiles=49152
    kern.maxfilesperproc=24576

    or

    kern.maxfiles=200000
    kern.maxfilesperproc=200000

    Alternately, run interactive commands:

    sudo sysctl -w kern.maxfiles=5242880

    The response:

    kern.maxfiles: 49152 -> 5242880
    sudo sysctl -w kern.maxfilesperproc=524288

    The response:

    kern.maxfilesperproc: 24576 -> 524288
  4. Restart the system for the new limits to take effect.

  5. After restarting, verify the new limits by running:

On RedHat and CentOS

  1. To enable PAM-based user limits so that non-root users, such as the riak user, may specify a higher value for maximum open files, edit (or create) file /etc/security/limits.conf to contain:

    * soft nofile 65536
    * hard nofile 200000

    If the file already exists, append the above to the bottom of the file.

  2. To activate limits, edit (or create) file /etc/pam.d/login to contain:

    session    required   pam_limits.so
  3. Restart the machine.
  4. Verify by “ulimit -a” again.

    ulimit -n
    ulimit -Hn # Hard limit
    ulimit -Sn # Soft limit

    If the response is “10032”