Python Resource Limitation

Did you know you can explicitly control system resources for your Python applications? By resources, I mean CPU time, memory, number of processes, open file descriptors, stack size — you name it. Python's resource module, hidden away in the standard library, makes this surprisingly straightforward.

Here's an example where we limit CPU time for a process:

import signal
import resource


def time_exceeded(signo, frame):
    raise SystemExit("Time's up!")

def set_max_runtime(seconds):
    _, hard = resource.getrlimit(resource.RLIMIT_CPU)
    resource.setrlimit(resource.RLIMIT_CPU, (seconds, hard))
    signal.signal(signal.SIGXCPU, time_exceeded)

if __name__ == '__main__':
    set_max_runtime(15)  # 15-second limit
    while True:
        pass

What happens when the time is up?

Run this script, and after 15 seconds, it terminates with the following message:

Time's up!

The resource.setrlimit() function allows you to define both soft and hard limits for a given resource:

• Soft limit: The current cap, which can be adjusted dynamically (within the hard limit). When the process hits this limit, the operating system usually intervenes with a signal.
• Hard limit: The absolute maximum. It can only be decreased (to match the soft limit or less) and never raised — unless you're a superuser using your privileges.

In the above example, we use RLIMIT_CPU to restrict the CPU time to a maximum of 15 seconds.

Here's another practical example — limiting the number of processes:

import resource
import multiprocessing

def set_max_processes(n):
    _, hard = resource.getrlimit(resource.RLIMIT_NPROC)
    resource.setrlimit(resource.RLIMIT_NPROC, (n, hard))

if __name__ == '__main__':
    set_max_processes(2)  # Max of 2 processes
    f = lambda x: x * x
    p = multiprocessing.Pool(5)  # Requesting 5 workers
    print(p.map(f, [1, 2, 3]))

The result? Chaos.

BlockingIOError: [Errno 11] Resource temporarily unavailable

Why? Because we capped the process count at 2, but the multiprocessing.Pool requested 5 workers. The system wasn't having it, so it threw an error. Nice try, though!

The resource module supports a range of resource controls. Some notable examples include:

• RLIMIT_AS: Virtual memory usage.
• RLIMIT_NOFILE: Number of open file descriptors.
• RLIMIT_STACK: Size of the program stack.
• RLIMIT_DATA: Maximum size of the process's data segment.

For the full list of resources you can tweak, check out the official documentation.

Setting resource limits is especially useful for:

• Sandboxing: If you're running untrusted code (hello, serverless functions!), you probably want to rein it in.
• Preventing system crashes: Avoid accidental overuse of critical resources, which could bring your server to its knees.
• Enforcing policies: Keep processes within company-mandated quotas (e.g., for multi-tenant systems).