Docker OOM kills and exit code 137

What exit code 137 means

Exit code 137 is 128 + 9: the process ended on SIGKILL. The out-of-memory (OOM) killer sends SIGKILL, but so does docker kill, and so does docker stop when a container ignores SIGTERM past its grace period. So 137 means “killed”, not necessarily “out of memory”. You need one more piece of evidence.

Step 1: ask Docker

docker inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}}' web
# true 137

If OOMKilled is true, the container was killed for exceeding memory. If it is false with exit code 137, something else sent the signal, or the OOM killer took a different process inside the container and the container’s main process carried on. That is why the kernel counter is more complete.

You can also look back through events:

docker events --since 24h --until "$(date +%s)" --filter event=oom --filter event=die

Step 2: read the cgroup’s memory events

cat memory.events
# low 0
# high 0
# max 14
# oom 3
# oom_kill 3

max counts how often usage hit the limit, and oom_kill counts processes the kernel killed in this cgroup. A rising max with no kills means the container is pressed against its limit. A rising oom_kill confirms kills. Compare memory.current with memory.max to see how close it runs normally.

Step 3: tell a container limit from a host shortage

dmesg -T | grep -iE 'out of memory|killed process'

Reading the kernel log can require extra privileges on some hosts. A line beginning Memory cgroup out of memory means a container limit was hit. A line beginning plain Out of memory means the host ran short, and the kernel may have picked a victim from any container. These call for different fixes.

Common causes and fixes

• The limit is too low for the workload. In Compose it is mem_limit or deploy.resources.limits.memory. Raise it if the host has memory to give.
• A leak or growth in the app. Memory climbs steadily until the kill and then starts again after the restart.
• Concurrency settings that multiply memory use, such as worker counts, connection pools or cache sizes.
• Too little host memory for everything on it, so the host-level OOM killer steps in.

Adding swap can delay a kill but also makes a slow service slower, so treat it as a trade-off and not a cure.

How Docktor reports it

Docktor’s OOM kill finding fires from the oom_kill counter in memory.events increasing, or a Docker oom event. It states the observed fact plainly, for example “postgres was OOM-killed at 14:32”, and it can use that observation from the first sample, before any baseline exists. Separate container memory pressure and host memory pressure findings use memory.current against memory.max, memory pressure, swap activity and reclaim. Docktor doesn’t restart anything, and it suggests what to check next.