The Root Cause Analysis Blueprint: Debugging OOMKilled (Exit Code 137) in Production
Last updated: July 7, 2026
Quick Answer: Exit code 137 means the kernel killed your container with SIGKILL (128 + 9) for exceeding its cgroup memory limit. The root cause analysis blueprint: confirm the kill from the orchestrator's event stream (not the app logs), compare the cgroup limit against what the process actually allocated, find which service in the topology started leaking first, and correlate that service's deployment tag with recent changes and past post-mortems.
Overview
- The evidence for an OOM kill lives in the orchestrator's event stream, because the killed process gets SIGKILL and dies before it can log anything useful
- A 137 is only a memory kill if the payload confirms it:
stopCode: OutOfMemoryin ECS,Reason: OOMKilledinkubectl describe, or theanon-rssline indmesg - Many "mystery" OOMs are configuration: heap plus off-heap (metaspace, thread stacks, direct buffers) exceeding the cgroup ceiling, or memory-backed volumes silently eating headroom
- The killed container is frequently a victim, not the cause: ranking services by memory growth rate (
derivin PromQL) finds the upstream service that leaked first - Most memory regressions ship with a deploy, so correlating the suspect's image tags against the growth window and searching old post-mortems usually surfaces a known fix
- Single-container threshold alerts page whoever owns the noisiest symptom; Vibe OnCall's Triage Agent runs this entire blueprint before a human is paged
The Evidence: A Raw OOMKilled Payload from Production
Start from the orchestrator's event stream, not the application logs — an OOM-killed process gets SIGKILL and dies mid-write, so its own logs rarely contain the evidence. Here is the anonymized CloudWatch payload we'll work through (Datadog's ECS integration forwards a near-identical structure):
{
"timestamp": "2026-07-07T21:04:12.893Z",
"logStream": "ecs-payment-service-cluster/payment-gateway/a1b2c3d4",
"message": "Task failed to start successfully.",
"containerNames": ["payment-container"],
"detail": {
"containerInstanceArn": "arn:aws:ecs:us-east-1:123456789012:container-instance/abc-123",
"exitCode": 137,
"lastStatus": "STOPPED",
"stoppedReason": "Essential container in task exited",
"stopCode": "OutOfMemory",
"reason": "ResourceInitializationError: failed to create new containerName"
},
"platformError": "OOMKilled: Process isolated and terminated by kernel due to cgroup memory limits exceeded."
}
Three fields do the diagnostic work:
detail.exitCode: 137+detail.stopCode: OutOfMemory— confirms a kernel OOM kill, not an application crash or manual stop. Without the OOM stop code, a 137 can also be adocker stoptimeout or a scale-in event.logStream— names the cluster, service, and task. This is the starting node for the investigation, not necessarily the root cause.stoppedReason— "Essential container in task exited" means ECS tore down the whole task, so sidecars (Envoy, log routers, agents) died with it and their metrics will gap out.
To find every 137 in the recent window, query the ECS task-state-change events with CloudWatch Logs Insights:
fields @timestamp, detail.group, detail.stoppedReason
| filter detail.containers.0.exitCode = 137
| sort @timestamp desc
| limit 20
On Kubernetes, confirm the same evidence directly:
kubectl describe pod payment-gateway-7d9f8c -n payments
# Look for:
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137
And on the node itself, the kernel log names the process and the memory that killed it:
dmesg -T | grep -iE "oom|killed process"
# Memory cgroup out of memory: Killed process 12345 (java)
# total-vm:4915204kB, anon-rss:1966080kB, file-rss:0kBanon-rss is the number that matters — it's the anonymous (heap + off-heap) memory the process actually held when the kernel pulled the trigger.
AWS documents the stopped-task codes in its ECS error reference; Kubernetes OOM behavior is covered in the official memory docs.
The Root Cause Analysis Blueprint, Step by Step
[Raw failure payload]
|
v
[Step 1: Limits vs. actual allocation] --> misconfigured? --> fix limit/heap ratio
|
v
[Step 2: Find the first mover] --> which service's memory climbed FIRST?
|
v
[Step 3: Correlate deployment tags] --> match spike to recent change
|
v
[Root cause + matching past post-mortems]
Step 1: Compare the cgroup limit against actual allocation
Pull the hard limit and put it next to what the process was configured to allocate:
# Kubernetes: the configured limit
kubectl get pod payment-gateway-7d9f8c -n payments \
-o jsonpath='{.spec.containers[*].resources.limits.memory}'
# Inside the container (cgroup v2): the kill line and live usage
cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current
For ECS, compare memory (hard limit) and memoryReservation (soft) in the task definition against the container's reported usage in Container Insights.
The classic failure is a JVM with -Xmx512m in a container with a 512 MiB limit. Heap is only part of the process: metaspace, thread stacks (~1 MiB each × hundreds of threads), direct byte buffers, and GC overhead all live outside -Xmx. Verify with native memory tracking:
# requires the JVM flag -XX:NativeMemoryTracking=summary
jcmd <pid> VM.native_memory summary
# Compare "Total: committed" against the cgroup limit — not just the heap line
Two accounting gotchas that make containers die "below" their limit:
- The kernel charges the whole cgroup, not just your process: page cache awaiting writeback, tmpfs mounts, and child processes all count against
memory.max. - Memory-backed
emptyDirvolumes (medium: Memoryin Kubernetes) count against the container's memory limit — a 300 MiB scratch file eats 300 MiB of your headroom.
Rule of thumb: heap at 70–75% of the container limit, or -XX:MaxRAMPercentage=70 (JDK 10+) so the JVM reads the cgroup itself. If limits and allocation are sane, the pressure came from load or a leak — go to Step 2.
Step 2: Find the first mover, not the loudest victim
The killed container is where memory ran out, not necessarily where the problem started. Rank services by memory growth rate in the 30 minutes before the kill — the steepest sustained climb is your suspect:
Everything in this blueprint starts from the orchestrator's event stream, not the application logs: an OOM-killed process gets SIGKILL and dies mid-sentence, so its own logs rarely contain the evidence. Here is the anonymized CloudWatch payload we'll work through (Datadog's ECS integration forwards a near-identical structure):
topk(5, deriv(container_memory_working_set_bytes{namespace="payments"}[30m]))
In Datadog, the equivalent is graphing kubernetes.memory.working_set (or container.memory.working_set) by pod over the same window. Use working set, not raw usage — it excludes reclaimable page cache and is what the kubelet's eviction logic actually tracks.
Then check whether the growth was pushed from upstream: queue depth (kafka_consumergroup_lag, SQS ApproximateNumberOfMessagesVisible), connection counts, and retry rates. A consumer that OOMs because an upstream producer doubled its event size is a victim; the producer is the root cause. The service that moved first is the suspect — everything that degraded after it is blast radius.
Step 3: Correlate deployment tags with recent changes
Match the suspect's deployment history against the memory curve:
kubectl rollout history deployment/payment-gateway -n payments
kubectl get replicasets -n payments --sort-by=.metadata.creationTimestamp \
-o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image,CREATED:.metadata.creationTimestamp
If a new image tag landed within the growth window, diff what shipped: dependency bumps, cache-size changes, a batch job that lost its pagination. Then search closed post-mortems for the same service and signature — in Jira:
project = OPS AND labels = post-mortem AND text ~ "payment-gateway OOM"
Repeat OOM kills on the same service are the norm, not the exception, and the old ticket usually contains the fix.
Why Standard Threshold Alerts Fail in Microservice Topologies
The Threshold Mirage: a single-container memory alert that looks like the incident but is actually a downstream symptom of a leak elsewhere in the topology. Paging on it sends the on-call engineer to the wrong service — the blueprint exists precisely to correct for it.
Traditional alerting watches one container cross a static line — say, 85% memory. In a microservice topology that signal is ambiguous: a leak in an upstream payment service backs up queues, inflates consumer buffers in three downstream services, and fires three "memory high" alerts on services that are perfectly healthy. The engineer gets paged for the symptom, triages the wrong service, and the real leak keeps growing.
A static threshold alert hands a human one ambiguous data point; blueprint-driven triage identifies the leaking service, its blast radius, and matching historical fixes before anyone is paged.
This is exactly the sequence Vibe OnCall's Triage Agent automates:
- Interception: the Triage Agent grabs the raw JSON payload — like the one above — directly from CloudWatch or Datadog, before the alert fans out into Slack noise.
- Context mapping: it maps the memory spike across the microservice topology to define the exact blast radius: which service leaked first, which are collateral.
- Historical correlation: it matches active deployment tags against past Jira post-mortems and surfaces incidents with the same signature, and their fixes.
- The result: the page arrives already root-caused. One mid-market Vibe OnCall customer cut MTTR 60% and handles incidents 70% faster largely because this blueprint runs before an SRE opens a laptop.
For the broader pattern: AI investigation running before a human is paged-see our guide to Tier 0 incident
Root Cause Analysis Anti-Patterns
- Don't fix the symptom and close the incident. Doubling the memory limit on a leaking service buys hours, not a fix — and makes the next kill slower to diagnose because the growth curve is longer.
- Don't raise the alert threshold to stop the noise. The noise is the Threshold Mirage; raising the line just delays detection of the real leak.
- Don't start the RCA from the alert text. Alerts describe symptoms in whichever service crossed a line first. Start from the raw orchestrator payload — it names the actual kill reason.
- Don't skip the post-mortem search. If the same service OOM-killed six months ago, the fix is probably written down. Improvising a fresh investigation each time is how MTTR stays flat.
Preventing the Next Incident
- Set ECS memoryReservation below the hard memory limit: the gap absorbs bursts without inviting the OOM killer.
- Tune JVM heap to the container, not the host: with -XX:MaxRAMPercentage=70 (JDK 10+).
- Set Kubernetes resource requests and limits: requests without limits invite node-level OOM kills that take out neighboring pods.
- Profile memory per deploy, not per incident: a memory delta in canary analysis catches the regression before it pages anyone.
FAQ
What does exit code 137 mean?
Exit code 137 means the process received SIGKILL (128 + 9), almost always from the kernel OOM killer after the container exceeded its cgroup memory limit. Confirm via stopCode: OutOfMemory (ECS) or reason OOMKilled (Kubernetes).
Why was my container OOMKilled below its configured memory limit?
Because the kernel charges the entire cgroup, not just your process: page cache pending writeback, memory-backed emptyDir volumes, and child processes all count against the limit. Separately, the kubelet can evict pods under node memory pressure before any single container hits its own limit.
How do I find which service actually caused an OOMKilled error?
Rank services by memory growth rate, not absolute usage — topk(5, deriv(container_memory_working_set_bytes[30m])) in PromQL. The killed container is often a downstream victim; the service whose memory climbed first is the suspect.
Can root cause analysis be automated?
The evidence-gathering steps can — reading the failure payload, mapping the blast radius, matching deployment tags to past post-mortems are deterministic and machine-executable. Tools like Vibe OnCall's Triage Agent run them before paging; judgment calls about the fix stay with the engineer, who now starts with the answer instead of the search.
Stop Running the Blueprint by Hand
Every step above is deterministic triage work that happens at 3 AM today because a human has to do it. Vibe OnCall's Triage Agent runs the entire root cause analysis blueprint the moment the failure lands, so the page that reaches your on-call engineer already says what broke, why, and what fixed it last time. Try the Triage Agent
Methodology note: the payload above is anonymized from a real ECS task failure event; commands and field behavior are verified against AWS ECS, Kubernetes, and OpenJDK documentation as of July 2026. MTTR figures are from a published Vibe OnCall mid-market customer case study.
