Diagnosing JVM Memory Leaks in WSO2 Enterprise Integrator

This white paper reconstructs, step by step, the diagnostic reasoning that took one engagement from "the JVM keeps crashing" to "this specific <aggregate> mediator is the bug". It is written as a method rather than a war story, so that the same funnel can be applied to a different incident on a different runtime.

The investigation followed a deliberate funnel: each step ruled in or out one class of cause, narrowing the search space until only one suspect remained.

Step 0: Establish what kind of failure this is

Question: Is this a JVM crash, a graceful shutdown, a wrapper restart, or something else?

Why this matters: Different failure modes have completely different diagnostic paths. An OOM crash needs heap analysis; a graceful shutdown needs to be traced back to whoever triggered it; a wrapper restart needs the wrapper log.

What we checked:

# OOM in carbon log
grep -iE "OutOfMemory|GC overhead|Java heap space" repository/logs/wso2carbon.log*

# Graceful shutdowns
grep -iE "Shutting down|Shutdown hook|Halting" repository/logs/wso2carbon.log*

# Heap dump presence
find . -name "heap-dump*.hprof" -o -name "java_pid*.hprof"

# Kernel OOM killer
sudo dmesg | grep -iE "killed process|out of memory"

What we found:

• OutOfMemoryError entries in the carbon log around each crash time
• A heap dump written by +HeapDumpOnOutOfMemoryError
• No graceful-shutdown messages
• No kernel OOM. The kernel did not kill the JVM; the JVM killed itself
• No native crash files

Conclusion: This is a clean Java-heap OOM. The JVM ran out of heap, wrote a dump, and +ExitOnOutOfMemoryError exited the process. The wrapper then restarted it.

Step 1: Get the GC log behaviour right

Question: Was this a sudden allocation spike, a gradual fill, or a GC pathology?

Why this matters: A gradual fill says "memory is being retained that shouldn't be", which points at a leak. A spike says "something allocated more than the heap could fit at once". A GC pathology (death spiral) often points at fragmentation or humongous-allocation issues.

What we checked:

# Last GC events before OOM
tail -200 repository/logs/gc.log.1

# Full GC frequency in the death window
grep "Full GC" gc.log.1 | tail -30

# Humongous allocation triggers
grep "Humongous Allocation" gc.log.1 | tail

What we found:

• Heap pinned at the -Xmx ceiling for many minutes before OOM
• Full GCs firing back to back, several within a few seconds of each other
• Each Full GC reclaiming a negligible fraction of the heap
• Multiple (G1 Humongous Allocation) triggers
• "To-space exhausted" messages

Interpretation at the time: Classic G1 death spiral on humongous allocations. At small heap sizes, G1's default region size is 1 MB, so any object larger than 512 KB becomes a humongous allocation. Humongous objects skip young generation, land in old, cannot be compacted, and pile up.

First-pass diagnosis: "Heap too small, and region size too small for the workload." Proposed fix: raise the heap substantially and set -XX:G1HeapRegionSize=16m.

Crucially, we did not deploy that fix yet. The GC log told us how the JVM died, not why. The humongous allocations could be a cause (genuinely large legitimate messages) or a symptom (a single object grown so large that its internal arrays are now humongous). Heap analysis was needed to tell which.

Step 2: Load the heap dump into MAT

Question: What is actually inside the heap at the moment of death?

Why this matters: A heap dump is the only artefact that shows the live object graph at the moment of OOM. Everything else (logs, metrics, traces) tells you about behaviour; the dump tells you about state.

Tooling used: Eclipse Memory Analyzer running headless in a GitHub Codespace, because the dump needed far more JVM heap to parse than any available laptop could give it. The setup is documented separately in Heap Analysis Without Leaving the Browser.

/workspaces/mat/ParseHeapDump.sh heap-dump.hprof org.eclipse.mat.api:suspects

Output: An HTML report (heap-dump_Leak_Suspects.zip) with a pie chart, identified problem suspects, and walkable references into the dominator tree.

Step 3: Read the Leak Suspects summary

Question: Is the heap pressure spread across many objects (a generalised pattern), or concentrated in one place (a specific leak)?

Why this matters: Generalised pressure means tuning. A specific leak means a single suspect that can be named and fixed.

What the report showed: Problem Suspect 1 held the clear majority of the live heap: one instance of org.apache.synapse.config.SynapseConfiguration, with the memory accumulated inside a single java.util.HashMap$Node[].

Conclusion: This is a concentrated leak. One SynapseConfiguration instance, one HashMap$Node[] inside it. The investigation is now a search for which HashMap inside SynapseConfiguration is bloated, and what is inside it.

The "Remainder" slice (everything else combined) was about what we would expect for a working EI runtime, which means that apart from this one leak the JVM was healthy.

Step 4: Walk the dominator tree

Question: Which specific child object of SynapseConfiguration owns the retained set?

Why this matters: SynapseConfiguration is the runtime's catalogue of everything deployed: proxies, sequences, endpoints, templates, registry entries. Each is a separate field with separate semantics. Knowing which field is bloated tells you which subsystem to look at.

Navigation in MAT: Click Details » on Problem Suspect 1 to expand the retention path, then read it top-down:

SynapseConfiguration
└─ localRegistry (ConcurrentHashMap)
   └─ ConcurrentHashMap$Node[]
      └─ ...one specific entry...
         └─ TemplateMediator
            └─ ArrayList of child mediators
               └─ Object[]
                  └─ AggregateMediator
                     └─ activeAggregates (SynchronizedMap)
                        └─ HashMap
                           └─ HashMap$Node[]
                              ← the retained set lives here

Each level down held very nearly the same share of the heap as the level above it, which is what tells you the chain is a single line of retention rather than a branching structure.

What this told us:

• The bloat is inside localRegistry, not proxyServices, messageStores, or the others
• One specific entry in localRegistry contains a TemplateMediator
• The TemplateMediator's mediator list contains an AggregateMediator
• The AggregateMediator's activeAggregates field holds the retained set

Critical inference: activeAggregates is a map of in-progress aggregations. Each entry is one aggregation that has started but not yet fired. That this map holds nearly all of it means one of two things:

• One aggregation has accumulated a massive amount of data and never completed, or
• Many aggregations are open simultaneously, each holding their accumulated state

The size distribution of the map's entries would tell us which.

Step 5: Check what is inside the bucket array

Question: Is this one giant aggregation, or many medium aggregations?

Why this matters: The two shapes have different root causes. One giant aggregation means the messageCount condition has been wrong and the aggregate has been collecting indefinitely. Many medium aggregations means the correlateOn expression is wrong, every input opens a new aggregate, and none of them ever complete.

What we looked at: the expanded view of the bucket array, showing the retained size of each entry.

What we saw: many occupied buckets, each retaining a substantial amount, and (this is the important part) all of the same order of magnitude as each other. No single entry stood out as the whole problem. The bucket array had also been resized well beyond its default capacity, which is itself evidence of how many aggregates were open at once.

Conclusion: This is the "many medium aggregations" case. So either the correlation expression is opening too many aggregates, or the aggregates cannot complete: count condition never satisfied, timeout never firing.

Step 6: Confirm what kind of objects are accumulated

Question: What is each aggregation holding inside it?

Why this matters: Different content tells different stories. byte[] arrays point at raw message payloads, often from PayloadFactory operations. OMElementImpl trees are XML message bodies. JsonObject and JsonArray are JSON bodies. Retained MessageContext objects point at whole message contexts being held, often transaction-related. ResultSet points at data service results.

What MAT's class histogram showed for the dominator: Axiom XML objects, top to bottom. OMElementImpl was the most numerous class by a wide margin, followed by OMTextImpl, then OMAttributeImpl and QName, with byte[] behind them.

Conclusion: This is XML payload retention, not payload buffers and not JSON. The ratio between elements, text nodes and attributes is the signature of structured records (many small elements, each with a value and an attribute or two) rather than a few large documents. The element count implied a very large number of accumulated records.

We are now looking for an aggregate that is collecting structured XML records, most likely from a REST API response, and never releasing them.

Step 7: Identify the anchor thread, and rule it out

Question: Which thread is keeping the retained graph alive?

Why this matters: The thread that holds the reference is often the thread that is responsible. But not always. GC root analysis names the thread with the shortest reference path, which can be incidental.

What MAT showed: The "Shortest Paths to the Accumulation Point" panel named a single worker thread, org.wso2.carbon.mediation.initializer.persistence.MediationPersistenceManager$MediationPersistenceWorker.

Initial reaction: this looked promising. MediationPersistenceManager is the subsystem that periodically serialises Synapse configuration to disk. If it were hanging on to a giant in-memory representation, that could plausibly cause this pattern.

What we then checked, and the inference that ruled it out:

• The thread's own retained heap was a few hundred bytes. It holds the reference but does not own the data. It is just an anchor.
• Its stack trace was two frames deep: java.lang.Thread.sleep called from MediationPersistenceWorker.run. The worker was sleeping when the dump was captured.
• Thread state: [alive, sleeping, waiting].

Conclusion: The persistence worker is innocent. It is the cheapest GC root holding the SynapseConfiguration reference, because it has an outer-instance pointer to it, so MAT names it as the shortest path. But it is not doing anything with the accumulated data. It is sleeping between persistence cycles, holding a reference so it can serialise the config when its timer fires.

This is the classic GC-root red herring. MAT correctly identifies what is holding the graph alive technically, but that is not the same as what is causing it to be alive. Always check whether the named thread is actually doing work that touches the suspect data.

Where the actual cause lies, by elimination: somewhere in the Synapse mediation flow, an AggregateMediator is being populated and never drained. The cause is therefore configuration-level: an <aggregate> element in a synapse XML file with a problematic configuration.

Step 8: Map the AggregateMediator back to source

Question: Which file in the codebase contains the runaway aggregate?

Why this matters: This is the moment the diagnosis becomes actionable. Up to here we have identified a class and a behaviour; now we need a filename.

Strategy used: MAT's HTML report does not expose object field values cleanly, so extracting the AggregateMediator's id field via OQL was judged slower than simply grepping the codebase. There are usually only a handful of aggregates in any given repository, so a direct grep is fast.

grep -rln "<aggregate" --include="*.xml" .

The matches were spread thinly across the repository except in one place: a scheduled synchronisation that enumerates cloud subscriptions and resources into a CMDB, where most of them were concentrated, structured as *.Aggregate.v1.0.sq.xml and *.Iterate.v1.0.sq.xml pairs.

Why that integration became the prime suspect:

• Much the highest concentration of aggregate usage in the repository
• The naming convention suggested an iterate-then-aggregate pattern, matching exactly what MAT showed: TemplateMediator → ArrayList → AggregateMediator
• The workload (paginating large REST responses while enumerating subscriptions and resources) fits the "structured XML records being accumulated" finding from Step 6

Listing that project's sequence folder returned one aggregate sequence file per entity type being synchronised.

Step 9: Read the aggregate configurations

Question: What does the <aggregate> element actually look like in each file?

for f in .../sequences/*Aggregate*.xml; do
  echo "=== $(basename $f) ==="
  awk '/<aggregate/,/<\/aggregate>/' "$f"
done

What we saw: every file had identical structure

<aggregate id="iterate-...">
    <completeCondition timeout="60">
        <messageCount max="-1" min="-1"/>
    </completeCondition>
    <onComplete enclosingElementProperty="..." expression="json-eval($.)">
        <log category="INFO" level="custom">
            <property name="..." value=" --- all ... processed --- "/>
        </log>
    </onComplete>
</aggregate>

What this told us about the failure mode: <messageCount min="-1" max="-1"/> means "no count condition". The aggregate fires only when the inactivity timer of 60 seconds elapses. Under sustained input, such as REST pagination producing message bursts, the inactivity timer is constantly reset. The aggregate never gets 60 seconds of silence, never fires, and accumulates forever.

This would already have been enough to prosecute the bug. But there was more.

Step 10: Trace what calls the aggregate

Question: How is the aggregate being dispatched, and what processes each message?

Reading the corresponding Iterate sub-sequence, its tail looked like this:

<call-template target="...Insert.v1.0.tp">
    <with-param name="resourceName" value="{$ctx:resourceName}"/>
    <!-- ... DB insert parameters ... -->
</call-template>
<property name="RESPONSE" value="true"/>
<!-- aggregate for resources after iterate steps -->
<sequence key="...Aggregate.v1.0.sq"/>

What this told us:

• The per-resource processing happens inside Iterate
• The actual database write happens via the Insert template (that is the real work)
• After the DB write, every iteration also dispatches into the Aggregate sequence
• So the Aggregate receives one sub-message per resource, on every iteration

Combined with Step 9 (no count bound, only an inactivity timer that never triggers under sustained flow), this confirmed that the aggregate accumulates indefinitely, the accumulated state has no upper bound, and the real work does not depend on the aggregate completing.

That last point was the most interesting. It raised an obvious question: what does the aggregate actually do with the accumulated data?

Step 11: Check what onComplete does

Question: When the aggregate eventually fires, after a sufficiently long idle period, what happens with the assembled result?

<onComplete enclosingElementProperty="resources" expression="json-eval($.)">
    <log category="INFO" level="custom">
        <property name="..." value=" --- all resources for current subscription processing completed --- "/>
        <property name="ACTIVITY_ID" expression="$ctx:activityID"/>
    </log>
</onComplete>

What this told us: enclosingElementProperty="resources" assembles all accumulated child messages into a property named resources. The <log> block emits one INFO line saying "completed". That is it. No further processing, no callback, no transformation, no publication.

Step 12: Confirm the assembled property is never read

Question: Does any downstream mediator consume the resources property?

grep -rn "ctx:resources\|get-property.*resources" . --include="*.xml"

What we saw: nothing. Empty output. No mediator anywhere in the integration reads $ctx:resources.

Conclusion, and the punchline of the entire investigation: the aggregate is functionally a no-op. It runs, consumes memory, eventually fires, builds a property, and that property is then discarded. The XML being held in memory exists for no business reason. The only observable effect of the aggregate completing is one INFO line in the log.

The integration was holding an enormous set of XML records in memory to print one line per subscription.

Step 13: Verify the pattern across every aggregate

Question: Is this an isolated configuration mistake in one file, or a pattern across the integration?

Reviewing each Aggregate XML file individually, every one had the same unbounded completeCondition, the same shape of onComplete block (log line only, no downstream consumption) and a different enclosingElementProperty value in each case. For every one of those property names, a grep showed nothing reads it.

Conclusion: the pattern is consistent. All of the aggregates are dead code. The one handling the most numerous entity type happened to be the largest by volume, so it filled the heap first. The others were quieter copies of the same defect, waiting for a long enough sync run to hit the limit.

Step 14: Confirm on a second independent dump

Question: Is this the same bug on a second affected node?

The same MAT pipeline was run against a dump from a different production node.

• An even larger share of the heap held by a single SynapseConfiguration instance
• Same retention chain: SynapseConfiguration → localRegistry → TemplateMediator → ArrayList → AggregateMediator → activeAggregates
• A bucket array one rehash step larger, indicating proportionally more open aggregates
• Same anchor thread, the sleeping persistence worker
• Same class composition, dominated by OMElementImpl

Conclusion: the bug is confirmed on a second independent production node, same signature, larger scale. This rules out node-specific causes (resource exhaustion, hardware fault, unique workload) and confirms the bug travels with the deployed integration artefact.

The complete reasoning chain in one sentence

The JVM died from heap exhaustion (Steps 0 to 1); most of the heap was held by one SynapseConfiguration instance (Step 3), specifically by an AggregateMediator's activeAggregates map (Step 4), which held many medium-sized aggregations (Step 5) containing XML records (Step 6); the thread holding the reference was incidental (Step 7); a single grep narrowed the suspect to one integration (Step 8); every aggregate in it had unbounded messageCount with only an inactivity timeout that never triggered under sustained flow (Step 9); the aggregate was dispatched per-iteration after the real work had already happened (Step 10); the assembled result was logged and then discarded (Step 11); no downstream code reads it (Step 12); the pattern was consistent across all of them (Step 13); and the bug reproduced on a second production node with the same fingerprint (Step 14).

Applying this method to other incidents

The funnel has a stable shape.

1. Confirm the failure type (Step 0). Different failures get different toolkits.
2. Read the GC log to understand the death mechanism (Step 1). Resist applying the textbook fix until heap analysis confirms the cause.
3. Identify the dominant retainer in MAT (Steps 2 to 3). One number (the share of heap held by Problem Suspect 1) is the most important signal. Concentrated retention of a third of the heap or more means a specific leak rather than general pressure.
4. Walk the retention chain down to the smallest meaningful subsystem (Step 4). Do not stop at "SynapseConfiguration"; that is the runtime root. Keep going until you find a class whose name maps to a configurable element.
5. Characterise what is accumulated, not just where (Steps 5 to 6). Distribution shape (one big versus many medium) and class composition each rule out different causes.
6. Be suspicious of anchor threads (Step 7). MAT's shortest path to GC root is a technical fact, not a behavioural one. Check whether the named thread actually does work.
7. Map the bug back to source via grep (Steps 8 to 9). For Synapse-based runtimes, the bug almost always lives in a mediator element in a synapse XML file.
8. Check whether the suspect code actually does anything useful (Steps 10 to 12). The most damaging bugs are often functionally no-ops that nonetheless retain state. Always verify that any output produced by the suspect is consumed by something downstream.
9. Verify on a second affected system if possible (Step 14). One dump tells you what happened. Two dumps tell you whether it is a pattern.

Appendix A: The JVM configuration change, and why it is not the fix

The node was running with a max heap of a few gigabytes on a host with far more RAM available than the JVM was allowed to use. The memory block was changed to size the heap to the host and raise the G1 region size:

JVM_MEM_OPTS="-Xms<N>g -Xmx<N>g -XX:G1HeapRegionSize=16m"

With a larger heap and a higher humongous threshold, the JVM absorbs the bad application behaviour for longer and produces a single clean OOM instead of a restart storm. That is a better failure mode, not an absence of failure. OOMs still occur eventually if the application bug is not addressed.

The application-side change capped the message count on every aggregate:

<messageCount min="1" max="5000" />

Pick a bound comfortably above the realistic batch size for the flow. The aggregate then force-fires at the ceiling instead of waiting for a period of silence that never comes. The proper long-term fix is to remove the aggregate dispatch entirely. The aggregates are dead code, and the business logic in the Iterate sub-sequences does not depend on them.

Always confirm the application-level cause before applying JVM tuning. The GC log alone pointed at humongous allocations and a heap that was too small. Raising -Xmx and setting G1HeapRegionSize would have stopped the crashes and looked like a complete fix, while the application bug continued silently and eventually filled the larger heap too.

Appendix B: Spotting this pattern again

In GC logs:

• (G1 Humongous Allocation) as the GC trigger
• Full GCs reclaiming a negligible share of the heap
• Heap pinned near -Xmx across many consecutive GC cycles
• Back-to-back Full GCs with growing pause times

In heap dumps:

• A single SynapseConfiguration instance dominating the dominator tree
• A retention path ending at AggregateMediator.activeAggregates
• OMElementImpl and OMTextImpl at the top of the class histogram

In aggregate XML configuration:

<!-- Dangerous pattern — no count bound -->
<messageCount max="-1" min="-1"/>

<!-- Safe pattern — bounded -->
<messageCount min="1" max="5000"/>

A detection script to find every unbounded aggregate in a codebase:

grep -rl 'max="-1"' --include="*.xml" . | while read f; do
  if grep -q "<aggregate" "$f"; then
    echo "UNBOUNDED: $f"
  fi
done

The narrative version of this engagement, written for a general reader, is The Aggregate That Did Nothing. If your organisation runs WSO2 Enterprise Integrator, Micro Integrator or API Manager in production and wants this kind of analysis on a node that will not stay up, Pinuno's consultancy practice takes that work on. Tell us what you are seeing.