Fails Somewhere Else, Part 2: Silent Runtime Traps in WSO2 EI

In the first part of this series we covered the build and deployment traps in WSO2 Enterprise Integrator 6.6 that produce a green result while leaving something broken. This part covers the runtime equivalents: expressions that evaluate to something other than what they appear to, message consumers that wait on nothing, and APIs whose status codes say success while no data is written. As before, the inclusion rule is narrow. Every entry fails silently, late, or in a place other than the one that is wrong.

Synapse expressions

boolean() is true for the string "false"

In XPath 1.0, boolean() on a string returns true for any non-empty string. A property holding "false" is therefore true. We once watched a run log MODE = DRY-RUN while it wrote to the target. Test the value, not its existence:

<filter source="$ctx:dryRun" regex="true">

For an empty check, use string-length($ctx:x) = 0.

The same trap exists outside XPath. In April 2026 pinuno.com.gh read YII_DEBUG from its .env file and cast it with (bool). "false" is a non-empty string, so PHP made it true, and debug mode was on in production, with raw exception pages showing in Facebook link previews. The fix was filter_var(..., FILTER_VALIDATE_BOOLEAN): parse the value, do not test that it exists.

json-eval() must be the whole attribute

Synapse decides whether an expression is JSONPath or XPath by checking whether it starts with json-eval(. Wrap it in anything, even boolean(), and the whole string goes to the XPath engine, which fails to parse $. as a variable reference. This one does fail loudly, but at deployment time and it takes the whole CApp with it, which is not where anyone looks for an expression error.

Curly-brace expressions do not resolve inside every attribute

<messageCount min="{$ctx:CHUNK_COUNT}"/> does not evaluate the property. The aggregate behaves as if the value were unset. The same shape inside a template parameter can arrive downstream as the literal text. Where an attribute must be dynamic, build the element with a payloadFactory or hardcode the value.

An aggregate starts a new message context

Properties set before an aggregate are not available after it unless they are deliberately carried through. Nothing errors. Downstream log lines simply come out blank, and correlation identifiers disappear exactly where they are needed most.

Deep recursion fails without an error

Paging through a large source by having a sequence call itself works until the call stack runs out. In our case that happened after a few hundred recursive pages, on a table of hundreds of thousands of rows, and no ERROR line was logged. The target API rejected larger batches, so the fix was to restructure into chunks rather than raise the batch size.

Vault lookups need the right form

A secure vault lookup in a property mediator needs expression=, not value=, and the wso2 namespace declared on the enclosing sequence or API. Without the namespace, the expression fails to compile and the property is silently empty. Against an LDAP directory, an empty password produces an anonymous bind, which returns an empty result rather than an authentication error. The integration reports that the directory has no users.

State left over between readers

When one mediation runs several readers on the same message context, every control property a reader sets must be cleared before the next one starts. A leftover "already truncated" flag truncates only the first table and appends a duplicate day to the rest. A leftover table name makes the second reader write into the first reader's table. Neither raises an error.

A task injecting into a sequence has no service name

A scheduled task that injects directly into a sequence produces monitoring events with an empty service name. The integration runs correctly and is invisible on the monitoring dashboard. Injecting into a proxy (injectTo=proxy with proxyName) gives it a name to report under.

File polling over VFS

• A proxy declared with transports="https" where vfs was intended starts without complaint and never receives a file.
• A VFS proxy reads files as text/plain unless transport.vfs.ContentType says otherwise, so a JSON file arrives as text and any attempt to treat it as JSON sees null. Set the content type on the transport, and set messageType and ContentType at axis2 scope after building the outgoing payload.
• ActionAfterFailure=NONE is not a recognised value and may fall back to deleting the file.

JMS

Message stores expect their own message type

A Synapse JMS message store consumer expects messages that the ESB itself stored. It cannot consume a plain text message produced by a different publisher. A message store and processor pair built on such a queue deploys, activates on schedule and fails on the first message.

Proxies on a cluster compete

Every node in a cluster deploys a JMS proxy and opens its own consumers, and they compete for messages. Inbound endpoints have a coordination setting because proxies do not. Where single-consumer behaviour matters, consumer.exclusive=true on the destination gives one active consumer and one on standby.

Two related settings are easy to misread. ConcurrentConsumers is a minimum, not a maximum, so set MaxConcurrentConsumers as well. And startOnLoad="false" means the proxy never starts unless something external starts it. Synapse has no scheduler for proxies, and the flag does not survive a redeploy or restart.

The wrong connection factory gives an empty queue, not an error

A consumer that names the local broker's connection factory, when it should name the one that points at a remote broker, deploys cleanly and listens on a queue nothing publishes to. The dashboard shows a started proxy and the run ends with zero rows.

Dequeued is not delivered

A queue's dequeue counter includes messages moved to the dead-letter queue. "Twenty enqueued, twenty dequeued" does not mean twenty delivered. Under client acknowledgement, an acknowledged message means the mediation returned, not that the downstream write succeeded.

APIs that say success

The pipelines in this project wrote through a shared ingestion API, and almost every part of its response proved unreliable as a signal:

• 202 Accepted with guaranteed delivery means the request was queued, not written.
• A database rejection inside the API, such as a value too long for its column, could discard a whole batch of rows while the API still answered 202.
• On one occasion a validation failure came back as 200 with "status": "ERROR" in the body.
• The validation failure itself was logged at INFO level, among hundreds of status lines.
• The API truncated the target table before validating the first batch, so a rejected load left the table empty rather than stale. Downstream consumers cannot tell an empty table from a genuine zero-row day.

The only check we trusted was a row count in the target after the run.

Reading the log

One log line deserves its own mention. A per-batch "total records in the table" figure looked like a running total and was not: concurrent threads read the table count and logged it as each returned, so the values arrived out of order. In one run the count went down between two consecutive lines. An intermediate value looks exactly like a completed one, and on one occasion it made a complete table look a thousand rows short. Use an explicit completion log line for what was read, and a count query for what landed.

A cache flush that emptied a queue

Consumers that wait on nothing do not need a misconfigured connection factory. In our own energy management product, WSO2 Micro Integrator 4.6 keeps its inbound queue and dead-letter queue in a Redis instance of its own, next to the application's Redis on the same Docker network. Both compose files named their service redis, and Compose registers the service name as a network alias, so both containers claimed the name and Docker's DNS answered with either. Measured on the server, the application resolved redis to Micro Integrator's instance. Every deploy then ran the application's cache flush, which issued a bare FLUSHDB, and emptied the integration's queues. No persistence setting would have saved them: a durable store replays a flush faithfully. The fix renamed the service, turned on append-only persistence, and made the application's cache delete only its own keys.

Healthy is a claim about one path

The same product supplied the plainest example. Micro Integrator's log carried 68 OutOfMemoryError entries while docker ps said the container had been up for 45 hours. An out-of-memory JVM holds its socket open, so a check on the port would have kept saying fine. The health check now calls an endpoint that runs a real mediation sequence, so a pass means Synapse is mediating, not merely that the port is bound.

One more from our own stack. We run Moodle in a container behind pinuno.com.gh. A command-line maintenance script run as root created cache directories owned by root, and from then on real page requests failed with HTTP 500 permission errors. The container's health check, which runs from inside the container, still passed, and the container showed as healthy. The fix was to run command-line scripts as the web user and restore ownership. The lesson carries straight back to integration runtimes: a health check proves only the path it exercises.

Conclusion

The runtime traps share one property with the build traps in part 1: the system has no reason to tell you about them, because from its point of view nothing went wrong. A filter evaluated, a consumer started, an API accepted a request. The defence is to verify outcomes rather than events: count the rows that arrived, test values rather than their existence, and treat a success code from a downstream API as a statement about the request, not about the data.

Our WSO2 course covers building and running integrations like these, taught by practitioners who do this work for a living.