Fails Somewhere Else, Part 1: Silent Build and Deployment Traps in WSO2 EI

Most integration defects announce themselves: a stack trace, a red deployment, a failed build. The expensive ones do not. They produce a green build, a healthy dashboard and a CApp that reports itself as deployed, and then fail at three in the morning on a scheduled run, or never fail at all and simply do nothing. Over several months of work on a set of WSO2 Enterprise Integrator 6.6 data pipelines we kept a list of every defect that failed silently, late, or somewhere other than the thing that was wrong. This first part covers the ones that live in the build, the packaging and the deployment.

Why a separate list is worth keeping

An ordinary bug costs the time it takes to read the error. A silent one costs the time it takes to realise there is an error at all, and then the time spent investigating the wrong layer, because every symptom points at the change not working rather than at the change not being there.

The inclusion rule for the list was deliberately narrow. An entry qualified only if the failure was invisible at the point where the mistake was made. That rule kept it short enough to be used as a release checklist rather than a reference nobody reads.

Packaging: the CAR is built from whatever matched

A CApp project in WSO2 Integration Studio has a config module containing the Synapse artefacts and an artifact.xml that lists them, plus a CAR module whose pom declares what goes into the archive. Each artefact appears three times:

Two details catch people out. The groupId carries a type suffix (.sequence, .task, .template), and the suffix is written explicitly in all three places; the plugin does not add it for you. And the dependency type for Synapse artefacts is xml, while zip is for registry resources and jar for bundles. The mapping comes from the parent pom's artifact.types property, which is the authority to check.

The Maven CAR plugin packages an artefact only when the dependency and the _._ property match. The failure modes are asymmetric:

• A dependency with no matching property is skipped. The build passes, the CAR ships without the artefact, and nothing reports it.
• A typo inside the property key, such as a double dot in the groupId, behaves exactly like a missing property.
• An artefact type that has no entry in the parent pom's artifact.types mapping is dropped the same way. An inbound endpoint would not ship until synapse/inbound-endpoint=xml was added.
• Retiring an artefact by commenting it out in artifact.xml and removing its property, while leaving its dependency in place, drops it the same way. Because the artefact is still in the local Maven repository from earlier builds, the dependency keeps resolving and the build output looks identical to a correct one.
• The reverse cases (a property with no dependency, an artifact.xml entry with no file) are either inert or fail the build loudly, which is why the dangerous direction is easy to overlook.

On one integration this left the orchestrator's own fault sequence out of the CAR for months. The integration ran with no fault handler of its own and nobody noticed, because nothing failed until something else did.

When an artefact is retired, remove the artifact.xml entry, the dependency and the property together, in one commit, so the three places never disagree. Note that artifact.xml is usually shared by every CAR module in the project, so commenting an entry out there removes it from all of them.

The IDE markers that mean nothing

The IDE is no help in finding these, and it raises alarms of its own. When we added a set of new artefacts to a CAR pom, the IDE showed "Missing artifact ...:xml:1.0.0" errors on almost all of them, yet a full mvn clean install from the repository root succeeded and listed every one. The markers come from the IDE's Maven integration, which resolves against the local repository and the workspace and does not understand plugin-generated artefacts: the build plugin creates one Maven artefact per artifact.xml entry at build time. The ones that resolved had simply been packaged by an earlier build. When every dependency of a kind fails except the ones that already existed, the problem is install state, not coordinates. Build from the repository root, not the CAR module, and reload the Maven project to clear the stale markers.

The only check that catches a dropped artefact

A successful build proves that the artefacts were produced. It does not prove they were packaged. The only reliable check is to count what is inside the built archive:

unzip -l <module>/target/*.car

Expect one entry per artefact. Fewer means a mismatch, not a build failure. To make sure the result is not standing on cached state, build against an empty local repository, which is the build a CI pipeline performs:

mvn -q clean install -Dmaven.repo.local=/tmp/m2-clean -DskipTests

The management console is not a substitute. It hides templates, so a count taken from the console is wrong by however many templates the CApp contains.

Deployment: green is not the same as working

References resolve at message time

A sequence key that points at an artefact in another CApp is resolved when a message arrives, not when the CApp deploys. A CApp with a dangling reference deploys green, shows as healthy, and fails on the first message. On a nightly schedule that first message arrives unattended.

Where integrations are split across network zones, there is a cheap version of this check: an artefact prefixed for one zone must never reference an artefact prefixed for another, because they run on different nodes and cannot resolve under any deployment order. A single grep for the prefix pair catches it before release.

A version change does not replace

Carbon identifies a CApp by name plus version. Deploying version 2.0.0 onto a node that holds 1.0.0 adds a second application rather than replacing the first, and both now claim every shared artefact name. Removing an artefact from the build also removes nothing from a running node until the old CApp version is explicitly undeployed.

Data services validate at deploy time

A .dbs data service checks its database connection when the CApp deploys and faults the whole CApp if the database is unreachable. Setting initialSize=0 does not help, and EI 6.6 has no flag to skip the validation. Worse, a datasource that times out can hang server startup entirely, which turns an unreachable database into an unavailable node.

Scheduled tasks and hot deployment

A CApp carrying a scheduled task that is hot-deployed onto a cluster, without a restart, can end up with the task running on every node. A separate post covers that case in detail, but it belongs on this list: the deployment succeeds and the symptom appears the next morning as a load exactly twice the expected size.

Bundles need a restart

A custom mediator delivered as an OSGi bundle is picked up at server startup. Deploy the CApp without restarting and it reports green, then every class mediator fails to load at the first scheduled run. A bundle that reports ACTIVE is also weaker evidence than it looks when its imports are declared optional: the only check that exercises a real class load is invoking the mediator.

The management console overrides the CApp

Editing an artefact in the Carbon management console writes a file into synapse-configs/default/. From then on, that file takes precedence over the version in the CApp, permanently. Reverting the content in the console does not remove the override. The only trace is a warning at startup:

grep -h "Ignoring CAPP's artifact" wso2carbon.log* | sort -u

Every line it returns is an artefact where the code in the repository is not the code that is running. On one production pair, three artefacts were overridden on one node and none on the other, so the two nodes ran different configurations under the same CApp version.

Because the warning is written only at startup, the check belongs after every restart, not only after a deployment. And the Application Dashboard is no help here: it lists what a CApp contains, not what is in effect.

The console also changes what it saves. Saving an iterate mediator from the console strips the <sequence key> from its target. Edit the source and redeploy by CAR instead.

Property overlays: literal find and replace

Environment-specific values are commonly injected at build time by a replacer plugin that substitutes tokens in the artefacts. It does a literal find and replace, which produces three quiet failure modes:

• The left-hand side must include the braces. some.token=value matches the bare text inside ${some.token} and deploys ${value}.
• Trailing whitespace is part of the value. On an LDAP distinguished name this produces a malformed-name error that points nowhere useful.
• A token in an artefact with no line in the properties file deploys as the literal text ${...} and fails at runtime. A line in the properties file that no artefact uses does nothing, but reads like live configuration for as long as it stays.

The last one cost us a full round of investigation. A cron property described a message-draining mechanism that had never been built. Nothing consumed it, but its presence made the mechanism appear to exist. A stale property is worse than a missing one, so dead properties should be deleted, not left in place.

A build-time token chained to a runtime vault lookup is a related trap: the token is replaced at build time, and if the replacement is wrong the vault lookup receives a string it cannot resolve. A CAR built against the wrong environment then fails exactly like a rotated password.

The same shape in our own projects

None of this is peculiar to Synapse. On pinuno.com.gh, a Yii2 application deployed with Deployer, we have met the same class of failure three times.

• A shared file the deploy never touches. config/params.php is a shared file: it persists across releases and a deploy never overwrites it. A new key added in the repository therefore never existed on production. The deploy was green and the code quietly fell back to its default. The fix is a defaults file that ships with every release and is merged underneath the shared one.
• One line that empties the environment. In August 2026 a single value with an unquoted space in the .env file made the dotenv parser reject the whole file, not just that line, so every variable vanished on the next environment upload. We now boot the application once after editing .env, before deploying.
• A deploy that never reaches the browser. Eight CSS and JavaScript files were served with an immutable one-year cache and no cache-busting query. In July 2026 we found that a deployed change to scripts injected into proxied course pages had never reached returning visitors. Every such reference now carries ?v= followed by the file's modification time.

Two more, where the deployment script itself said success:

• A recreate that reported success when it had failed. Our energy management product runs WSO2 Micro Integrator 4.6 in Docker. Its deploy step ran up -d ... || true followed by an unconditional success line, so a failed recreate looked identical to a successful one. When a renamed service hit a container name conflict, the deploy printed "MI stack recreated" twice while the server kept running the old container. Earlier the same step had used docker compose restart, which reuses a container's configuration from whenever it was last created, so compose changes never applied however often it ran. Both steps now check the result.
• A migration step that could not fail. On another of our sites, the migration task ended in || echo "Migration completed", discarding the exit code. A deploy against a database the application user held no grant on printed its success line twice and finished green, leaving a release that returned 500 on every request. The health check had the same shape: it warned on an unexpected status code and then declared success anyway. Both now fail the deploy.

In each case the build and the deployment succeeded. What failed was the assumption that a successful deployment means the new version is what runs.

Conclusion

None of these traps is exotic, and each one is cheap to detect once you know where to look: count the artefacts in the CAR, grep the startup log for overridden artefacts, diff the tokens used against the tokens defined, and restart after deploying bundles or tasks. The common thread is that a successful build or deployment proves only that nothing objected. The checks that matter are the ones that confirm what actually arrived on the node. Part 2 covers the runtime equivalents: expressions, messaging and APIs that report success while doing nothing.

Our WSO2 course covers building, packaging and deploying integrations like these, taught by practitioners who do this work for a living.