Exactly Twice: Scheduled Tasks, Hot Deployment and Clustered WSO2 EI Nodes

A nightly inventory load on a test environment delivered exactly twice as many rows as the same integration delivered in production. Nothing in the source had doubled, nothing in the code had changed between the environments, and the run logged no errors. The ratio was the clue: when a load comes out at exactly twice its expected size, the likeliest explanation is not bad data but a job that ran twice. This short post explains how that happens with scheduled tasks on a clustered pair of WSO2 Enterprise Integrator nodes, and how to build the check into the deployment.

The symptom

The test environment's load wrote twice as many rows into a staging table as production loaded from the same source. Every row appeared twice, with the same keys, a second or so apart. The integration's logs on each node showed a complete, successful run.

That last detail is what gave it away. Each node's log looked normal on its own. Only reading both nodes together showed two runs at the same time.

The cause

The CApp containing the integration and its scheduled task had been hot-deployed onto both nodes of the pair, without a restart.

In a clustered EI 6.x setup, scheduled tasks are meant to run on one node at a time, with the cluster deciding which. In our experience, a task that arrives through hot deployment does not reliably join that coordination, and each node can start running it independently until the nodes are restarted. Both nodes then fire the task on schedule, both read the full source, and both deliver it.

The load design made the result visible as exact doubling rather than as an error. Each run appended to the staging table, and the table was truncated once a day by a separate load in another zone. Two runs between truncates meant two full copies.

The fix

Restarting both nodes resolved it. After the restart, the task ran on one node. The duplicates cleared at the next scheduled truncate, so no manual clean-up was needed in this case. In a design without a regular truncate, the duplicated rows would have to be removed by hand.

Building it into the deployment

The lasting fix is procedural. For any CApp that carries a scheduled task, the deployment steps now include:

• Deploy the CApp to every node in the cluster.
• Restart the nodes, one at a time if availability matters.
• After the first scheduled run, confirm that the task ran on exactly one node by searching both nodes' logs for the run's start line.
• Compare the delivered row count with the source count, not just with yesterday's figure.

The same logic applies to manual reruns. If a scheduled run has already appended its data, a manual run on the same day appends it again. Before rerunning by hand, we check the target for rows from that day's run and wait for the next truncate if any are present.

The same class of trap on Micro Integrator

Our own energy management product runs WSO2 Micro Integrator 4.6 in a single Docker container, not a cluster, so it says nothing about how cluster nodes share a task. It did teach us two related lessons about trusting what hot deployment appears to have done.

• Sequences hot-deploy; APIs do not. Measured on 4.6.0: editing a sequence logs a redeploy within seconds, while editing an API file produces no redeploy line at all and the previous resource keeps serving. That looks exactly like your change having no effect. Our deploy task restarts the runtime for this reason.
• A task that registers is not a task that fires. A scheduled task deployed and registered cleanly, with normal-looking registration logging, and never sent anything. Enabling debug logging on the class that executes the task showed nothing across several intervals. The message injector needed injectTo set to sequence as well as the sequence name; without it the injector's default target is a proxy service, so the task had nowhere to send its message.

Both point the same way as the double run above: after deploying anything scheduled, confirm the run from its effect, not from the deployment log.

Conclusion

A load that is exactly double is rarely a data problem. It is almost always a job that ran twice, and on a WSO2 EI cluster, a scheduled task that was hot-deployed without a restart is the first thing to check. Treat the restart as part of deploying a task, not as an optional step after it.

Our WSO2 course covers building, deploying and running scheduled integrations like this one, taught by practitioners who do this work for a living.