0
  • No products in the cart.
0
  • No products in the cart.
mobile logo

Reliability pattern in MuleSoft

Reliability pattern in MuleSoft

by admin
Media
July 11, 2022

Overview 

Delivering a message only once to ensure the data integrity and delivering the message without failure completes the data reliability of nonfunctional requirement of any enterprise solution which is often overlooked during the architecture and solutioning. I am trying to provide a reliable solution using MuleSoft for 100% reliability. 

In such scenarios, reliability pattern can be implemented by introducing a reliable acquisition flow in the following ways.

Reliability Pattern implementations

In this article explaining the reliability pattern implementation using Queues and Persistent Storage 

Queues:

           By introducing queues(MQ/JMS/VM) in the flow, the reliability and the retry mechanism are taken care of by  default. The reliable acquisition flow consumes the message from source and drops the message on the queue which will be consumed by the Application logic flow. 

Once the message enters the Application logic flow, it must be acknowledged only on successful processing. If the process encounters any failure, the message will not be acknowledged so that it remains in the same queue and can be retried.

<flow name=”Reliable_Acquisition_Flow” doc:id=”d5d1d621-de54-4b7c-ab66-754f2709b7a9″ >

<http:listener doc:name=”Listener” doc:id=”8d926367-125f-433a-8b2c-30a5ee3f4a00″ config-ref=”HTTP_Listener_config” path=”/rel”/>

<logger level=”INFO” doc:name=”Logger” doc:id=”a75c80d0-5ac3-45b0-9cd9-a4d1ea3a6239″ />

<anypoint-mq:publish doc:name=”Publish to Application Logic Flow” doc:id=”6c83ab15-892e-4394-83de-61380abce0c5″ config-ref=”Anypoint_MQ_Config” destination=”test”/>

<set-payload value=”#[payload]” doc:name=”Map Response” doc:id=”6e25a465-da88-43a2-a7cc-38bc6923571a” />

</flow>

<flow name=”Application_Logic_Flow” doc:id=”671e12cb-10b2-4c77-a38d-20b4c7a016de” >

<anypoint-mq:subscriber doc:name=”Subscriber” doc:id=”baf5c096-04a1-48f2-a4ea-70618c94bbf7″ config-ref=”Anypoint_MQ_Config” destination=”asd”/>

<logger level=”INFO” doc:name=”Logger” doc:id=”08129db4-9e56-40c0-b310-b4a3a4ef16de” />

<set-variable value=”” doc:name=”Set Ack Token” doc:id=”f40767fa-55ca-4bf9-a4bf-a2dcab6bcf5a” variableName=”1″/>

<flow-ref doc:name=”Business Logic flow” doc:id=”a1a0f350-46ad-47b2-bc92-35e94eec5a66″ name=”testFlow1″/>

<anypoint-mq:ack doc:name=”Manual Ack” doc:id=”51953e51-e0c4-4f03-95b4-dd29010bd20b” config-ref=”Anypoint_MQ_Config” ackToken=”q”/>

<error-handler >

<on-error-propagate enableNotifications=”true” logException=”true” doc:name=”On Error Propagate” doc:id=”947603de-0ddc-493f-afe3-842b707a2513″ >

<logger level=”INFO” doc:name=”Error Log” doc:id=”ed44df2d-af80-4618-b253-a9c755c9ff90″ />

<anypoint-mq:nack doc:name=”Manual Nack” doc:id=”5a99c625-b2b3-4497-9ae1-5febb3728932″ config-ref=”Anypoint_MQ_Config” ackToken=”1″/>

</on-error-propagate>

</error-handler>

</flow>

 

The type of queue used will have its own merits and demerits.

JMS / MQ VM
Highly reliable. Application status does not affect data. Data is lost if the application is down
Feature rich; Highly flexible Limited features
Additional software is required Easy to use as it is shipped with Mule
Slower than VM as it is a different process Faster than JMS as everything is in-memory

Persistent Store

Another approach is by using a persistent store like a database or an object store to persistently hold the data. This method is relatively more flexible than the queues approach as we have better control of data while using a store like database. Based on the need, the processing can be done in real-time or in a delayed manner.

Real time processing:

  1. Upon receiving an incoming request, the system attempts to process it instantly.
  2. In case of a happy path, the thread ends normally as no additional processing is required.
  3. If there is any failure in processing, the exception will be caught and the request message will be written to a database / object store.
  4. The unsuccessful requests are retried in a different thread that is triggered by a scheduler configured to a desired frequency.
  5. If a record is processed successfully, then it can be deleted from the database / object store. In case of failure, it’s retry count can be updated.

<flow name=”Realtime_Processing” doc:id=”2da184ff-0b21-472c-85fa-454845a92830″ >

<http:listener doc:name=”Listener” doc:id=”7d601a07-125a-4986-bac2-f2b2b0ca4f5b” config-ref=”HTTP_Listener_config” path=”/real”/>

<logger level=”INFO” doc:name=”Logger” doc:id=”909b1d81-8780-47e1-ba4c-5e54b52121dc” />

<flow-ref doc:name=”Business Logic Flow” doc:id=”903d29bc-f5f3-46ab-a2d5-ca6e75a6e141″ name=”testFlow1″/>

<set-payload value=”#[payload]” doc:name=”Map Response” doc:id=”b39b3143-78b6-4e33-b065-13e94db7d308″ />

<error-handler >

<on-error-continue enableNotifications=”true” logException=”true” doc:name=”On Error Continue” doc:id=”a4f1ab14-e47a-4f24-99c9-6d2e87a70ed9″ >

<logger level=”INFO” doc:name=”Error log” doc:id=”0ab7ae74-2f3b-47da-b899-04a31b0dbf3c” />

<os:store doc:id=”af6c2a6d-e9f6-4e2e-a9bb-0e928b3f15bf” key=”1″ doc:name=”Store Data to Object Store”/>

</on-error-continue>

</error-handler>

</flow>

 

Delayed processing:

  • Upon receiving an incoming request, the system creates an entry in the database / object store with the request received and closes the thread.
  • A scheduler must be configured with a desired frequency, that picks the messages and processes them.
  • If a record is processed successfully, then it can be deleted from the database / object store. In case of failure, it’s retry count can be updated.

This approach has many advantages like request tracking, flexible processing order instead of FCFS and high reliability.

<flow name=”Delayed_Processing” doc:id=”f51c18f1-6dec-42ab-9a8b-722796b53b03″ >

<http:listener doc:name=”Listener” doc:id=”d4ca7a94-e89e-476e-94f9-8ba90fbf3104″ config-ref=”HTTP_Listener_config” path=”/delay”/>

<logger level=”INFO” doc:name=”Logger” doc:id=”79afd773-f143-41fc-b44d-cd4b2ef6f596″ />

<os:store doc:name=”Store Data to Object Store” doc:id=”283f2d05-2512-4158-9516-b5de4a7a8b50″ key=”1″ />

<set-payload value=”#[payload]” doc:name=”Map Response” doc:id=”0f23d04d-07c4-4319-b2c5-3ff17f0ef7e2″ />

</flow>

 

Scheduler_Flow is common for both real-time and delayed processing.

<flow name=”Scheduler_Flow” doc:id=”007e39c6-31b7-414b-815e-a0aeb2058292″ >

<scheduler doc:name=”Scheduler” doc:id=”68f718eb-ff19-4c02-811e-bc1fd52a0c80″ >

<scheduling-strategy >

<fixed-frequency startDelay=”110″ timeUnit=”DAYS”/>

</scheduling-strategy>

</scheduler>

<logger level=”INFO” doc:name=”Logger” doc:id=”9cbbdd6e-2e17-46e6-9176-3ac1c7c03a85″ />

<os:retrieve-all-keys doc:name=”Retrieve all keys from Object Store” doc:id=”3189dcf9-529e-44a5-8404-2a0d9b000146″ />

<foreach doc:name=”For Each Key” doc:id=”05de884e-4b93-4fb5-8b18-8083706691ef” >

<try doc:name=”Try” doc:id=”3b337c67-637e-42af-8887-1ba159607b21″ >

<flow-ref doc:name=”Business Logic Flow” doc:id=”336ac245-cb79-4856-b9a9-98b92a421ac2″ name=”testFlow1″ />

<os:remove doc:name=”Remove Key on Success” doc:id=”84b262cc-8c7d-4f09-a2b0-4b0b35eb6483″ key=”l” />

<error-handler >

<on-error-continue enableNotifications=”true” logException=”true” doc:name=”On Error Continue” doc:id=”fae14780-4a68-4f7e-871b-2695900795f9″ >

<logger level=”INFO” doc:name=”Error Log” doc:id=”e0ad4366-0ae2-4e9c-b2bb-4cf92dde2b1d” />

</on-error-continue>

</error-handler>

</try>

</foreach>

</flow>

 

Conclusion

Based on the need for reliability, out of the box VM feature can be leveraged to implement the reliability and similarly it is a good idea to use storage like Object Store or Database to implement the solution. If the customer is looking for an end to end solution for 100% reliability then it is recommended to leverage Messaging based solutions like Any point MQ or Kafka or Amazon SQS and etc. as part of the solution. 

admin

No Comments

Post a Comment