Spring boot with Apache Ignite fail fast distributed map reduce closures

Here we are going a cover a case need in Apache ignite , what if you want to do distributed compute jobs that do data computations or external service calls using Apache Ignite distributed closures that has map reduce nature and fail fast once of the computations fail or it has the unexpected results , how to do that ? below we are going to explain that .

mapReduce

  1. The main node will submit a collection of Ignite callable plus the custom fail fast reducer that we will explain into details later
  2. The list of jobs will be distributed between the server nodes in the current cluster topology with the same cluster group for actual execution and to use the distributed parallel map reduce nature execution of Ignite compute grid in synchronous or asynchronous non blocking way
  3. each single Job will return the result or error to the fail fast reducer which upon receiving the results of each single compute task , it will determine if it can keep collection other results before reducing the final aggregated result or fail fast immediately once one of the jobs failed or has the unexpected  results

So how it is  implemented ?

  • The fail fast Ignite compute grid reducer :

  • Generic Ignite compute utility to trigger the map reduce tasks in synchronous or asynchronous non blocking :

  • The custom aggregated reducer response class:

  • The single task response class:

  • Example service for calling the Ignite compute grid with the distributed closures and we will use the synchronous way for testing the execution :

  • Unit test for fail fast and successful cases using spring boot integration test:

References :

  1. Ignite compute grid : https://apacheignite.readme.io/docs/compute-grid
  2. The code is on GitHub : https://github.com/Romeh/spring-boot-ignite

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s