Spring boot with Apache ignite persistent durable memory storage plus sql queries over ignite cache

In this post we will show how we can do the following :

  1. Integrate spring boot with Apache Ignite
  2. How to enable and use persistent durable memory feature of Apache Ignite which can persist your cache data to the file disk to survive crash or restart so you can avoid data losing.
  3. How to execute SQL queries over ignite caches
  4. How to unit test and integration test ignite with spring boot
  5. Simple Jenkins pipeline reference
  6. Code repository in GitHub : GithubRepo

ignitedurablememory

what is Ignite durable memory ?

Apache Ignite memory-centric platform is based on the durable memory architecture that allows storing and processing data and indexes both in memory and on disk when the Ignite Native Persistence feature is enabled. The durable memory architecture helps achieve in-memory performance with the durability of disk using all the available resources of the cluster

What is ignite data-grid SQL queries ?

Ignite supports a very elegant query API with support for Predicate-based Scan Queries, SQL Queries (ANSI-99 compliant), and Text Queries. For SQL queries ignites supports in-memory indexing, so all the data lookups are extremely fast. If you are caching your data in off-heap memory, then query indexes will also be cached in off-heap memory as well.

Ignite also provides support for custom indexing via IndexingSpi and SpiQuery class.

more information on : https://apacheignite.readme.io/docs/cache-queries

So to have Apache Ignite server node integrated and started in your spring boot app we need to do the following :

  1. Add the following maven dependencies to your spring boot app pom file
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-core</artifactId>
<version>${ignite.version}</version>
</dependency>
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-spring</artifactId>
<version>${ignite.version}</version>
</dependency>
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-indexing</artifactId>
<version>${ignite.version}</version>
</dependency>
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-slf4j</artifactId>
<version>${ignite.version}</version>
</dependency>

view raw
igniteConfig.xml
hosted with ❤ by GitHub

  1. Define ignite configuration via java DSL for better portability and management as a spring configuration and the properties values will be loaded from the application.yml file :
@Configuration
public class AlertManagerConfiguration {
@Value("${mail.service.baseUrl}")
private String baseUrl;
@Value("${mail.service.user}")
private String user;
@Value("${mail.service.password}")
private String password;
@Value("${enableFilePersistence}")
private boolean enableFilePersistence;
@Value("${igniteConnectorPort}")
private int igniteConnectorPort;
@Value("${igniteServerPortRange}")
private String igniteServerPortRange;
@Value("${ignitePersistenceFilePath}")
private String ignitePersistenceFilePath;
@Bean
IgniteConfiguration igniteConfiguration() {
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setClientMode(false);
// durable file memory persistence
if(enableFilePersistence){
PersistentStoreConfiguration persistentStoreConfiguration = new PersistentStoreConfiguration();
persistentStoreConfiguration.setPersistentStorePath("./data/store");
persistentStoreConfiguration.setWalArchivePath("./data/walArchive");
persistentStoreConfiguration.setWalStorePath("./data/walStore");
igniteConfiguration.setPersistentStoreConfiguration(persistentStoreConfiguration);
}
// connector configuration
ConnectorConfiguration connectorConfiguration=new ConnectorConfiguration();
connectorConfiguration.setPort(igniteConnectorPort);
// common ignite configuration
igniteConfiguration.setMetricsLogFrequency(0);
igniteConfiguration.setQueryThreadPoolSize(2);
igniteConfiguration.setDataStreamerThreadPoolSize(1);
igniteConfiguration.setManagementThreadPoolSize(2);
igniteConfiguration.setPublicThreadPoolSize(2);
igniteConfiguration.setSystemThreadPoolSize(2);
igniteConfiguration.setRebalanceThreadPoolSize(1);
igniteConfiguration.setAsyncCallbackPoolSize(2);
igniteConfiguration.setPeerClassLoadingEnabled(false);
igniteConfiguration.setIgniteInstanceName("alertsGrid");
BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
binaryConfiguration.setCompactFooter(false);
igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
// cluster tcp configuration
TcpDiscoverySpi tcpDiscoverySpi=new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder=new TcpDiscoveryVmIpFinder();
// need to be changed when it come to real cluster
tcpDiscoveryVmIpFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);
igniteConfiguration.setDiscoverySpi(new TcpDiscoverySpi());
// cache configuration
CacheConfiguration alerts=new CacheConfiguration();
alerts.setCopyOnRead(false);
// as we have one node for now
alerts.setBackups(0);
alerts.setAtomicityMode(CacheAtomicityMode.ATOMIC);
alerts.setName("Alerts");
alerts.setIndexedTypes(String.class,AlertEntry.class);
CacheConfiguration alertsConfig=new CacheConfiguration();
alertsConfig.setCopyOnRead(false);
// as we have one node for now
alertsConfig.setBackups(0);
alertsConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
alertsConfig.setName("AlertsConfig");
alertsConfig.setIndexedTypes(String.class,AlertConfigEntry.class);
igniteConfiguration.setCacheConfiguration(alerts,alertsConfig);
return igniteConfiguration;
}
@Bean(destroyMethod = "close")
Ignite ignite(IgniteConfiguration igniteConfiguration) throws IgniteException {
final Ignite start = Ignition.start(igniteConfiguration);
start.active(true);
return start;
}
}

view raw
igniteConfig.java
hosted with ❤ by GitHub

  1. then you can just inject ignite instance as a Spring bean which make unit testing much easier
@Component
public class IgniteAlertConfigStore implements AlertsConfigStore {
private static final Logger logger = LoggerFactory.getLogger(IgniteAlertConfigStore.class);
// here it will be injected as a spring bean
@Autowired
private Ignite ignite;
@Override
public AlertConfigEntry getConfigForServiceIdCodeId(String serviceId, String codeId) {
return Optional.ofNullable(getAlertsConfigCache().get(serviceId + "_" + codeId))
.orElseThrow(() > new ResourceNotFoundException(String.format("Alert config for %s with %s not found", serviceId,codeId)));
}
@Override
public void update(String serviceId, String codeId, AlertConfigEntry alertConfigEntry) {
getAlertsConfigCache().put(serviceId + "_" + codeId, alertConfigEntry);
}
@Override
public Optional<AlertConfigEntry> getConfigForServiceIdCodeIdCount(String serviceId, String codeId) {
return Optional.ofNullable(getAlertsConfigCache().get(serviceId + "_" + codeId));
}
public Cache<String, AlertConfigEntry> getAlertsConfigCache() {
return ignite.getOrCreateCache(CacheNames.AlertsConfig.name());
}
}

view raw
InjectIgnite.java
hosted with ❤ by GitHub

How to enable Ignite durable memory :

// durable file memory persistence
if(enableFilePersistence){
PersistentStoreConfiguration persistentStoreConfiguration = new PersistentStoreConfiguration();
persistentStoreConfiguration.setPersistentStorePath("./data/store");
persistentStoreConfiguration.setWalArchivePath("./data/walArchive");
persistentStoreConfiguration.setWalStorePath("./data/walStore");
igniteConfiguration.setPersistentStoreConfiguration(persistentStoreConfiguration);
}

How to use Ignite SQL queries over in memory storage:

@Override
public List<AlertEntry> getAlertForServiceId(String serviceId) {
final String sql = "serviceId = ?";
// create the sql query object with entity type of the value part of the key value cache
SqlQuery<String, AlertEntry> query = new SqlQuery<>(AlertEntry.class, sql);
// set the query params
query.setArgs(serviceId);
//then execute it over the cache
return Optional.ofNullable(getAlertsCache().query(query).getAll().stream().map(stringAlertEntryEntry > stringAlertEntryEntry.getValue()).collect(Collectors.toList()))
.orElseThrow(() > new ResourceNotFoundException(String.format("Alert for %s not found", serviceId)));
}

view raw
SqlQuery.java
hosted with ❤ by GitHub

How to do atomic thread safe action over the same record via cache invoke API:

@Override
public void updateAlertEntry(String serviceId, String serviceCode, AlertEntry alertEntry) {
final IgniteCache<String, AlertEntry> alertsCache = getAlertsCache();
// update the alert entry via cache invoke for atomicity
alertsCache.invoke(alertEntry.getAlertId(), (mutableEntry, objects) > {
if (mutableEntry.exists() && mutableEntry.getValue() != null) {
logger.debug("updating alert entry into the cache store invoke: {},{}", serviceId, serviceCode);
mutableEntry.setValue(alertEntry);
} else {
throw new ResourceNotFoundException(String.format("Alert for %s with %s not found", serviceId, serviceCode));
}
// by api design nothing needed here
return null;
});
}

How to unit test Apache ignite usage in spring boot service :

@RunWith(MockitoJUnitRunner.class)
public class IgniteAlertsSoreTest {
@Mock
private Ignite ignite;
@Mock
Cache<String, List<AlertEntry>> cache;
@Mock
IgniteCache IgniteCache;
@InjectMocks
private IgniteAlertsStore igniteAlertsStore;
//simulate the needed behaviour for the mocked ignite cache
@Before
public void setUp() throws Exception {
when(ignite.getOrCreateCache(anyString())).thenReturn(IgniteCache);
List<AlertEntry> entries=new ArrayList<>();
entries.add(AlertEntry.builder().errorCode("errorCode").build());
when(IgniteCache.get(anyString())).thenReturn(entries);
}
@Test
public void getAllAlerts() throws Exception {
assertEquals(igniteAlertsStore.getAlertForServiceId("serviceId").get(0).getErrorCode(),"errorCode");
}
}

view raw
IgniteTest.java
hosted with ❤ by GitHub

How to trigger integration test with Ignite, check test resources as well :

@RunWith(SpringRunner.class)
@SpringBootTest(classes = AlertManagerApplication.class,webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ActiveProfiles("INTEGRATION_TEST")
public class AlertManagerApplicationIT {
@LocalServerPort
private int port;
@Autowired
private TestRestTemplate template;
private URL base;
@Before
public void setUp() throws Exception {
this.base = new URL("http://localhost:" + port + "/");
}
// then add your integration test which will include real started ignite server node whoch will be closed once the integration test is done
@Test
public void contextLoads() {
assertTrue(template.getForEntity(base+"/health",String.class).getStatusCode().is2xxSuccessful());
}
}

view raw
integrationTest.java
hosted with ❤ by GitHub

How to run and test the application over swagger rest api :

  • build the project via maven : mvn clean install
  • you can run it from IDEA via AlertManagerApplication.java or via java -jar jarName

Screen Shot 2017-11-17 at 16.28.03.png

  • swagger which contain the REST API and REST API model documentation will be accessible on the URL below where you can start triggering different REST API calls exposed by the spring boot app:

   http://localhost:8080/swagger-ui.html#/

Screen Shot 2017-11-17 at 16.24.11

  • if you STOP the app or restart it and do query again , you will find all created entities from last run so it survived the crash plus any possible restart
  • you can build a portable docker image of the whole app using maven Spotify docker plugin if you wish

 

References :

 

 

5 thoughts on “Spring boot with Apache ignite persistent durable memory storage plus sql queries over ignite cache

  1. roussep April 26, 2018 / 2:45 pm

    Thanks for the great article about Apache Ignite.
    I’ve a remark about the unit Test (IgniteAlertsSoreTest). I’ve tried it from the Git code and I got a NullPointerException with the query in IgniteAlertsStore. Is it actually working on your side?

    Like

    • mromeh May 2, 2018 / 10:35 am

      Thanks for your comment , indeed i forgot to properly update the unit test , now it is fixed and pushed into github repo , thx a lot again for that catch !

      Like

    • mromeh January 24, 2019 / 8:07 pm

      Hey , i have commented on github issue , please try out my comment and let me know the output !

      Like

  2. Alonso January 24, 2019 / 12:06 pm

    Hi Sir, i am trying to use the github project in order to learn but when i try to run it using java -jar, i got an exception. I have opened an issue in github, if you could help, i will apreciate it!
    Thank you sir.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s