In this post we will show how we can do the following :
- Integrate spring boot with Apache Ignite
- How to enable and use persistent durable memory feature of Apache Ignite which can persist your cache data to the file disk to survive crash or restart so you can avoid data losing.
- How to execute SQL queries over ignite caches
- How to unit test and integration test ignite with spring boot
- Simple Jenkins pipeline reference
- Code repository in GitHub : GithubRepo
what is Ignite durable memory ?
Apache Ignite memory-centric platform is based on the durable memory architecture that allows storing and processing data and indexes both in memory and on disk when the Ignite Native Persistence feature is enabled. The durable memory architecture helps achieve in-memory performance with the durability of disk using all the available resources of the cluster
What is ignite data-grid SQL queries ?
Ignite supports a very elegant query API with support for Predicate-based Scan Queries, SQL Queries (ANSI-99 compliant), and Text Queries. For SQL queries ignites supports in-memory indexing, so all the data lookups are extremely fast. If you are caching your data in off-heap memory, then query indexes will also be cached in off-heap memory as well.
Ignite also provides support for custom indexing via IndexingSpi and SpiQuery class.
more information on : https://apacheignite.readme.io/docs/cache-queries
So to have Apache Ignite server node integrated and started in your spring boot app we need to do the following :
- Add the following maven dependencies to your spring boot app pom file
| <dependency> | |
| <groupId>org.apache.ignite</groupId> | |
| <artifactId>ignite-core</artifactId> | |
| <version>${ignite.version}</version> | |
| </dependency> | |
| <dependency> | |
| <groupId>org.apache.ignite</groupId> | |
| <artifactId>ignite-spring</artifactId> | |
| <version>${ignite.version}</version> | |
| </dependency> | |
| <dependency> | |
| <groupId>org.apache.ignite</groupId> | |
| <artifactId>ignite-indexing</artifactId> | |
| <version>${ignite.version}</version> | |
| </dependency> | |
| <dependency> | |
| <groupId>org.apache.ignite</groupId> | |
| <artifactId>ignite-slf4j</artifactId> | |
| <version>${ignite.version}</version> | |
| </dependency> |
- Define ignite configuration via java DSL for better portability and management as a spring configuration and the properties values will be loaded from the application.yml file :
| @Configuration | |
| public class AlertManagerConfiguration { | |
| @Value("${mail.service.baseUrl}") | |
| private String baseUrl; | |
| @Value("${mail.service.user}") | |
| private String user; | |
| @Value("${mail.service.password}") | |
| private String password; | |
| @Value("${enableFilePersistence}") | |
| private boolean enableFilePersistence; | |
| @Value("${igniteConnectorPort}") | |
| private int igniteConnectorPort; | |
| @Value("${igniteServerPortRange}") | |
| private String igniteServerPortRange; | |
| @Value("${ignitePersistenceFilePath}") | |
| private String ignitePersistenceFilePath; | |
| @Bean | |
| IgniteConfiguration igniteConfiguration() { | |
| IgniteConfiguration igniteConfiguration = new IgniteConfiguration(); | |
| igniteConfiguration.setClientMode(false); | |
| // durable file memory persistence | |
| if(enableFilePersistence){ | |
| PersistentStoreConfiguration persistentStoreConfiguration = new PersistentStoreConfiguration(); | |
| persistentStoreConfiguration.setPersistentStorePath("./data/store"); | |
| persistentStoreConfiguration.setWalArchivePath("./data/walArchive"); | |
| persistentStoreConfiguration.setWalStorePath("./data/walStore"); | |
| igniteConfiguration.setPersistentStoreConfiguration(persistentStoreConfiguration); | |
| } | |
| // connector configuration | |
| ConnectorConfiguration connectorConfiguration=new ConnectorConfiguration(); | |
| connectorConfiguration.setPort(igniteConnectorPort); | |
| // common ignite configuration | |
| igniteConfiguration.setMetricsLogFrequency(0); | |
| igniteConfiguration.setQueryThreadPoolSize(2); | |
| igniteConfiguration.setDataStreamerThreadPoolSize(1); | |
| igniteConfiguration.setManagementThreadPoolSize(2); | |
| igniteConfiguration.setPublicThreadPoolSize(2); | |
| igniteConfiguration.setSystemThreadPoolSize(2); | |
| igniteConfiguration.setRebalanceThreadPoolSize(1); | |
| igniteConfiguration.setAsyncCallbackPoolSize(2); | |
| igniteConfiguration.setPeerClassLoadingEnabled(false); | |
| igniteConfiguration.setIgniteInstanceName("alertsGrid"); | |
| BinaryConfiguration binaryConfiguration = new BinaryConfiguration(); | |
| binaryConfiguration.setCompactFooter(false); | |
| igniteConfiguration.setBinaryConfiguration(binaryConfiguration); | |
| // cluster tcp configuration | |
| TcpDiscoverySpi tcpDiscoverySpi=new TcpDiscoverySpi(); | |
| TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder=new TcpDiscoveryVmIpFinder(); | |
| // need to be changed when it come to real cluster | |
| tcpDiscoveryVmIpFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509")); | |
| tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder); | |
| igniteConfiguration.setDiscoverySpi(new TcpDiscoverySpi()); | |
| // cache configuration | |
| CacheConfiguration alerts=new CacheConfiguration(); | |
| alerts.setCopyOnRead(false); | |
| // as we have one node for now | |
| alerts.setBackups(0); | |
| alerts.setAtomicityMode(CacheAtomicityMode.ATOMIC); | |
| alerts.setName("Alerts"); | |
| alerts.setIndexedTypes(String.class,AlertEntry.class); | |
| CacheConfiguration alertsConfig=new CacheConfiguration(); | |
| alertsConfig.setCopyOnRead(false); | |
| // as we have one node for now | |
| alertsConfig.setBackups(0); | |
| alertsConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC); | |
| alertsConfig.setName("AlertsConfig"); | |
| alertsConfig.setIndexedTypes(String.class,AlertConfigEntry.class); | |
| igniteConfiguration.setCacheConfiguration(alerts,alertsConfig); | |
| return igniteConfiguration; | |
| } | |
| @Bean(destroyMethod = "close") | |
| Ignite ignite(IgniteConfiguration igniteConfiguration) throws IgniteException { | |
| final Ignite start = Ignition.start(igniteConfiguration); | |
| start.active(true); | |
| return start; | |
| } | |
| } |
- then you can just inject ignite instance as a Spring bean which make unit testing much easier
| @Component | |
| public class IgniteAlertConfigStore implements AlertsConfigStore { | |
| private static final Logger logger = LoggerFactory.getLogger(IgniteAlertConfigStore.class); | |
| // here it will be injected as a spring bean | |
| @Autowired | |
| private Ignite ignite; | |
| @Override | |
| public AlertConfigEntry getConfigForServiceIdCodeId(String serviceId, String codeId) { | |
| return Optional.ofNullable(getAlertsConfigCache().get(serviceId + "_" + codeId)) | |
| .orElseThrow(() –> new ResourceNotFoundException(String.format("Alert config for %s with %s not found", serviceId,codeId))); | |
| } | |
| @Override | |
| public void update(String serviceId, String codeId, AlertConfigEntry alertConfigEntry) { | |
| getAlertsConfigCache().put(serviceId + "_" + codeId, alertConfigEntry); | |
| } | |
| @Override | |
| public Optional<AlertConfigEntry> getConfigForServiceIdCodeIdCount(String serviceId, String codeId) { | |
| return Optional.ofNullable(getAlertsConfigCache().get(serviceId + "_" + codeId)); | |
| } | |
| public Cache<String, AlertConfigEntry> getAlertsConfigCache() { | |
| return ignite.getOrCreateCache(CacheNames.AlertsConfig.name()); | |
| } | |
| } |
How to enable Ignite durable memory :
| // durable file memory persistence | |
| if(enableFilePersistence){ | |
| PersistentStoreConfiguration persistentStoreConfiguration = new PersistentStoreConfiguration(); | |
| persistentStoreConfiguration.setPersistentStorePath("./data/store"); | |
| persistentStoreConfiguration.setWalArchivePath("./data/walArchive"); | |
| persistentStoreConfiguration.setWalStorePath("./data/walStore"); | |
| igniteConfiguration.setPersistentStoreConfiguration(persistentStoreConfiguration); | |
| } |
How to use Ignite SQL queries over in memory storage:
| @Override | |
| public List<AlertEntry> getAlertForServiceId(String serviceId) { | |
| final String sql = "serviceId = ?"; | |
| // create the sql query object with entity type of the value part of the key value cache | |
| SqlQuery<String, AlertEntry> query = new SqlQuery<>(AlertEntry.class, sql); | |
| // set the query params | |
| query.setArgs(serviceId); | |
| //then execute it over the cache | |
| return Optional.ofNullable(getAlertsCache().query(query).getAll().stream().map(stringAlertEntryEntry –> stringAlertEntryEntry.getValue()).collect(Collectors.toList())) | |
| .orElseThrow(() –> new ResourceNotFoundException(String.format("Alert for %s not found", serviceId))); | |
| } |
How to do atomic thread safe action over the same record via cache invoke API:
| @Override | |
| public void updateAlertEntry(String serviceId, String serviceCode, AlertEntry alertEntry) { | |
| final IgniteCache<String, AlertEntry> alertsCache = getAlertsCache(); | |
| // update the alert entry via cache invoke for atomicity | |
| alertsCache.invoke(alertEntry.getAlertId(), (mutableEntry, objects) –> { | |
| if (mutableEntry.exists() && mutableEntry.getValue() != null) { | |
| logger.debug("updating alert entry into the cache store invoke: {},{}", serviceId, serviceCode); | |
| mutableEntry.setValue(alertEntry); | |
| } else { | |
| throw new ResourceNotFoundException(String.format("Alert for %s with %s not found", serviceId, serviceCode)); | |
| } | |
| // by api design nothing needed here | |
| return null; | |
| }); | |
| } |
How to unit test Apache ignite usage in spring boot service :
| @RunWith(MockitoJUnitRunner.class) | |
| public class IgniteAlertsSoreTest { | |
| @Mock | |
| private Ignite ignite; | |
| @Mock | |
| Cache<String, List<AlertEntry>> cache; | |
| @Mock | |
| IgniteCache IgniteCache; | |
| @InjectMocks | |
| private IgniteAlertsStore igniteAlertsStore; | |
| //simulate the needed behaviour for the mocked ignite cache | |
| @Before | |
| public void setUp() throws Exception { | |
| when(ignite.getOrCreateCache(anyString())).thenReturn(IgniteCache); | |
| List<AlertEntry> entries=new ArrayList<>(); | |
| entries.add(AlertEntry.builder().errorCode("errorCode").build()); | |
| when(IgniteCache.get(anyString())).thenReturn(entries); | |
| } | |
| @Test | |
| public void getAllAlerts() throws Exception { | |
| assertEquals(igniteAlertsStore.getAlertForServiceId("serviceId").get(0).getErrorCode(),"errorCode"); | |
| } | |
| } |
How to trigger integration test with Ignite, check test resources as well :
| @RunWith(SpringRunner.class) | |
| @SpringBootTest(classes = AlertManagerApplication.class,webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) | |
| @ActiveProfiles("INTEGRATION_TEST") | |
| public class AlertManagerApplicationIT { | |
| @LocalServerPort | |
| private int port; | |
| @Autowired | |
| private TestRestTemplate template; | |
| private URL base; | |
| @Before | |
| public void setUp() throws Exception { | |
| this.base = new URL("http://localhost:" + port + "/"); | |
| } | |
| // then add your integration test which will include real started ignite server node whoch will be closed once the integration test is done | |
| @Test | |
| public void contextLoads() { | |
| assertTrue(template.getForEntity(base+"/health",String.class).getStatusCode().is2xxSuccessful()); | |
| } | |
| } |
How to run and test the application over swagger rest api :
- build the project via maven : mvn clean install
- you can run it from IDEA via AlertManagerApplication.java or via java -jar jarName
- swagger which contain the REST API and REST API model documentation will be accessible on the URL below where you can start triggering different REST API calls exposed by the spring boot app:
http://localhost:8080/swagger-ui.html#/
- if you STOP the app or restart it and do query again , you will find all created entities from last run so it survived the crash plus any possible restart
- you can build a portable docker image of the whole app using maven Spotify docker plugin if you wish
References :
- Apache Ignite : https://apacheignite.readme.io/docs
- Code repository in GitHub : GithubRepo
Thanks for the great article about Apache Ignite.
I’ve a remark about the unit Test (IgniteAlertsSoreTest). I’ve tried it from the Git code and I got a NullPointerException with the query in IgniteAlertsStore. Is it actually working on your side?
LikeLike
Thanks for your comment , indeed i forgot to properly update the unit test , now it is fixed and pushed into github repo , thx a lot again for that catch !
LikeLike
Hi sir, i am trying to run the github project but i got this exception: https://pastebin.com/TGktvtEA
could you help me? I opened a github issue.
thanks
LikeLike
Hey , i have commented on github issue , please try out my comment and let me know the output !
LikeLike
Hi Sir, i am trying to use the github project in order to learn but when i try to run it using java -jar, i got an exception. I have opened an issue in github, if you could help, i will apreciate it!
Thank you sir.
LikeLike