Insightful Applications

By Anthony Golia, Chief Architect, Red Hat & Josh West, Principal Solution Architect, Red Hat

The delivery style of data analytics is evolving. Currently, business intelligence (BI) tools are separate from business applications. This is evolving into embedded delivery of contextual insight within business applications, which have been called “insightful applications.” Frameworks that provide introspection into business applications’ data in motion will be key to this convergence. The rest of this article describes the benefits of this convergence as well as what’s causing it to happen now.

The need for insightful applications
Traditionally, the typical architecture for data analytics is made up of a centralized database (data warehouse, datamart, or data lake) that is simultaneously read and written to by multiple business applications and BI tools. This is sometimes called a monolith system. This architecture provides benefits such as: atomicity, consistency, isolation, and durability (ACID) transactions; a single place to look; being well understood; and providing a single point of manageability. But the shared database model has one major maintenance requirement that has become harder to execute over time: it is hard to make changes to it. For example, it’s hard to get all the constituents to agree on a work window to add a feature, upgrade, or even more difficult, a schema change to that shared database. However, traditional databases will continue to be a key component in the world of insightful applications. They will provide the current state of record—a materialized view of a persistent log/stream of events that happened in the past.

Increased competition in the financial community is the key reason data analytic systems now require agility. Furthermore, it’s important that these changes not impact business processes—they should not decrease stability or impede the progress of ancillary systems. Some examples of the increased competition are less-regulated broker/dealers, FinTechs, and roboadvisors.

Insightful applications and microservices
The microservices model is also an impetus for more introspection into our applications’ data in motion. Jonas Boner says in his Reactive Microservices Architecture, “Creating a system from a collection of small, isolated services, each of which owns its data, and is independently isolated… Services integrate with other services in order to form a cohesive system that’s far more flexible than the typical enterprise systems we build today.”1

Given that any microservice is part of a larger system, those services need to communicate with each other. This is typically done via APIs or message passing. Either way, it’s via some form of data in motion mechanism. But how do we reconcile the need for consistent holistic reporting (e.g., for regulatory reasons such as audit logging, data lineage) with this highly distributed system with streaming data? Data warehouses, datamarts, and/or data lakes can still be used in this use case, but instead of being a runtime dependency for business applications, they will be a repository for events emitted by business applications, also known as event sourcing. Each service publishes events (immutable, point in time, state changes) to an event store (e.g., directly to a database or to an event bus). The event store is used for reporting and/or analytics only. It is not used as a communications mechanism among the microservices, otherwise it would be a runtime dependency and you’d be back to it being hard to make changes to the database.

Insightful applications now
Why insightful applications now? Two main reasons: 1) multiple frameworks for stream processing are now maturing thanks to the increased pace of innovation in the open source community (e.g., Spark, Kafka, Debezium, Camel, data grids, and OpenShift), and 2) organizations now have easy access to computing power at commodity prices, thanks to private and public cloud.

Due to these factors, we can now deliver on the promise of predictive analytics and real-time operational analytics for use cases such as insider threat and recommendation systems.

 

References:

http://blog.christianposta.com/microservices/the-hardest-part-about-microservices-data/

1Bonér, Jonas. Reactive Microservices Architecture: Design Principles for Distributed Systems: O’Reilly, 2016. Book.

 

About Red Hat

Red Hat is the world’s leading provider of open source solutions, using a community-powered approach to provide reliable and high-performing cloud, virtualization, storage, Linux, and middleware technologies.Red Hat also offers award-winning support, training, and consulting services. Red Hat is an S&P company with more than 70 offices spanning the globe, empowering its customer’s businesses.

Please visit www.redhat.com for more information.

Advertisement

Follow Us:

Sitemap | Privacy | Copyright © © 2017, WSTA®, All Rights Reserved.