Native Apps At The Client & Cloud

Srinivasan Sundara Rajan

Subscribe to Srinivasan Sundara Rajan: eMailAlertsEmail Alerts
Get Srinivasan Sundara Rajan: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Cloud Computing, IBM Journal, SOA & WOA Magazine, Big Data on Ulitzer


Big Data OLTP with IBM DB2 BLU and DB2 pureScale

Big Data and analytical processing

Big Data as we know it today is more aligned to the analytical processing of large quantities of data. All the predominant use cases identified by the Big Data product vendors are more aligned with analytical processing. For example one of the major use cases of Big Data is about utilizing social media data to get into advertisement targeting. Naturally this kind of processing analyzing lots of unstructured data and come up with predictions on customer preferences and this use case is aligned with analytical processing. To support these kinds of analytical processing Columnar databases have emerged as a natural extension to Big Data processing. Columnar databases only reads columns involved in the query and not the entire row, making it a perfect fit for analytical processing.

Online Transaction Processing
On the other side of the Enterprise Data Access pattern we have the most important OLTP pattern. Most of the real life events that are important for the survival of an enterprise like an online ecommerce transaction need to follow ACID property, where by the Atomicity, Consistency, Isolation and Durability of the transaction needs to be maintained. Due to this need OLTP applications and their natural allies, the relational databases, continue to be popular and an essential part of enterprise data access patterns.

OLTP + Big Data
Advent of Big Data and the associated analytics initially looked like a separate stream from OLTP , mainly because of the data integration challenges. For example the tools meant for big data processing were from the new era like Hadoop, where as invariably OLTP applications like banking were built out of legacy platforms, mainly platforms like Mainframe. Also the response time needs for OLTP applications are very critical and most of the batch oriented big data processing platforms cannot cater to them.

However for use cases like fraud detection of Online transactions, we needed to combine the ACID nature of the transactions with the analytical capabilities enabled by the Big Data Platform, so far we did not have very many unified platforms to achieve this combination.

However the traditional RDBMS platforms which all along cater to the OLTP needs have started to release native Big Data Integration features as part of their offering this has facilitated easier integration of Big Data Analytics with OLTP.

We find that recently IBM DB2 which is one of the popular relational databases catering to the OLTP needs in Mainframe as well as Unix, Linux, Windows has announced the BLU Acceleration on top of the traditional database offering. The rest of the sections covers about DB2 BLU Acceleration and how it can fit a Big Data OLTP processing need.

DB2 with BLU Acceleration
Recently IBM Announced the offering information about DB2 10.5 for Linux, Unix and Windows. There are some major components within the DB2 10.5, which caters to the Big data OLTP Needs.

  • DB2 With BLU Acceleration
  • DB2 PureScale Clustering Technologies

DB2 with BLU Acceleration is dynamic in-memory technology that yields faster analytics without the costs or limits of in-memory only systems. It speeds analytics and reporting with a combination of in-memory and column store data retrieval, maximized CPU processing, and data skipping that allows for faster input/output.

IBM DB2 pureScale® database clustering technology. Helps to ensure transaction processing is highly available and extremely scalable.

The following are the important Analytical Aspects of DB2 With BLU Acceleration.

  • Fully Integrated Solution as part of the base DB2 database which enables the combination of OLTP with Big Data Analytics
  • Utilizes the Column Organized Table Architecture so that the I/O Operations on the Analytical queries are fully optimized
  • Advanced Compression Technologies further optimize the columnar storage
  • In Memory Database storage further optimizes already compressed columnar data
  • A new concept of Data Skipping further skips the data that is not of interest resulting in further efficiencies
  • All the above optimizations work in compliment with the Parallel Processing nature of the solution

Another important aspect is that BLU Acceleration works with the familiar regular DB2 environment such that all the commands like LOAD, IMPORT and other DML commands work seamlessly with it. The regular DDL commands have been extended to support BLU Acceleration.

On the other hand, IBM DB2 PureScale is a multi instance shared served model similar to PARALLEL SYSPLEX environment in Mainframe and can be an ideal candidate for large scale OLTP processing. A DB2 pureScale environment is ideal for short transactions where there is little need to parallelize each query. Queries are automatically routed to different members, based on member workload. The ideal scenario for a DB2 pureScale environment includes workloads that handle online transaction processing (OLTP) or enterprise resource planning (ERP).

With the possibility to combine operations, especially JOINS between traditional OLTP work load row organized tables and Analytical work load column organized tables within the same DB2 environment powered by PureScale & BLU Acceleration technologies, the below architecture provides a blue print for a Big Data OLTP environment.

There is not a lot of documentation available on DB2 BLU Acceleration at this time; however, from my understanding it looks like at present DB2 BLU Scale Up Vertically within a powerful server with multiple cores rather than Scale Out using multiple horizontal servers, while this understanding may be wrong, but availability of both Scale UP and Scale Out would be bigger boost in enterprise scenarios. In other words the ability to use the DB2 BLU in conjunction with DB2 Data Partitioning Feature will further enhance the scalability of the Analytical component.

Combining the Big Data features with traditional RDBMS makes enterprises easily embrace Hybrid Big Data much like the success of Hybrid Cloud we see a increasing adoption here. The above mentioned combination of BLU Acceleration In DB2 together with traditional features like PureScale is a good example of Big Data OLTP. As I write this article I find Sql Server 2014 announcements are flashing on various sites which talks about features like In Memory OLTP which will provide further choices for enterprises.

More Stories By Srinivasan Sundara Rajan

Highly passionate about utilizing Digital Technologies to enable next generation enterprise. Believes in enterprise transformation through the Natives (Cloud Native & Mobile Native).