David Menninger's Analyst Perspectives

At one point, analytics and business intelligence were considered non-mission critical activities. One of the primary concerns in designing analytics systems was to ensure they didn’t interfere with or draw computing resources away from operational systems. But today, analytical systems are integral to many aspects of operations. More than 9 in 10 participants in our Analytics and Data Benchmark Research reported analytics had improved activities and processes. However, most analytics and BI software products are still designed as read-only systems, and the process of using their output is either a manual process or involves custom coding or scripting to provide inputs to the operational systems. The most common complaint about analytics and BI technologies is that they are hard to integrate with business processes. In this perspective, we’ll look at how the recent trend of “reverse ETL” helps organizations utilize the output of analytic processes more effectively in operations.  

When data warehouses emerged, ETL – extract, transform and load – became the pattern for data integration. Source data is extracted from operational systems, transformed in flight and loaded into the data warehouse. As data lakes arrived on the scene, ELT – extract, load and transform – became another process where data is extracted from source systems, loaded into a highly scalable data platform and processed there rather than in a separate system.

Now, several solutions, including a new software category called reverse ETL, are emerging for moving information from analytical data warehouses and BI tools into the operational systems they are supporting. Reverse ETL applications are based on the premise that information stored in data warehouses is valuable for operational activities and some of it should be moved into those systems. For example, data warehouses often enrich customer data with metrics such as segmentation scores, propensity to respond to different offers and external data such as demographic information. If this data is available within operational systems, it can be used directly in customer service interactions or marketing campaigns.

Over the past few years, I’ve noticed analytics vendors starting to recognize the need to move information into operational systems. Most of the solutions are limited, but others are more systematic. The more limited solutions invoke custom scripts from the analytics systems to move information. The more systematic solutions involve two-way connectors with the source systems. Many BI tools have connectors to gather data from source systems. Several vendors now offer two-way connectors to either pull information from source systems or push information to those systems. Simultaneously, a new category of vendors is developing reverse ETL tools independent of the analytics and BI systems. As the name implies, reverse ETL systems extract data from the data warehouse, perhaps perform light transformation and load it into operational systems.

There are some challenges with each approach. Custom scripts or application programming interface calls may be useful for moving small amounts of information, but often, large volumes of data must be moved into the source systems. Two-way connectors associated with BI tools can move larger amounts of information but require careful governance so data from the operational system is not corrupted. Reverse ETL systems are primarily designed to work with data from the data warehouse. Often, valuable information exists in the analytics and BI tools that is materialized via calculations. That information wouldn’t be available to the reverse ETL tools.

I welcome the developments in this area. BI needs to be actionable, but actions are rarely taken from the BI tools. They are implemented in systems for marketing, customer service, human resources, supply chain and other operational systems. Keep an eye on this area as it develops. Explore the options associated with your information architecture. As applications become available to replace custom coding or scripts, consider switching as a way to scale the value of analytics and BI investments.

Regards,

David Menninger