A Deep Dive into the Data Governance Ecosystem (1/3)— What it is, how its evolved, and what’s on the horizon
The first installation of this series will be focused on demystifying data governance (DG), exploring its history, and sharing views on where the space may be heading. My hope is that this will be a helpful guide for operators, investors and entrepreneurs who find themselves burrowing down the data governance rabbit hole!
Introduction — in a world where we are all ‘data driven…’
The proliferation of big data has presented multiple avenues for businesses to boost their bottom line. Almost every company today is positioning itself as a “data-driven” organisation. Everything from product innovation to cost optimization is fuelled by insights derived from data. However, big data going “mainstream” has come at a cost. Increased data volumes, amplified data consumption needs, and new legislative/regulatory frameworks, have presented multiple challenges for executives in overseeing their data operations.
A survey conducted by Capital One and Forrester earlier this year examined some of the data management challenges corporations are facing, and… SURPRISE, SURPRISE, among the list was data governance.
An overwhelming majority of respondents (80%+) cited tracking and enforcing adherence to governance policies, governing data estates at scale, and managing internal access to specific data assets as fundamental challenges for their organisations.
It is undeniably clear that implementing a successful data governance program is mission critical for any enterprise. From outlining roles and responsibilities, to deciding which products adopt to enable the process, the topic has become a top-level concern for data professionals.
What is Data Governance? Everyone has a different view!
A quick search online will yield no dearth of definitions for data governance. When I first started reading on the topic, unearthing a consistent description was incredibly frustrating. Depending on who you ask, chances are you will get different views on the concept of data governance. Some think it is all about complying with regulations, whilst others see it as more about data security and quality. In actuality, it covers all of these topics and more.
Data governance is responsible for the development and implementation of policies and standards for managing data within an organization.
At its core, it is concerned with managing the availability, usability, integrity, and security of data used in an organization. It is a set of processes, roles, standards, and metrics that ensure data is used in a consistent and controlled manner.
DG defines who is authorised to take what actions, with what data, in what conditions, and using which procedures.
Key benefits of governance programs include:
- Compliance with regulatory and industry regulations: Arguably the focal reason behind organisations devising governance programs is compliance. Minor violations of GDPR or PDPA regulations can lead to hefty fines. Good DG tools enable automated compliance with such regulations and laws.
- Increased data quality and reliability: An effective program will give decision makers and users greater confidence in their data, and thus more confidence in making decisions based on it.
- Enables effective data collaboration: Having a clear understanding of who should/shouldn’t have access to certain data reduces the friction between data practitioners.
- Cost reduction: DG prevents decisions being made on outdated or faulty information, therefore reducing wasted resources. Audits will be quicker and simpler, and the day-to-day operations more efficient and productive.
Where we were → Data Governance 1.0 — reactive & I.T. led
Although “data governance” was popularized in the 2000s, the notion of protecting and taming data emerged the decade prior. DG in its early days was solely managed by the IT function, and focused largely on data cataloguing, that was seldom looked at again or used by the business.
Data governance was a retrospective exercise. Practitioners would deploy data programs with little to no regard for its principles, only then to go back and make adjustments that were dictated by top-down mandates, which in turn were enforced by external regulations/policies. During this period, DG had one purpose — to keep the executives out of jail.
The tooling available at this point was very limited. Vendors largely consisted of legacy ERP businesses and legacy software companies whose offerings were fundamentally consulting/services based.
Where we are → Data Governance 2.0 — proactive and cross-functional
The new millennium brought with it the age of big data and the cloud computing boom. Data is seen as a superpower and an asset, and businesses have become more dependent on data generated insights to influence key business decisions.
Simultaneously, the “assetisation” of data sparked the growth in the number of high-profile data breaches, or as the Wall Street Journal aptly put it, [the] global reckoning on data governance. From antiquated data security systems to shambolic data management practices, poor data governance methodologies were being exposed everywhere. Data protection laws like GDPR in Europe, CCPA in the US, as well as industry specific regulations like the BCBS 239 for financial services came into effect. It quickly became evident that data had evolved into a whole new beast, and further proved that governance would be more critical than ever before.
The democratisation and cross-geographical usage of data within organisations meant that data governance could no longer be an IT led initiative but required a cross-functional one.
The sheer scale of data estates presented an insurmountable task for IT teams to govern alone. Anyone who engaged with data, had to share the brunt of the data governance responsibilities.
Moreover, implementing governance programs was no more a retroactive exercise, but a priority from the onset. Executives had not only woken up to the risks of taking the topic haphazardly (breaches, fines and tarnished a reputation), but finally recognised the benefits from a successful governance program, like increased data quality and enhanced data intelligence.
Data governance had transitioned from something executives had to do, to something they wanted to do… or at least were motivated to do beyond the threat of jail time and/or hefty fines.
Data governance 2.0 has been catalysed in part by a refresh of the product landscape. Legacy vendors productised their consulting businesses (or at least tried to), whilst a new wave of emerging vendors came to market with their tech-first offerings.
The main trend we saw here was that vendors were addressing specific sub-categories of data governance e.g. Alation and Atlan with data cataloguing, and Ataccama and Datactics with data quality. However, as the market developed, we saw a number of breakout players expand their product offering to become fully-fledged data management/governance platforms. For instance, Collibra, started out in data cataloguing, but now covers cataloguing, lineage, discovery, access/policy management and master data management.
Where we’re going → Data Governance 3.0 — verticalization
- We will continue to see this new class of category-specific vendors rise farther and faster, with increasing adoption and the categories becoming clearer and better articulated
- The data operations and governance layer will stabilise and the first big winners to clearly establish themselves as “must haves” for large enterprises
- Verticalization of the toolkit. Currently, most solutions are industry-agnostic and therefore require organisation-wide behavioural change for full adoption. We expect tools focusing on industry-specific challenges will be easier to adopt, especially for companies operating in regulated markets like financial services, healthcare, defence and government.
In the next installment of this series, we’ll breakdown the data governance software landscape, define its sub-categories, and share our views on where we think the key opportunities lie.
Illuminate Financial is a thesis-driven venture capital firm dedicated to fintech and enterprise software companies building technology solutions for financial services. If you are building something in this space, or have any thoughts to share, please we’d love to hear from you!