What is Data Mesh and why is it revolutionising the data world?
To understand what Data Mesh is, we must first enter the world of Big Data.
When we talk about ETL processes, we understand that the Data Warehouse (DW) is a store of large volumes of data.
These processes and warehouses helped companies to reach certain customers in a way that was not possible before. But, as we know, innovations are continuous and grow rapidly, and this means that on several occasions the processes of extraction, transformation and Data loading become too slow.
To achieve a faster pace in data management, Data Lakes were born.
What is a Data Lake?
A Data Lake is a data warehouse, similar to a Data Warehouse, but with the difference that it admits any type of data and keeps it as it has been loaded.
The advantage of the Data Lake is that it quickly adapts to changes, because they are stored in a flat architecture, and not in the form of files as it happens in the DW.
Users access data before it has been modified and segmented, which makes the process easier.
So… What is Data Mesh?
A Data Mesh divides the data stored within the Data Lake into various business domains that will be used to create data products, making them reach various areas of the company.
Some of its features:
- It allows the customization of products or services offered to the customer
- It allows data hierarchy
- It helps to generate more agile processes for the digital needs of the market
- It enables browsing from multiple devices and allows quick synchronization between them.
A Data Mesh allows data products to be joined between domains granting data exchange without a store.
Data Mesh Principles
To have a better understanding of how Data Mesh can revolutionize the data world, we will explain the 4 basic principles that define it:
Independent data management –
As there is no warehouse, it offers a simpler, more comfortable and less expensive data management, through domains, without a warehouse. In other words, data management is independent.
Usable and simple data as if it were a product –
The data that the domain provides should be treated as a product, and those who consume that data should be treated as customers.
Its correct operation is based mainly on the fact that the owner of the domain data product must know in detail who the users are, what their use is and the methods with which they feel comfortable consuming them. This knowledge enables you to design a data product interface that meets your needs.
Data self-service platform –
Members of each domain can access and use the data. A complex technology is required if you want to access them successfully.
Federated governance –
We have explained that data sets are independent and do not need a warehouse. But, for this to happen, they must be able to interoperate with each other and allow large-scale operations.
We can say that Data Mesh is neither an architecture nor a platform, but rather a solution in continuous development that adopts different interpretations. It is the Data Mesh principles that define the concept at its best: the domain as the basis of the design, the data as the product, the platform as self-service, and federated governance.
You must abandon the concept of thinking about and using data as a resource, and start thinking of data as a product. The focus should shift from the producer as the data collector, to the producer as the data server. When an organisation manages to transform this concept, it will achieve a better experience in data consumption.
What do you think about Data Mesh? –
If you want to contact us directly, send us a message.