{"id":23784,"date":"2023-09-01T05:00:00","date_gmt":"2023-09-01T03:00:00","guid":{"rendered":"https:\/\/sii.pl\/blog\/?p=23784"},"modified":"2023-08-27T07:09:58","modified_gmt":"2023-08-27T05:09:58","slug":"microsoft-purview-introduction-and-review-of-a-data-governance-solution","status":"publish","type":"post","link":"https:\/\/sii.pl\/blog\/en\/microsoft-purview-introduction-and-review-of-a-data-governance-solution\/","title":{"rendered":"Microsoft Purview \u2013 introduction and review of a data governance solution"},"content":{"rendered":"\n

In September 2021, Microsoft launched a beta version of a cloud-native data governance<\/strong> service called Microsoft Purview<\/strong>. The tool promised users the ability to govern their on-premises, multicloud and SaaS data. It\u2019s been 2 years since that date, and I have been wondering if this \u201cnew\u201d and promising solution, although now aged, can still be called promising and what the service provides.<\/p>\n\n\n\n

Together with some friends from Sii, we conducted a PoC for our client. I want to share some of our findings with you.<\/p>\n\n\n\n

Getting started<\/strong><\/h2>\n\n\n\n

You can create a Microsoft Purview account in the Azure Portal site by simply creating a new resource and searching for Purview in the Marketplace. During the creation process, you will be asked to specify the Resource Group<\/strong> and Purview Account<\/strong> name. Additionally, you can configure the network connection and the platform size of your account.<\/p>\n\n\n\n

For testing purposes, \u201c4 capacity units\u201d is more than sufficient in terms of platform size. On the network tab, it would be good to provide a \u201cPrivate endpoint\u201d<\/strong> for security reasons, but \u201cAll networks\u201d<\/strong> set up in a non-production environment is fine.<\/p>\n\n\n\n

To visualize the data in the organization, we have to create a few collections<\/strong> and subcollections<\/strong> in a tree structure to depict the layout of the architecture holding our data. Below is an example of what we came up with for the PoC.<\/p>\n\n\n\n

As you can see, at the top we have the name of the account we created, then a collection called \u201cSII\u201d, and under that we have several subcollections showing various subsidiaries of the company. Each subsidiary has an application running and a department that uses it. In our example \u201cDzia\u0142_1\u201d in \u201cJednostka_1\u201d is based on Power BI reports, so we created a data source<\/strong> that will scan<\/strong> all Power Bi<\/strong> reports on that tenant<\/strong> to depict what reports hold what data. For a full list of data sources that can be created directly from the purview and scan, follow the link.<\/a><\/p>\n\n\n\n

That same department in \u201dJednostka_2\u201d gets its data from a SQL database,<\/strong> so this is the data source we created and scanned to find out what assets<\/strong>(data) are stored there.<\/p>\n\n\n\n

For the last subsidiary, we import<\/strong> assets <\/strong>directly from the Excel template<\/strong> using a custom application<\/strong> in Python<\/strong> that we are building. This is a solution for importing information on custom flat file reports and outputs that are not covered by the scans of data sources. We will discuss both scanning and template<\/strong> import topics in more detail later.<\/p>\n\n\n\n

\"Collections<\/a>
Fig. 1 Collections and subcollections in Microsoft Pureview<\/figcaption><\/figure>\n\n\n\n

Data scans<\/strong><\/h2>\n\n\n\n

One way to automatically obtain data information from the environment is to add a data source to a particular collection and run a scan on it. In our example, we have created a Power BI<\/strong> data source that can be used to scan the same tenant where we have our purview account on for resources. The result of the scan in this case will show not only Power Bi reports,<\/strong> but also workspaces<\/strong>, datasets<\/strong> and dashboards<\/strong>. <\/p>\n\n\n\n

In the details of a given asset, you will be able to see under which collection path<\/strong> and in which hierarchy<\/strong> in the Power Bi data source it is located. It will also be possible to check the lineage<\/strong> of the data and even the schema,<\/strong> if the asset can be displayed as a table.<\/p>\n\n\n\n

\"Lineage<\/a>
Fig. 2 Lineage of the data<\/figcaption><\/figure>\n\n\n\n

Detailed instructions for creating this configuration you can find at this link<\/a>.<\/p>\n\n\n\n

Bulk import of assets<\/strong><\/h2>\n\n\n\n

Since one of the most common things, you can find in any IT environment is an Excel-based report or a flat file of data that someone uses in their daily work, there must be a way to introduce this information and data structure into the purview. To do this, some coding in Python is required.<\/p>\n\n\n\n

While doing this, you will find that underneath purview actually uses apacheatlas <\/strong>to import assets. The coding is fairly simple, and a tutorial on how to do this with examples can be found here.<\/a><\/p>\n\n\n\n

For our PoC, we built a small application for loading Excel templates using apacheatlas<\/strong> and a graphical interface using customtkinter<\/strong>. The application shows what environment you are connected to, allows you to browse your PC for files\/templates to upload and perform the upload itself.<\/p>\n\n\n\n

\"Purview
Fig. 3 Purview \u2013 Bulk Loader<\/figcaption><\/figure>\n\n\n\n

What is really worth noting at this point is that the Excel template that must be used in this process requires a significant amount of work from the uploader. We need to take the report apart and know all the details about it to fill in the template, such as:<\/p>\n\n\n\n