Skip to main content Link Menu Expand (external link) Document Search Copy Copied
Fabric w/ Azure Databricks

Task 03: Read from the Silver layer, create a data model (Facts, Dimensions) on the data lake from Azure Databricks, and write data into Gold layer as external tables. (Z-ordered within ADLS Gen2 as the Gold layer)

The last step is to develop and implement a robust data model from the Silver layer, moving critical data into the Gold layer, thus bolstering Contoso’s analytics and decision-making capabilities with structured and refined data sets.

  1. Switch back to the Databricks browser tab.

  2. On the Databricks page, in the left navigation pane, select Workspace.

  3. In the Users workspace, select the Silver to Gold notebook.

  4. In the first code cell, populate the ENTERHERE placeholders with the following information.

    Name Value Comment
    scope scopesecret  
    key appServiceRegistrationSecret You can also retrieve the secret from the Keys page for the key vault
    storage_account [Your Storage Account Name] You can also retrieve the Storage Account name from the Storage Account resource page
    application_id [Your Application ID] You can also retrieve this ID from the Microsoft Entra > App registration Overview page
    directory_id [Your Directory (Tenant) ID] You can also retrieve this ID from the Microsoft Entra > App registration Overview page

  5. In the second code cell, replace the ENTERHERE placeholder with medallion.

  6. In the sixth code cell, replace the <ENTERHERE> placeholder with the following code:

     "select dp.ProductIDKey, ds.CustomerIDKey, soh.*, sod.OrderQty, sod.ProductID, sod.UnitPrice, sod.UnitPriceDiscount, sod.LineTotal from salesorderheader soh join salesorderdetail sod on soh.SalesOrderID = sod.SalesOrderID LEFT JOIN dimProduct dp ON sod.ProductID = dp.ProductID LEFT JOIN dimCustomer ds ON soh.CustomerID = ds.CustomerID"

  7. In the thirteenth code cell, populate the ENTERHERE placeholder to create the df_factSales dataframe:

     select dp.ProductIDKey, ds.CustomerIDKey, soh.*, sod.OrderQty, sod.ProductID, sod.UnitPrice, sod.UnitPriceDiscount, sod.LineTotal from salesorderheader soh join salesorderdetail sod on soh.SalesOrderID = sod.SalesOrderID LEFT JOIN dimProduct dp ON sod.ProductID = dp.ProductID LEFT JOIN dimCustomer ds ON soh.CustomerID = ds.CustomerID

  8. Select Run all.

  9. Review the output for each cell.