Pooja Dak

Background

About Pooja Dak

I'm a highly motivated and results-oriented Databricks platform/technical Architect/Lead with extensive experience in building and managing complex data lake solutions on Microsoft Azure. At Zensar Technologies, I played a pivotal role in architecting a federated data lake for Derivco, a leading gaming organization. My contributions spanned across various aspects, including:• Data Lake Architecture: Designed a robust data lake solution for Derivco, incorporating Azure Data Bricks, Azure Data Factory, ADLS Gen2, and Terraform for efficient data management.• Unified Governance: Implemented a Unity Catalog solution to provide centralized access control and governance for all data objects within the data lake, ensuring data security and compliance.• Serverless Databricks Exploration: Conducted technical evaluations to assess the feasibility of adopting serverless compute in Databricks for cost optimization.• End-to-End Delivery: Owned the entire solution lifecycle, from initial design and proof of concept (POC) to implementation. This included creating virtual networks, subnets, Databricks SQL, SQL warehouses, service principals, and security configurations (SCIM connectors, PAT tokens, PIM).• Metrics and Data Modeling: Leveraged Unity Catalog system tables and operational metrics to build comprehensive data processing, object size, usage, lineage, and cost metrics dashboards.• AI and Power BI Integration: Designed and implemented an AI and Power BI integration solution for data visualization, enabling Derivco to gain deeper insights from their data.• Automation Expertise: Streamlined processes using PowerShell scripts, including data lake storage account access controls (ACLs and RBAC), and automated tasks for infrastructure provisioning and management.• Databricks Cluster Management: Defined T-shirt size cluster policies (General Purpose and Job clusters) to optimize resource allocation based on workload requirements.• Security Champion: Conducted POCs for data encryption/decryption using Immuta and championed secure data access practices.• Delta Table Creation: Utilized Databricks Jobs and policy APIs to create Delta tables for efficient reporting jobs.• Data Modeling for Gaming Analytics: Developed data models for tournament and free spins reports, empowering Derivco with actionable insights into their gaming operations.• Cost Optimization: Implemented strategies to optimize Azure costs associated with Databricks usage.