Prabhat Diwaker

Background

About Prabhat Diwaker

Principal Data Engineer with 14+ years of experience designing and building petabyte-scale data platforms, real-time streaming systems, and cloud-native analytics solutions that drive data-informed decision-making.•Proven track record of architecting modern data lake and lake-house ecosystems, leading cross-functional engineering teams, and delivering high-impact, large-scale data transformation initiatives.•Hands-on expertise across distributed data processing, streaming architectures, data modeling, governance, and cloud-native big data technologies. •Successfully led enterprise-scale modernization programs including on-prem to Google Cloud migration, legacy scheduler to Airflow transformation, Hive-to-Spark re-engineering, and Spark 3 upgrades—ensuring performance optimization, data accuracy, and seamless production cutovers. •Technical Lead and Architect for a team of 20+ engineers, focused on Last-Mile data for Walmart’s eCommerce platform. •Designed and implemented Google Cloud infrastructure for the team managing GCP projects, Cloud Storage, IAM policies, Dataproc clusters, Big Query, and BigLake tables. •Built robust and scalable pipelines to ingest and integrate data from over 15 distinct source systems, including Kafka, RDBMS, APIs, Webhooks, Splunk, Files, and Azure blobs, among others, into a data lake for advanced analytics use cases. •Worked in close collaboration with the business and data science teams, supporting data and pipelines for critical initiatives, including Driver Fraud & Scoring, Traffic Modeling, and AlwaysOn among other strategic initiatives. •Implemented data quality rules across multiple dimensions such as completeness, accuracy, consistency, and conformity, ensuring healthy DQ scores across lake tables. •Implemented data lake practices in a cloud environment such as data security measures, routine audits for cost efficiency & job performance, and regular clean-up to ensure a lean data lake. •Developed numerous scripts and frameworks for essential housekeeping activities like monitoring the overall health of the data lake, overseeing long-running jobs, tracking user base, query & adoption patterns, validating data, and optimizing cloud- related costs. •Implemented self-service solutions for business operations utilising Google BigQuery with Bl engine, BigLake, GenAI & LLMs, alongside data visualisation tools such as Looker and Tableau.