Arpit Solanki

Background

About Arpit Solanki

Currently building a Data Virtualization Platform to support querying over hetroginous data sources with central authorization and caching for faster querying with less cost. I am using open source projects such as Apache's Calcite, Ranger, Spark etc to build it. Most recent work: Built data quality platform at Atlan so users can setup checks and calculate metrics such as median, null values, sensitive data (PII detection credit card phone number etc) on their data to ensure only right data goes consumption. I used Spark as processing engine and kubernetes as infrastructure to build this. I am a seasoned Data Engineer, solving Data Enginering problems at Atlan. I am currently working on technologies such as Presto, Airflow, Spark, Kafka, Apache Hudi, Hive, k8s. At Atlan I have - Built data pipelines (ETL) using Airflow- Built data lakes and data versioning in lakes using Spark, Presto and Hive- Built developer tooling for Atlan data platform users- Built plugin based generic data ingestion framework with change data capture (CDC). I am passionate about working on open source technologies and contributed to PyData's Dask and Presto. if you'd like to work together or just discuss anything, drop me an email on solankiarpit1997@gmail.com. Github: https://github.com/arpit1997/ Stackoverflow: https://stackoverflow.com/users/5250746/arpit-solanki Codementor: https://codementor.io/arpitsolanki My interview on Presto at Community broadcast: https://youtu.be/X77FAfIf1Qo