Manik Soni

Background

About Manik Soni

With 14 years of experience as a Data Engineer, I have built my expertise on Spark, PySpark, Python, SQL, and databricks —transitioning from classic Hadoop clusters to modern Lakehouse architecture to drive data accuracy above 99% and slash operational costs. I architect and optimize ELT pipelines that convert raw data into structured, high-quality information using Databricks, Delta Lake, Spark Structured Streaming, Auto Loader, HDFS, and Kafka for near real-time ingestion. In the cloud, I harness Azure Data Factory, ADLS Gen2, Azure Synapse Analytics, Big Query, GCS Storage, GCP Data Catalog, and GCP DataPlex to migrate and scale massive workloads. My orchestration toolkit includes Ab Initio, Informatica, Airflow, Autosys, and Unity Catalog, while CI/CD pipelines run on GitHub Actions and Jenkins. Highlights of my impact:* Migrated 150 TB of Teradata data and 200+ Ab Initio jobs to Databricks and ADLS Gen2, cutting pipeline costs by 25% and reducing failures from 8% to under 1%.* Designed Medallion-style ingestion frameworks on Databricks with Spark Structured Streaming, Auto Loader, and Kafka—processing 1 M+ records/day with ≤ 5-minute latency.* Optimized Spark ETL on Azure (Azure Data Factory, Azure Databricks), slashing runtimes by 30% and reclaiming 35 compute hours monthly.* Automated workflow orchestration with Informatica, Airflow, and Autosys, and enforced data governance via Unity Catalog and GCP DataPlex for compliance and efficiency.* Mentored engineers to deliver 10+ KPI dashboards, championing best practices in CI/CD, code reviews, and testing. I hold certifications as a Databricks Certified Data Engineer Associate, Azure Certified Data Engineer, AWS Certified Cloud Practitioner, and Teradata Certified Professional. What fuels me is solving complex data challenges, mentoring teams, and driving proactive ownership from ideation through production. If you need a Data Engineer who turns data chaos into clear, measurable business impact, let’s connect.