Deepak Singh

Background

About Deepak Singh

8 + years of experience in Development role working in Agile model. My key skills include designing and building scalable and optimized ETL data pipelines. Over the years, I've gathered extensive experience in Data Engineering, designing and building products, Data Modeling, Requirement gathering / analysis, Client interaction, data pipelines optimization, problem solving, performance monitoring, POC work. Technical expertise includes Spark, Databricks, Azure Data Lake storage, Python, MongoDB, Airflow/ Google composer, SQL, AWS, Azure, Hadoop, Confluent Kafka, Hive, SonarQube, Splunk, Micro services, Python Django, Fast API, Azure Web App, OOPs, C++, MPP, SCD, CDC, unity catalog, data governance, SQS, Glue, Lambda, EMR, DynamoDB, Athena, S3 with basic Data structure. Work Summary: a) Evaluate and assess user risk factors and develop strategies to mitigate potential threats using trainings and tips. b) Design & develop data Pipelines to process user events that impact risk scores, enabling the identification of high-risk users for timely intervention ( AWS ). c) Built ETL data pipelines using Spark in Azure Databricks to process terabytes of data. d) Have done Orchestration using Google composer / Airflow and Databricks workflow. f) Worked on Batch and Spark streaming with Azure Data Lake storage (ADLS). g) Developed micro-services application using FastAPI. h) Worked on Django Rest framework to provide PDF extraction feature for client. i) Implemented unit test cases for code coverage and integrated it with SonarQube. Cloud- a) Microsoft Azure b) AWS Quality: a) Unit Testing, Feature Testing. b) API validation using PostMan. c) Data Pipeline optimization and performance testing. d) Unit test and integration with SonarQube. Devops: a) CI/CD b) Git POC work: a) Global Mongo DB b) Kafka MongoDB connector Domains: a) Cybersecurity b) Retail c) HealthCare d) Background Verification e) Telecom AI tools: a) ChatGPT b) Github Copilot