Avijit Deb

Background

About Avijit Deb

Hello! A Senior Data Engineer, having over a decade of experience in Software Industry, I design, develop, and optimize large-scale data solutions to drive strategic insights. My expertise spans ETL/ ELT, data warehousing, cloud platforms and advanced data processing, enabling robust, scalable pipelines and data-driven decision-making. Seeking Global Opportunities Passionate about technology and growth, I’m eager to relocate for impactful projects, diverse collaboration, and cultural immersion. Ready for the next adventure! Data Engineering Specialist My expertise centers on Big Data and AWS Data Engineering technologies, with skills and experience in the following areas: I excel in creating and optimizing data pipelines with AWS Glue and AWS Data Pipeline for ETL transformations, using HDFS, HIVE, SQOOP, PySpark, Spark SQL, and Scala for efficient data processing. From ingesting raw data to executing complex transformations, I ensure smooth and reliable data flows: My experience includes deploying large-scale Data Lakes on AWS, leveraging services such as AWS S3, EMR, AWS RDS, AWS Redshift and Glue for scalable, high-performance data storage and processing. I’m well-versed in managing Hadoop clusters, handling data integration, and performing deep data analytics with AWS-native tools. & : Ensuring data accuracy and integrity is a core part of my work. I design robust checks and validations within data pipelines to maintain high-quality data outputs, giving stakeholders confidence in the data-driven insights they rely on- Boosted system performance by 25% through Big Data optimization techniques- Streamlined ETL processes at PwC with DBT and Apache Airflow, improving efficiency and performance- Built scalable data pipelines on AWS and GCP, optimizing Spark jobs for high-performance, cost-effective solutions. Beyond Work I'm a continuous learner who enjoys staying current with the latest technology trends. When I'm not working, you can find me enjoying nature, traveling to new places, and exploring different cultures. Let's connect! Whether you’re interested in discussing Big Data solutions, cloud deployments, or simply sharing insights, I’d love to hear from you.