Jay Varma

Background

About Jay Varma

Results-driven Data Engineer with 6+ years of experience in designing and implementing robust data solutions to drive business insights and enhance data-driven decision-making. Expertise in building batch and real-time data processing systems leveraging AWS services (S3, Redshift, EMR, Lambda, DynamoDB) and Azure services (ADLS, ADF, Databricks, Synapse Analytics). ETL pipelines in and out of data warehouse using Python with Pandas and Spark. Hands-on experience with Azure SQL Database, Azure SQL Data Warehouse, Azure Analysis Services, HDInsight, Azure Data Lake and Data Factory. Implemented Big Data solutions using Hadoop technology stack, including Pyspark, Hive, Sqoop, Avro and Thrift. Hands on experience in GCP, Big Query, GCS bucket, G - cloud function, cloud dataflow, Data Proc, Stack driver. Practical experience using KTables, Global KTables, and KStreams in Apache Kafka and Confluent environments to work with Kafka streaming. Created Python code to collect data from HBase and develop the PySpark implementation of the solution. Experience in Performance Monitoring, Security, Trouble shooting, Backup, Disaster recovery, Maintenance and Support of Linux systems. Hands-on experience with Spark, Databricks, and Delta Lake. Experience working with various SDLC methodologies like Agile Scrum, RUP and Waterfall model. Experienced with Dimensional modelling, Data migration, Data cleansing, Data profiling, and ETL Processes features for data warehouses.