Job description

• Design, develop, and maintain ETL/ELT data pipelines using Azure Data Factory and Databricks.
• Build and optimize PySpark scripts and Databricks notebooks for large-scale data processing.
• Perform data ingestion, transformation, and aggregation from multiple sources (SQL, APIs, ADLS, Blob, SFTP, etc.).
• Implement Delta Lake architecture for data versioning, ACID transactions, and optimized queries.
• Work with Azure Data Lake Storage (Gen2) for managing and structuring raw, processed, and curated data layers.
• Develop and manage CI/CD pipelines for Databricks using Azure DevOps or GitHub Actions.
• Implement data validation, quality checks, and monitoring mechanisms across data pipelines.
• Collaborate with business and analytics teams to deliver clean, structured data for reporting and machine learning.
• Optimize PySpark jobs using techniques like broadcast joins, Z-Ordering, partitioning, and caching.
• Ensure compliance with data governance, security, and regulatory requirements.
• Troubleshoot performance bottlenecks and improve resource utilization on Databricks clusters.

Skills Required

Key Skills

• DATA ENGINEER
• DATABRICKS
• PY SPARK

Other Information