Azure Databricks Data Engineer – London

Your Role:

• Design and implement robust ETL/ELT pipelines on Databricks.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality solutions.
• Optimize data workflows for performance, scalability.
• Develop and maintain data lake and data warehouse architectures.
• Ensure data quality, governance, and security standards are met.
• Lead code reviews, mentor junior engineers, and contribute to best practices in data engineering.
• Integrate Databricks with other enterprise systems and tools (e.g., Delta Lake, MLflow, Power BI, etc.).
• Monitor and troubleshoot production data pipelines and jobs.

Your Profile:

• Lead the design, development, and optimization of scalable data pipelines using Databricks.
• Drive data modernization initiatives to support advanced analytics and machine learning.
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions.
• Implement best practices for data engineering, including performance tuning and cost optimization.
• Ensure data quality, reliability, and consistency across all data platforms.
• Leverage Databricks features such as Delta Lake, notebooks, and MLflow for end-to-end workflows.
• Apply knowledge of data governance frameworks and tools, including Unity Catalog.
• Maintain and enhance data security, access controls, and compliance standards.
• Mentor junior engineers and contribute to the development of engineering standards.
• Hold relevant Databricks certifications (e.g., Databricks Certified Data Engineer Professional).