Databricks Data Engineer Consultant

About this role

Role -Databricks Data Engineer Consultant

About the Role

We are seeking experienced Databricks Data Engineers to join our Data & AI. You will play a pivotal role in designing, developing, and maintaining scalable data pipelines and transformation workflows on Azure Databricks, enabling advanced analytics and data engineering initiatives.

Key Responsibilities

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.
• Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment to support Data Science use cases.
• Collaborate closely with Data Scientists to understand data requirements and ensure timely delivery of clean, structured datasets.
• Implement data quality checks, validation rules, and monitoring mechanisms.
• Integrate data from various sources including structured, semi-structured, and unstructured formats.
• Ensure data security, governance, and compliance with banking regulations.
• Optimize Spark jobs for performance and cost-efficiency in Azure.
• Participate in code reviews, design discussions, and agile ceremonies.
• Document data pipelines, workflows, and technical decisions.
• .

Required Skills & Qualifications

Technical Skills

• Experience in data engineering, with at least 3+ years hands-on experience in Azure Databricks.
• Strong proficiency in Apache Spark, especially PySpark for distributed data processing.
• Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake.
• Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts.
• Proficiency in Python and/or Scala for data processing.
• Experience with CI/CD pipelines using tools like Azure DevOps, Git, and Terraform (preferred).
• Familiarity with SQL, NoSQL, and data lake architectures.
• Knowledge of data governance, security, and compliance in financial services.

Soft Skills

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work collaboratively in cross-functional teams.
• Agile mindset and experience working in Scrum/Agile environments.

Preferred Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field.
• Databricks Certified Data Engineer Associate/Professional certification is a plus.
• Experience working in banking or financial services domain is highly desirable.
• Australian citizenship is required, and candidates must be eligible to obtain security clearance.
• Location is flexible across Brisbane, Sydney, and Melbourne.
• .