Location
Sydney/Melbourne
About the Role
We are looking for an experienced Data Engineer with strong expertise in Microsoft Azure and Databricks to design, develop, and optimize data solutions. You will be responsible for building scalable data pipelines, managing big data environments, and enabling advanced analytics for business insights.
Key Responsibilities
- Design and implement data pipelines using Azure Data Factory, Databricks, and related services.
- Develop and optimize ETL/ELT workflows for structured and unstructured data.
- Build and maintain data lakes and data warehouses on Azure.
- Implement data governance, security, and compliance standards.
- Collaborate with data scientists and analysts to deliver high-quality datasets for analytics and machine learning.
- Monitor and troubleshoot data workflows for performance and reliability.
- Work with Spark and PySpark for big data processing in Databricks.
- Integrate data from multiple sources including APIs, streaming data, and on-prem systems.
Required Skills & Qualifications
- Bachelor's degree in Computer Science, Engineering, or related field.
- Hands-on experience with Azure services:
- Strong experience with Databricks, Apache Spark, and PySpark.
- Proficiency in Python and SQL for data engineering tasks.
- Knowledge of data modeling, data governance, and security best practices.
- Familiarity with CI/CD pipelines and DevOps practices for data solutions.
Preferred Qualifications
- Experience with real-time data streaming (Azure Event Hub, Kafka).
- Knowledge of Delta Lake architecture and Lakehouse concepts.
- Familiarity with Power BI or other visualization tools.
- Understanding of machine learning workflows in Databricks.
Seniority Level
Mid-Senior level
Employment Type
Full-time
Job Function
Information Technology
Industries
Software Development
Compensation
A$140,000.00-A$180,000.00
Data Engineer – Azure & Databricks employer: N2S.Global
Join a dynamic team that values innovation and collaboration, where your expertise in Golang will be instrumental in building scalable backend services. Our vibrant work culture fosters continuous learning and growth, offering ample opportunities for professional development while working with cutting-edge technologies in a supportive environment. Located in a thriving tech hub, we provide a unique chance to engage with like-minded professionals and contribute to impactful projects that make a difference.
