Job Overview
Location: Sydney
Note: Dependent visa with full work permit OR any visa with unlimited work rights are eligible
Key Responsibilities
- Design, build, and optimise batch and streaming data pipelines using approved frameworks and tooling.
- Ensure pipelines are resilient, performant, and cost optimised.
- Implement monitoring, alerting, and automated recovery processes.
- Apply data quality checks and reconciliation processes to ensure accuracy and completeness.
- Maintain metadata, lineage, and documentation to support governance requirements.
- Adhere to role based access control, data privacy, and security standards.
- Contribute to the development and maintenance of ingestion frameworks, layers, and data models.
- Implement and maintain CI/CD pipelines for automated testing and deployment.
- Optimise data storage and retrieval strategies to meet performance needs.
- Work closely with Technical Leads, Data Architects, Analysts, and business stakeholders to deliver fit for purpose solutions.
- Participate in code reviews and knowledge sharing sessions to improve team capability.
- Explore and adopt new tools, frameworks, and techniques to enhance engineering capability.
- Identify opportunities to automate and streamline workflows.
Key Skills & Experience
- Proficient in SQL for data wrangling, transformation, and analysis.
- Strong programming skills in Python, Scala, or Java.
- Experience building ETL/ELT pipelines in cloud or distributed environments.
- Familiarity with cloud platforms and services.
- Knowledge of data modeling concepts.
- Experience with data orchestration tools.
- Familiar with version control and CI/CD pipelines.
- Understanding of data quality, validation, and observability practices.
- Works independently with minimal supervision on well‐scoped tasks.
- Collaborates effectively with stakeholders.
- Communicates clearly and documents work for others to maintain or build upon.
- Open to code reviews, feedback, and continuous learning.
- Able to troubleshoot and resolve data pipeline failures or performance issues.
Qualifications & Experience
- 4 years of experience in a data engineering or closely related role.
- Has delivered production‐ready pipelines end‐to‐end.
- Demonstrated ability to improve code quality, reliability, or performance.
- Experience working with real‐time or near real‐time datasets.
Seniority level
- Mid‐Senior level
Employment type
- Contract
Job function
- Information Technology
Industries
- IT Services and IT Consulting
Senior Data Engineer employer: Resource Corner
As an Azure Databricks AI/ML Engineer, you'll join a dynamic team that values innovation and collaboration in the heart of the tech industry. Our company offers competitive benefits, a supportive work culture that fosters professional growth, and unique opportunities to work on cutting-edge projects in a vibrant location. We are committed to empowering our employees with the tools and resources they need to thrive in their careers while making a meaningful impact in the field of AI and MLOps.
