Role Overview
The Senior Data Engineer will be responsible for designing, building, and operating scalable, secure, and high‐performance data platforms. The role involves critical data pipelines and architectures that power analytics, reporting, and advanced data use cases such as machine learning and real‐time insights. The candidate will collaborate closely with analytics, data science, product, and business teams while mentoring engineers and driving engineering best practices across the data organization. The role combines technical leadership, architecture ownership, and business impact.
Key Responsibilities
- Design & own scalable batch and streaming data architecture, choose right storage, compute, and processing patterns, ensure scalability, reliability, fault tolerance, and cost efficiency.
- Build reliable, high‐quality data pipelines, develop and maintain ETL/ELT pipelines at scale, handle incremental loads, CDC, and failure recovery, implement data quality checks and monitoring.
- Provide technical leadership & collaboration: review code, data models, and designs, translate business requirements into technical solutions, communicate trade‐offs to stakeholders and leadership.
Required Skills
- Strong hands‐on experience with Azure Data Lake Gen2, Synapse, Databricks.
- Solid experience in PySpark / Spark SQL, Databricks performance tuning & job optimization.
- Proficient in complex SQL (window functions, CTEs, execution plans).
- Strong understanding of mentoring junior and mid‐level engineers, communicating trade‐offs to architects and product teams.
- Experience working with Azure DevOps or GitHub Actions.
Nice to Have Skills
- Experience with streaming & real‐time data processing on designing low‐latency pipelines.
- Exposure to data governance & metadata management in data classification & sensitivity labels.
- Familiarity with advanced Python engineering practices on performance‐aware Python design.
- Experience working in analytics & BI awareness.
- Knowledge of cloud cost optimization & FinOps.
Minimum Experience
6 years.
Certifications
No certifications required.
Senior Data Engineer employer: XPT Software Australia
XPT Software Australia is an exceptional employer that fosters a collaborative and innovative work culture, perfect for those passionate about learning and development. Located in a vibrant tech hub, we offer comprehensive employee growth opportunities, including access to cutting-edge training resources and mentorship programs. Join us to make a meaningful impact while enjoying a supportive environment that values your contributions and encourages professional advancement.
