Contract position (8 weeks) Strong chance of extension
Our client is seeking an experienced Databricks Data Engineer for an 8-week contract to bring the Databricks Proof of Technology (PoT) to completion and stand up a simulated MVP environment. This role supports the Global Optimisation Platform Project, which is creating an innovative optimisation and bidding solution.
About the role:
- Be part of a core Data Engineering team, this engagement adds the specialist Databricks capacity to close that gap across the team.
- Contributing direct Databricks engineering effort within the existing team, with the goal of clearing the PoT gate on schedule and handing Quant Research a workable simulated MVP.
- Stand up the core Databricks environment for the simulated MVP, with security built in through Unity Catalog, access controls, and compute policies.
- Create simulated asset telemetry and align the market data, giving the Quant Research team a complete, end-to-end modelling pipeline
- Use real effort from reference pipelines as a baseline to refine the delivery-phase estimate for the data ingestion and integration workstream.
Required Experience:
- Over five years of data engineering experience, including a minimum of two years building and running Azure Databricks platforms in production.
- Hands‑on experience designing and governing Unity Catalog, covering catalog structure, access permissions, external locations, storage credentials and system tables.
- Deep knowledge of Azure Databricks security, including VNet injection, Private Link, Entra ID with SCIM provisioning, service principals and Key Vault-backed secret scopes.
- Proven experience building real‑time pipelines with Structured Streaming on Event Hubs or Kafka, along with Delta Lake and medallion architecture.
- Practical experience deploying with Databricks Asset Bundles and running CI/CD pipelines in Azure DevOps or GitHub, plus working with Git folders.
- Advanced Python skills, including building wheel packages and managing library dependencies in privately networked environments.
#J-18808-Ljbffr
Data Engineer in Sydney employer: Source Point Strategic Sourcing
Source Point Strategic Sourcing offers a dynamic work environment that prioritises quality, innovation, and team success. Employees benefit from ongoing learning and development opportunities, as well as a hybrid working model that allows for flexibility. The company fosters personal and professional growth alongside a forward-thinking leadership team.
Contact Details:
Source Point Strategic Sourcing Recruitment Team
