Data Engineer | Data Platform & Analytics

Job Description

Job Responsibilities • Design, develop and maintain efficient and reliable ETL processes and data pipelines for data ingestion, transformation and integration from multiple source agencies. • Develop and maintain modular, production-quality code for statistical production processes while preserving existing business logic and ensuring outputs are accurately reconciled against current production results. • Support the migration, modernisation and optimisation of legacy processes to improve efficiency, maintainability and reliability. • Design and maintain database schemas, data models and storage structures to support the efficient ingestion, validation, transformation and retrieval of large datasets. • Develop reusable data engineering components, including common libraries, configuration frameworks, logging standards and utilities to streamline production workflows. • Implement and maintain CI/CD pipelines for data engineering workflows, including automated testing and deployment. • Manage workflows across Development, UAT and Production environments to support safe testing, deployment and synchronisation. • Design and configure complex data pipeline workflows and schedules, including dependencies, failure handling, recovery and operational monitoring. • Establish and maintain version control structures using GitLab, including branching strategies, merge requests, code reviews and technical documentation. • Monitor and troubleshoot data pipelines, conduct root-cause analysis and continuously improve system reliability and performance. • Collaborate with technical teams, business users and stakeholders to translate business and statistical processing requirements into robust technical solutions. Show More