Job Description

Data Engineering Processing

Design and implement scalable data ingestion pipelines from diverse sources including APIs SharePoint onpremise systems and filebased sources

Perform data cleansing validation and transformation to produce highquality reliable datasets

Develop and maintain data migration and archival strategies ensuring accuracy integrity and compliance

Build and optimise logical and physical data models

Handle diverse data structures and formats including BAK MDF CSV JSON XML and Parquet

Automation Orchestration

Automate ingestion and processing workflows using Python PowerShell and orchestration tools

Apply an automationfirst mindset including experience integrating AI agents for workflow automation

Cloud Data Platforms Azure

Build and maintain solutions using Azure Data Factory Blob Storage ADLS and Azure SQL

Implement data lineage cataloguing and governance capabilities

Data Quality Security Compliance

Oversee data quality frameworks ensuring accuracy consistency and integrity

Implement audit logging data lineage and compliance practices

Maintain strong security and governance controls

Performance Scalability

Optimize pipelines SQL queries and storage layers

Troubleshoot performance issues across compute and storage

MustHave Skills DataFocused Essentials

Strong experience in ingestion migration archival pipeline automation and largescale data processing

Advanced SQL expertise including stored procedures indexing and performance tuning

Handson experience with Python and PowerShell for ETLELT and automation

Experience with Azure Data Factory ADLS Blob Storage Azure SQL

Expertise in data modelling logical physical quality frameworks and optimisation

Ability to work with structured semistructured and unstructured data formats

Strong knowledge of audit logging lineage cataloguing metadata management and security

Demonstrated automation mindset including use of AI agents

Handson experience or strong understanding of modern cloud data warehousing including Snowflake fundamentals such as virtual warehouses micropartitioning query optimization and rolebased access control

GoodtoHave Skills Infrastructure Advanced Platforms

Infrastructure DevOps

Experience with Terraform Azure DevOps YAML pipelines and cloud automation

Exposure to Azure Function Apps serverless compute and orchestration

Understanding of infrastructure as code and cloud deployment patterns

Exposure to DockerKubernetes

Advanced Data Platforms Tools

Deep or handson exposure to Snowflake including

Creating and managing Snowflake objects databases schemas roles

Using Snowpipe for automated ingestion

Performance tuning using clustering caching and micropartitioning

Understanding Snowflake cost optimization and storagecompute separation

Experience with Databricks for advanced data engineering workflows

Familiarity with Denodo or Microsoft Purview

Certifications

Azure or Snowflake certifications are strong advantages

Personal Attributes

Analytical thinker with strong problemsolving capabilities

Proactive adaptable and committed to continuous learning

Excellent communication and collaboration skills

Strong commitment to ethical data usage

Job Tags

Similar Jobs