Data Engineer

We are looking for a skilled Data Engineer with 3–4 years of experience in designing, developing, and maintaining cloud-based data integration solutions on Microsoft Azure. The ideal candidate should have hands-on experience building Azure Data Factory (ADF) pipelines, integrating data from REST APIs, developing SQL solutions, and creating Power BI reports. The role requires someone who can independently handle ad hoc data requests, troubleshoot production issues, and maintain code using Azure DevOps.

Key Responsibilities

• Design, develop, and maintain data ingestion pipelines using Azure Data Factory (ADF).
• Develop pipelines to extract data from REST APIs, including handling authentication mechanisms, pagination, incremental and historical data loads, and error handling.
• Load and transform data into Azure target systems such as Azure Blob Storage and Azure SQL Database.
• Develop and optimize SQL queries, stored procedures, views, and data transformations.
• Handle ad hoc data extraction, transformation, and reporting requests from business stakeholders.
• Monitor scheduled pipelines, investigate failures, and perform production support activities.
• Maintain source code, pipelines, and deployment artifacts using Azure DevOps with proper version control and branching strategies.
• Collaborate with business users to understand reporting requirements and build interactive Power BI dashboards and reports.
• Ensure data quality, consistency, and completeness across ingestion and reporting processes.
• Participate in code reviews and follow development best practices.

Required Skills

Azure

• 3–4 years of hands-on experience with Microsoft Azure.
• Strong experience with Azure Data Factory (ADF).
• Experience creating and scheduling pipelines, triggers, parameterized pipelines, and reusable datasets.
• Good understanding of linked services and integration runtimes.
• Experience integrating data from REST APIs, working with JSON payloads, and handling pagination, rate limiting, authentication, historical backfills, and incremental loads.
• Experience maintaining code repositories in Azure DevOps, including Git branching strategies, pull requests, and deployment pipelines.
• Experience working with Azure Blob Storage and Azure SQL Database.

SQL

• Strong hands-on experience in SQL.
• Experience writing complex queries, stored procedures, views, joins, window functions, and performance tuning.

Power BI

• Experience developing Power BI dashboards and reports.
• Ability to create data models, measures (DAX), and visualizations.
• Experience connecting Power BI to Azure SQL and Azure Blob data sources.

Preferred Qualifications

• Experience working with Python for data processing is a plus.
• Familiarity with Azure Functions or Logic Apps is an advantage.
• Understanding of ETL/ELT design principles.
• Experience working in Agile/Scrum environments.
• Knowledge of data warehousing concepts and dimensional modelling.
• Strong analytical and problem-solving skills.
• Ability to independently own development tasks.
• Good communication and stakeholder management skills.
• Ability to manage multiple priorities and ad hoc requests.
• Proactive approach to troubleshooting and production support.

Experience

• 3–4 years of experience as a Data Engineer or ETL Developer.
• Hands-on experience with Azure data integration technologies.
• Experience delivering end-to-end data pipelines and reporting solutions.

Who we are

Glencore is one of the world’s largest global diversified natural resource companies and a major producer and marketer of more than 60 commodities. Through a network of assets, customers and suppliers that spans the globe, we produce, process, recycle, source, market and distribute the commodities that advance everyday life.

With over 140,000 employees and contractors and a strong footprint in over 30 countries in both established and emerging regions for natural resources, our marketing and industrial activities are supported by a global network of offices.

Glencore’s customers are principally industrial consumers, such as those in the automotive, steel, power generation, battery manufacturing and oil sectors. We also provide financing, logistics and other services to producers and consumers of commodities.