Data Engineer (ADF, Databricks, Azure):
On behalf of our Banking client, Procom is searching for a Data Engineer (ADF, Databricks, Azure) for a 6-month role. This position is a hybrid position with 1 day onsite at our client’s Toronto office.
Data Engineer (ADF, Databricks, Azure) - Job Description:
The Data Engineer will be working on a project focused on advancing the functionality of Databricks within a business undergoing a transformation journey. This role involves optimizing data architecture and implementing advanced data governance and security features.
Data Engineer (ADF, Databricks, Azure) - Responsibilities:
- Develop and manage data pipelines using ADF, Databricks, and Azure Functions
- Implement data governance setup and configuration with Unity Catalog
- Load datasets from various sources into Databricks using Medallion architecture
- Convert Databricks workspace from hive metastore to Unity Catalog and set up associated permissions
- Create and manage delta table schemas with appropriate design and security permissions
- Collaborate with Data Engineer Lead and Business Analysts to achieve project goals
Data Engineer (ADF, Databricks, Azure) - Mandatory Skills:
- 7+ years experience with ADF, Databricks, and Azure Functions (C#)
- In-depth development experience with Python/Notebook on Databricks/Spark/Panda
- Expert knowledge of Unity Catalog and data governance setup/configuration
- Experience with Azure DevOps/GitHub Enterprise for deployment pipelines
- Proficiency in working with Azure PaaS resources
Data Engineer (ADF, Databricks, Azure) – Nice-to-Have Skills:
- Experience with Delta Live Table
- Financial Services Industry experience
- Previous experience with CIBC or other financial institutions
- Familiarity with Azure Entra ID for SSO/RBAC setup
- Ability to work in a highly regulated industry environment
Data Engineer (ADF, Databricks, Azure) – Assignment Length:
This is a 6-month contract position with the potential to extend or convert to full-time.
Data Engineer (ADF, Databricks, Azure) - Start Date:
ASAP
Data Engineer (ADF, Databricks, Azure) - Assignment Location:
Toronto, Ontario, Canada. The position is hybrid with 1 day in the office.
