Technical Lead – Data Platform in Toronto, Ontario

Job Description

Insight Globa is hiring for a Technical Lead for our Data Platform team. This role requires a strong technical background in building data pipelines and infrastructure for internal. The downstream teams will use the pipelines and data for data lakes, visualization and analytics.

Responsibilities

 Design and implement robust data pipelines for data ingestion and processing.

 Build and maintain data lakes, ensuring data is cleaned and structured for use by other teams.

 Collaborate with data engineers and scientists to create infrastructure that supports data visualization and analytics.

 Utilize CI/CD practices to automate deployment and integration of data solutions.

 Ensure the effective monitoring of data flows using tools like Datadog and other cloud monitoring solutions.

 Manage databases predominantly on AWS (with some Azure), including relational (PostgreSQL) and non-relational (MongoDB) databases.

 Develop automation scripts using Python for orchestration tasks, including workflow management with tools like Apache Airflow.

 Work with real-time streaming technologies such as Kafka to support both batch and real-time data processing.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] .

To learn more about how we collect, keep, and process your private information, please review Insight Global’s Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .

Skills and Requirements

8+ years of experience as a data engineer focused on building pipelines for internal data ingestion from data lakes / raw formats

 Advance Python user

 Strong experience with AWS cloud services; familiarity with Azure is a plus.

 Knowledge of CI/CD practices and tools for automation.

 Experience in building and maintaining data pipelines, data lakes, and ETL processes.

 Understanding of database technologies PostgreSQL (relational) and MongoDB (non-relational)

 Familiarity with orchestration tools like Airflow or Dagster

 Cloud Monitoring Datadog ro CloudWatch

 Terraform, ansible experience

 Proven ability to lead projects and mentor team members. Experience with streaming technologies like Kafka for handling real-time data.

 Understanding of cloud infrastructure best practices and tools like Terraform or Ansible.

 Appreciation for data visualization concepts and tools.

 Realtime data flow automation null

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].