Databricks Engineer

Cynet Systems Voir toutes les offres

  • Toronto, ON
  • Permanent
  • Temps-plein
  • Il y a 12 heures
  • Postuler facilement
Job Overview:Responsibilities:Data Engineering And Pipeline Development:
  • Designs, develops, and maintains scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL).
  • Implements Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake.
  • Builds and optimizes batch and real-time data pipelines using Structured Streaming.
  • Develops data ingestion frameworks for structured and semi-structured data sources.
Performance Optimization And Data Management:
  • Optimizes Spark jobs for performance, scalability, and cost efficiency (partitioning, caching, joins, etc.).
  • Manages and optimizes Delta Lake tables, including ACID transactions, time travel, and schema evolution.
  • Performs debugging, troubleshooting, and performance tuning of data pipelines and jobs.
Data Quality, Governance And Security:
  • Implements data quality checks, validation frameworks, and monitoring solutions.
  • Ensures adherence to data governance, security, and access control best practices.
  • Applies knowledge of semantic layer concepts to support reporting and analytics use cases.
Collaboration And Operations:
  • Works with cross-functional teams including data analysts, engineers, and business stakeholders.
  • Utilizes Databricks workflows and scheduling tools for orchestration.
  • Troubleshoots and resolves issues across pipelines, clusters, and data workflows.
Education:
  • Bachelor's degree in Computer Science, Information Technology, or a related field preferred.
Required Skills And Experience:
  • 4 5 years of experience in Data Engineering or Big Data environments.
  • Strong hands-on experience with Databricks platform.
  • Expertise in Python/PySpark, Spark SQL, and advanced SQL.
  • Strong understanding of Apache Spark internals and distributed data processing.
  • Experience with Delta Lake and batch/streaming architectures.
  • Experience working with large-scale datasets.
  • Strong debugging, troubleshooting, and performance tuning skills.
  • Experience with Salesforce data or system integrations.
  • Knowledge of semantic layer concepts.
Preferred Qualifications:
  • Experience with cloud platforms (GCP preferred).
  • Familiarity with data orchestration tools.
  • Experience with data warehousing solutions such as Snowflake.
  • Understanding of data modeling concepts.
  • Experience with Unity Catalog or similar governance tools.
  • Exposure to real-time streaming technologies.
Soft Skills:
  • Strong analytical and problem-solving abilities.
  • Ability to work effectively in a fast-paced Agile environment.
  • Strong communication and stakeholder collaboration skills.

Cynet Systems