Overview

We are seeking a skilled Data Engineer/Developer to join our team. The ideal candidate will have a passion for data, strong technical expertise, and the ability to create efficient and reliable data solutions. You will work closely with data scientists, analysts, and other stakeholders to ensure seamless data processing, storage, and retrieval.

Responsibilities

Data Pipeline Development

• Design, build, and maintain data pipelines to extract, transform, and load (ETL) data from various sources.
• Utilize the following technologies:
  • Pyspark : Develop ETL workflows and data transformations.
  • Delta Parquet/Delta Tables : Optimize data storage and management.
  • Azure Databricks : Host and manage data pipelines.
• Collaborate with data architects to ensure data quality, consistency, and reliability.

Database Management

• Work with relational databases (eg, Azure SQL Database ):
  • Design schema and optimize database performance.
  • Write efficient SQL queries for data retrieval and manipulation.
• Implement data warehousing solutions using Azure Synapse Analytics .

Data Integration

• Integrate data from various systems and APIs:
  • Develop connectors and APIs for seamless data flow.
  • Ensure data security and compliance.

Monitoring and Optimization

• Monitor data pipelines and database performance.
• Troubleshoot issues and optimize data processing.
• Implement automated testing and quality checks.

Behavior-Driven Design (BDD)

• Utilize BDD practices in collaboration with stakeholders:
  • Write Gherkin tests to define behavior.
  • Implement BDD frameworks (eg, Behave ).
  • Ensure alignment between business requirements and technical implementation.

CI/CD and Infrastructure

• Manage CI/CD pipelines using Python and Bash scripts .
• Utilize Terraform or other infrastructure-as-code tools for provisioning.
• Collaborate with DevOps teams to ensure smooth deployment and scaling.

Qualifications Required Skills

• Proficiency in Pyspark and working with Delta Parquet/Delta Tables .
• Experience with Azure Databricks or other cloud-based data processing platforms.
• Familiarity with Azure SQL Database and writing efficient SQL queries.
• Strong programming skills in Python and Bash Scripting .
• Understanding of Java-based Lambda functions .
• Knowledge of Terraform for infrastructure provisioning.
• Ability to optimize costs using Step Functions .

Preferred Qualifications

• Exposure to other big data technologies (eg, Azure HDInsight, Azure Data Factory, or Azure Event Hubs ).
• Familiarity with data streaming and Real Time processing.
• Experience with data visualization tools (eg, Power BI, Tableau, or Looker ).

How to Apply If you are passionate about data engineering, enjoy solving complex problems, and thrive in a collaborative environment, we encourage you to apply. Please submit your resume and a cover letter detailing your relevant experience to (see below)