Senior Data Engineer (Databricks, Lakehouse & AI-BI)

Title: Senior Data Engineer (Databricks Lakehouse & AI-BI)
Location: ABU DHABI
Exp: 6 Years

Job Description:
Key Responsibilities
1. Pipeline Orchestration: Design and build robust data ingestion and orchestration pipelines with Databricks (Lakeflow Jobs/Workflows) as the primary platform and Azure Data Factory supporting legacy/external needs taking full ownership of pipeline reliability and the transition toward Databricks-native orchestration.
2. Lakehouse Data Engineering: Build and maintain scalable Delta Lake tables under Unity Catalog schema design access controls lineage tracking and data quality across bronze/silver/gold layers.
3. Lakeflow Pipeline Development: Build declarative ETL pipelines using Databricks Lakeflow incorporating CDC incremental processing and data quality validations.
4. AI/BI & Genie Enablement: Design and maintain semantic layers metrics and certified datasets to support Genie spaces and AI/BI dashboards enabling natural-language querying and self-service analytics.
5. SharePoint/External Source Integration: Configure and maintain Lakeflow Connect (or similar) integrations for ingesting SharePoint and other external data sources into the Lakehouse with minimal manual effort.
6. Governance & Best Practices: Apply Unity Catalog governance controls access policies data masking audit logging and uphold strong engineering practices (CI/CD version control testing).

Must-Have Skills
Strong hands-on Databricks expertise PySpark/SQL Workflows/Jobs orchestration Delta Lake Unity Catalog
Working knowledge of Azure Data Factory
Experience with Lakeflow (Declarative Pipelines/Connect) including CDC and data quality validations
Exposure to Databricks AI/BI or Genie (semantic layers certified datasets)
Familiarity with CI/CD and version control (Git Databricks Repos/Asset Bundles)
Demonstrated ability to own deliverables fully and resolve issues independently
Strong communication skills for working with business/analytics stakeholders