Resident Solution Architect Databricks

Unison Consulting Pte Ltd

Apply to this job
Sydney, New South Wales, Australia on site Until 8/21/2026 First posted March 20, 2026 Last posted March 20, 2026
Job description
  • Design and implement large-scale Azure and Databricks Data Lakehouse solutions.
  • Build and optimize ETL/ELT pipelines for batch and real-time streaming workloads using Azure Data Factory, Databricks, and Apache Spark.
  • Develop scalable data ingestion frameworks and integrate diverse structured and unstructured data sources.
  • Optimize data storage and query performance using Delta Lake, Parquet format, partitioning, and Spark performance tuning techniques.
  • Implement robust data governance, security, and access control using Unity Catalog, Azure Key Vault, and least-privilege principles.
  • Build and maintain data quality frameworks using Great Expectations or similar validation tools for batch and streaming data.
  • Automate pipeline deployment and CI/CD processes using Azure DevOps, Git, and configuration management tools.
  • Enable advanced analytics and ML workflows by preparing curated datasets and integrating with ML platforms like MLflow.
  • Develop real-time streaming solutions using Kafka, Azure Event Hubs, and Databricks Structured Streaming.
  • Create reusable PySpark libraries and frameworks for data curation, reconciliation, notifications, and Delta table automation.
  • Migrate legacy Hive or on-prem data sources to cloud platforms while ensuring security, compliance, and performance.
  • Monitor pipeline performance, troubleshoot issues, and implement alerting and logging for proactive management.
  • Collaborate with cross-functional teams and stakeholders to translate business requirements into scalable data solutions.
  • Establish best practices, standards, and documentation for enterprise-wide data engineering and analytics initiatives.
  • Mentor junior engineers and promote knowledge sharing within the data engineering team.
About this role

Summary

Designs and implements large-scale data solutions, pipelines, and ML workflows on cloud.

Job title

Resident Solution Architect Databricks

Experience level

not specified

Industry

software

Location requirements

Sydney, Australia; remote work not specified

Salary

Not specified

Management role

Yes

Skills & keywords

Required skills

AzureDatabricksSparkPythonApache KafkaAzure Data FactoryUnity CatalogAzure Key VaultCI/CD

Preferred skills

AzureDatabricksSparkMLflowAzure Data Factory

Specializations

cloud data engineeringETL/ELT pipelinesstreamingML workflowsdata governance
Locations

Structured locations inferred from the posting.

Sydney NSW, Australia

On-site City
Related searches