Python QA - (Pyspark with Databricks)

About the Role

We are looking for an experienced QA Automation Engineer with strong expertise in Python, PySpark, and Databricks to support enterprise data engineering and analytics initiatives. The ideal candidate should have hands-on experience in validating large-scale data pipelines, ETL processes, and data platforms while building robust automation frameworks to ensure data quality and reliability.

Key Responsibilities

• Design, develop, and maintain automated test frameworks using Python.
• Validate ETL pipelines, data transformations, and data ingestion processes built using PySpark and Databricks.
• Perform data validation, reconciliation, and quality checks across structured and semi-structured datasets.
• Develop automated test scripts for functional, integration, regression, and end-to-end testing of data platforms.
• Validate large datasets using SQL and PySpark.
• Build and maintain automated data quality checks and monitoring processes.
• Collaborate with Data Engineers, Developers, and Business Analysts to understand data requirements and ensure high-quality deliveries.
• Analyse defects, perform root cause analysis, and support issue resolution.
• Integrate automated tests into CI/CD pipelines and support Agile delivery.
• Contribute to QA best practices, automation strategy, and continuous improvement initiatives.

Skills & Qualifications

• 8+ years of experience in Software Testing or Test Automation.
• Strong hands-on experience with Python.
• Solid expertise in PySpark and Databricks.
• Strong experience in SQL and ETL/data validation.
• Experience testing large-scale data pipelines and data engineering workflows.
• Hands-on experience with data quality frameworks or libraries such as Great Expectations, Soda, Deequ, dbt Tests, or similar.
• Strong understanding of data validation, reconciliation, and quality assurance methodologies.
• Experience working in Agile/Scrum environments.
• Excellent analytical, troubleshooting, and problem-solving skills.