Job Description

Job Summary

We are seeking an experienced Architect with 10 to 12 years of experience to join our dynamic team. The ideal candidate will have extensive expertise in Spark in Scala Synapse Pipelines Azure CLI Azure PowerShell Structured Streaming Kafka Azure Monitor Azure Data Lake Store Azure Key Vault Azure DevOps Azure Event Hub Python Azure Synapse SQL Pool Azure Synapse Spark Pool SQL and PySpark. This role is hybrid with day shifts and does not require travel.

Responsibilities

• Lead the design and implementation of scalable data architectures using Spark in Scala and Synapse Pipelines
• Oversee the development and deployment of data solutions leveraging Azure CLI and Azure PowerShell
• Provide expertise in Structured Streaming and Kafka to ensure real-time data processing
• Implement robust monitoring solutions using Azure Monitor to maintain data pipeline health
• Manage and optimize data storage solutions with Azure Data Lake Store and Azure Key Vault
• Collaborate with DevOps teams to streamline CI/CD processes using Azure DevOps
• Integrate and manage event-driven architectures with Azure Event Hub
• Develop and maintain data processing scripts using Python and PySpark
• Optimize SQL queries and manage data warehousing solutions with Azure Synapse SQL Pool
• Utilize Azure Synapse Spark Pool for large-scale data processing and analytics
• Ensure data security and compliance through effective use of Azure Key Vault
• Provide technical guidance and mentorship to junior team members
• Collaborate with cross-functional teams to align data architecture with business objectives
• Drive continuous improvement in data processing and storage solutions
• Conduct performance tuning and optimization of data pipelines
• Ensure high availability and disaster recovery planning for critical data systems
• Stay updated with the latest industry trends and best practices in data architecture
• Document architectural designs and technical specifications for future reference

Qualifications

• Possess a strong background in Spark in Scala and Synapse Pipelines
• Demonstrate proficiency in Azure CLI and Azure PowerShell
• Have hands-on experience with Structured Streaming and Kafka
• Show expertise in Azure Monitor and Azure Data Lake Store
• Be skilled in using Azure Key Vault for data security
• Have experience with Azure DevOps for CI/CD processes
• Be knowledgeable in Azure Event Hub for event-driven architectures
• Be proficient in Python and PySpark for data processing
• Have experience with Azure Synapse SQL Pool and Azure Synapse Spark Pool
• Possess strong SQL skills for data querying and management
• Demonstrate excellent problem-solving and analytical skills
• Have strong communication and collaboration abilities
• Be committed to continuous learning and professional development.

Certifications Required

• Azure Solutions Architect Expert Microsoft Certified: Azure Data Engineer Associate Databricks Certified Associate Developer for Apache Spark

Job Tags

Similar Jobs