Data Engineer

Plan A Technologies is looking for an experienced Data Engineer to join our growing team. This is a fast-paced job with room for significant career growth. Please note: you must have at least 5+ years of experience as a Data Engineer to be considered for this role.

Job Responsibilities:

• Data Pipeline Management:
  • Design, build, and manage Azure Data Factory pipelines for data ingestion.
  • Handle both batch and near real-time data ingestion using Azure Service Bus.
  • Integrate structured, semi-structured, and unstructured data sources using formats such as Parquet, Avro, JSON, etc., and manage loading into the Bronze layer.
  • Build and manage data engineering pipelines using Databricks (PySpark, SparkSQL).
• Data Transformation:
  • Implement Spark-based data transformation jobs from Bronze to Silver and Gold layers.
  • Establish and manage Data Quality Framework using Databricks and Data Factory.
• Data Security and Monitoring:
  • Set up data security using RBAC and ACLs to manage permissions on sensitive datasets.
  • Monitor pipelines using Azure Monitor, set up alerts, and ensure SLAs are met.
  • Optimize storage in ADLS Gen2 and use Azure Key Vault to secure credentials.
  • Optimize costs using Azure Cost Management.

Experience Requirements:

• BS in Computer Science, Software Engineering, or equivalent practical experience.
• 5+ years of experience with Azure Data Factory for ETL/ELT operations and Databricks (PySpark, SparkSQL) for data transformations.
• Proficiency in Azure Data Lake Storage (ADLS Gen2), SQL Server, PostgreSQL, Cosmos DB, CRM, and ERP systems.
• Knowledge of Data Governance, RBAC, and ACLs for managing permissions.
• Experience with CI/CD tools such as Azure DevOps, GitHub Actions, and Terraform.
• Familiarity with Azure Monitoring, Alerting, and Logging tools, and hands-on experience with PowerBI.
• Excellent organizational, problem-solving, and analytical abilities coupled with outstanding communication and interpersonal skills.
• Initiative and drive to excel in your role.

About The Company/Benefits:

Plan A Technologies is an American software development and technology advisory firm that brings top-tier engineering talent to clients around the world. Our software engineers tackle custom product development projects, staff augmentation, major integrations and upgrades, and much more. The team is far more hands-on than the giant outsourcing shops, but still big enough to handle major enterprise clients.

Location: Work From Home 100% of the time, or come into one of our global offices—it's up to you.

We foster a great colleagues and an upbeat work environment: You'll join an excellent team of supportive engineers and project managers who work hard but don’t ever compete with each other.

Benefits: You’ll receive a generous vacation schedule, a brand new laptop, and other perks.

If this sounds like you, we'd love to hear from you!