We are seeking a skilled and experienced Data Engineer to join our team at M3. The Data Engineer will be responsible for designing, developing, and maintaining data infrastructure, storage solutions, and pipelines. The ideal candidate will possess expertise in data management, security, and optimization, along with proficiency in ETL/ELT, batch and near real-time workloads, and CI/CD practices. Collaboration with cross-functional teams and effective communication are essential for success in this role.
Essential Duties:
- Develop and maintain data infrastructure, including databases, data warehouses, and data lakes to support business needs.
- Design and build scalable and efficient data storage solutions for large volumes of structured and unstructured data.
- Design, develop, and deploy data pipelines for ingesting, processing, and transforming data from various sources.
- Implement data security measures, including encryption, access controls, and auditing, to protect sensitive data.
- Manage data quality by implementing monitoring and validation processes to ensure accuracy, completeness, and consistency.
- Optimize data performance through tuning queries, optimizing storage, and implementing caching and indexing strategies.
- Develop and maintain data models to organize and structure data for efficient storage, retrieval, and analysis.
- Integrate data from multiple sources, including internal databases, external APIs, and third-party data providers, to support analytical and reporting needs.
- Utilize Microsoft Azure DevOps Repos and Release Pipelines and GitHub for version control, collaboration, and deployment of data solutions.
- Communicate effectively with stakeholders to understand data requirements and provide technical expertise and support.
- Other duties as assigned.
Education/Training/Experience:
- 3+ years of experience as a Data Engineer or similar role.
- Bachelor's Degree preferred in IT, MIS, or Computer Science, or an equivalent combination of education and experience.
- Strong proficiency in data infrastructure, data management, and SQL for querying and manipulating data.
- Experience with ETL and ELT for data pipeline development, including designing and implementing batch and near real-time processing workloads.
- Proficiency in Python for data analysis, manipulation, and automation is a plus.
- Familiarity with CI/CD practices and tools for automated deployment, as well as Microsoft Azure DevOps Repos, Release Pipelines, and GitHub for version control and collaboration.
- Knowledge of cloud data lake services and data management best practices.
- Excellent communication and collaboration skills.
Physical Requirements:
- Ability to sit and/or stand for extended periods.
- Ability to perform work on a computer for extended periods.
- Ability to attend work per assigned schedule, with excellent attendance and punctuality.
- Ability to bend and lift up to 25 lbs.
.png)
