Data Engineer

We are seeking a skilled individual to design, implement, document, and automate scalable production-grade end-to-end data pipelines, incorporating API ingestion, transformation, processing, monitoring, and analytics. Key Responsibilities: - Develop data-intensive applications using cloud platforms like AWS, focused on distributed computing, lake house architecture, and real-time streaming. - Enforce best practices for data modeling and engineering throughout the development process. - Collaborate with the infrastructure engineering team to establish optimized infrastructure for data extraction, transformation, and loading. - Work independently while effectively communicating and collaborating with cross-functional teams to complete projects. - Lead the integration of technical components across teams as necessary. Technical Skills: - Proficient in one or more advanced programming languages for cloud-native data solutions. - Experience with distributed processing systems (e.g., Apache Spark) and orchestration frameworks (e.g., Apache Airflow). - Familiarity with open table formats such as Hudi for efficient data management. Qualifications: - Strong commitment to quality, ethics, integrity, and compliance in all work activities. - Exceptional decision-making and problem-solving abilities, with keen attention to detail in managing multiple tasks and meeting deadlines.