Data Engineer

Company Overview
TechGencia Private Limited embodies excellence, passion, and flexibility. As a premier software development company based in Kerala, India, we deliver world-class IT consulting, offshore software development, testing, and user interface design services. Our diverse projects range from simple information systems and websites to complex enterprise architectures, both desktop and web-enabled. With a commitment to on-time, within-scope, and cost-effective service delivery, TechGencia upholds a quality-focused approach, making us one of the best software development companies in Kerala. Visit us at www.techgencia.com

Job Overview
We are seeking a Junior Data Engineer with 1 to 3 years of work experience to join our dynamic team on a freelance, remote basis. The role is open to candidates located in Thiruvananthapuram, Kollam, Cochin, Kozhikode, Malappuram, and Palakkad districts. The ideal candidate will have experience in data pipeline development and toolsets, specifically using technologies such as Apache Spark, Airflow, Kubernetes, SQL, Progress, and relational databases.

Qualifications and Skills
We are looking for candidates with:

• Experience in developing and maintaining data pipelines, ensuring data integrity and performance.
• Proficiency in using Apache Spark for data processing and analytics tasks.
• Hands-on experience with Apache Airflow for orchestrating complex data workflows.
• Strong knowledge of Kubernetes for container orchestration and efficient large-scale data processing.
• Proficiency in SQL for querying relational databases and managing data sets.
• Experience with Progress and Vector databases for high-performance data management.
• Excellent problem-solving skills and a collaborative attitude towards team-oriented tasks.

Roles and Responsibilities
The successful candidate will be responsible for:

• Designing, developing, and maintaining scalable data pipelines for optimal data integration and quality.
• Developing and maintaining platform development using Python and related technologies.
• Collaborating with data scientists and other engineering teams to integrate new data sources.
• Ensuring the reliability and performance of data processing systems by monitoring and optimizing workflows.
• Implementing best practices for data security, governance, and compliance.
• Automating manual processes and optimizing data delivery for business intelligence initiatives.
• Staying updated with emerging technologies and industry best practices in data engineering.
• Participating in code reviews, ensuring code quality while sharing knowledge with peers.
• Contributing to continuous improvements of data engineering processes and infrastructure.
• Implementing and managing data processing, pipelines, and analytical frameworks.
• Developing and integrating APIs for data interchange between different systems or services.
• Training, retraining, and monitoring machine learning systems and models as needed.
• Constructing optimized data pipelines to feed machine learning models and extending existing machine learning libraries and frameworks.