Cloud Data Engineer with Databricks

Diaconia is looking for a talented Cloud Data Engineer to join our Amazing team! If you're seeking a company that truly values your talents, look no further! At Diaconia, we are committed to serving and caring for our colleagues, clients, and community. Our team comprises talented individuals who appreciate the opportunity to contribute their knowledge and experience to further the growth and development of our industry. Our ideal candidates embrace diverse thinking, enjoy partnering with others, and seek to make a difference! We are currently searching for a full-time member for the position of: Cloud Data Engineer with Databricks. **U.S. Citizenship is REQUIRED.**

A Cloud Data Engineer with Databricks experience is responsible for managing, developing, and maintaining data infrastructure and pipelines, with a focus on Databricks, to support the agency's data-driven initiatives and regulatory responsibilities. The CFPB is a U.S. government agency tasked with regulating and overseeing financial products and services to protect consumers.

Job Summary: The Cloud Data Engineer with Databricks at the Consumer Financial Protection Bureau (CFPB) plays a crucial role in designing, building, and maintaining data infrastructure and ETL pipelines using Databricks and cloud-based technologies, enabling data-driven decision-making and ensuring data accuracy and accessibility for regulatory purposes.

Key Responsibilities: - Collaborate with cross-functional teams to contribute to the architecture, design, development, and maintenance of large-scale data and analytics platforms including system integrations, data pipelines, and data models. - Develop data pipelines using Databricks, Apache Spark, and other cloud-based technologies for ingesting, transforming, and loading data from various sources. - Implement data transformation processes to ensure quality and integrity, and create transformation paths for migrating data from on-premise sources to AWS. - Integrate data from diverse sources, develop optimized data models for regulatory analysis, and ensure performance optimization of data pipelines. - Maintain data governance and regulatory compliance, including documentation for audits and reporting purposes. - Troubleshoot data-related issues, ensuring reliability and compliance with regulatory standards. - Stay updated with regulatory changes, industry trends, and advancements in cloud technologies to implement best practices in data engineering.

Minimum Qualifications: - U.S. Citizens ONLY per Federal Requirements. - Bachelor's or higher degree in computer science, data engineering, or a related field. - Databricks Certified Data Engineer Professional certification (NO EXCEPTIONS). - Minimum of 3 years of experience with: - Strong understanding of data lake, lakehouse, and data warehousing architectures in a cloud-based environment. - Hands-on experience with Databricks, data ingestion, transformation, and analysis. - Proficiency in Python for data manipulation, scripting, and automation, along with in-depth knowledge of AWS services like Amazon S3, EC2, RDS, etc. - Designing and building ETL processes and data pipelines using Python/PySpark and SQL, including automation tools like Apache Airflow or AWS Step Functions. - Data quality management and governance principles. - Experience with GitHub or similar tools and leveraging Postgres in a parallel processing environment.

Preferred Skills: - Experience with financial or regulatory data management. - Familiarity with Agile or DevSecOps environments and related collaboration tools. - Knowledge of regulatory frameworks in the financial industry, as well as familiarity with DevOps and CI/CD practices. - Experience with machine learning and AI technologies.

Clearance Requirements: Must be able to obtain and maintain a Public Trust clearance and be a verifiable U.S. Citizen for this Federal support position.

A Cloud Data Engineer with Databricks at the CFPB plays a vital role in ensuring data accuracy, integrity, and compliance with regulatory standards, supporting the agency's mission to protect consumers in the financial sector. The role demands expertise in Databricks, cloud technologies, and a deep understanding of data engineering principles within a regulatory context. Applicant selected will be subject to a government security investigation and must meet eligibility requirements for access to classified information.

Diaconia is an Equal Opportunity Employer, Minorities/Females/Veterans/Disabled. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, or national origin.