Senior Data Engineer

ComplexCare Solutions is a leading technology company that combines advanced cloud-based data analytics and data-driven intervention platforms to achieve meaningful insights. Our platform requires the ingestion of large amounts of varied data from multiple clients. Timely and accurate ingestion of business data is crucial for the platform's success. You have the opportunity to create a configurable and robust data ingestion platform and work on cutting-edge cloud and big data technologies.

Duties and Responsibilities:

• Participate actively with the agile team in ceremonies like grooming, planning, standups, retrospectives, and demos. Contribute to grooming and standup meetings, creating and updating tasks, estimates, and status.
• Design and develop modern data ingestion pipelines using tools like Azure Data Factory (ADF), Databricks, and MS-SQL, and extensively work on the Azure Cloud Platform.
• Write complex queries, stored procedures, and functions for job execution and data manipulation. Create highly configurable designs to apply various business rules, filters, and mappings based on different clients.
• Collaborate with data architects and business analysts to create logical data models and DDL scripts for physical model creation. Read data files in various formats (flat, excel, csv, json) from SFTP servers and validate the data using various business rules and configurations.
• Design and code various data architecture components like data validation, cleansing, de-duping, and the Symantec layer. Work on large data sets to ensure configurable ingestion, dynamic rule validation, data cleansing, transformation, and loading into the data warehouse.
• Implement data validation and quality checks to ensure the accuracy and completeness of the data in the data warehouse.
• Perform query and data processing performance optimization, identify and resolve any issues by examining query plans, creating appropriate indexes, resolving deadlocks, and creating table hints.
• Participate in design discussions, data analysis, data model creation, and produce quality design diagrams (using MS-Visio, Draw.io, etc.) and documentation (MS-Word, Excel, etc.)
• Maintain compliance with ComplexCare Solutions policies, procedures, and mission statement; adhere to confidentiality and HIPAA requirements in all aspects of data handling and services rendered.

Job Requirements:

• Minimum of 10 years of industry experience working in the data and reporting area; knowledge of healthcare data is a plus.
• 10+ years of experience with data processing/pipeline, ETL, Data Workflow, Query Plans, and Optimization; expertise in MS SQL, T-SQL, and ETL Jobs.
• 5+ years of experience with Databricks, PySpark/SparkSQL, data frames, and unity catalog.
• 5+ years of experience in writing query plans and managing indexes. Familiarity with Azure Cloud is preferred, including ADF, Azure Delta Lake, Azure SQL Server, Data Sync, Log Insights, and Analytics.
• Experience working with role-based security at database, ETL jobs, and data export levels. Experience with HIPPA and PHI is beneficial.
• Strong verbal and written communication skills, ability to effectively communicate with internal and external customers.
• Excellent computer proficiency (MS Office – Word, Excel, and Outlook) and ability to work independently, under pressure, and meet deadlines.

Education: BS degree in Computer Science, Computer Engineering, Business, or equivalent experience.

Physical Demands and Work Environment:

• Sedentary work (i.e., sitting for long periods); exerting up to 10 pounds of force occasionally.
• Frequently or constantly lifting, carrying, pushing, pulling, or otherwise moving objects and performing repetitive motions; subject to inside environmental conditions.
• Some travel (less than 10%) may be required for training and collaboration purposes.

We encourage applicants from diverse backgrounds to apply, even if not meeting every qualification. ComplexCare Solutions seeks the best candidate and values diversity, equity, and inclusion, fostering a culture where everyone can bring their full, authentic selves to work.

ComplexCare Solutions is proud to be an equal opportunity workplace and affirmative action employer, committed to equal employment regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, or veteran status. We also consider qualified applicants with criminal histories, consistent with legal requirements.

The company maintains a drug-free work environment. Manufacturing, selling, distributing, dispensing, possessing, or using any controlled substance or marijuana in the workplace is unlawful and will result in disciplinary action, including termination.