Job Description

Lead Data Engineer

Location: 100% Remote (Preferred: Chicago, IL / Chicagoland Area) Client Location: Chicago, IL Duration: 6+ Months Contract (Contract-to-Hire) Start Date: 07/20/2026 End Date: 12/31/2026 Position Type: Contract (C2C) No. of Positions: 1

Position Overview: We are seeking a highly skilled Lead Data Engineer to join a growing Data & Analytics team responsible for building and modernizing an enterprise-scale AWS + Databricks Lakehouse platform. This is a hands-on technical leadership role where you will architect, develop, and optimize scalable data solutions that power analytics, reporting, and AI initiatives across the organization. The ideal candidate has extensive experience designing enterprise data platforms, building robust ETL/ELT pipelines, and working with cloud-native technologies such as Databricks, Apache Spark, Delta Lake, Unity Catalog, AWS, SQL, and Python. This position offers the opportunity to influence enterprise data architecture while mentoring engineering teams and collaborating closely with business stakeholders.

Key Responsibilities

• Design, develop, and maintain scalable enterprise data pipelines using Databricks and AWS.
• Lead the architecture and implementation of modern Lakehouse solutions.
• Build high-performance ETL/ELT pipelines supporting both batch and streaming workloads.
• Develop data ingestion, transformation, and orchestration frameworks.
• Optimize Spark workloads for performance, scalability, and cost efficiency.
• Implement Delta Lake best practices for reliable and governed data management.
• Design and maintain enterprise data models supporting analytics and reporting.
• Collaborate with business stakeholders to translate requirements into scalable technical solutions.
• Establish engineering standards, coding best practices, and data governance processes.
• Monitor pipeline performance and troubleshoot production issues.
• Mentor junior engineers while remaining actively involved in hands-on development.
• Support CI/CD automation and DevOps best practices across the data platform.

Required Technical Skills

• Databricks
• Apache Spark
• Delta Lake
• Unity Catalog
• Databricks Workflows
• Lakehouse Architecture
• Notebook Development
• Performance Optimization
• AWS Cloud
• Amazon S3
• IAM
• VPC & Networking
• Cloud Storage
• Security & Access Management
• Cloud-native Architecture
• Programming
• Python
• SQL
• PySpark
• Spark SQL
• Data Processing Frameworks
• Data Engineering
• ETL / ELT Development
• Data Pipelines
• Data Integration
• Data Transformation
• Batch Processing
• Streaming Data Processing
• Distributed Computing
• Data Architecture
• Lakehouse Architecture
• Medallion Architecture
• Data Modeling
• Data Warehousing
• Metadata Management
• Data Governance
• Data Quality
• DevOps & Automation
• CI/CD Pipelines
• Git
• Azure DevOps or GitHub Actions
• Infrastructure Automation
• Monitoring & Logging

Required Qualifications

• 10+ years of overall IT experience.
• 5+ years of hands-on Data Engineering experience.
• Strong experience designing enterprise-scale ETL/ELT solutions.
• Expert-level experience with Databricks, Apache Spark, Delta Lake, and Unity Catalog.
• Strong AWS experience including S3, IAM, networking, and cloud architecture.
• Advanced SQL and Python programming skills.
• Experience building batch and streaming data pipelines.
• Strong understanding of distributed computing concepts.
• Experience implementing Lakehouse and Medallion architecture.
• Experience with Agile software development methodologies.
• Strong problem-solving and analytical skills.
• Excellent communication and stakeholder management abilities.

Preferred Qualifications

• Experience leading Data Engineering teams.
• Experience supporting AI, Machine Learning, or Advanced Analytics platforms.
• Knowledge of enterprise data governance frameworks.
• Experience with infrastructure automation and cloud optimization.
• Exposure to modern analytics platforms and reporting ecosystems.
• Experience working in highly collaborative Agile environments.

Soft Skills

• Strong leadership and mentoring abilities.
• Excellent communication and presentation skills.
• Ability to translate business needs into technical solutions.
• Self-driven with strong ownership and accountability.
• Comfortable working in fast-paced, evolving environments.
• Strong organizational and time-management skills.
• Collaborative mindset with a focus on delivering business value.

Job Tags

Similar Jobs