Lead Data Engineer
Windfall
Responsibilities:
- Lead the design, development, and optimization of robust and scalable data pipelines to ingest and merge billions of individual entities into Windfall’s core data asset.
- Provide technical guidance and mentorship to other data engineers, fostering a culture of best practices, innovation, and continuous improvement.
- Collaborate closely with data science, product, and other engineering teams to translate business needs into technical requirements and deliver impactful data solutions.
- Work closely with our data science team to enable the effective running of ML models on top of massive data points.
- Architect and build supporting data services and applications to orchestrate, monitor, and ensure the reliability of our data systems.
- Drive the adoption of new technologies and methodologies to enhance our data infrastructure and processes.
Some technology you will use:
- Cloud platform - GCP
- Programming languages - Java, Python, and Kotlin
- Data warehouse & databases - BigQuery, Postgres
- Distributed processing frameworks - Dataflow (Apache Beam) and Apache Spark
- Orchestration - Airflow
Requirements:
- Bachelor’s Degree in Computer Science, Engineering, or a related field.
- 8+ years of progressive professional data engineering experience, with at least 2-3 years in a technical lead or senior-level role.
- Exceptional understanding and practical application of object-oriented programming principles in a JVM language.
- Demonstrated experience designing, building, and maintaining large-scale, high-performance data systems.
- Experience at a sub-200 person company (or similar fast-paced startup environment).
- Expert SQL knowledge and experience with complex database operations.
- Proven ability to communicate complex technical concepts effectively to both technical and non-technical stakeholders.
- Strong problem-solving skills with a track record of simplifying complex challenges into elegant, scalable solutions.
- A strong sense of ownership and responsibility for your work, coupled with excellent collaboration and team alignment skills.
- A pragmatic approach to making trade-offs between quality, complexity, and speed-of-delivery, with a focus on business impact.
Preferred Qualifications:
- Significant experience in a player-coach capacity, mentoring junior engineers, and contributing to team growth.
- Demonstrated success in leading and delivering large-scale data projects from ideation through to production, including architectural decisions.
- Proven experience leading greenfield data initiatives.
- Deep understanding of the underlying principles and trade-offs of databases, data warehouses, and distributed computing frameworks, enabling optimal technology selection and system optimization.
- Extensive working knowledge and hands-on experience with cloud-native data engineering infrastructure (GCP preferred).