Data Engineer (Architect) - Remote
Braintrust
IT, Data Science
Latin America
Posted on Jan 30, 2025
Job Description
What You’ll Do
What You’ll Do
- Snowflake & AWS: Own and optimize our Snowflake data warehouse, ensure efficient scaling on AWS, and maintain performant, reliable data stores.
- Build Core Data Infrastructure (0 → 1): Design and implement Neon Blue’s foundational data architecture and pipelines using tools like Airbyte/FiveTran for external data ingestion (Shopify, Klaviyo, Amplitude, etc.).
- Pipeline Orchestration: Develop and manage DAG-based workflows for batch and near-real-time data processing in Airflow or Dagster.
- Data Transformation & Modeling: Use dbt and Python to build robust, maintainable transformations that power reporting, analytics, and ML workflows.
- Machine Learning Platform Integration: Collaborate with Data Science teams to integrate data into cloud ML platforms (Google Vertex AI or Azure ML) for advanced analytics and modeling.
- Observability & Security: Implement end-to-end monitoring, alerting, and logging strategies to ensure data quality, reliability, and compliance. Champion security best practices and SOC-2 standards.
- Compliance & Controls: Work alongside legal and DevOps teams to meet data governance requirements, enforce data access controls, and support compliance audits.
- Scalability & Performance: Architect solutions for rapid, large-scale data ingestion and transformation. Design automated frameworks to handle real-time event streams.
- Culture & Collaboration: Be part of a small, nimble, and passionate engineering team—contributing to a collaborative, ownership-driven environment.
- Experience in 0-1 Data Systems: Proven track record of building data infrastructure from scratch at a startup or similar environment.
- Advanced Cloud Warehouse Skills: Expert-level Snowflake knowledge (data ingestion, optimization, security, RBAC controls, etc.).
- Python, dbt, Airflow/Dagster: Strong proficiency in scripting and orchestration tools for both batch and near-real-time pipelines.
- Cloud Services (AWS/GCP/Azure): Hands-on experience setting up and optimizing data pipelines and ML workflows on major cloud platforms.
- Data Observability & Security: Familiarity with logging, monitoring, alerting, and end-to-end data governance, including SOC-2 compliance.
- Performance & Scaling: Deep understanding of how to handle large, rapid data ingestion while ensuring reliability and performance.
- Strong Communication & Collaboration: Ability to work closely with cross-functional teams (Data Science, Product, Ops) to deliver data-driven insights.
- Impact & Autonomy: Be a key architect who shapes the entire data landscape in a fast-growing startup.
- Experienced Founders: Work alongside a leadership team that has successfully built and scaled a $100M/year e-commerce venture.
- Cutting-Edge Tech: Get hands-on with modern tooling and cloud platforms for machine learning and real-time analytics.
- Competitive Compensation & Benefits: We offer a generous salary, equity, healthcare, and a flexible time-off policy.