Senior Data Engineer
Viafoura
About Viafoura
About the Role
Key Responsibilities
- Design and implement scalable data pipelines using modern tools like Apache Spark, Kafka, or Airflow
- Build and maintain data warehouses and data lakes using cloud platforms (AWS, Snowflake)
- Optimize data storage solutions and implement effective data partitioning strategies
- Ensure data architecture follows best practices for security, reliability, and performance
- Develop ETL/ELT processes to ingest data from various sources including APIs, databases, and streaming platforms
- Monitor and troubleshoot data pipeline performance, implementing automated alerting systems
- Collaborate with software engineers to integrate data systems with production applications
- Maintain data quality through validation frameworks and testing protocols
- Provide technical guidance on complex projects
- Evaluate and recommend new technologies and tools for the data stack
- Lead architecture discussions and contribute to technical decision-making processes
- Document data systems and establish engineering best practices
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, or related technical field
- 5+ years of experience in data engineering or related roles
- Strong programming skills in Python, Scala, or Java
- Extensive experience with SQL and database technologies (PostgreSQL, MySQL, etc.)
- Proficiency with cloud platforms and services (AWS, Snowflake)
- Experience with distributed computing frameworks (Spark) and workflow orchestration tools (Airflow, Dagster)
- Knowledge of data modeling concepts and dimensional modeling techniques
Preferred Qualifications
- Experience with streaming technologies (Kafka, Kinesis, Pub/Sub)
- Familiarity with containerization and orchestration (Docker, Kubernetes)
- Background in machine learning operations (MLOps) and feature engineering
- Experience with data governance and compliance frameworks
- Knowledge of infrastructure as code (Terraform, CloudFormation)
- Experience with Apache Iceberg