Job Summary:
We are seeking a skilled Data Engineer to join our team. The ideal candidate will have a strong technical background in designing and developing data pipelines, integrating and managing data systems, and collaborating with various stakeholders to deliver high-quality data solutions. This role is vital to ensuring our data infrastructure is robust, scalable, and optimized to meet the business’s evolving needs.
Location: Bali, Indonesia (in office)
Key Responsibilities:
Design and Develop Data Pipelines:
- Create and maintain optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional and non-functional business requirements.
Data Integration and ETL:
- Build the infrastructure required for the optimal extraction, transformation, and loading (ETL) of data from various data sources.
- Develop processes and tools to monitor and analyze data pipeline performance.
- Experience in data migration projects involving large-scale datasets.
- Build custom transformation logic to map and convert raw source data into structured formats.
Database Management:
- Work with data and analytics experts to strive for greater functionality in our data systems.
- Implement, maintain, and optimize database solutions, including both SQL and NoSQL databases.
Collaboration and Communication:
- Collaborate with data scientists, business analysts, and other stakeholders to understand data needs and provide technical solutions.
- Communicate project progress, issues, and solutions effectively to the team and management.
Data Quality and Governance:
- Ensure data quality and data governance standards are maintained.
- Implement data security measures and compliance policies.
Qualifications:
Education and Experience:
- Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
- Proven experience as a Data Engineer or in a similar role.
- Experience working with large-scale datasets (e.g., ~200 million rows) and designing scalable data pipelines.
Technical Skills:
- Proficient in SQL and experienced with relational databases.
- Experience with big data tools such as Hadoop, Spark, or Kafka.
- Knowledge of data pipeline and workflow management tools like Airflow or Luigi.
- Experience with cloud services (AWS, Azure, GCP) and containerization technologies (Docker, Kubernetes).
- Experience with data warehouses such as BigQuery, Databricks, or Snowflake.
- Experience implementing DBT transformations within medallion architecture frameworks.
Programming Skills:
- Strong programming skills in Python, Java, Scala, or similar languages.
Soft Skills:
- Strong problem-solving skills and attention to detail.
- Excellent communication and teamwork abilities.
- Ability to work in a fast-paced, dynamic environment.
Preferred Qualifications:
- Experience with machine learning and data mining techniques.
- Familiarity with data visualization tools such as Tableau or Power BI.
- Knowledge of data privacy regulations and best practices.