As a Data Engineer, you will be responsible for designing, building, and maintaining our data infrastructure, which includes streaming pipelines for applications and data lakes. You will collaborate with cross-functional teams, including business intelligence, analysts, product managers, and software engineers, to ensure the availability, reliability, and scalability of our data systems
Responsibilities:
- Leading the development and implementation of streaming and batch data pipelines to extract, transform, and load data from various sources into our data lakes/data warehouse.
- Implement self-service tools in the data platform (orchestration tools and data quality tools)
- Implement a data validation process to improve data quality in data pipelines
- Monitor the data platform cluster (Kubernetes)
- Implement a data observability process to monitor data pipelines
- Implement data lineage to tracking data movement
- Implement data dictionary, relationship and its interpretation
- Developing and maintaining data processing and analysis frameworks
- Ensuring data security, privacy, and compliance with relevant regulations and policies
- Collaborating with data analysts to understand data requirements and perform database design based on data understanding and requirements with consideration for performance
- Manipulating complex data from different sources (MySQL, PostgreSQL, MongoDB, Excel, CSV, and Google Sheets)
- Monitoring data quality and troubleshooting data-related issues
- Identifying and implementing process improvements to enhance the efficiency and effectiveness of our data systems
Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, or related field
- 3-5 years of experience in data engineering or related field
- Experience with streaming and batch data processing, ETL, and data warehousing
- Experience in Cloud-based data platforms (Azure, AWS, GCP, or others)
- Experience in Git, Docker, BigQuery, Airflow
- Experience with databases (e.g., MySQL, PostgreSQL, MongoDB)
- Strong programming skills (SQL, Python)
- Strong understanding of coding standards and testing
- Experience in managing projects and leading teams
- Have a growth mindset and a willingness to learn new things or share knowledge with others
- Ability to self-learn
- Understand Kubernetes is a plus
- Understanding the data lineage concept is a plus
- Understand data observability concept is a plus
- Understand data mesh concept is a plus
"To apply for this opportunity, please send your resume"
Email : hr@skilllane.com
More info : 061-419-0701 (Team People)