Design, build, and operate large-scale distributed data pipelines and platforms (Spark, Kafka, data warehouses/lakes). Optimize performance, establish architecture and tooling, and support ETL, deployment, monitoring, and scalability.
About Appier
Appier is a software-as-a-service (SaaS) company that uses artificial intelligence (AI) to power business decision-making. Founded in 2012 with a vision of democratizing AI, Appier’s mission is turning AI into ROI by making software intelligent. Appier now has 17 offices across APAC, Europe and U.S., and is listed on the Tokyo Stock Exchange (Ticker number: 4180). Visit www.appier.com for more information.
About the role
We're looking for an experienced data engineer to build and scale our data infrastructure. You'll work on mission-critical systems that power our data-driven decision making, designing robust pipelines and optimizing distributed platforms that process massive datasets.
Responsibilities
- Design, develop, and maintain production data pipelines processing hundreds of terabytes of data daily across distributed systems
- Build, manage, and optimize large-scale data platforms including Spark clusters, Kafka clusters, and related big data infrastructure
- Develop and operate data warehouse systems and ETL pipelines for data collection, processing, storage, and analytics
- Profile and tune performance of critical components to optimize throughput, reduce latency, and improve resource efficiency
- Establish architectural foundations and engineering best practices for the platform
- Design frameworks and tooling to streamline software development, deployment, monitoring, and observability
- Provide technical expertise and guidance on big data system design, performance optimization, and scalability challenges
About you
[Minimum qualifications]
- BS/MS degree in Computer Science or related technical field
- 2+ years of hands-on experience building and operating large-scale distributed systems
- Proficiency in Java and/or Scala for production-level development
- Proven experience building ETL pipelines and data processing workflows using Apache Spark
- Experience managing and optimizing data lakes or data warehouses at scale
- Ability to work independently and adapt quickly in fast-paced, dynamic environments
- Self-motivated learner who takes ownership, manages multiple priorities effectively, and thrives in collaborative team settings
- Experience deploying and managing infrastructure on public cloud platforms (AWS, GCP, or Azure)
[Preferred qualifications]
- Proficiency in Python and/or Golang for building production systems
- Experience profiling and optimizing JVM performance for high-throughput applications
- Hands-on experience managing and operating data platforms such as Hadoop, Kafka, Flink, Hive, Trino, Presto, or ClickHouse
- Knowledge of modern open table formats including Apache Iceberg, Delta Lake, or Hudi
- Active contributions to open source projects (please include your GitHub profile if applicable)
#LI-AK1
Similar Jobs
Artificial Intelligence
Design and operate high-performance data backend systems at scale, focusing on workflow optimization, reliability, and collaboration across teams.
Top Skills:
DatabricksDelta LakeGCPGoKafkaKubernetesPythonSparkSQLTerraform
Aerospace • Information Technology • Software • Cybersecurity • Design • Defense • Manufacturing
The Finance Business Partner will assist in financial management for government contracts, focusing on cost control, financial reporting, and proposal development.
Top Skills:
ExcelMS Office
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
The Technical Consultant will configure ServiceNow solutions, conduct customer workshops, advise on platform capabilities, and guide documentation efforts, ensuring customer satisfaction and operational efficiency.
Top Skills:
AIServicenow Platform
What you need to know about the Melbourne Tech Scene
Home to 650 biotech companies, 10 major research institutes and nine universities, Melbourne is among one of the top cities for biotech. In fact, some of the greatest medical advancements were conceptualized and developed here, including Symex Lab's "lab-on-a-chip" solution that monitors hormones to predict ovulation for conception, and Denteric's vaccine for periodontal gum disease. Yet, the thousands of people working in the city's healthtech sector are just getting started, to say nothing of the tech advancements across all other sectors.
