The Staff Software Engineer will design, develop, and maintain scalable data ingestion pipelines, integrating various data sources and ensuring data quality.
The Staff Software Engineer, Data Ingestion will be a critical individual contributor responsible for designing collection strategies, developing, and maintaining robust and scalable data pipelines. This role is at the heart of our data ecosystem, deliver new analytical software solution to access timely, accurate, and complete data for insights, products, and operational efficiency.
Key Responsibility
- Design, develop, and maintain high-performance, fault-tolerant data ingestion pipelines using Python.
- Integrate with diverse data sources (databases, APIs, streaming platforms, cloud storage, etc.).
- Implement data transformation and cleansing logic during ingestion to ensure data quality.
- Monitor and troubleshoot data ingestion pipelines, identifying and resolving issues promptly.
- Collaborate with database engineers to optimize data models for fast consumption.
- Evaluate and propose new technologies or frameworks to improve ingestion efficiency and reliability.
- Develop and implement self-healing mechanisms for data pipelines to ensure continuity.
- Define and uphold SLAs and SLOs for data freshness, completeness, and availability.
- Participate in on-call rotation as needed for critical data pipeline issues.
Required Skills
- 6+ years experience in software development industry from computer science background
- Extensive Python Expertise: Extensive experience in developing robust, production-grade applications with Python.
- Data Collection & Integration: Proven experience collecting data from various sources (REST APIs, OAuth, GraphQL, Kafka, S3, SFTP, etc.).
- Distributed Systems & Scalability: Strong understanding of distributed systems concepts, designing for scale, performance optimization, and fault tolerance.
- Cloud Platforms: Experience with major cloud providers (AWS or GCP) and their data-related services (e.g., S3, EC2, Lambda, SQS, Kafka, Cloud Storage, GKE).
- Database Fundamentals: Solid understanding of relational databases (SQL, schema design, indexing, query optimization). OLAP database experience is a plus (Hadoop)
- Monitoring & Alerting: Experience with monitoring tools (e.g., Prometheus, Grafana) and setting up effective alerts.
- Version Control: Proficiency with Git.
- Containerization (Plus): Experience with Docker and Kubernetes.
- Streaming Technologies (Plus): Experience with real-time data processing using Kafka, Flink, Spark Streaming
Top Skills
AWS
Docker
GCP
Grafana
GraphQL
Kafka
Kubernetes
Prometheus
Python
Rest Apis
S3
Sftp
SQL
Similar Jobs
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Develop and design applications on the ServiceNow platform, collaborate with business teams, and ensure high-quality deliverables. 5+ years of development experience is required.
Top Skills:
Angularjs 1.5Angularjs 2+CSSCSS3D3JsHighchartHTML5JavaScriptJqueryJSONReactRest ServicesServicenowXML
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Lead security discussions, conduct product security testing, audit source code, and identify vulnerabilities across ServiceNow products.
Top Skills:
JavaJavaScriptPHPPythonRuby
Agency • Digital Media • eCommerce • Professional Services • Software • Analytics • Consulting
Oversee accounting, billing, and invoicing processes, ensuring compliance with revenue recognition (ASC 606), leveraging OpenAir, and optimizing operations.
Top Skills:
Microsoft Office SuiteNetSuiteOpenair
What you need to know about the Melbourne Tech Scene
Home to 650 biotech companies, 10 major research institutes and nine universities, Melbourne is among one of the top cities for biotech. In fact, some of the greatest medical advancements were conceptualized and developed here, including Symex Lab's "lab-on-a-chip" solution that monitors hormones to predict ovulation for conception, and Denteric's vaccine for periodontal gum disease. Yet, the thousands of people working in the city's healthtech sector are just getting started, to say nothing of the tech advancements across all other sectors.
