Andromeda (andromeda.ai) Logo

Andromeda (andromeda.ai)

Customer Reliability Engineer

Reposted 17 Days Ago
In-Office or Remote
Hiring Remotely in Australia
Senior level
In-Office or Remote
Hiring Remotely in Australia
Senior level
The Site Reliability Engineer will provision and manage Kubernetes clusters, build automation tools, debug customer issues, and improve infrastructure reliability.
The summary above was generated by AI

Customer Reliability EngineerLocation: Remote/SF-Hybrid · Full-Time

About Andromeda

Andromeda is a market and infrastructure platform to buy, sell, and operate compute.
We believe demand for compute will grow exponentially. So fast that a handful of vertically integrated providers won't be able to scale across operations, capital, supply chains, and politics to serve it. The result is a massive wave of fragmentation, with AI factories of every shape and size coming to market to fill this demand. Our job is to enable all of that fragmented compute to flow through one platform, delivering reliable capacity to model builders, research labs, and inference providers when they need it. We believe every spare electron should be made productive for AI and we're building the platform that makes that possible.


We sit at the center of three forces:

  • Companies that need reliable, high-performance compute fast

  • A fragmented global supply of GPUs across hyperscalers, neoclouds, and independent data centers

  • Capital, risk, and operational complexity that most teams are not equipped to manage

When we succeed, trillions of dollars of compute will flow through Andromeda. Builders get capacity when they need it. Providers get a reliable way to monetize, operate, and finance infrastructure at scale. Capital gets an easy way to deploy, hedge, and underwrite.
In five years, Andromeda won't just participate in the AI infrastructure market. We will shape it.

The Role

Our customers run large AI training and inference workloads on GPU clusters we source from providers worldwide. When a node goes dark or a job dies eight hours into a run, the Customer Reliability Engineer is who they hear from, and who gets it sorted.

The job has three parts. You triage incoming issues and debug them at the Linux and Kubernetes layer. You work provider-side to figure out whose fault something actually is and push external providers to fix it. And you build the monitoring and scripts that catch problems before a customer has to tell us.

You need to be comfortable in a Linux shell and know how Kubernetes works. You don't need GPU or HPC experience. Most people pick that up here.

What You’ll Do

Triage and fix customer issues

  • Own issues start to finish: reproduce, diagnose, fix or escalate, close the loop

  • Debug at the Linux layer: processes, networking, storage, kernel logs, resource contention, systemd, journald

  • Dig into Kubernetes problems like pods stuck pending or crash-looping, node conditions, scheduling failures, resource limits

  • Work GPU failures: driver and device-plugin issues, XID errors, thermal throttling, nodes that need cordoning or draining, jobs failing across multiple nodes

  • Escalate when you're past your depth, with the evidence already gathered

Handle incidents

  • Take part in a 24/7 on-call rotation

  • First response on alerts and customer-reported outages: assess impact, set severity, pull in the right people

  • Keep customers updated during incidents. Clear status, honest unknowns, no silence

  • Write up what happened, then turn it into a runbook, an alert, or a fix so it costs less next time

Push providers to resolution

  • Work out whether a fault is provider-side, ours, or the customer's before it gets handed anywhere

  • Open tickets with compute providers and chase them down rather than waiting

  • Track recurring provider failures and flag the patterns to the people making sourcing decisions

Build the tooling

  • Write Python or Bash to automate the checks you'd otherwise run by hand

  • Build and improve monitoring: cluster and node health checks, GPU telemetry, dashboards, alerts that fire on real problems

  • Keep runbooks and customer docs current as you go

What We’re Looking For

  • Real Linux troubleshooting ability from the command line. You can work a problem through logs, processes, networking, and disk without a script to follow

  • Working knowledge of Kubernetes: pods, nodes, deployments, services, scheduling, and how to investigate when one of those breaks

  • Can write a script in Python or Bash to automate something repetitive

  • Strong writing. You can explain a technical problem to a frustrated customer clearly and without condescension

  • Good judgment under pressure. You know what to check first, when to escalate, and how to keep people informed while you're still working it out

  • Willing to join a 24/7 on-call rotation

Strong Candidates May Have

  • Hands-on time with NVIDIA GPUs in production: drivers, CUDA, DCGM, the Kubernetes device plugin

  • Experience with high-performance networking (InfiniBand, RoCE) or NCCL

  • Experience with HPC or batch schedulers like Slurm

  • A previous customer-facing technical role: support engineering, TAM, solutions, professional services

  • Knowledge of Prometheus, Grafana, Datadog, or similar

  • IAC: Terraform, Ansible, or Helm

  • Genuine interest in AI infrastructure and how big training jobs behave

Why You’ll Love It Here

  • High-growth environment: Get in early at a company at the center of the AI infrastructure boom

  • Competitive compensation: + meaningful equity

  • Comprehensive benefits: for you and your dependents, including healthcare, dental, and vision coverage, 401(k), and unlimited PTO

Andromeda Cluster is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

Similar Jobs

28 Days Ago
Remote
Senior level
Senior level
Software
Lead and resolve high-severity post-sales technical escalations through deep-dive investigation, reproduction, and engineering handoff. Own customer calls, collect logs/evidence, file bug reports, monitor patterns, and improve tooling and runbooks in partnership with Support and Engineering.
Top Skills: Ci/CdCloud IaasContainerized ApplicationsDnsFirewallsGoJSONKubernetesLinuxLoad BalancersmacOSNatPacket-Level AnalysisPprofPythonRoutingStunTcp/IpUdp Hole PunchingVpnWindowsWireguard
Yesterday
Easy Apply
Remote
Easy Apply
Senior level
Senior level
Cloud • Security • Software • Cybersecurity • Automation
Conduct advanced application security research across GitLab’s DevSecOps and AI-powered platforms. Identify and exploit novel, systemic, and chained vulnerabilities; develop proof-of-concept exploits, testing methodologies, and research automation; assess AI attack vectors and open-source dependencies; drive remediation and security improvements; advise engineering teams, contribute to roadmaps, mentor researchers, and share findings with the security community.
Top Skills: Duo Agent PlatformGitlabGitlab DuoGoPythonRubyRustTypescript
Yesterday
Easy Apply
Remote
Easy Apply
Expert/Leader
Expert/Leader
Cloud • Security • Software • Cybersecurity • Automation
Conducts advanced application security research across GitLab’s DevSecOps and AI-powered platforms. Responsibilities include discovering and exploiting systemic vulnerabilities, penetration testing, developing proof-of-concept exploits, researching AI and agentic security threats, building automated research tooling, coordinating remediation, mentoring security professionals, and communicating findings to engineering and security communities.
Top Skills: Ai FrameworksDevsecopsDuo Agent PlatformGitlabGitlab Duo ChatGoPythonRubyRustTypescript

What you need to know about the Melbourne Tech Scene

Home to 650 biotech companies, 10 major research institutes and nine universities, Melbourne is among one of the top cities for biotech. In fact, some of the greatest medical advancements were conceptualized and developed here, including Symex Lab's "lab-on-a-chip" solution that monitors hormones to predict ovulation for conception, and Denteric's vaccine for periodontal gum disease. Yet, the thousands of people working in the city's healthtech sector are just getting started, to say nothing of the tech advancements across all other sectors.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account