Infrastructure AI Engineer (SRE)

Trine Partners is hiring an Infrastructure AI Engineer for its Site Reliability Engineering team. The job is to design, build and tune the compute that AI and LLM workloads actually run on, across cloud and bare metal, and to keep training and inference environments fast, efficient and reliable at scale.

This is a remote position working United States Eastern Time, on a rotational shift pattern.

What you will do

AI compute and cluster infrastructure

• Architect and run compute platforms built for deep learning training, pipelines and real time inference.

• Manage GPU and TPU cluster allocation, multi node networking, and high throughput distributed storage.

• Get more out of the hardware through scheduling, using Kubernetes, Ray, Slurm or a custom orchestrator.

AIOps and system reliability

• Wire models and ML pipelines into the monitoring stack, with Prometheus, Datadog and OpenTelemetry, for anomaly detection and automated root cause analysis.

• Build self healing workflows and remediation agents that bring mean time to resolution down.

• Define and hold service level objectives and error budgets, for infrastructure uptime and for inference latency.

Performance, scaling and MLOps infrastructure

• Provision and maintain high performance storage, such as NVMe arrays and parallel file systems, and high bandwidth networking such as InfiniBand and RoCE.

• Work with the data science and MLOps teams on vector databases, model registries and prompt execution frameworks.

• Build CI/CD and MLOps pipelines for continuous model validation and zero downtime serving updates.

Cost and capacity

• Analyse and reduce GPU and CPU compute cost, memory efficiency and cloud spend across multi cloud or hybrid environments.

• Run capacity planning and load testing for heavy concurrent inference traffic.

What we are looking for

• 6 or more years in SRE, systems engineering, infrastructure engineering or cloud platform engineering, with a focus on AI and ML workloads.

• Hands on Kubernetes, container runtimes and GPU node provisioning.

• Infrastructure as code with Terraform, Pulumi, Ansible or CloudFormation.

• Python, Go or C++ for automation and tooling.

• Deep familiarity with OpenTelemetry, Prometheus, Grafana and modern log and trace platforms.

Nice to have

• Inference frameworks such as vLLM, TensorRT-LLM or Triton Inference Server.

• Distributed training and orchestration with DeepSpeed, Megatron-LM or Ray.

• Low level networking and GPU interconnects such as NVLink and InfiniBand, and memory techniques such as quantisation and FlashAttention.

• Vector databases such as Pinecone, Milvus or Qdrant, and LLM application frameworks.

Working pattern

• Fully remote.

• United States Eastern Time, rotational shifts.

• We are looking for people who can start immediately or within 15 to 20 days.

Compensation

Up to 20 LPA. There is room to discuss for the right person, so apply if the role fits and the number is close.