Principal Engineer – Distributed Systems (GPU Edge + Inference)

Elloe Ai

Apply to this job
Austin, Texas San Francisco remote Until 8/21/2026 First posted May 25, 2026 Last posted May 25, 2026
Job description
Full-time | Remote | Infrastructure | Reports to CTO

About Elloe
Elloe is the trust layer for AI.
We sit between the world’s most powerful language models and the institutions that can't afford to get it wrong — hospitals, banks, regulators. We trace and block failures in real time. That’s not marketing — we’re deployed at the European Commission, with NIH clinical trials, and inside a Top-5 EU bank catching GDPR violations live.

This is the enforcement layer GenAI has been missing. We're not visualizing problems — we're fixing them.

About the Role
You’ll lead our GPU-edge inference systems. From chaos-resilient deployment to SHAP-driven compliance metrics, you’ll own global infra that makes AI safe and performant.

What You’ll Own
1. Global Edge Routing
  •   Design zone-routing that ensures <50ms SLA in 10+ regions
  •   Build fallback orchestration to handle compliance-aware rollbacks
2. GPU Infra Ops
  •   Maximize utilization across 100K+ GPUs via mesh & load prediction
  •   Integrate compliance overlays with VaultChain and SHAP triggers
3. Reliability Telemetry
  •   Ship `/vault/audit`, `/inference/predict`, `/compliance/log` endpoints
  •   Trace every edge request across governance and model layers

Who You Are
  • Senior systems engineer with GPU fleet experience (KubeRay, Istio, Envoy)
  • Operated real-time AI infra with 10M+ QPS loads
  • Comfortable with compliance observability and infra governance

Why This Matters
Our competitive edge isn’t just AI — it’s defensible enforcement. This role turns that into product.

You’ll Leave This Role With
  • Referenceable contributions to enforcement infra that’s live in EU and US institutions
  • First-hand product work across legal, engineering, and GTM teams
  • Influence over how regulatory primitives become systems people trust

Logistics & Application
  • Start Date: Flexible (Q3–Q4 ideal)
  • Location: Remote-first; timezone overlap with NY or EU preferred
  • Compensation: Top of market salary + equity
  • To Apply: Send your resume and a sentence on the hardest infra problem you'd want to own at scale.
About this role

Summary

Lead GPU-edge inference systems, manage global infrastructure, ensure AI safety and performance

Job title

Principal Engineer – Distributed Systems (GPU Edge + Inference)

Experience level

senior level

Industry

software

Location requirements

Remote work allowed, with timezone overlap with NY or EU

Salary

Not specified

Management role

No

Skills & keywords

Required skills

GPUKubeRayIstioEnvoyreal-time AI infrastructurecompliance observability

Preferred skills

None specified

Specializations

distributed systemsGPU infrastructureedge routingcompliance observability
Locations

Structured locations inferred from the posting.

Texas, USA

Remote State

San Francisco, CA, USA

Remote City
Related searches