Kubernetes on Oxide: How customer needs shaped our integrations

stevehipwell

I'm interested to see how the `oxide-cloud-controller-manager` is being built for "modern" Kubernetes and if it leads to any signify difference compared to CCMs that originated in-tree. Given the way Oxide engineer their solutions this could be really interesting. FYI I've got `karpenter-provider-oxide` on my bingo card...

overflowy

I would absolutely kill for them to open source their documentation system.

pianoben

I have seldom wanted anything as much as I want an Oxide rack at home. Maybe in 40 years we'll start to see the first ones show up in surplus auctions...

moondev

Love to see the CAPOx provider and buy in to Cluster API

whazor

Kubernetes is where Oxide, from my armchair, doesn't yet quite match the public cloud. On AWS EKS Fargate, each pod runs in its own dedicated VM. With Oxide each k8s node is a VM, so you still need something like Talos. On networking it looks like it is getting closer. Where you can have external subnet give pod routed IPs without overlay. But the gap, as marked by the article, is also the load balancing. It would be nice if Kubernetes were a native feature out-of-the-box. Also integrated within the existing user/access control.

wolttam

So you’d be attaching a new volume to the running worker VM for each PVC? This seems a little odd to me. Could you attach a single large volume, and do path-based provisioning on that?

bitlad

I am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal? At first look, it feels like oxide is equivalent to proxmox or some virtualization tool, may be it uses qemu stuff underneath. Just curious. The reason I am asking this is that, we have lot onprem scenarios in our business. We are tightly coupled with k8s, to solve this we started building an internal project that is kubernetes API compatible [1] but runs containerd or WASM or our platform natively. Just curious how oxides work in this scenario [1] https://github.com/debarshibasak/superkube

redwood

How are folks managing stateful workloads like databases on Oxide?

lars_francke

Disclaimer: I'm totally biased here. I talked with a colleague from Oxide in 2024 about your Kubernetes story and he said back then "not yet but soon-ish". Seems like soon-ish is now :) We said we'd talk again when that happens but he's since left Oxide. If you (or well...your customers) are interested in a Kubernetes native data platform 100% open source we'd be very happy to talk about how that could work easily. As it's "just" Kubernetes it should be trivial but we'd be happy to test and add you to our list: https://docs.stackable.tech/home/stable/kubernetes/#supporte... The offer stands. If you're interested, my mail is in my HN profile. https://stackable.tech

donatj

Bug report I guess, but the pages top navigation seems completely inoperable to me on my iPad. Clicking with my finger does not work. Clicking with my Apple Pencil does not work. Any sort of hover state that might be there does not work (with the pencil)

preisschild

I think it'd be cooler if you could just run kubernetes directly on their bare metal hardware without their hypervisor in between

ninkendo

I’ve tried so hard to figure out what the hell oxide even is, but the website is utterly impenetrable. I guess it’s an “integrated platform” and does “AI” and it’s “purpose built for frontier workloads”, but… I have absolutely no idea what it is , still. I guess it’s a bunch of computers. It says “AMD” in a bunch of places so, I guess they’re x86. Where are the GPU’s? If it’s for “frontier workloads”, wouldn’t those be sorta important? Is it its own OS? No idea. I see people adjacent to Oxide mention IllumOS sometimes[0], so does that mean it’s a solaris-like OS? Why would I want to run k8s (presumably with linux containers) on a not-linux OS? Or does it virtualize linux instances? Do actual CTO’s go for this type of marketing, with zero details and nothing but pure fluff about “solutions”? [0] This of course is from HN comments, I see no mention of any operating system whatsoever on the website, so if it is IllumOS, it’s not like you can easily find that info anywhere.

nunez

Great post. ClusterAPI never got the love that it should. I spent a good amount of time on it at VMware, as Tanzu heavily leverages it for cluster deployment (mostly CAP-A and CAP-V). It's basically kubeadm + the spirit of Terraform, Kubernetes controller edition. There's lots of great enterprise-ready options for centralized k8s cluster fleet management these days, but it works really well for folks that are all-in on GitOps and such.