Migrating from Proxmox to NixOS and Incus

cassianoleal

I'm also considering migrating from Proxmox to Incus, but I'd look into IncusOS rather than having to manage the host OS myself.

EnigmaCurry

NixOS has transformed my use of Proxmox. I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. Proxmox is just a target, and I've abstracted things enough to where I can deploy the same machines to libvirt on a local machine too. Why would I need to let my agent into my PVE box? I haven't looked at incus, but if I wanted to run the full stack declaratively, nixos and LLMs are so powerful now that I would probably just say to run libvirt and ZFS on nixos natively.

kennywinker

> But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm. Uhh, whut? It provides a button-y interface, but you can do everything via config files and `pct` on the command line if you prefer. I know that’s not full nix-style declarative, but you don’t have to mislead to sell me on the advantages of declarative infra.

iotapi322

I've been using incus for a while now and actually run it on a side project in production for the better part of a year. Rock solid performance.

daishi55

This seems very cool and I will probably try it, but I think I’m missing something. I run Proxmox so that I can have multiple VMs running on my NUC. This doesn’t really solve that right? I cant spin up a windows 11 vm one weekend for a random experiment.

sbstp

Incus is great. I've been trying to revive an unmaintained ansible collection to manage incus resources https://github.com/sbstp/ansible-collection-incus

whalesalad

I haven't abandoned Proxmox yet, but the take here resonates with me. I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I have had pretty good luck managing Proxmox clusters with the Proxmox API ( https://pve.proxmox.com/pve-docs/api-viewer/index.html ) or just letting the agent shell in as root (lol). I built a very simple provisioning tool called vmfactory that takes some really somple config on disk, bakes a fresh qcow image, pushes it to proxmox and then configures networking and boots it. It's extremely rudimentary but has been working well for me. I did abandon TrueNAS, however. It really is a locked-down appliance. Good luck installing custom software on the base OS. I have a domain-joined Ubuntu/ZFS box that inherits a lot of policy from FreeIPA and/or Ansible config that is all backed by files on disk. It's been really easy to orchestrate what many would consider overkill in my homelab because literally everything is represented in a single Github repo. I yanked vmfactory out and into a standalone repo if anyone is interested: https://github.com/whalesalad/vmfactory

redlewel

I get the advantage of fully declarative systems such as Nix, but everything you do in proxmox via GUI can be done via CLI: https://pve.proxmox.com/pve-docs/#_command_line_interface CLI is first class in proxmox, I use the qm command for managing vms all the time. The networking is also just a file in `/etc/network/interfaces` that I modify with vim as needed.

sfRattan

Another proxmox-esque project I've been watching is Sylve , a control plane for FreeBSD that provides a web interface to jails, bhyve VMs, and containers [1]. It's new-ish, but it looks like a possible sweet spot replacement for both proxmox and TrueNAS (which was originally also built on FreeBSD before they switched to Linux IIRC), at least for my homelab-ing use case. Potentially eventually for environments at greater scale also. The company behind it is a software consultancy and Sylve is built with their actual business needs in mind, and is BSD-licensed like the OS it runs atop. One of the developers building Sylve gave a talk last year [2]. [1]: https://sylve.io/ [2]: https://youtu.be/wo4oD5UON30

linsomniac

It's really hard to tell what the benefits of Incus over Ganeti are. The Incus site doesn't really lead with features or benefits that I can see. About the only things I could figure out are that Incus can manage containers and can manage instance creation through cloning a base volume, where ganeti (afaik) always creates volume and populates it. Well, and Incus seems to be MUCH more popular than Ganeti. I searched the documentation but it wasn't really clear what its live migration and ZFS migration story is, but when I asked Claude to research it, it tells me that it supports live migration via ZFS snapshot replication, which is exactly what I'm looking for. I implemented a ganeti storage driver that does the same thing and am just getting ready to start testing it, but if Incus supports it I might look at moving that direction. Anyone use Incus live migration with ZFS?

HackerThemAll

The amount of b.s. in this article is staggering.

linsomniac

>The Agentic Multiplier Oh man, you buried the lede there. I switched over to NixOS around a month ago from Ubuntu and it's been just a dream. I expected there to be some friction with installing things that aren't already built for NixOS, but honestly it's been easier with LLM+NixOS than it was with Ubuntu. edit: Thinko

evanjrowley

I've recently migrated to Incus and am enjoying it so far. Hopefully the good feelings last. Roughly ten years ago, my homelab consisted of a dozen virtual machines running on SmartOS. I was not familiar with Illumos, and this was before it had a widely available web UI, but it was simple enough to use that these challenges didn't matter much. SmartOS was designed to boot reliably from USB flash storage, allowed me to use all my SATA ports for VM storage, and was my first "immutable" operating system. The primary focus on ZFS storage was another great quality of SmartOS. Two moves and several years later, it was time to rebuild the lab, and I decided to go with Proxmox because it had decent ZFS support. Experience with Proxmox has been very good too. The GUI, many more virtualization features (in addition to the key ones I care about), and better hardware support through the Linux kernel have kept me on Proxmox for a long time. Customizing my Proxmox installation always gave me anxiety. How could I defend my hypervisor from configuration drift? I wished there could be an immutable version of Proxmox. Later on, I learned about govulcheck, which offers a novel dynamic/static analysis hybrid approach to vulnerability management. Nothing else out there does this (without teaming up with some huge company). I began to think that I should favor software solutions based on golang. Ultimately, Incus (and IncusOS) fit this need very well. My IncusOS hosts excellent and I'm glad I can run Incus itself on most Linux distros - including NixOS! I'll keep a small Proxmox host around for experimenting with new kernel features (Intel GVT-g / SR-IOV graphics) and old operating systems like Windows XP or anything else that needs special QEMU options.

h4kunamata

>By moving to NixOS, my entire infrastructure is defined in text files. This means my AI agents can read, understand, and even safely modify my infrastructure You lost me there!! Firstly, NixOS is hype, like everything being moved to Rust and failing miserably. Secondly, "AI ... can .... safely modify my infrastructure", OP is either being a troll or haven't seem how the whole IT world is upside down because of those very same statements. Thirdly, "my entire infrastructure is defined in text files", you clearly never heard of Ansible. All my Proxmox LXC containers from DNS servers, to NGINX firewall aliases feeding OPNSense firewall rules, from Forgejo hosting my repos to PostgreSQL database, from Semaphore running my Ansible playbooks on schedule to *Arr collection, everything is fully infrastructure as code, there is no GUI. I do not log into Proxmox to deploy my stuff, I enjoy CLI and Ansible makes everything like a walk in the park. I use Proxmox CLI tool "pct" for everything, even snapshots are CLI via "vzdump" and its config file. My take from that post and comments resume in "hype" "not understanding processes" "seeing problems where there isn't one" I only run Linux here, even my 3D printer runs Debian Netinst Linux. I am missing something here.

arikrahman

Very happy to see more people Nixpilled. Haven't used incus but seems like an interesting tool for when I want to leave the comforts of my dotfiles.

apitman

I just started playing with Incus this week. So far really impressed. It's one of those tools that just feels well designed and high quality. I keep expecting the abstractions to leak and so far they haven't.

monksy

I've been using Incus to support coding agents to run wild. I've been pretty happy about it, but I wish I understood it more.

daft_pink

I find I have a lot of trouble with updating with nixos. I really like the OS but I find that when I have something like Claude Code and I need to update to the newest version, I just have trouble. It just seems like a lot of work updating versions otherwise NixOS is really awesome and great.

meitham

I am satisfied with pve and qm commands on proxmox, but I wish there was a nix answer to pfsense where my network and firewall are defined declaratively