I’m thinking about moving my router to be a VM on a server in my homelab. Anyone have any experience to share about this? Any downsides I haven’t thought of?
Backstory: My current pfSense router box can’t keep up with my new fibre speeds because PPPOE is single threaded on FreeBSD, so as a test, I installed OpenWRT in a VM on a server I have and using VLANs, got it to act as a router for my network. I was able to validate it can keep up with the fibre speeds, so all good there. While shopping for a new routerboard, I was thinking about minimizing power and heat, and it made me realize that maybe I should just keep the router virtualized permanently. The physical server is already on a big UPS, so I could keep it running in a power outage.
I only have 1 gbps fibre and a single GbE port on the server, but I could buff the LAN ports if needed.
Any downsides to keeping your router as a VM over having dedicated hardware for it?
I did it for a few years, it looks interesting on paper, but in practice, it’s a nightmare.
At home, you’ll be getting real sick of asking for change windows to reboot your hypervisor.
At work, you will rue the day you convinced mgmt to let it happen, only to now have hypervisor weirdness to troubleshoot on top of chasing down bgp and TCP header issues. If it’s a dedicated router, you can at least narrow the scope of possible problems.
Gotta disagree, for home use at least. I have found it to be the opposite of a nightmare.
Moving my home routing and firewall to a VM saved me hours, and hours, and hours of time in the long run. I have a pretty complex home network and firewall setup with multiple public IPs, multiple outbound gateways, and multiple inbound and outbound VPN setups for various purposes. I’m also one of those loons that does outbound firewall with deny by default on my network, except the isolated guest VLAN. With a complex setup like that, being in a VM means it’s so easy to tweak stuff safely and roll back if you mess something up or it just doesn’t work the way you expected. Turns what would be a long outage rebuilding from scratch into a 30 second outage while you roll back the VM. And being able to snapshot your setup for backup is incredibly useful when your software doesn’t behave properly (looking at you, PFsense).
All that said, I run redundant, synced hypervisors which takes care of a lot of the risk. A person who is not well versed in hypervisor management might not be a good fit for this setup, but if you have any kind of experience with VM management (or want to), I think it’s the way to go.
For sure, if your thing is leaning into network configs, nothing wrong with it, especially if you have proper failover set up.
I think virtualized routing looks fun to the learning homelabber, and it is, but it does come with some caveats.
For home use, if used in an HA setup, the change window issue should disappear. Do you see any other issues that might crop up?
HA… Do you mean failover? It would need some consideration, either a second wan link or accepting that a few TCP sessions might reset after the cutover, even with state sync. But it’s definitely doable.
I’m currently in a state of ramping down my hardware from a 1u dual Xeon to a more appropriate solution on less power-hungry gear, so I’m not as interested in setting up failover if it means adding to my power consumption simply for the uptime. After 25 years in IT, its become clear to me that the solutions we put in place at work come with some downsides like power consumption, noise, complexity and cost that aren’t offset by any meaningful advantage.
All that said, i did run that setup for a few years and it does perform very well. The one advantage of having a router virtualized was being able to revert to a snapshot if an upgrade failed, which is a good case for virtualizing a router on its own.
Yea either failover or an active/active virtual switch… I’ve been toying with hyperconverged infrastructure and I wanted to bring my network infra into the fold, been looking at OVS. Not for any particular use case, just to learn how it works and I really like the concept of horizontally scaling out my entire infra just by plugging in another box of commodity hardware. Also been toying with a concept of automatically bootstrapping the whole thing.
OVS is fine, you can make live changes and something like spanning port traffic is a bit less hassle than using tc, but beyond that, it’s not really an important component to a failover scenario over any other vswitch, since it has no idea what a TCP stream is.