Hi All!
New to the Fediverse from the reddit exodus, I’ve gotten into Self-hosting around January this year and have been loving learning about Networks and how they are structured and communicate and I love the projects that come out of managing a home lab.
As I’ve built up my home lab, from a single node to 3, I’ve been trying to think of how to structure my network to segment it in such a way that my homelab is on its own segmented network, whether this be VLAN or separate LAN (though I’ve heard terrible things about double NAT), and have that whole segment of traffic be pushed through a VPN tunnel. Unless that is not necessary? Part of the reason I want to make this post is so that people who are around the same point in their home lab adventure who might have similar questions can come to this thread to discuss particulars about the manner at hand. I’ll probably be structuring future questions in such a manner that allows people to discuss and nail down topics they may be struggling with wrapping their head around.
Back to my question, so let me give a better lay of the land. I am running Proxmox as my hypervisor on all these nodes, I have a generic ISP-provided router (ActionTek T3200 if interested in the model) that handles LAN routing and WiFi. Currently all my nodes are hard-lined to the router and I rely on proxmox default firewall atm, I haven’t dug into how to properly configure any of that since I wanted a separate solution, not sure the security implications of just using proxmox’s firewall so chime in if you know.
So all my nodes are hooked up to my router, but I have a Layer 2 switch I got for free (supports basic VLANs as well as some other basic features) and I want to configure my 3rd node to run OPNsense for my routing and VLAN tagging. The 3rd node will sit on the edge of the Router and the Switch, meaning Router connects to Ethernet port 1 on node 3 and Ethernet port 2 on node 3 connects to the switch and would be providing the LAN and internet access from my understanding. Node 3 will also be running a VPN tunnel to provide remote access as well as providing protection for my *Arr downloads. So the routing for my homelab should go from this:
Node 1, Node 2, Node 3 ------- ISP Router ------ Internet
to this:
Node 1, Node 2–Switch/Node3—ISP Router --Internet
Now my understanding is that structuring it in such a way means that if, for example, Node 1 which hosts my *Arr network were to pull a compromised download then its damage would only go as far as where the VLAN ends, ie would only affect my homelab network because its segmented in such a way that my devices connected to wifi would be unaffected.
I’ve just started to tinker around with configuring OPNsense when I got a sense for how to structure my network. I’m trying to virtualize it, which is a bit unconventional but not unheard of. I was able to spin up a VM running the installer and it gets through the install just fine but I am unable to reach the address provided. It’s 192.168.1.1 which is off my IP range. This might be where I need a bit of help understanding, but shouldn’t it give me an IP address that’s in my Routers IP Range? Maybe not, maybe its a sort of DMZ type thing? I’m not all too familiar so give a shout if you know something. I am thinking that the WAN and LAN ports are just getting switched during install and I need to interrupt the install and manually delegate those ports and then I’ll be able to connect? I haven’t had some solid time to dig deep on this so I figured rambling on a forum with tech savvy individuals might at the very least provide me with some insight and more understanding.
So I suppose my question is what do you think of my thought process? Am I missing anything major in my understanding? How should you think about configuring your firewall and VPN? How do you setup VLANs to allow communication where necessary between VLANs and Wifi network? Apologies that this got so long, I was trying to keep it brief but also give enough info on my environment. Let me know if there’s any questions. I’d also be interested in resources if my topics just point to needing a better understanding of networking generally. Thanks for your time
Dude this is awesome, thanks for your comment! It’s exactly the type of engagement I was hoping for, someone to help with a few key concepts. I’ll definitely be taking your advice to heart it sounds like you’ve been through the ringer with your own setup
I agree with you that running OPNsense in a VM is less than ideal but I figured as it would only affect my lab, I can deal with the occasional outage and this is more so I can learn first before going all in on a solution such as a netgate, I appreciate the suggestion there too. I’m not dead set on OPNsense I just figured a gui would be easier to navigate and it looked nice. I haven’t heard of suricata and mirroring WAN connection to pfsense, that definitely makes sense.
As for the VLANs I’m not dead set on that in particular either, the switch I have supported it and it made the most sense as to how to segment my network in that way. So you are saying you can also segment a LAN connection with just a firewall? It would make sense as you can set rules as to how a device can communicate with other devices on the network, is it rather cumbersome to set it up in that way? And on VLAN performance I would assume that would not be a problem as I am the only user who would be connecting to my media server, unless I had a bunch of services sending and receiving shitloads of data essentially?
I really appreciate all your help! You definitely sound like you’ve been doing this for a decade haha, very useful stuff. I might pop back and ask more questions later if you don’t mind
For sure, love talking about my home lab! And it’s an easy way to help get a bit of content on Beehaw, albeit a bit scatterbrained haha
First, the OSI model! Yeah, this is a very “first you must invent the universe” moment, sorry (not sorry). Basically for you the important bits are layer 2 and layer 3 (though I HIGHLY recommend knowing the OSI model forwards and backwards in its entirety).
Layer 2, the datalink layer, uses MAC addresses and is how to computers on a LAN communicate directly with one an other. Layer 3, the IP layer, uses (you guessed it) IP addresses and is what allows you to communicate between networks (it’s where we get the term internet: inter-network). Layer 2 is where switching takes place, layer 3 is where routing takes place.
So, if two computers are in a single LAN (or vLAN), they will communicate through a switch only. If two computers are not in the same LAN/vLAN, they can only have connectivity through a router. It’s really important to understand a switch only sees MAC addresses, so you can’t firewall at the switch level.
If some of this is new to you, I highly recommend reading networking 101 and 201 from this site. It seemed like a good resource.
Okay, with that in mind, next you really need to understand bridges. Any *nix box can be act as a switch and/or router. By default when you install Proxmox it creates a bridge on the machine’s default network interface. This bridge interface is effectively a switch. Every VM then “plugs” into this switch, giving it access to your network’s router, and fetches its IP via DHCP. You can use
ip linkandbrctl showto see it. Take a look at the related debian docs too.Setting up OPN in Proxmox
So with OPNSense, you have a few ways of setting it up, but you always need at least two LANs. One which is shared by OPNSense and your edge router (the ActionTek), and the other is shared by OPNSense and the VMs running in Proxmox. I’d recommend just using two vLANs as the best solution for now. One vLAN will be for your edge router, and the other will be for your OPN router.
First I’d setup your new switch so that every Proxmox host is on a trunk port. This allows tagged traffic from the host, meaning you can specify a vLAN for each VM network interface. Make sure you get this working first, here’s a decent guide, and make sure to read the proxmox network docs too. Now everything should be on the native vLAN (special vlan 1, containing untagged traffic).
Next just add a second interface to your OPNSense with a different vLAN (maybe 100), this’ll be your OPN vLAN. Now you can move VMs to your OPN network by changing their vLAN to the OPN network’s vLAN in Proxmox. Easy peasy!
Finally you’ll want to setup proper connectivity between your routers. If you were running a cooler edge router you could setup BGP, but for now static routes will suffice. Setup a static route for your OPN network in your edge router, and point it to your OPN VM’s IP address (the IP the edge router assigned it, 192.168.1.X). Finally create the opposing route in your OPN router for the edge router network. This way you don’t need a double NAT.
Okay, really sorry if none of that made sense. I really did try to keep it concise :S
I really appreciate all your input, its really helped me connect a few dots that needed connecting so I really do appreciate it my man. I’m aware of the OSI model and I was hoping my work with my homelab would help me understand better, but I’ve still got a ways to go. I really appreciate your explanations on Layer 2 and Layer 3, I understood the definitions but your explanation helps me connect the idea to real world examples. I also understood bridges somewhat but relating it to a switch makes it simple to understand and configure.
Thanks for the links as well, I will be messing around with this config tonight so I will be using your comments as a resource, so really, thank you! Can I ask for permission to PM you if I have any questions related to homelab/networking? You and I seem very similar in our scatterbrain-ness lol, so I figured it might make things easier if that makes sense. I will definitely be doing more reading and research, the first link you provided seems like an absolute blessing of a resource.
Oh on my first point I misunderstood, this will not replace my router, it will only be used by my proxmox network