Proxmox + mikrotik / OPNsense help needed

Asim

I have been banging my head against the wall regarding proxmox with mikrotik (and later tried opnsense) setup on a VPS/VDS. It's pretty straight forward when I do it on a dedicated server (like Hetzner or OVH) but can't really make it to work on the VPS/VDS

Facts
* I have used two different providers for this, same issue
* Using mikrotik initially and then switched to opnsense as firewall. I know, I am not using both of them at the same time

Configuration
vmbr0 -> primary IP for the proxmox (works brilliant)
vmbr1 -> additional public IP for the firewall (mikrotik / opnsense)
vmbr2 -> LAN interface

Problem?
vmbr0 works brilliantly, maybe because it's BridgePort to ens18 (primary)
vmbr1 doesn't have internet even though the IP address, Subnet and Gateway are correct. Can't really BridgePort it to ens18 as well

Any pointers? please do share

RapToN

Why do you need vmbr1?

If I am right you want to install a Firewall in a VM, so use vmbr0 as bridge interface and set the Firewall IP inside opnsense/microtik.

havoc

@Asim said: Can't really BridgePort it to ens18 as well

Its gonna need a interface too.

You sure you don't have a 2nd one for the 2nd ip already? If not presumably one can create a virtual one somehow

Asim

@RapToN Using vmbr0 also doesn't help. The provider has assured me that he has done everything at his end

I have tried all sorts of configs, this is the latest one

iface ens18 inet manual
        dns-nameservers 8.8.8.8 8.8.4.4
        dns-search dmzng.com

auto vmbr0
iface vmbr0 inet static
        address <public_ip1>/24
        gateway <the gateway>
        bridge-ports ens18
        bridge-stp off
        bridge-fd 0
# WAN

auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
        post-down echo 0 > /proc/sys/net/ipv4/ip_forward
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
#LAN

So far I have tried pfSense, OPNsense and Mikrotik but even with the correct network details, the public_ip2 (second ip) doesn't activate

What am I missing?

nullnothere

Did you check if you need a special MAC address (quite common across providers) to route the extra public IPv4?

I've typically setup pfSense/OPNSense to be on the same public IP bridge (same bridge as that of the host node's internet interface) and forced the MAC address to the "virtual MAC" (or whatever it's called - you need to generate it to support virtualized hosts and to route the extra public IPv4 to that MAC address). After that it's quite straightforward and all within the pfSense/OPNSense config for WAN.

Another quick way to check is to run a container and have it use vmbr0 (in your case) as the bridge interface and then setup the correct GW, MAC and IP (within the container) and see if that works. Easier to debug/fix the container than going all the way through pfSense/whatever else.

EDIT: I'm assuming a typical dedicated server configuration which has only ONE public ethernet equivalent internet interface which will be having multiple (public) IPv4 assigned (though very likely the alternate/extra IPv4s will need special MAC settings to route correctly).

Asim

@nullnothere I suspected the same but It seems the provider has allowed access to my NIC. Still doesn't work

tjn

This is what I use for my Proxmox installations on Hetzner - it's been working really well!

pfSense WAN on vmbr0 and LAN on vmbr100. When setting up pfSense, I assign it one of the additional IPs.
The below assumes that your additional IPs are on the same subnet as your primary IP.

auto lo
iface lo inet loopback

auto enp2s0
iface enp2s0 inet static
        address xxx.xxx.xxx.xxx /32 # <PRIMARY IP>
        gateway xxx.xxx.xxx.xxx # <GATEWAY>
        pointopoint xxx.xxx.xxx.xxx # <GATEWAY>

auto vmbr0
iface vmbr0 inet static
        address xxx.xxx.xxx.xxx/32 # <PRIMARY IP>
        bridge-ports none
        bridge-stp off
        bridge-fd 0 
        post-up /bin/echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up /sbin/ip route add xxx.xxx.xxx.xxx/32 dev vmbr0 # <Additional IP>
        post-up /sbin/ip route add xxx.xxx.xxx.xxx/32 dev vmbr0 # <Additional IP>
        post-up /sbin/ip route add xxx.xxx.xxx.xxx/32 dev vmbr0 # <Additional IP>

auto vmbr100
iface vmbr100 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094