Proxmox + mikrotik / OPNsense help needed

AsimAsim OGServices Provider
edited July 2022 in Help

I have been banging my head against the wall regarding proxmox with mikrotik (and later tried opnsense) setup on a VPS/VDS. It's pretty straight forward when I do it on a dedicated server (like Hetzner or OVH) but can't really make it to work on the VPS/VDS

Facts
* I have used two different providers for this, same issue
* Using mikrotik initially and then switched to opnsense as firewall. I know, I am not using both of them at the same time

Configuration
vmbr0 -> primary IP for the proxmox (works brilliant)
vmbr1 -> additional public IP for the firewall (mikrotik / opnsense)
vmbr2 -> LAN interface

Problem?
vmbr0 works brilliantly, maybe because it's BridgePort to ens18 (primary)
vmbr1 doesn't have internet even though the IP address, Subnet and Gateway are correct. Can't really BridgePort it to ens18 as well

Any pointers? please do share

Tagged:

Comments

  • RapToNRapToN Hosting ProviderOG

    Why do you need vmbr1?

    If I am right you want to install a Firewall in a VM, so use vmbr0 as bridge interface and set the Firewall IP inside opnsense/microtik.

    Thanked by (1)Asim
  • @Asim said: Can't really BridgePort it to ens18 as well

    Its gonna need a interface too.

    You sure you don't have a 2nd one for the 2nd ip already? If not presumably one can create a virtual one somehow

    Thanked by (1)Asim
  • AsimAsim OGServices Provider

    @RapToN Using vmbr0 also doesn't help. The provider has assured me that he has done everything at his end

    I have tried all sorts of configs, this is the latest one

    iface ens18 inet manual
            dns-nameservers 8.8.8.8 8.8.4.4
            dns-search dmzng.com
    
    auto vmbr0
    iface vmbr0 inet static
            address <public_ip1>/24
            gateway <the gateway>
            bridge-ports ens18
            bridge-stp off
            bridge-fd 0
    # WAN
    
    auto vmbr1
    iface vmbr1 inet static
            address 10.10.10.1/24
            bridge-ports none
            bridge-stp off
            bridge-fd 0
            post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
            post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
            post-down echo 0 > /proc/sys/net/ipv4/ip_forward
            post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
    #LAN
    

    So far I have tried pfSense, OPNsense and Mikrotik but even with the correct network details, the public_ip2 (second ip) doesn't activate

    What am I missing?

  • edited July 2022

    Did you check if you need a special MAC address (quite common across providers) to route the extra public IPv4?

    I've typically setup pfSense/OPNSense to be on the same public IP bridge (same bridge as that of the host node's internet interface) and forced the MAC address to the "virtual MAC" (or whatever it's called - you need to generate it to support virtualized hosts and to route the extra public IPv4 to that MAC address). After that it's quite straightforward and all within the pfSense/OPNSense config for WAN.

    Another quick way to check is to run a container and have it use vmbr0 (in your case) as the bridge interface and then setup the correct GW, MAC and IP (within the container) and see if that works. Easier to debug/fix the container than going all the way through pfSense/whatever else.

    EDIT: I'm assuming a typical dedicated server configuration which has only ONE public ethernet equivalent internet interface which will be having multiple (public) IPv4 assigned (though very likely the alternate/extra IPv4s will need special MAC settings to route correctly).

    Thanked by (2)RapToN Asim
  • AsimAsim OGServices Provider

    @nullnothere I suspected the same but It seems the provider has allowed access to my NIC. Still doesn't work :s

  • tjntjn
    edited August 2022

    This is what I use for my Proxmox installations on Hetzner - it's been working really well!

    pfSense WAN on vmbr0 and LAN on vmbr100. When setting up pfSense, I assign it one of the additional IPs.
    The below assumes that your additional IPs are on the same subnet as your primary IP.

    auto lo
    iface lo inet loopback
    
    auto enp2s0
    iface enp2s0 inet static
            address xxx.xxx.xxx.xxx /32 # <PRIMARY IP>
            gateway xxx.xxx.xxx.xxx # <GATEWAY>
            pointopoint xxx.xxx.xxx.xxx # <GATEWAY>
    
    auto vmbr0
    iface vmbr0 inet static
            address xxx.xxx.xxx.xxx/32 # <PRIMARY IP>
            bridge-ports none
            bridge-stp off
            bridge-fd 0 
            post-up /bin/echo 1 > /proc/sys/net/ipv4/ip_forward
            post-up /sbin/ip route add xxx.xxx.xxx.xxx/32 dev vmbr0 # <Additional IP>
            post-up /sbin/ip route add xxx.xxx.xxx.xxx/32 dev vmbr0 # <Additional IP>
            post-up /sbin/ip route add xxx.xxx.xxx.xxx/32 dev vmbr0 # <Additional IP>
    
    auto vmbr100
    iface vmbr100 inet manual
            bridge-ports none
            bridge-stp off
            bridge-fd 0
            bridge-vlan-aware yes
            bridge-vids 2-4094
    
    Thanked by (1)ehab
Sign In or Register to comment.