Proxmox NAT & CSF

edited November 2019 in Help

I have the above setup, at long last nearing the point of using it, rather than idling. I can connect to a VM from the 'net via redirected ports and the VM is able to connect to the world.
When I ssh in to the VM, it is being logged as coming from the host server, not my home IP, so I'm obviously missing something in the setup. Any pointers for me?
Ta.

It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)

Tagged:

Comments

  • InceptionHostingInceptionHosting Hosting ProviderOG
    edited November 2019

    The NAT les servers just use simple SNAT per external IP e.g.
    iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o br0 -j SNAT --to 104.105.106.107
    Then port forwarding with DNAT e.g for ssh access
    iptables -t nat -A PREROUTING -p tcp -d 104.105.106.107 --dport 321 -j DNAT --to 192.168.20.3:22
    and then a port range through DNAT for the rest:
    iptables -t nat -A PREROUTING -p tcp -d 104.105.106.107 --dport 301:320 -j DNAT --to-destination 192.168.20.3

    But that assumes a few things, its been an age since I used proxmox properly but I remember it had a number of different options on how to pass traffic through when adding an interface type for the VM's the above would only work in a bridged network, I do seem to remember (going back maybe 7 - 8 years) that the default NAT setup in proxmox was to masquerade which may be the cause of your problem, I think a bond would do the same so you need bridged or a seperate VLAN for guest networks using a manually configured bridge would be my guess.

    Could be wrong, just guessing without seeing.

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • I'm using a bridged network on an OVH host. I have the following if that makes things clearer.
    Part of my iptables-set.sh/csfpost.sh

    • echo 1 >/proc/sys/net/ipv4/ip_forward
    • echo 1 > /proc/sys/net/ipv4/conf/vmbr1/proxy_arp
    • /sbin/iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80
    • ...
    • iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
    • /sbin/iptables -F FORWARD
    • /sbin/iptables -P FORWARD ACCEPT

    In /etc/network/interfaces

    • post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
    • post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80

    There appears to be a duplication above but it's taken me ages to get packets traversing the host. Seems like I'm missing SNAT entries .. I'm sure I spotted them someplace. :-S

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • InceptionHostingInceptionHosting Hosting ProviderOG

    I think the issue is the MASQUERADE, so it is well, masquerading :) if i were you I would strip it all back and follow the examples above, its actually a lot more simple than it seems to achieve what you want.

    Thanked by (2)AlwaysSkint mikho

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • I was following a mashup of 'instructions' on the web and got pretty confused along the way. I also have nginx reverse proxy and webserver VMs on this host, so I'll take my time over the reconfiguration. Thanks!

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • mikhomikho AdministratorHosting ProviderOG

    I was about to write something useful, @AnthonySmith beat me to it.
    SNAT is the way to go unless you either want to MASQUERADE the original IP or in some cases uses a dynamic IP.

    Thanked by (1)AlwaysSkint
  • edited November 2019

    Latest configuration:
    /etc/network/interfaces (snippet)

    • post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    • post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx
    • post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx
    • post-up iptables -t nat -A PREROUTING -d 178.32.xxx.xxx -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80
    • post-down iptables -t nat -D PREROUTING -d 178.32.xxx.xxx -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80

    I couldn't get outbound (eg. apt update) from the VM when the host had CSF active, until I placed this in csfpost.sh

    • #!/bin/bash
    • echo 1 >/proc/sys/net/ipv4/ip_forward
    • echo 1 > /proc/sys/net/ipv4/conf/vmbr1/proxy_arp
    • /sbin/iptables -F FORWARD
    • /sbin/iptables -P FORWARD ACCEPT
    • /sbin/iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx

    I dunno the consequences of those iptables FORWARD directives, however.. will need to do a search.

    EDIT: VM still shows host IP for my laptop doing an ssh to it. :'( In other words the originating IP is being stripped during the forwarding.

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • kmmmkmmm OG
    edited November 2019

    I routed all traffic to the bare metal nginx instance before passing traffic to corresponding virtual network interfaces (created and assigned by Proxmox, ofc). This is dead simple, and I do not need to mess with iptables since I am also using block IPs based on ipset. My setup is for personal use, so no idea how it really effects the performance of Proxmox host under load.

    Thanked by (1)AlwaysSkint
  • What I do is simpler
    Access to the NAT vm's is via a http(s) load balancer KVM(dedicated IPv4)

    The individual KVMs serial terminals can be 'accessed' by ' qm terminal 101' in a tmux ssh/mosh session to the proxmox host.

    My vm templates don't have openssh-server. I install it when required

    Thanked by (1)AlwaysSkint
  • edited November 2019

    A proxmox console is one way around it but doesn't address other services. I used ssh as an example because it is relatively simple to monitor activity to it. You also assume that an additional IP is available (I know I didn't cover that aspect) but that's not an option on Kimsufi servers, unfortunately. I have another proxmox instance elsewhere with /29, so that this subject is never considered, except for IPv6 trials.

    TLDR; Originating IP is replaced by host IP at the VM

    Thanked by (1)vimalware

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • Use IPv6 B)

    The all seeing eye sees everything...

  • edited March 2020

    Sorry to drag up this old thread again.. I'm back trying to get NAT working, though I am a bit further forward than previously.
    On a different server, again with only one public IP, with Proxmox handling VMs; it's nginx that I've been tearing my hair out with. I'm trying to use it on the main server to redirect to the appropriate VMs.
    I can access VMs from the outside world by using the server defined ports but I can't access using the VMs internal ports. There's also the issue of how to handle SSL certs.
    Example:
    http://tx200.vmdomain.com:80 <-- 502 Bad Gateway
    http://tx200.vmdomain.com:42080 <-- Testing 123 (nginx default page)
    https://tx200.vmdoamin.com:443 <-- PR_END_OF_FILE_ERROR
    https://tx200.vmdomain.com:42443 <-- (Firefox:) Unable to connect
    https://tx200.vmdomain.com:10000 <-- Error code: SSL_ERROR_RX_RECORD_TOO_LONG
    https://tx200.vmdomain.com:42000 <-- webmin loads, with self-certified, correct FQDN SSL warning

    This is what I've cobbled together for the nginx tx200.vmdomain.com.conf

    I've tried without proxy_redirect but made no odds.
    Obviously I have the IP addresses defined in the server hosts file and the firewall has a block of ports opened.
    VMs can access the outside world and I can ssh in using the appropriate redirected port.
    As well as the KVM above, I have a LXC that unsurprisingly displays similar symptoms.
    Next thing I'm about to try is copy the self-cert SSL from webmin over to the default Apache location to see if it resolves the 443 issue.
    I feel that I'm close but no cigar. :'(

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • NeoonNeoon OG
    edited March 2020

    HAProxy takes care of TLS/Plain inspection with SNI and forwards it to the right machine:
    https://gist.github.com/Ne00n/b52e112afa5d587178920e0164691083
    Maybe helpful.
    https://github.com/Ne00n/NanoKVM-Tools

    Thanked by (1)Ganonk
  • edited March 2020

    @Neoon Thanks! Will take a look.

    Edit: I'm definitely missing something there.. The 'tools' appear to use DHCP which I specifically don't want use - will only have about 4 VMs max.
    The HAProxy SNI TLS + HTTP looks promising, assuming I can extend to other services. Goes off to read about yet another prog. ;)

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • @AlwaysSkint said:
    @Neoon Thanks! Will take a look.

    Edit: I'm definitely missing something there.. The 'tools' appear to use DHCP which I specifically don't want use - will only have about 4 VMs max.
    The HAProxy SNI TLS + HTTP looks promising, assuming I can extend to other services. Goes off to read about yet another prog. ;)

    The HAProxy config is just copy & paste, also DHCP is just optional for every network, its not mandatory.
    Same goes for radvd for IPv6.

    Thanked by (1)AlwaysSkint
  • Think I'm too stupid for this game. :-(
    Same errors as above, apart from port 10000 which now has PR_END_OF_FILE_ERROR.
    (that Testing 123 page is Apache on the VM, not nginx.)

    • frontend http
    • bind *:80
    • mode http
    • #Define hosts
    • acl host_tx100.domain.com hdr(host) -i tx100.domain.com
    • use_backend tx100.domain.com if host_tx100.domain.com
    • acl host_tx200.domain.com hdr(host) -i tx200.domain.com
    • use_backend tx200.domain.com if host_tx200.domain.com
    • backend tx100.domain.com
    • server tx100.domain.com 10.0.0.100:80
    • mode http
    • backend tx200.domain.com
    • server tx200.domain.com 10.0.0.120:80
    • mode http
    • frontend https
    • bind *:443
    • mode tcp
    • acl tls req.ssl_hello_type 1
    • tcp-request inspect-delay 5s
    • tcp-request content accept if tls
    • #Define hosts
    • acl host_tx100.domain.com_https req.ssl_sni -i tx100.domain.com
    • use_backend tx100.domain.com_https if host_tx100.domain.com_https
    • acl host_tx200.domain.com_https req.ssl_sni -i tx200.domain.com
    • use_backend tx200.domain.com_https if host_tx200.domain.com_https
    • backend tx100.domain.com_https
    • server tx100.domain.com 10.0.0.100:443
    • mode tcp
    • backend tx200.domain.com_https
    • server tx200.domain.com 10.0.0.120:443
    • mode tcp
    • frontend webmin
    • bind *:10000
    • mode tcp
    • acl tls req.ssl_hello_type 1
    • tcp-request inspect-delay 5s
    • tcp-request content accept if tls
    • #Define hosts
    • acl host_tx100.domain.com_webmin req.ssl_sni -i tx100.domain.com
    • use_backend tx100.domain.com_webmin if host_tx100.domain.com_webmin
    • acl host_tx200.domain.com_webmin req.ssl_sni -i tx200.domain.com
    • use_backend tx200.domain.com_webmin if host_tx200.domain.com_webmin
    • backend tx100.domain.com_webmin
    • server tx200.domain.com 10.0.0.100:12321
    • mode tcp
    • backend tx200.domain.com_webmin
    • server tx200.domain.com 10.0.0.120:10000
    • mode tcp

    Am I right in thinking there's no need to have a httpd service running now, as haproxy takes over?

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • NeoonNeoon OG
    edited March 2020

    HAProxy does forward HTTP/HTTPS to a specific IP:PORT, you need to have a webserver running at that destination.
    If you have no webserver running, then you get errors.

    Try plain, if plain works, setup LetsEncrypt, then https will work also.
    You can setup https like on any normal webserver since HAProxy forward encrypted https traffic.

    Use pastebin if you post any configs, I wont read that otherwise.
    The formating is painful.

    Thanked by (1)AlwaysSkint
  • edited March 2020

    Yes, of course the VMs have a httpd running, I meant at the server. ;)
    Yep the code flag on the forum goes mental - I'll try a pastebin.
    https://pastebin.com/gDhXQZ7W

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • I see no error does plain work?

  • @Neoon
    Thanks for bearing with me. Turnkey NextCloud/LAMP confused issues a bit due to redirections at the VM.
    Current situation: port 80 requests work fine. 433 works only after I access webmin via the NAT 41000/42000 ports and request LE SSL certificates, plus add them manually to the webmin defined virtual hosts. A very convoluted way of doing things but at least I'm getting there. ;)

    It wisnae me! A big boy done it and ran away.
    NVMe2G for life! until death (the end is nigh)

  • NeoonNeoon OG
    edited March 2020

    Yes, SSL won't work until you setup valid certificates.
    You just need a for example and nginx with LetsEncrypt and HAProxy forwards the requests fine.

    I do not know your exact setup, you need to debug and check what's wrong. eg log files.
    As far as I know it should work as expected.

    Thanked by (1)AlwaysSkint
Sign In or Register to comment.