Proxmox NAT & CSF
AlwaysSkint
OG
I have the above setup, at long last nearing the point of using it, rather than idling. I can connect to a VM from the 'net via redirected ports and the VM is able to connect to the world.
When I ssh in to the VM, it is being logged as coming from the host server, not my home IP, so I'm obviously missing something in the setup. Any pointers for me?
Ta.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Comments
The NAT les servers just use simple SNAT per external IP e.g.
iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o br0 -j SNAT --to 104.105.106.107
Then port forwarding with DNAT e.g for ssh access
iptables -t nat -A PREROUTING -p tcp -d 104.105.106.107 --dport 321 -j DNAT --to 192.168.20.3:22
and then a port range through DNAT for the rest:
iptables -t nat -A PREROUTING -p tcp -d 104.105.106.107 --dport 301:320 -j DNAT --to-destination 192.168.20.3
But that assumes a few things, its been an age since I used proxmox properly but I remember it had a number of different options on how to pass traffic through when adding an interface type for the VM's the above would only work in a bridged network, I do seem to remember (going back maybe 7 - 8 years) that the default NAT setup in proxmox was to masquerade which may be the cause of your problem, I think a bond would do the same so you need bridged or a seperate VLAN for guest networks using a manually configured bridge would be my guess.
Could be wrong, just guessing without seeing.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
I'm using a bridged network on an OVH host. I have the following if that makes things clearer.
Part of my iptables-set.sh/csfpost.sh
In /etc/network/interfaces
There appears to be a duplication above but it's taken me ages to get packets traversing the host. Seems like I'm missing SNAT entries .. I'm sure I spotted them someplace. :-S
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I think the issue is the MASQUERADE, so it is well, masquerading if i were you I would strip it all back and follow the examples above, its actually a lot more simple than it seems to achieve what you want.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
I was following a mashup of 'instructions' on the web and got pretty confused along the way. I also have nginx reverse proxy and webserver VMs on this host, so I'll take my time over the reconfiguration. Thanks!
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I was about to write something useful, @AnthonySmith beat me to it.
SNAT is the way to go unless you either want to MASQUERADE the original IP or in some cases uses a dynamic IP.
https://clients.mrvm.net
Latest configuration:
/etc/network/interfaces (snippet)
I couldn't get outbound (eg. apt update) from the VM when the host had CSF active, until I placed this in csfpost.sh
I dunno the consequences of those iptables FORWARD directives, however.. will need to do a search.
EDIT: VM still shows host IP for my laptop doing an ssh to it. In other words the originating IP is being stripped during the forwarding.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I routed all traffic to the bare metal nginx instance before passing traffic to corresponding virtual network interfaces (created and assigned by Proxmox, ofc). This is dead simple, and I do not need to mess with iptables since I am also using block IPs based on ipset. My setup is for personal use, so no idea how it really effects the performance of Proxmox host under load.
What I do is simpler
Access to the NAT vm's is via a http(s) load balancer KVM(dedicated IPv4)
The individual KVMs serial terminals can be 'accessed' by ' qm terminal 101' in a tmux ssh/mosh session to the proxmox host.
My vm templates don't have openssh-server. I install it when required
A proxmox console is one way around it but doesn't address other services. I used ssh as an example because it is relatively simple to monitor activity to it. You also assume that an additional IP is available (I know I didn't cover that aspect) but that's not an option on Kimsufi servers, unfortunately. I have another proxmox instance elsewhere with /29, so that this subject is never considered, except for IPv6 trials.
TLDR; Originating IP is replaced by host IP at the VM
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Use IPv6
The all seeing eye sees everything...
Sorry to drag up this old thread again.. I'm back trying to get NAT working, though I am a bit further forward than previously.
On a different server, again with only one public IP, with Proxmox handling VMs; it's nginx that I've been tearing my hair out with. I'm trying to use it on the main server to redirect to the appropriate VMs.
I can access VMs from the outside world by using the server defined ports but I can't access using the VMs internal ports. There's also the issue of how to handle SSL certs.
Example:
http://tx200.vmdomain.com:80 <-- 502 Bad Gateway
http://tx200.vmdomain.com:42080 <-- Testing 123 (nginx default page)
https://tx200.vmdoamin.com:443 <-- PR_END_OF_FILE_ERROR
https://tx200.vmdomain.com:42443 <-- (Firefox:) Unable to connect
https://tx200.vmdomain.com:10000 <-- Error code: SSL_ERROR_RX_RECORD_TOO_LONG
https://tx200.vmdomain.com:42000 <-- webmin loads, with self-certified, correct FQDN SSL warning
This is what I've cobbled together for the nginx tx200.vmdomain.com.conf
I've tried without proxy_redirect but made no odds.
Obviously I have the IP addresses defined in the server hosts file and the firewall has a block of ports opened.
VMs can access the outside world and I can ssh in using the appropriate redirected port.
As well as the KVM above, I have a LXC that unsurprisingly displays similar symptoms.
Next thing I'm about to try is copy the self-cert SSL from webmin over to the default Apache location to see if it resolves the 443 issue.
I feel that I'm close but no cigar.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
HAProxy takes care of TLS/Plain inspection with SNI and forwards it to the right machine:
https://gist.github.com/Ne00n/b52e112afa5d587178920e0164691083
Maybe helpful.
https://github.com/Ne00n/NanoKVM-Tools
Free NAT KVM | Free NAT LXC
@Neoon Thanks! Will take a look.
Edit: I'm definitely missing something there.. The 'tools' appear to use DHCP which I specifically don't want use - will only have about 4 VMs max.
The HAProxy SNI TLS + HTTP looks promising, assuming I can extend to other services. Goes off to read about yet another prog.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
The HAProxy config is just copy & paste, also DHCP is just optional for every network, its not mandatory.
Same goes for radvd for IPv6.
Free NAT KVM | Free NAT LXC
Think I'm too stupid for this game. :-(
Same errors as above, apart from port 10000 which now has PR_END_OF_FILE_ERROR.
(that Testing 123 page is Apache on the VM, not nginx.)
Am I right in thinking there's no need to have a httpd service running now, as haproxy takes over?
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
HAProxy does forward HTTP/HTTPS to a specific IP:PORT, you need to have a webserver running at that destination.
If you have no webserver running, then you get errors.
Try plain, if plain works, setup LetsEncrypt, then https will work also.
You can setup https like on any normal webserver since HAProxy forward encrypted https traffic.
Use pastebin if you post any configs, I wont read that otherwise.
The formating is painful.
Free NAT KVM | Free NAT LXC
Yes, of course the VMs have a httpd running, I meant at the server.
Yep the code flag on the forum goes mental - I'll try a pastebin.
https://pastebin.com/gDhXQZ7W
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I see no error does plain work?
Free NAT KVM | Free NAT LXC
@Neoon
Thanks for bearing with me. Turnkey NextCloud/LAMP confused issues a bit due to redirections at the VM.
Current situation: port 80 requests work fine. 433 works only after I access webmin via the NAT 41000/42000 ports and request LE SSL certificates, plus add them manually to the webmin defined virtual hosts. A very convoluted way of doing things but at least I'm getting there.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Yes, SSL won't work until you setup valid certificates.
You just need a for example and nginx with LetsEncrypt and HAProxy forwards the requests fine.
I do not know your exact setup, you need to debug and check what's wrong. eg log files.
As far as I know it should work as expected.
Free NAT KVM | Free NAT LXC