Using part of the IPv6 /64 block to provide public ips to wireguard clients
Thanks to @MaxKVM for providing an awesome hosting service. I have a ticket with them that they and their upstream provider have not been able to resolve, and I would like to get a second opinion here.
I get a /64 block of IPv6 address of which 1 is allocated to the eth0 interface on my VPS. I then allocate a /112 block to Wireguard outside of the eth0 address, and statically assign IPv6 address from this block to wireguard clients.
MaxKVM does not do routed IPv6, but uses on-link IPv6, so I have to enable proxy_ndp on my VPS so that the eth0 interface would respond to neighbor solication (NS) messages with a neighbor advertisement (NA) for addresses in the /112 block.
sudo sysctl -w net.ipv6.conf.all.proxy_ndp = 1
sudo ip -6 neigh add proxy 2402:xxxx:xxxx:xxxx::200:4 dev eth0
When I try to ping an external IPv6 address on my wireguard client, the upstream router of the VPS would then ask who has the 2402:xxxx:xxxx:xxxx::200:4 address so that it knows where to route the response to. The issue though is that the upstream router is sending NS messages with a fe80::xxxx:xxxx:xxxx:fdc0 (IPv6 EUI-64 address) and expecting a reply back to that fe80 address. See tcpdump output below.
jon@max1 /etc: sudo tcpdump -i eth0 -v 'icmp6[icmp6type]=icmp6-neighborsolicit or icmp6[icmp6type]=icmp6-neighboradvert'
04:32:07.414482 IP6 (class 0xc0, hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::xxxx:xxxx:xxxx:fdc0 > ff02::1:ff00:4: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2402:xxxx:xxxx:xxxx::200:4
source link-address option (1), length 8 (1): xx:xx:xx:xx:fd:c0
04:32:07.482930 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::yyyy:yyyy:yyyy:2d51 > fe80::xxxx:xxxx:xxxx:fdc0: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2402:xxxx:xxxx:xxxx::200:4, Flags [solicited]
destination link-address option (2), length 8 (1): xx:xx:xx:xx:2d:51
04:32:07.550926 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::yyyy:yyyy:yyyy:2d51 > fe80::xxxx:xxxx:xxxx:fdc0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::xxxx:xxx:xxxx:fdc0
source link-address option (1), length 8 (1): xx:xx:xx:xx:2d:51
Since I enabled ndp proxying, my VPS tries to respond back to the router's fe80 address with a NA, but determines that it cannot, and sends a NS asking for how to route to that address. As a result, my wireguard client gets a host unreachable error because it gets no response.
However, if I ping the global IPv6 address that is the IPv6 gateway (which is also the router) from the wireguard client, I will see a NS coming from that global IPv6 address. And because it is the gateway, my VPS has no problems with responding with a NA and IPv6 starts working on my wireguard client.
04:39:34.124527 IP6 (class 0xc0, hlim 255, next-header ICMPv6 (58) payload length: 32) 2402:zzzz:zzzz::1 > ff02::1:ff00:4: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2402:xxxx:xxxx:xxxx::200:4
source link-address option (1), length 8 (1): xx:xx:xx:xx:fd:c0
04:39:34.718943 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) <My Public IPv6 address> > 2402:zzzz:zzzz::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2402:xxxx:xxxx:xxxx::200:4, Flags [solicited]
destination link-address option (2), length 8 (1): xx:xx:xx:xx:2d:51
Is it normal to block ICMPv6 access to the fe80 address of the upstream router?
For now though, I have switch to NATed IPv6 for my wireguard clients, but what a waste of the /64 block though.
Thanks
Jonathan
Comments
This is a really interesting idea
Any guides pls
A few points:
accept_ra=2on eth0I'm familiar with something like this in the Proxmox context (eth0 external, non-routed IPv6/64, internal vmbr1, IPv6's from the /64) and except that you're using WG there's no other difference and it works. There's a IPv6 setup for vmbr1 (and that is the gateway for the rest of the internal VMs sitting on vmbr1).
IPv4 and IPv6 forwarding is enabled in sysctl:
Setting accept_ra to 2 does not help. My VPS instance is not able to route to the upstream router's link local address IPV6 address (fe80::xxxx:xxxx:xxxx:fdc0), so it is unable to reply to the NS messages coming from that IPv6 address.
Hmmm... I'm thinking aloud here.
ip -6 route show?gateway <-> eth0 <-> wg0gateway <-> eth0 <-> vmbr1 <-> container/vmeverything very similar to what you have except that I have an IPv6 (say a /80) for vmbr1 which is the gateway for the container. I'm just wondering because (I assume) wg* is essentially a tun interface if there's something different to watch out for.2/3. Output from ip -6 route below. 2402:zzzz:zzzz::1 is the gateway provided by MaxKVM and is a /48 address. 2402:zzzz:zzzz:yyyy::/64 is the /64 block assigned to the VPS and 2402:zzzz:zzzz:yyyy::200:0/112 is the /112 block I have assigned to wg0.
gateway <-> eth0 <-> wg0would be the path that data going to and leaving the wg interface would take.First, I assume the wg interface does not have a MAC. That is likely going to be one of the reasons you are not having a fe80:: address for wg0 (key difference in my case with a vmbr* interface).
Can you also confirm that your wg0 interface's IP is reliably/consistently reachable from the outside? This is the one proxy entry you have added (and not of the clients). Are you having troubles reaching only your other clients on the wg* interfaces or even the primary wg* IPv6 that you've explicitly proxied on the VM?
If you wish to dynamically support various client IPv6s in your /112 subnet, you'll have to either run an ndp proxy on your VM or else statically add all desired IPs to via neigh proxy.
Do also dump (masked)
ip -6 neigh show proxy.That missing MAC is getting me thinking...
Maybe try disabling ebtables for your VPS once, power off & restart the VM.
I had a slightly similar case earlier.
https://webhorizon.net
Disabling ebtables did not help in this case.
Perhaps our resident networking Ph.D can help where we, SolusVM, and HIVELOCITY failed. @yoursunny do you have any ideas?
AMD EPYC powered Performance NVMe VPS - Los Angeles, Dallas, New York, Amsterdam, Singapore | Support | Status
I have this working on a VPS from skb-enterprise, but when I tried to recreate the same setup on tetahost it didn't work.
While tetahost was still up tcpdump showed that their side was constantly soliciting, getting advertise in response, then soliciting again as if it ignored it.
I would suggest validating your setup on a different network.
You need to have routed IPv6.
When we teach networking class, we need to send IPv4 traffic into a simulation system, and I just asked lab staff to setup static routes for a /16 subnet on the L3 switches.
Webhosting24 aff best VPS; ServerFactory aff best VDS; Cloudie best ASN; Huel aff best brotein.
Yes, the wg interface has no MAC address:
I do not proxy the wg0 interface's IP, so it will not be reachable from outside. wg clients can access it though, for dns resolution.
I mention in the original post that I am proxying ndp, testing with just 1 client IP now. The server and clients can ping each other though.
Sadly, not many hosts provide routed IPv6, and that is why I have to use proxy_ndp.
Aaha. OK. Suggestion. Proxy the wg0 IPv6 as well and setup the default route for the clients via that IPv6 and you should be all set.
If need be setup an explicit gateway via your eth0 IPv6 address on the wg side (remember that because there's no MAC, there no possibility of using the fe80 interface to discover routes on that part of the network and so you need to have the explicit route setup.
Do post your findings as this is definitely an interesting case (I have no issues with a virtual bridge which does have a MAC but otherwise everything is identical).
Edit Add: Also (from my earlier post): If you wish to dynamically support various client IPv6s in your /112 subnet, you'll have to either run an ndp proxy on your VM or else statically add all desired IPs to via neigh proxy.
On the client side, there is already a route created when wireguard starts the connection.
I will try proxying the wg0 IPv6 address too though - UPDATE: that does not help. Clients can already access the VPS wg0 interface through the tunnel. On the VPS side, tcpdump shows the same NS and NA loop for the interface address now, if i try pinging it from outside.
I am not exactly sure what you mean by
setup an explicit gateway via your eth0 IPv6 address on the wg side? Do you mean setting up an explicit route on the client? I tried doing something similar, but I still get the NS/NA loop on the VPS.Just to confirm the routing on the client side is working fine, I tried doing a traceroute on the wireguard client and it is using the expected IPv6 gateway. The same result occurs whether I use the wireguard created route, or add the explicit routing.
Interesting note is that after the traceroute, IPv6 will start working, because the 2nd hop pings the IPv6 gateway, generating a NS that my VPS can respond to.
edit: More info to the traceroute
The NS/NA loop that you mention looks very similar to the issue I have.
Anyway, this is the second host I am using NDP to try and get around the on-link IPv6. IPv6 completely stopped working on the first host after a few days of testing, so I cancelled it.
However, I have successfully tested without NDP on other services before:
1. Linode provides a /128 IPv6 address and /64 block (or larger) on request. They handle the routing of the block to the instance's /128 address so no NDP proxying is required.
2. AWS EC2 allows for multiple IPv6 addresses to be added to an instance. By disabling DHCPv6, I can use one of the IPv6 addresses as the public IP, and allocate the rest to wireguard clients without having to proxy ndp.
3. If I use tunnelbroker, HE will route the assigned /64 or /48 subnet to the client IPv6 address, so public IPv6 addresses can be allocated from the subnet to wireguard clients without having to proxy ndp. The IPv6 routing is not as good though and geolocation is broken too (if outside US).
@jnraptor
Hosters here mentioned on multiple ocasions ipv6 has low usage, so I think it's quite likely it simply doesn't get enough attention.
Not a guide but a working setup for reference (I hope I didn't forget anything):
I'm just going back to the basics now:
Some thoughts (and bear with me and any stupid ideas)
I hope this helps and irrespective I'm very curious on the root cause and assuming this is reasonably run-of-the-mill upstream, I'm beginning to suspect some wg trickery that is confounding things.
Yes, my suspicion is that this is the root case, hence my original question. I was not able to get an answer to this from MaxKVM or Hivelocity.
That works if I can ping fe80::1 gateway from eth0 and if upstream router is configured to listen to that. Unfortunately, that does not work.
I am in US, and the VPS is in Singapore.
I can try that over the weekend.
Reading online, it is likely because wireguard operates at layer 3 (IP tunnel), instead of layer 2 (ethernet mac tunnel).
However, as part of the starting fresh, I could try to bring up an OpenVPN tunnel and route a different /112 block.
Right and I'd say no. Lots of bad things happen when (some) ICMPv6 packets are disallowed.
Seems odd because if you have an upstream /48 gateway, I'd think that things are routed and of course you shouldn't need any fe80 magic.
Sure, that and of course there is no MAC address here... but I've seen other instances where IIRC the tun device also did get fe80::/64 - and I'm not very sure on the impact here and so just ignore it for now.
Please post findings. Very interested in this as well.
Got it.
Logically I'm in the clear here and can't figure out why this isn't working as expected. In any case, do keep an eye on the iptables and if possible check the LOG target and counters. I'm just suspicious that something is coming in the way and I don't yet have a clue.
If I think of something, I'll post my thoughts and tag you @jnraptor - definitely an interesting puzzle.
1) do not expect proxy_ndp to work;
2) https://github.com/DanielAdolfsson/ndppd may have a better chance of working, but it often won't either, because of this: https://github.com/DanielAdolfsson/ndppd/issues/55
Ask @Neoon, he had a ton of experience with this on NanoKVM.
The proper solution is to have (beg / demand / ask for) routed subnets. Anything else is basically a set of horrible hacks to work around the provider being incompetent and giving you unusable networking setup.
ADD35 may unlock routed IPv6.
Webhosting24 aff best VPS; ServerFactory aff best VDS; Cloudie best ASN; Huel aff best brotein.
Yes, SolusVM & Virtualizor ARE BEST FOR THIS!
RECOMMENDED 10/10
Besides, you need to ask the provider to disable security features to make a full /64 work in Virtualizor, if its routed.
And as @rm_ said you need at least one routed /64 anything else will be pain in the arse or close to impossible.
Also sometimes, you need to set the Router NDP entry on static, to make it work.
Because the provider has a broken setup.
Free NAT KVM | Free NAT LXC
Geolocation of US PoPs is also broken: Ashburn VA goes to Fremont CA, causing suboptimal CDN node selection in some cases.
However, this isn't HE's fault. Blame the location database provider.
Truth. It's also what I said earlier.
As a stallion coder, I don't do dirty hacks.
Webhosting24 aff best VPS; ServerFactory aff best VDS; Cloudie best ASN; Huel aff best brotein.
@MaxKVM noted that they disabled ebtables. Do you know what about security features are required to be disabled?
I tried doing a permanent neighbor entry for the upstream router's fe80 address.
My VPS no longer sends out the NS entry below asking for who has the router's fe80 address, but my VPS is still unable to communicate over ICMPv6 with the router's fe80 address, so tcpdump will just show a constant NS/NA loop.
Edit: Add first question
I was able to get public IPv6s from a /112 block to work on @Abdullah's WebHorizon though.
What is interesting is that with userland wireguard, a fe80 address is generated on the wireguard interface, most probably because it is using a tun interface behind the scenes:
I tried using the same boringtun userland setup on MaxKVM too, and though a fe80 address was generated, I still had the same problem where my VPS instance could not communicate with the upstream router's fe80 address, so it could not reply properly to NS messages.
Exactly.
Well, it does not always fix stuff, In a specific case, when you check the neighbours discovery entries and the entry of the router goes to FAIL, you can try it if it fixes the issue.
Free NAT KVM | Free NAT LXC
Thanks to @MaxKVM for being patient with me and following this thread too. They had me reboot the VPS after verifying the networking changes on their end and it is working now!
My VPS is able to ping the upstream router's fe80 address now and thus able to respond back with NAs, and IPv6 starts working on wireguard clients immediately. No config changes other than what has been specified above (ip forwarding in sysctl and iptables, proxy_ndp enabled and wireguard ipv6 addresses added to neigh proxy).