Then I remembered this 2020 Proxmox forum thread where Wolfgang from the Proxmox staff explains that "network configuration is out of scope" so that even buying a Proxmox subscription "would help nothing" for a network configuration question.
Is possible that dude who have updated it found solution (add hwaddress param into config file)
When i will have any time i will try/test it then i'll be back with result
@AlwaysSkint said: .. and adding hwaddress to the respective entry in /etc/network/interfaces.
Truth.
I assume seems be solution. And apologies for my blindness.
interesting still, that this was not needed before Proxmox 7 and maybe related to sticking to ifupdown even with the newer kernel? also according to your findings simply restarting the networking later in the process seemed to work even without providing hwaddress in that config, so...
Can confirm that adding the hwaddress definitely fixed this issue. Just did a bunch of new installs as well as test migrations from pve 6.4 to 7 to confirm. No idea why that didn't work for me the first time!
1) use 'ip -c link' to determine the mac address of the physical interface (eg. ens3)
2) edit /etc/network/interfaces and specify this mac (using hwaddress ) on the bridge (eg. vmbr0)
Installing 'ifupdown2' on an existing pve 6.4 install also worked without having to muck around with the hwaddress
@bdl said: Installing 'ifupdown2' on an existing pve 6.4 install also worked without having to muck around with the hwaddress
only so I undestand this right (as it will be very helpful): so you had the same problems with networking after upgrading from 6.4 to pve7 but just installing this package before the migration fixed it as well, without having to touch any configs?
that'd be the preferred upgrade path then I'd say.
can you recall if your pve6 has been initially installed from iso or on top of debian?
@bdl said: Installing 'ifupdown2' on an existing pve 6.4 install also worked without having to muck around with the hwaddress
only so I undestand this right (as it will be very helpful): so you had the same problems with networking after upgrading from 6.4 to pve7 but just installing this package before the migration fixed it as well, without having to touch any configs?
that'd be the preferred upgrade path then I'd say.
can you recall if your pve6 has been initially installed from iso or on top of debian?
When 7 was first released, tried two 6.4->7 upgrades on boxes from MaxKVM. All single IP machines, installed with PVE 6 .iso, with a bridged nic to the outside world. From memory I think I'd installed ifupdown2 on these, but can't remember for sure. Had no problems whatsoever so thought I'd try it on one of my Chicago Hosthatch box. Chicago Hosthatch box had also been installed from a Pve 6.x iso, so I did the 6 to 7 Proxmox upgrade test like I'd done for the MaxKVM boxes, it passed, so I went ahead.
After the upgraded completed successfully, I lost network access on reboot. From memory, I tried renaming nics and re-establishing the bridges, nothing worked. After reinstalling 6.4 from iso and having networking working, and then also repeating the process a second time with the same result, I just wrote it off as PVE 7 teething problems and was going to wait for it to be patched out. The Proxmox documentation describing the 'macaddress' option wasn't as detailed as it is now and (I think) I did gloss over it but it didn't seem applicable. I had private networking set up on that box too, so wasn't sure if that was causing more problems.
I purchased a new Hosthatch service (in a different location) which was provisioned today, and I was going to install PVE 6.x on it given the issues I'd had with 7 previously (again, waiting for 7 to be updated and migrating at a later date), but thought I'd do more testing as some of the comments in this thread seemed to be inconclusive. Tried an install from .iso for 7, no networking. Reinstalled with 6.x .iso, networking worked perfectly.
I then installed ifupdown2 on the 6.x .iso install, ran the upgrade script and it upgraded no problem.
Then, I thought, "well, that works but I can't be stuffed installing 6.x and then upgrading for future installs", so tried a 7 .iso installation. Again, no networking post upgrade. I checked the mac's with 'ip -c link' and noticed that my bridge had a different macaddress to the physical interface. I then edited /etc/network/interfaces and defined 'hwaddress' for the bridge, then reloaded the network using 'ifreload -a' and could verify I had Internet connectivity. Tested and it survived a reboot.
Was curious to see ifupdown2 had been installed by default on 7.x and it had, I definitely had to add the macaddress for my bridge to come up.
I'm going to do a fresh install on the Hosthatch Chicago box I referred to above. Going to install 6.4 from iso, update, install ifupdown2 and then do the inplace upgrade to 7. Will report back if it works OK.
Apologies for the waffle, hope it helps. Let me know if you have any other questions.
@bdl said: Installing 'ifupdown2' on an existing pve 6.4 install also worked without having to muck around with the hwaddress
only so I undestand this right (as it will be very helpful): so you had the same problems with networking after upgrading from 6.4 to pve7 but just installing this package before the migration fixed it as well, without having to touch any configs?
that'd be the preferred upgrade path then I'd say.
can you recall if your pve6 has been initially installed from iso or on top of debian?
When 7 was first released, tried two 6.4->7 upgrades on boxes from MaxKVM. All single IP machines, installed with PVE 6 .iso, with a bridged nic to the outside world. From memory I think I'd installed ifupdown2 on these, but can't remember for sure. Had no problems whatsoever so thought I'd try it on one of my Chicago Hosthatch box. Chicago Hosthatch box had also been installed from a Pve 6.x iso, so I did the 6 to 7 Proxmox upgrade test like I'd done for the MaxKVM boxes, it passed, so I went ahead.
After the upgraded completed successfully, I lost network access on reboot. From memory, I tried renaming nics and re-establishing the bridges, nothing worked. After reinstalling 6.4 from iso and having networking working, and then also repeating the process a second time with the same result, I just wrote it off as PVE 7 teething problems and was going to wait for it to be patched out. The Proxmox documentation describing the 'macaddress' option wasn't as detailed as it is now and (I think) I did gloss over it but it didn't seem applicable. I had private networking set up on that box too, so wasn't sure if that was causing more problems.
I purchased a new Hosthatch service (in a different location) which was provisioned today, and I was going to install PVE 6.x on it given the issues I'd had with 7 previously (again, waiting for 7 to be updated and migrating at a later date), but thought I'd do more testing as some of the comments in this thread seemed to be inconclusive. Tried an install from .iso for 7, no networking. Reinstalled with 6.x .iso, networking worked perfectly.
I then installed ifupdown2 on the 6.x .iso install, ran the upgrade script and it upgraded no problem.
Then, I thought, "well, that works but I can't be stuffed installing 6.x and then upgrading for future installs", so tried a 7 .iso installation. Again, no networking post upgrade. I checked the mac's with 'ip -c link' and noticed that my bridge had a different macaddress to the physical interface. I then edited /etc/network/interfaces and defined 'macaddress' for the bridge, then reloaded the network using 'ifreload -a' and could verify I had Internet connectivity. Tested and it survived a reboot.
Was curious to see ifupdown2 had been installed by default on 7.x and it had, I definitely had to add the macaddress for my bridge to come up.
I'm going to do a fresh install on the Hosthatch Chicago box I referred to above. Going to install 6.4 from iso, update, install ifupdown2 and then do the inplace upgrade to 7. Will report back if it works OK.
Apologies for the waffle, hope it helps. Let me know if you have any other questions.
Argh! Don't know why i typed "macaddress" instead of "hwaddress" above..
Anyway, did some more testing with fresh installs of PVE 6.4 from iso.
First install, installed 6.4, apt-upgraded, ran the update script - then lost connectivity after I rebooted into PVE 7. Running 'ip -c link' verified that the bridge's MAC was different to that of the physical interface it was meant to bridge. I edited /etc/network/interfaces and added 'hwaddress' for the bridge and set the MAC to be the same as the physical interface. Then ran '/etc/init.d/networking' to restart networking and the network came back.
Second install, installed 6.4, apt-upgraded, installed 'ifupdown2', then ran the update script - didn't lose connectivity after rebooting into PVE 7. Verified that MAC addresses for bridge and physical interfaces were the same using 'ip -c link'.
Final test was a clean install with PVE 7 .iso. No networking after install and reboot. Confirmed bridge and and physical interface MAC addresses were different using 'ip -c link'. Modified /etc/network/interfaces accordingly, and after a 'service networking restart' networking was good to go!
@bdl thanks a lot for all the information and tests, that helps a lot. could well be that with (old) ifupdown the bridge does not get a new/seperate hwaddress and that this is only introduced with the new version. that said, if one updates to ifupdown2 via apt it might run some post-inst script that detects and pins the hwaddress, while the pve udpate script somehow seems to miss that...
I am still waiting with the updating as for now I don't see the need... but at some point obviously need to do that and will make sure to switch to ifupdown2 beforehand then. though, my installation are all not from ISO but on top of debian.
I thought I'd just add my random $0.02 - one of the funky things with (I think) newer kernels and/or systemd (or some crazy lunar configuration between them) is that the there's some dynamic sorcery going on for automatic bridge devices (at least at startup). What seems to happen is that the mac address is dynamically picked to be the lowest (sorted) MAC address of all the devices that are part of the bridge. This is where there seems to be some shenanigans going on. I generally avoid systemd but with Proxmox, that is (so far) not possible.
Effectively by forcing the hw address to be set, you are changing this unpredictability and everything works automagically.
@Falzo I'm actually surprised the upgrade checking script doesn't check for the presence of ifupdown2, that would make more sense to me. I'm like you, I didn't really want to be pfutzing around with hwaddress entries if the installation of a package would resolve the issue, hence my testing to try and figure it out once and for all Until 7.1
@nullnothere said:
I thought I'd just add my random $0.02 - one of the funky things with (I think) newer kernels and/or systemd (or some crazy lunar configuration between them) is that the there's some dynamic sorcery going on for automatic bridge devices (at least at startup). What seems to happen is that the mac address is dynamically picked to be the lowest (sorted) MAC address of all the devices that are part of the bridge. This is where there seems to be some shenanigans going on. I generally avoid systemd but with Proxmox, that is (so far) not possible.
Effectively by forcing the hw address to be set, you are changing this unpredictability and everything works automagically.
This is all theory of course
My Hosthatch Chicago machine has two ens devices too which further complicated things when I first attempted an upgrade, from memory I think there there was funny buggers with the bridge grabbing the mac address of the physical interface not in use. I ended up cracking it and just re-installing 6.4 at the time
You're all very welcome, glad I could contribute something small back to the community. I am still going to hold back upgrading most of my infrastructure, however the newer Hosthatch service is a 10Tb storage server and I didn't really want to have to try and juggle 10Tb around for a failed upgrade in the future so that's staying at 7.x.
As I upgrade my other boxen, if I notice anything not in line with what I posted above, I'll post again to the thread.
I installed PVE 6.4 on top of Debian 10, tried to reboot, but I unable to add any additional network, it was solved with install ifupdown2 and restart the vps
⭕ A simple uptime dashboard using UptimeRobot API https://upy.duo.ovh
⭕ Currently using VPS from BuyVM, GreenCloudVPS, Gullo's, Hetzner, HostHatch, InceptionHosting, LetBox, MaxKVM, MrVM, VirMach.
Comments
@miu
I checked your post on the Proxmox Forum to see whether Proxmox replied. I found no reply.
Then I remembered this 2020 Proxmox forum thread where Wolfgang from the Proxmox staff explains that "network configuration is out of scope" so that even buying a Proxmox subscription "would help nothing" for a network configuration question.
Further up in the same thread Wolfgang said "We do no configuration."
Here is a link to the 2020 Proxmox forum thread as archived on archive.org.
Maybe network configuration being "out of scope" might explain why Proxmox seems not to have replied to your post on their forum?
Friendly greetings from the Sonoran Desert! ☀️
MetalVPS
Hi Tom, Check proxmox thread
Is possible that dude who have updated it found solution (add hwaddress param into config file)
When i will have any time i will try/test it then i'll be back with result
Have a great weekend everyone!
Good day and Goodbye
I already mentioned that, on page one:
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Truth.
I assume seems be solution. And apologies for my blindness.
Good day and Goodbye
interesting still, that this was not needed before Proxmox 7 and maybe related to sticking to ifupdown even with the newer kernel? also according to your findings simply restarting the networking later in the process seemed to work even without providing hwaddress in that config, so...
Can confirm that adding the hwaddress definitely fixed this issue. Just did a bunch of new installs as well as test migrations from pve 6.4 to 7 to confirm. No idea why that didn't work for me the first time!
1) use 'ip -c link' to determine the mac address of the physical interface (eg. ens3)
2) edit /etc/network/interfaces and specify this mac (using hwaddress ) on the bridge (eg. vmbr0)
Installing 'ifupdown2' on an existing pve 6.4 install also worked without having to muck around with the hwaddress
only so I undestand this right (as it will be very helpful): so you had the same problems with networking after upgrading from 6.4 to pve7 but just installing this package before the migration fixed it as well, without having to touch any configs?
that'd be the preferred upgrade path then I'd say.
can you recall if your pve6 has been initially installed from iso or on top of debian?
hey @Falzo,
Hope this helps:
When 7 was first released, tried two 6.4->7 upgrades on boxes from MaxKVM. All single IP machines, installed with PVE 6 .iso, with a bridged nic to the outside world. From memory I think I'd installed ifupdown2 on these, but can't remember for sure. Had no problems whatsoever so thought I'd try it on one of my Chicago Hosthatch box. Chicago Hosthatch box had also been installed from a Pve 6.x iso, so I did the 6 to 7 Proxmox upgrade test like I'd done for the MaxKVM boxes, it passed, so I went ahead.
After the upgraded completed successfully, I lost network access on reboot. From memory, I tried renaming nics and re-establishing the bridges, nothing worked. After reinstalling 6.4 from iso and having networking working, and then also repeating the process a second time with the same result, I just wrote it off as PVE 7 teething problems and was going to wait for it to be patched out. The Proxmox documentation describing the 'macaddress' option wasn't as detailed as it is now and (I think) I did gloss over it but it didn't seem applicable. I had private networking set up on that box too, so wasn't sure if that was causing more problems.
I purchased a new Hosthatch service (in a different location) which was provisioned today, and I was going to install PVE 6.x on it given the issues I'd had with 7 previously (again, waiting for 7 to be updated and migrating at a later date), but thought I'd do more testing as some of the comments in this thread seemed to be inconclusive. Tried an install from .iso for 7, no networking. Reinstalled with 6.x .iso, networking worked perfectly.
I then installed ifupdown2 on the 6.x .iso install, ran the upgrade script and it upgraded no problem.
Then, I thought, "well, that works but I can't be stuffed installing 6.x and then upgrading for future installs", so tried a 7 .iso installation. Again, no networking post upgrade. I checked the mac's with 'ip -c link' and noticed that my bridge had a different macaddress to the physical interface. I then edited /etc/network/interfaces and defined 'hwaddress' for the bridge, then reloaded the network using 'ifreload -a' and could verify I had Internet connectivity. Tested and it survived a reboot.
Was curious to see ifupdown2 had been installed by default on 7.x and it had, I definitely had to add the macaddress for my bridge to come up.
I'm going to do a fresh install on the Hosthatch Chicago box I referred to above. Going to install 6.4 from iso, update, install ifupdown2 and then do the inplace upgrade to 7. Will report back if it works OK.
Apologies for the waffle, hope it helps. Let me know if you have any other questions.
Argh! Don't know why i typed "macaddress" instead of "hwaddress" above..
Anyway, did some more testing with fresh installs of PVE 6.4 from iso.
First install, installed 6.4, apt-upgraded, ran the update script - then lost connectivity after I rebooted into PVE 7. Running 'ip -c link' verified that the bridge's MAC was different to that of the physical interface it was meant to bridge. I edited /etc/network/interfaces and added 'hwaddress' for the bridge and set the MAC to be the same as the physical interface. Then ran '/etc/init.d/networking' to restart networking and the network came back.
Second install, installed 6.4, apt-upgraded, installed 'ifupdown2', then ran the update script - didn't lose connectivity after rebooting into PVE 7. Verified that MAC addresses for bridge and physical interfaces were the same using 'ip -c link'.
Final test was a clean install with PVE 7 .iso. No networking after install and reboot. Confirmed bridge and and physical interface MAC addresses were different using 'ip -c link'. Modified /etc/network/interfaces accordingly, and after a 'service networking restart' networking was good to go!
@bdl thanks a lot for all the information and tests, that helps a lot. could well be that with (old) ifupdown the bridge does not get a new/seperate hwaddress and that this is only introduced with the new version. that said, if one updates to ifupdown2 via apt it might run some post-inst script that detects and pins the hwaddress, while the pve udpate script somehow seems to miss that...
I am still waiting with the updating as for now I don't see the need... but at some point obviously need to do that and will make sure to switch to ifupdown2 beforehand then. though, my installation are all not from ISO but on top of debian.
again thanks a lot for the efforts and report!
@bdl Well done, for the time/effort/perseverance.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I thought I'd just add my random $0.02 - one of the funky things with (I think) newer kernels and/or systemd (or some crazy lunar configuration between them) is that the there's some dynamic sorcery going on for automatic bridge devices (at least at startup). What seems to happen is that the mac address is dynamically picked to be the lowest (sorted) MAC address of all the devices that are part of the bridge. This is where there seems to be some shenanigans going on. I generally avoid systemd but with Proxmox, that is (so far) not possible.
Effectively by forcing the hw address to be set, you are changing this unpredictability and everything works automagically.
This is all theory of course
@Falzo I'm actually surprised the upgrade checking script doesn't check for the presence of ifupdown2, that would make more sense to me. I'm like you, I didn't really want to be pfutzing around with hwaddress entries if the installation of a package would resolve the issue, hence my testing to try and figure it out once and for all
Until 7.1 
My Hosthatch Chicago machine has two ens devices too which further complicated things when I first attempted an upgrade, from memory I think there there was funny buggers with the bridge grabbing the mac address of the physical interface not in use. I ended up cracking it and just re-installing 6.4 at the time
You're all very welcome, glad I could contribute something small back to the community. I am still going to hold back upgrading most of my infrastructure, however the newer Hosthatch service is a 10Tb storage server and I didn't really want to have to try and juggle 10Tb around for a failed upgrade in the future so that's staying at 7.x.
As I upgrade my other boxen, if I notice anything not in line with what I posted above, I'll post again to the thread.
Just tried another server:
Existing 6.4 iso install, verified that ifupdown2 was installed before commencing upgrade check and upgrade itself.
Proxmox upgraded while maintaining network connectivity - no manual tweaking of hwaddress required!
I installed PVE 6.4 on top of Debian 10, tried to reboot, but I unable to add any additional network, it was solved with install
ifupdown2and restart the vps⭕ A simple uptime dashboard using UptimeRobot API https://upy.duo.ovh
⭕ Currently using VPS from BuyVM, GreenCloudVPS, Gullo's, Hetzner, HostHatch, InceptionHosting, LetBox, MaxKVM, MrVM, VirMach.