@vyas said: @virmach Tech support promptly closed the ticket, no explanation offered, some kind soul here on LES tried to suggest I try what I had already tried, and nearly 3 weeks later, no mas.
It's an outage report ticket. All we do for those is read the title and decide if a network status needs to be opened for it or if anything needs to be done and we're already working on it. We didn't add it to outage report because it's not actually offline, it's just a panel issue, and the outages on network status page are already very cluttered so we're reserving it just for actual outages right now.
Also you're not in Amsterdam, it's in Frankfurt, unless I looked at it wrong? Anyway FFME001 has uptime of 49 days and most VMs are running. Yours may potentially be offline and controls are broken though still but I'm just answering on why the report outage ticket was closed.
Indeed, VPS is in Frankfurt, (FFME001 prefix).
it's just a panel issue
I ask out of genuine curiosity and not disdain- can this be fixed at user level (ie from my side)? Or do I have to raise another ticket highlighting this specific issue? Thanks-
View Ticket #232870
Subject: assigned ip not working on storage server
If it's urgent make sure it's a priority department ticket. We're almost done catching up and completing all those by today/tomorrow.
Tyoc040 has been offline for a long time. Can I have a refund?
So many people are creating tickets in every different department with various titles and custom requests for TYOC040 and TYOC035. If we take them offline to try to fix it, more people will do the same. Once we catch up to all these tickets being created and I have enough time to set aside to try to have a more permanent solution, and we send out e-mails warning people of the maintenance window while I can get someone else to answer all the tickets that are going to be created, then we could work on it.
Until then it seems like many people on those nodes would rather have work orders about it than have it work, as we're getting 10-20x more tickets, confusion, ignoring of posted network status, etc, than when other nodes face a similar issue.
We have been working in the background over the last two months to resolve any network issues resulting in packet loss. If you are still facing packet loss today, please reply back to this ticket and provide MTR results from and to your VPS, with at least 500 iterations and we will compile and forward any remaining issues to our datacenter partners.
You sent out a bulk ticket response to reply back if the ticket remains valid, but it isn't actually possible to reply.
I can appreciate that you have a lot to work through and have been working hard on these issues for months, but I don't really even get if there's a point in submitting tickets anymore since they keep being closed without resolution. I'm sure there will be a bulk fix in the future but it's already been some months and don't really see a clear course of action for broken VMs.
Just out of curiosity, and to confirm it is not a problem on my end, is anybody using IPv6 sit tunnels on AMSD026 or MIAZ012, or any other node, that was working fine and then has suddenly stopped working for no apparent reason? I use two different IPv6 tunnel providers and have tried two different tunnel servers each. I get nothing now when they were working fine for weeks. It's like a firewall suddenly went up blocking all IPv6 networking. Which I have double checked is not on my end. Anybody having a similar experience ?
EDIT: MIAZ012 is working again after ~1 hour down . AMSD026 has been this way for over a week.
@FrankZ said:
Just out of curiosity, and to confirm it is not a problem on my end, is anybody using IPv6 sit tunnels on AMSD026 or MIAZ012, or any other node, that was working fine and then has suddenly stopped working for no apparent reason? I use two different IPv6 tunnel providers and have tried two different tunnel servers each. I get nothing now when they were working fine for weeks. It's like a firewall suddenly went up blocking all IPv6 networking. Which I have double checked is not on my end. Anybody having a similar experience ?
EDIT: MIAZ012 is working again after ~1 hour down . AMSD026 has been this way for over a week.
Seeing it in LAX yeah, other locations seem fine. Tried it with Route48.
We have been working in the background over the last two months to resolve any network issues resulting in packet loss. If you are still facing packet loss today, please reply back to this ticket and provide MTR results from and to your VPS, with at least 500 iterations and we will compile and forward any remaining issues to our datacenter partners.
You sent out a bulk ticket response to reply back if the ticket remains valid, but it isn't actually possible to reply.
I can appreciate that you have a lot to work through and have been working hard on these issues for months, but I don't really even get if there's a point in submitting tickets anymore since they keep being closed without resolution. I'm sure there will be a bulk fix in the future but it's already been some months and don't really see a clear course of action for broken VMs.
We only do these when it's absolutely necessary and we have been doing them very specifically recently.
If you genuinely believe you created specifically a ticket marked as a connection issue with packet loss, and you were one of the very few people that genuinely followed all instructions before creating the ticket, and you are still facing said issues, then give me your ticket # and I'll take a look.
As for not being able to reply to it, that was someone's mistake. I'll see if we can re-open them but submitting one with that information should work as well.
@VirMach
For some reason, for the past few days, I'm not receiving the 2FA email. I thought I had saved the backup code but it appears to be incorrect (now on last attempt.)
Perhaps someone else knows the correct answer to the following:
What format does the backup code take? (Four sets of hexadecimal/alpha-numeric with spaces in between?)
@AlwaysSkint said:
Perhaps someone else knows the correct answer to the following:
What format does the backup code take? (Four sets of hexadecimal/alpha-numeric with spaces in between?)
For time-based 2fa, yes. It would look something like 0123 4567 89ab cdef
@AlwaysSkint said: @VirMach
For some reason, for the past few days, I'm not receiving the 2FA email. I thought I had saved the backup code but it appears to be incorrect (now on last attempt.)
Perhaps someone else knows the correct answer to the following:
What format does the backup code take? (Four sets of hexadecimal/alpha-numeric with spaces in between?)
Issue with mailing on our end which I should have resolved yesterday.
That's confused me. At least I know that the format is correct but if a time-restricted backup code then it ain't much good as a backup, unless I'm being senile again. :-/
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
@VirMach Still waiting on my ticket for the right OS (#218112). I'm not really in any rush, but now I got an invoice, even though this month was supposed to be free?
@VirMach said: Issue with mailing on our end which I should have resolved > > @jtk said: For time-based 2fa..
That's confused me. At least I know that the format is correct but if a time-restricted backup code then it ain't much good as a backup, unless I'm being senile again. :-/
People who don't need a mobile phone beeping at them, when 95% of the time they sit at a laptop!
Login device is always the same, nearly always from the same network/location, so why the fcuk should a f'kin mobile phone be involved? (rhetorical)
Sane people don't manage (critical/secure/multiple) services from an 8", or less screen, nor play games/take selfies with the same device.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
@VirMach
Is SJCZ004 still facing partial outage (no more outage message in network status)? I have 6 servers there but 2 of them still remaining offline.
@VirMach said: If you genuinely believe you created specifically a ticket marked as a connection issue with packet loss, and you were one of the very few people that genuinely followed all instructions before creating the ticket, and you are still facing said issues, then give me your ticket # and I'll take a look.
My concern with that statement is that my ticket is about no public network and I don't ever remember marking my recent ticket as being about packet loss.
I've already done the troubleshooting, from reinstalling the system, setting IP configurations manually, and using reconfigure networking on a fresh install (which is its present state). Ticket #677100.
At this point, if there's no resources to look into it, I'd rather just the cost of the service refunded as account credit so I can just consolidate what services that are still functional.
@haynhat said: @VirMach
Is SJCZ004 still facing partial outage (no more outage message in network status)? I have 6 servers there but 2 of them still remaining offline.
Generally what would happen after an outage like that is any VMs that were semi-broken before it would not have been fixed. It's in our queue to redo the fixes we did for any broken VMs for the nodes we skipped due to them being offline. It's very possible your offline services are part of those.
@VirMach said: If you genuinely believe you created specifically a ticket marked as a connection issue with packet loss, and you were one of the very few people that genuinely followed all instructions before creating the ticket, and you are still facing said issues, then give me your ticket # and I'll take a look.
My concern with that statement is that my ticket is about no public network and I don't ever remember marking my recent ticket as being about packet loss.
I've already done the troubleshooting, from reinstalling the system, setting IP configurations manually, and using reconfigure networking on a fresh install (which is its present state). Ticket #677100.
At this point, if there's no resources to look into it, I'd rather just the cost of the service refunded as account credit so I can just consolidate what services that are still functional.
Looks like someone made a mistake then, I'm speaking to the person that ran the closing script for that batch to see what happened and make sure we restore any incorrect tickets closed. That shouldn't have been closed.
@VirMach 23.95.68.151 is always offline a long time,can you help me to slove the problem?In additionally,the migration of Ticket #160581 is also handled together,thank a lot.
By the way,is there time compensation for being offline for too long?
@hmx27 said: @VirMach 23.95.68.151 is always offline a long time,can you help me to slove the problem?In additionally,the migration of Ticket #160581 is also handled together,thank a lot.
These are being planned out and done very soon. We just want to make sure we do it properly at this point, to where people go on a stable node, have functional networking, operating systems, and we communicate it properly and credit everyone accordingly.
I almost sent out an email a few days ago outlining everything but it wasn't clear enough so I didn't send it out yet. I'm trying to get everything down 100% because people have already waited so long, I want to make sure we do a perfect job and don't confuse people further or drag it out by providing incorrect dates, vague information, etc.
@haynhat said: @VirMach
Is SJCZ004 still facing partial outage (no more outage message in network status)? I have 6 servers there but 2 of them still remaining offline.
@VirMach said: If you genuinely believe you created specifically a ticket marked as a connection issue with packet loss, and you were one of the very few people that genuinely followed all instructions before creating the ticket, and you are still facing said issues, then give me your ticket # and I'll take a look.
My concern with that statement is that my ticket is about no public network and I don't ever remember marking my recent ticket as being about packet loss.
I've already done the troubleshooting, from reinstalling the system, setting IP configurations manually, and using reconfigure networking on a fresh install (which is its present state). Ticket #677100.
At this point, if there's no resources to look into it, I'd rather just the cost of the service refunded as account credit so I can just consolidate what services that are still functional.
@VirMach
My Ticket also got closed without any proper response,
Ticket #262275
My Node: FFME001.VIRM.AC
I know you already mentioned the node is up in one of your replies here. But still my vps is down. Unable to connect. Earlier I didn't had solusvm panel control access for this node. So I thought it must have been panel's fault but now the panel is up and I am able to browse it. Still it's offline, I tried to reboot/reconfigure network but no success.
I thought it was just me but one of my friend who is on same node facing exact same issue.
@Kaito said: @VirMach
My Ticket also got closed without any proper response,
Ticket #262275
My Node: FFME001.VIRM.AC
I know you already mentioned the node is up in one of your replies here. But still my vps is down. Unable to connect. Earlier I didn't had solusvm panel control access for this node. So I thought it must have been panel's fault but now the panel is up and I am able to browse it. Still it's offline, I tried to reboot/reconfigure network but no success.
I thought it was just me but one of my friend who is on same node facing exact same issue.
This isn't closed. If it was closed it already got unclosed and re-flagged. I see the title's also changed so it's marked correctly for us to include it in the proper queue for that specific fix.
That would be these:
Generally what would happen after an outage like that is any VMs that were semi-broken before it would not have been fixed. It's in our queue to redo the fixes we did for any broken VMs for the nodes we skipped due to them being offline. It's very possible your offline services are part of those.
There are nodes in bursts of times where it may have had a specific problem that caused the VMs not to generate properly and that's what you're facing an issue on, for FFME001.
I've already cleared a few of these up today. We might get to FFME001 today as well.
Looks like SJC2008 is reachable from billing panel again, which is great, it had been unreachable before. Is this node still having network issues? Billing and control panels say my VPS is offline, attempting reboot doesn't change it, it's unreachable in rescue system, and VNC address refuses connections. It does say it's a Ryzen node, i.e. the VPS got migrated from the old Intel hardware. I guess that is nice.
@willie said:
Looks like SJC2008 is reachable from billing panel again, which is great, it had been unreachable before. Is this node still having network issues? Billing and control panels say my VPS is offline, attempting reboot doesn't change it, it's unreachable in rescue system, and VNC address refuses connections. It does say it's a Ryzen node, i.e. the VPS got migrated from the old Intel hardware. I guess that is nice.
I'm going to just make a wild guess based on all your previous terrible luck & experiences and say that you're probably one of the 10 people affected on this node by a missing disk. I have no idea what happened and won't know without looking back at all our logs later. They're all back to back, just 10 VMs that don't have a proper disk.
Right now they're on the queue to get created (the LVMs.)
Comments
Controls were recently fixed.
So many people are creating tickets in every different department with various titles and custom requests for TYOC040 and TYOC035. If we take them offline to try to fix it, more people will do the same. Once we catch up to all these tickets being created and I have enough time to set aside to try to have a more permanent solution, and we send out e-mails warning people of the maintenance window while I can get someone else to answer all the tickets that are going to be created, then we could work on it.
Until then it seems like many people on those nodes would rather have work orders about it than have it work, as we're getting 10-20x more tickets, confusion, ignoring of posted network status, etc, than when other nodes face a similar issue.
(edit) So to answer you, no.
You sent out a bulk ticket response to reply back if the ticket remains valid, but it isn't actually possible to reply.
I can appreciate that you have a lot to work through and have been working hard on these issues for months, but I don't really even get if there's a point in submitting tickets anymore since they keep being closed without resolution. I'm sure there will be a bulk fix in the future but it's already been some months and don't really see a clear course of action for broken VMs.
Just out of curiosity, and to confirm it is not a problem on my end, is anybody using IPv6 sit tunnels on AMSD026 or MIAZ012, or any other node, that was working fine and then has suddenly stopped working for no apparent reason? I use two different IPv6 tunnel providers and have tried two different tunnel servers each. I get nothing now when they were working fine for weeks. It's like a firewall suddenly went up blocking all IPv6 networking. Which I have double checked is not on my end. Anybody having a similar experience ?
EDIT: MIAZ012 is working again after ~1 hour down
. AMSD026 has been this way for over a week.
LES • About • Donate • Rules • Support
Seeing it in LAX yeah, other locations seem fine. Tried it with Route48.
Ok, thanks. Not a big issue, I just wanted to make sure it was not just me.
LES • About • Donate • Rules • Support
We only do these when it's absolutely necessary and we have been doing them very specifically recently.
If you genuinely believe you created specifically a ticket marked as a connection issue with packet loss, and you were one of the very few people that genuinely followed all instructions before creating the ticket, and you are still facing said issues, then give me your ticket # and I'll take a look.
As for not being able to reply to it, that was someone's mistake. I'll see if we can re-open them but submitting one with that information should work as well.
@VirMach
For some reason, for the past few days, I'm not receiving the 2FA email. I thought I had saved the backup code but it appears to be incorrect (now on last attempt.)
Perhaps someone else knows the correct answer to the following:
What format does the backup code take? (Four sets of hexadecimal/alpha-numeric with spaces in between?)
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
For time-based 2fa, yes. It would look something like
0123 4567 89ab cdefDataplane.org's current server hosting provider list
Issue with mailing on our end which I should have resolved yesterday.
Doesn't appear to have helped.
That's confused me.
At least I know that the format is correct but if a time-restricted backup code then it ain't much good as a backup, unless I'm being senile again. :-/
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
@VirMach Still waiting on my ticket for the right OS (#218112). I'm not really in any rush, but now I got an invoice, even though this month was supposed to be free?
Thanks,
John
Acronym soup. Time-based aka TOPT (Time-based One-time Password Token)
Dataplane.org's current server hosting provider list
Who the hell uses 2FA via e-mail o.O
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
People who don't need a mobile phone beeping at them, when 95% of the time they sit at a laptop!
Login device is always the same, nearly always from the same network/location, so why the fcuk should a f'kin mobile phone be involved? (rhetorical)
Sane people don't manage (critical/secure/multiple) services from an 8", or less screen, nor play games/take selfies with the same device.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Authy has desktop apps and is probably still more secure/reliable than email 2FA
@VirMach
Is SJCZ004 still facing partial outage (no more outage message in network status)? I have 6 servers there but 2 of them still remaining offline.
My concern with that statement is that my ticket is about no public network and I don't ever remember marking my recent ticket as being about packet loss.
I've already done the troubleshooting, from reinstalling the system, setting IP configurations manually, and using reconfigure networking on a fresh install (which is its present state). Ticket #677100.
At this point, if there's no resources to look into it, I'd rather just the cost of the service refunded as account credit so I can just consolidate what services that are still functional.
Generally what would happen after an outage like that is any VMs that were semi-broken before it would not have been fixed. It's in our queue to redo the fixes we did for any broken VMs for the nodes we skipped due to them being offline. It's very possible your offline services are part of those.
Looks like someone made a mistake then, I'm speaking to the person that ran the closing script for that batch to see what happened and make sure we restore any incorrect tickets closed. That shouldn't have been closed.
@VirMach 23.95.68.151 is always offline a long time,can you help me to slove the problem?In additionally,the migration of Ticket #160581 is also handled together,thank a lot.
By the way,is there time compensation for being offline for too long?
I think they mentioned credits for being offline for too long
These are being planned out and done very soon. We just want to make sure we do it properly at this point, to where people go on a stable node, have functional networking, operating systems, and we communicate it properly and credit everyone accordingly.
I almost sent out an email a few days ago outlining everything but it wasn't clear enough so I didn't send it out yet. I'm trying to get everything down 100% because people have already waited so long, I want to make sure we do a perfect job and don't confuse people further or drag it out by providing incorrect dates, vague information, etc.
These just got fixed.
These got fixed.
Thanks for the nice surprise this morning @Virmach .
@VirMach
My Ticket also got closed without any proper response,
Ticket #262275
My Node: FFME001.VIRM.AC
I know you already mentioned the node is up in one of your replies here. But still my vps is down. Unable to connect. Earlier I didn't had solusvm panel control access for this node. So I thought it must have been panel's fault but now the panel is up and I am able to browse it. Still it's offline, I tried to reboot/reconfigure network but no success.
I thought it was just me but one of my friend who is on same node facing exact same issue.
This isn't closed. If it was closed it already got unclosed and re-flagged. I see the title's also changed so it's marked correctly for us to include it in the proper queue for that specific fix.
That would be these:
There are nodes in bursts of times where it may have had a specific problem that caused the VMs not to generate properly and that's what you're facing an issue on, for FFME001.
I've already cleared a few of these up today. We might get to FFME001 today as well.
Looks like SJC2008 is reachable from billing panel again, which is great, it had been unreachable before. Is this node still having network issues? Billing and control panels say my VPS is offline, attempting reboot doesn't change it, it's unreachable in rescue system, and VNC address refuses connections. It does say it's a Ryzen node, i.e. the VPS got migrated from the old Intel hardware. I guess that is nice.
I'm going to just make a wild guess based on all your previous terrible luck & experiences and say that you're probably one of the 10 people affected on this node by a missing disk. I have no idea what happened and won't know without looking back at all our logs later. They're all back to back, just 10 VMs that don't have a proper disk.
Right now they're on the queue to get created (the LVMs.)