smallweb down and out in Los Angeles

uptime

Just a heads up for people using smallweb shared hosting in Los Angeles: a series of unfortunate events has resulted in the untimely demise of service in that location.

As per the notification email:

Dear Customer,

Earlier today the server hosting Los Angeles website data suffered a networking failure which prevented it from connecting to a network and performing standard tasks. Attempts to restore the network were made but ultimately after the server being offline for such a prolonged period of time we attempted to re-install the server and possibly migrate backups elsewhere.

However it was discovered that our system backups for this location were not suitable to be restored and thus unfortunately we are announcing the immediate closure of Shared-LAX1.

I am extremely disappointed in myself for having to deliver this news to you and this is not how I wanted 2020 to start but I would like to learn from this experience going forward to improve our other available locations.

Because we can no longer deliver your service you are entitled to a full refund. To claim a refund, please respond to this email or open a support ticket. Your refund will be processed within 7 days and your LAX service will no longer be accessible through your client area.

If you have any questions, please let me know. I am extremely sorry for this disastrous event.

Regards,

Michael

SmallWeb Ltd.

Apparently a typo in a backup script - and not enough testing to catch it.

Seems like a painful lesson learned - hopefully not too many bridges burned.

I'm personally more inclined to offer my condolences etc rather than get too bent out of shape over a glitch in a £5 per year service. And I can't help but appreciate the generally direct and transparent communication from @Mic-hael so far (compare with for example a similar situation at Gandi) ... Hopefully we'll enjoy more supportive discussion and suggestions for end-user backup best practices here. (Or whatever.)

Might also suggest service in Melbourne may be an interesting alternative for some customers, if that's a viable option. Though I'll miss the LA location - life goes on ...

PHP_Backend

However it was discovered that our system backups for this location were not suitable to be restored

This has bitten me before. sadly, everyone learns in hard way.

uptime

@PHP_Backend indeed ... I think ultimately better if these lessons get learned sooner rather than later!

Also - anyone who really needs the LA location might want to get in touch with @sanvit, who has generously offered to provide some free service (as an individual, not a provider) from their (previously idling) MyW.pt reseller account there.

vyas

@SmallWeb 's alright- rightfully he said he will take a break from forums to focus on the business.
Sucks for those who had the LA location as their only service, hope they had backups.

@PHP_Backend said:

However it was discovered that our system backups for this location were not suitable to be restored

This has bitten me before. sadly, everyone learns in hard way.

OT, but man, you do have your share of hosting related stories, it seems.

sanvit

@uptime said:
Also - anyone who really needs the LA location might want to get in touch with @Sanvit, who has generously offered to provide some free service (as an individual, not a provider) from their (previously idlig) MyW.pt reseller account there.

Yup, if anyone needs LA specifically, you can PM me. More details are available on this LET comment
https://www.lowendtalk.com/discussion/comment/3073629/#Comment_3073629

deank

tl;tr

The dude put in "LUX" instead of "LAX" in setting.
He never checked / verified that backup was there.

But then, considering its price, I wouldn't feel entitled if I were a customer.

uptime

I heard he started a land war in asia - and went against a Sicilian when derp was on the line.

Lee

@deank said: But then, considering its price, I wouldn't feel entitled if I were a customer.

I would. The price providers ask me to pay is to provide the service they offer. This whole, because it's such a small amount being justification is bollocks quite frankly. I mean not even checking the backups are functioning..

That said, your data is only as safe as the backups you take and check.

At least a full refund is being offered.

deank

A buck or 2 a year service is not something I'd bother ranting about.

At the same time, a host being quite professional raises expectations, so basically he dug his own grave. In that sense, I think @cociu_with_sisters is doing a good job keeping expectations in line.

Lee

@deank said: A buck or 2 a year service is not something I'd bother ranting about.

It's not, but then that is not the point. It is the justifying of an avoidable issue like this with "it only cost.." is not acceptable in my view.

Backups at a provider should never be seen as your only route to recovery but;

Provider backups mean restoring all client accounts after a failure saves them time even if they do have their own.
It could also be that the provider has a newer backup point which saves the client re-work.

The most important take away here is that if the provider didn't even notice the backups were not happening in one location then they are not even being tested in any other location so assume they don't exist at all.

deank

Promises are easy to make with words. Delivering them is different no matter what the price point a product has.

That's probably why many (don't know exactly how many) are pissed at politicians because barely anyone keeps their words once elected.

uptime

"entitled" may be a somewhat loaded term (lol) ...

as far as reasonable expectations go well I dunno ... hope for the breast, plan for the wurst?

and if your data matters to you ... then you're already doing your own backups offsite in a serious way, ammirite?

I am a bit surprised to see service in a given location discontinued altogether - though this does seem to happen more often with new-ish providers offering low-priced services based on other low-priced providers, so not exactly shocked to see happening in this case. But I am disappoint. Maybe some room for improvememt in managing this sort of worst-case scenario going forward. Maybe not.

I'm guessing that once the dust settles we might get an incident report with a bit more detail and maybe even some (hopefully diplomatic and professional "with all due respect") backstory about the network issue or other motivations to abandon ship in LA. Or not ... I'm certainly not paying enough to expect more than "best efforts" all around, but hope springs eternal so ¯\_(ツ)_/¯

jvnadr

@uptime said: if your data matters to you ... then you're already doing your own backups offsite in a serious way, ammirite?

The thing with shared hosting is that usually, it is a pita to keep constant automated backups and/or incremental ones. Backup and transfer whole cpanel accounts in another location cannot be done easily in most of shared providers, especially if there are not installed external plugins and/or there is not any jailed ssh access to make life easier...
For a small web site with minimum to none updates, though, it's easy to back it up and download it locally just in case. The worst scenario is with email accounts, as the only viable option for constant backups without ssh access, is imapsync - and for that, you have to have access to a server with root privileges to install it...
Of course, if you have valuable data, then, you will not host them in a 10$/y shared cpanel hosting without more than one backup scenarios

Mic-hael said: The script I use to transfer backups to a remote location included incorrect labelling which transferred backups of "LUX" to the "LAX" destination. Unfortunately there was not a deep enough confirmation before the reload. Because of this, more reliable methods for backup retention need to be implemented. For those servers powered by the Cloud, I will be setting up automatic drive backups direct with my providers where applicable. As well as that, I will be increasing my redundancy for these backups and implementing stricter and clearer policies going forward. The events of yesterday are extremely disappointing and ones I never wish to repeat. For affected clients, I offer a new service in our other locations or a full refund and thank you to those whom I have spoke with personally for your support. I hope to do better.

If backups are essential for a customer, it is 100 x essential for a provider... I do not have services with @SmallWeb , and Mic-hael seem a really nice guy, but the structure of his shared environment as exposed due to the failure, is extremely poor. A single backup with a custom script that had never being verified if worked? Not even a simple check that the size of the tarball archive was the same in destination as in the original server? It's not about the money of the affected services, it's just ringing bells on the whole structure of his servers.

Also, he claims that there was a network failure that was difficult to be restored in a proper time window and he tried to restore services to another location due to that, when he discovered that the backup was not valid. There is a plot hole here: if there is an issue with networking, he could try to resolve it much more after he discovered that the backups were none. Why did he decide to abandon a server (or, better, a whole location!) that -as he claimed- the issue with it is networking and not corrupted disks / raid etc. Smells like he has a fight with the server provider and he is not able to get access to the node...

Anyway, the statement that it is a big failure from his side, the full refunds and the apologies to his customers can reduce the bad reputation that such an incident creates. And his claim that he will learn from his mistakes to make a better infrastructure and backups, is totally valid - only such a failure can really make someone to triple-check the next time everything.

sonic

I still go with SmallWeb when I need shared hosting. The best support ever!
Lesson learned, SmallWeb will be stronger!

uptime

@jvnadr - that is some insightful analysis. Thanks for sharing it here.

I hesitate to speculate too much about exact motivation for "abandoning ship" in Los Angeles but at this point I would just say it is a big question mark for me too. Hopefully we'll get some clarification on this after some time has passed - I think it's wise to recognize how much stress can distort perspective when the "downtime" clock is still ticking.

As for expectations about learning ... there's aptitude, and there's attitude. The attitude often comes from experience. And experience sometimes comes from making mistakes. So this may be the proverbial cheap tuition in the school of hard knocks.

deank

It all just means the end is always right around a corner and you guys should join Nigh sect.

MikePT

@uptime said:
@PHP_Backend indeed ... I think ultimately better if these lessons get learned sooner rather than later!

Also - anyone who really needs the LA location might want to get in touch with @sanvit, who has generously offered to provide some free service (as an individual, not a provider) from their (previously idling) MyW.pt reseller account there.

Definitely.

Anyone who needs help please let me know.

@SmallWeb this too applies for you mate. Feel free to PM me if you need help or if you think there still is a chance to recover the data.
Such events happen in our industry. We always advise our clients to backup their data.

MikePT

@sanvit said:

@uptime said:
Also - anyone who really needs the LA location might want to get in touch with @Sanvit, who has generously offered to provide some free service (as an individual, not a provider) from their (previously idlig) MyW.pt reseller account there.

Yup, if anyone needs LA specifically, you can PM me. More details are available on this LET comment
https://www.lowendtalk.com/discussion/comment/3073629/#Comment_3073629

I have upgraded both of your lifetime reseller accounts to the double of diskspace free of charge so you can help our forum members.

MichaelCee

Thank you for the messages.

As Lee says, the price point can not be an excuse. I never offered services with the upfront statement that "it's cheap so don't have any hopes". The results of my actions led to the integrity of the service being damaged.

For @jvnadr
The testing of the backup script was abysmal and did not cover all locations. Since this, it has been validated for all remaining locations.

Regarding the server, there's no correlation with the provider in question. No upstream support was offered and agreed to by me when I launched the location. They have a very respectable service that I aspire to be even half as good as.

The closure was extremely abrupt. I had not thought about starting from fresh, this is my first experience of such a catastrophic failure and my first instinct was to inform customers their data was gone. I am overlooking potential options regarding this.

WSS

@SmallWeb

Sometimes you're the guy stuck in the barrel. Other than offering a full refund, there's virtually little else you should be required to do. Yes, it would be nice if there were other options, but sometimes shit happens and it all goes wrong.

You seem to have accepted it with grace, humility, and hopefully don't get too many chargebacks-out-of-spite from this.

deank

Of course, you can either blame Trump or Brexit. Or even both.

Some seem to blame those two for everything that is apparently wrong in their worlds.

Harambe

Well that's a good reminder for me to verify some backups

WSS

Dicks out for HaramLA.

MichaelCee

As a continuation of my last point. The server has now been rebuilt and configured. I have reprovisioned 99% of LAX services bar those who have been refunded or transferred already. Those individuals will be contacted seperately.

Please expect official communication about this.

seriesn

@SmallWeb said:
As a continuation of my last point. The server has now been rebuilt and configured. I have reprovisioned 99% of LAX services bar those who have been refunded or transferred already. Those individuals will be contacted seperately.

Please expect official communication about this.

MichaelCee

LET, LES, HB or other, it is truly the community that keeps this industry alive and motivated. As vyas mentioned, I will be taking a leave from the forum end of things for a while to focus on correctly building my business. This was not something to be taken lightly, so of course I will be available to answer questions regarding this.

havoc

Ouch. Happens I guess

I'd sorta expect the backup software to have verification included? e.g. I know one of the drives in my NAS is bad because duplicati has been complaining about hashsums failing (It randomly picks one archive to verify on each run)

jvnadr

@SmallWeb said: The testing of the backup script was abysmal and did not cover all locations. Since this, it has been validated for all remaining locations.

Mistakes do happen, sorry if I was too harsh earlier. After all, this was a good lesson to avoid such mistakes in the future, as I said before. And, many of LE providers would not approach the issue like you but they would just say "hey, there was a catastrophic issue in a server, your data was lost, sorry but after all, it's clients responsibility to take backups, so, f** off"! Your attitude on this makes me think that this first big failure will turn you into a better provider.

@SmallWeb said: Regarding the server, there's no correlation with the provider in question. No upstream support was offered and agreed to by me when I launched the location. They have a very respectable service that I aspire to be even half as good as.

If you can share it atm, understanding that there are bigger priorities now: Apart from the backup failure, what was the catastrophic issue on the original server that caused the data loss and made the node impossible to be restored ?

MichaelCee

@jvnadr said:

@SmallWeb said: The testing of the backup script was abysmal and did not cover all locations. Since this, it has been validated for all remaining locations.

Mistakes do happen, sorry if I was too harsh earlier. After all, this was a good lesson to avoid such mistakes in the future, as I said before. And, many of LE providers would not approach the issue like you but they would just say "hey, there was a catastrophic issue in a server, your data was lost, sorry but after all, it's clients responsibility to take backups, so, f** off"! Your attitude on this makes me think that this first big failure will turn you into a better provider.

@SmallWeb said: Regarding the server, there's no correlation with the provider in question. No upstream support was offered and agreed to by me when I launched the location. They have a very respectable service that I aspire to be even half as good as.

If you can share it atm, understanding that there are bigger priorities now: Apart from the backup failure, what was the catastrophic issue on the original server that caused the data loss and made the node impossible to be restored ?

If I were to go through the process again, I would have never overwrote the data (regardless of backups) until a possible solution was found to preserve all original data. My original intention was to set up a second server, in which I would have discovered the backup issue however the timing required for relicensing and reconfiguring IPs (assuming backups were available) were my main concern in my wrong split decision to overwrite.

During the time spent trying to solve the network issue, I did multiple tests and reconfigurations of the network interfaces and discovered missing commands and thus packages which make the network function correctly. The downtime at this point and after further attempts was already disappointing, leading to that wrong split decision.

I don't take your comments to be "too harsh" at all. I think all of the opinions on this matter are vital. Thank you.

poisson

Smallweb stays on The Whitelist. That's my endorsement based on a reasonable assessment of the issue and its handling.

However, it is a good time to take stock of everything, (not just technical) because the next issue may not be met with such warm reception, depending on the nature of the issue (fried switches are understandable).

yoursunny

@SmallWeb said:
During the time spent trying to solve the network issue, I did multiple tests and reconfigurations of the network interfaces and discovered missing commands and thus packages which make the network function correctly.

When you have only serial console access and no network, but you need some packages, it's possible to "upload" files via console.

1. Download file to your laptop.
2. Base64 encode the file.
3. Use cat to paste the Base64 into a text file on the server.
4. Base64 decode the file. This needs base64 command on the server.

If you have lost cat or base64 on the server, you can use echo -ne but it's more difficult.