Awesome, thank you! I have added you as a moderator.
Sal
Amazing!!
AAAHHHH 😭 Such a relief. This was stressful.
The websites makes use of Contabo's object storage service for storing and serving images. Earlier during the month, I got an e-mail saying that there would be a 12-hour downtime window due to a migration. Alright, no problem.
A bit before this users reported problems with uploads. I figured this was related to the migration.
After the "migration", images still caused problems due to some "administrative rule" preventing uploads. I sent messages daily to Contabo. At first they were responding that "their engineers were looking into it, that this was related to the migration, and that they had no time estimate". OK, I can be patient....
After a few days I lost some of my patience and pressed them more often, until they started sending me generic messages and eventually saying that "the problems were already fixed" and suggesting it may be some configuration error on my side.
This is where it got stressful, because my knowledge of the implementation of the images back-end is superficial. So... That meant I would need to study the details and either fix it myself or show proof that it was their fault.
These past few days I have tried to do some reading and tested configuration changes, but I am the kind of person that puts too much on their plate and I have been quite busy, so this has been an additional stress point.
Today, images suddenly started working (until it came down again due to some logs piling up). This tells me that it was not a configuration issue and it was with high probability something on Contabo's side that they fixed today. Uploads are working as well now. Contabo support never shared any information about the specifics of what went wrong, so I don't know.
This doesn't mean that I don't need to study the details, but at least now the pressure is lower. My plan now is to do some studying on how images are stored and served and once I have a good understanding I can migrate the instance to a dedicated server. I will also look into techniques for making a server more resilient when things do go wrong.
Yes, it is pretty frustrating.
I keep sending them messages every day. Today's reply:
This is connected to the migration and there is no ETA currently.
Not very happy with them. I am looking into how to move the images somewhere else.
Thanks! <3 Cost is not the issue. The reason why I selected this option was because it seemed a good choice at the time. Moving the image database around is a bit of a hassle so I chose to just keep it as it was even after learning of other options. Last time it took several days of downtime to move the images over because the transfer speed into/out of the object storage is very low.
If I would have known that this would happen now, I might have taken this opportunity to move all the images over. But I was not expecting this amount of downtime. Now I worry that trying to move images from one place to another when the object storage is acting up is risky, and also I am busy.
I sent another message to support this morning and received:
We are sorry for the inconvenience caused. Please note that our technicians are working diligently on this matter, there is no ETA.
Sorry 😓
They are cheap. But I think that at the current total cost I would be able to improve performance while keeping cost about the same by migrating the instance to a dedicated server instead of a VPS + object storage.
Only that I am frustrated at the object storage provider (Contabo), and now I see why their reputation is not great. Their reply was that they sent a warning about the "12 hour downtime during March 10th".... but obviously these effects have extended both to before and after this "12 hour downtime". At least I got confirmation that this problem is due to their migration.
I contacted support this morning as the issue persists. They confirmed that it is related to their migration, but they have not been able to give me a specific time for when it will be resolved.
I think it is the object storage provider... I see that there is a scheduled migration for 12 hours March 10th, but I suspect that they may have already started moving things around.
Thanks. I have not changed anything recently. I am also blocked from uploading. Not sure yet what the problem is, I am looking into it.
Great, added. Thank you!