Go Back   IceInSpace > General Astronomy > Astronomy Books and Media

Reply
 
Thread Tools Rate Thread
  #1  
Old 17-02-2020, 08:36 AM
Andy01's Avatar
Andy01 (Andy)
My God it's full of stars

Andy01 is offline
 
Join Date: Mar 2012
Location: Melbourne
Posts: 3,253
Astrobin meltdown - Catastrophic data loss...

D'oh! https://www.astrobin.com/users/Andys_Astropix/

Dear all,
it appears that something serious happened on AstroBin today. I've now been investigating and trying to mitigate for 12 hours straight, but unfortunately I am certain that a currently unknown number of you have suffered the loss of some (many, in some cases) original uploads.

The data associated to images, on AstroBin database (which is a different system than the place where the image files are stored), is safe.
I don't know the reason yet, but in the meanwhile I have changed all the passwords used by AstroBin (database, storage, servers, etc). Note that this has nothing to do with your passwords you use to log in to AstroBin: they are encrypted.

I will update with my findings on this forum thread:
https://www.astrobin.com/forum/c/ast...of-thumbnails/

For now if you see the message "We are very, very sorry, but it appears that your image suffered a catastrophic data corruption", then that image is gone. Luckily you don't need to start over and lose the data, the comments, likes etc. You only need to replace the image file on that form, and hopefully you still have it on your computer.

Please visit your gallery after log-in, and the image thumbnails with the sad face are affected. If you are not logged-in, affected images will be hidden.
I can't even begin to explain how much I'm currently in shock, literally, and this is why this message is intentionally brief.
I will continue to investigate, and I’m awaiting help from the Amazon Web Services tech support.

In the meantime, if you are heavily affected, I want to say that I am incredibly sorry, though at this time I don’t know where the fault lies.
After the situation settles and I know more, I will offer compensation for the data loss with free subscription upgrades.

Thank you for your understanding and please keep an eye on that forum post for update.

Thanks!
Salvatore
Reply With Quote
  #2  
Old 17-02-2020, 09:44 AM
The_bluester's Avatar
The_bluester (Paul)
Registered User

The_bluester is offline
 
Join Date: Feb 2011
Location: Kilmore, Australia
Posts: 3,342
Looks like 50% of my public images and 100% loss in my staging area. I don't think I could be bothered working out which was which and re uploading them, I reckon I will just delete what is lost, some are images quite early in my AP journey and I may not have even bothered to keep the originals.

Edit. I appear to be among the lucky ones, I have only lost one original image, the rest were dead thumbnails. Regardless of that I think it will take a fair while for Astrobin to rebuild trust, particularly as a paid service.

Last edited by The_bluester; 17-02-2020 at 10:07 AM.
Reply With Quote
  #3  
Old 17-02-2020, 10:18 AM
multiweb's Avatar
multiweb (Marc)
ze frogginator

multiweb is offline
 
Join Date: Oct 2007
Location: Sydney
Posts: 22,060
I quickly browsed through the status report and update thread with the updates and that sounds like déjà vu having been in the same situation years ago. In my case a room containing high voltage electrical exploded. So it was physical damage. Bit extreme I know but the result is the same. So you're stuck between a rock and a hard place. It's not the astrobin guy's fault. He's got data corruption on his HD. He pays somebody to look after the servers (amazon I believe) so they should have redundancy in place for him. Truth is that it never works in the real world. There's always SLA and fine prints. If you don't do your own back up plan then you're left hanging like a piñata with everybody having a shot at you while you're still trying to fix stuff.
Reply With Quote
  #4  
Old 17-02-2020, 10:33 AM
The_bluester's Avatar
The_bluester (Paul)
Registered User

The_bluester is offline
 
Join Date: Feb 2011
Location: Kilmore, Australia
Posts: 3,342
As hard a sentiment as this sounds, the timing is probably good for Salvatore given the recent thread about him sounding out users about a price rise and making it his primary job and source of income. Imagine if you had upped the price, told the boss where to go and THEN had an issue like this come up.

I am just in the process of uploading a dummy image to the slot where the original has been lost so that I can delete it.
Reply With Quote
  #5  
Old 17-02-2020, 11:32 AM
Greg Bock's Avatar
Greg Bock (Greg Bock)
Registered User

Greg Bock is offline
 
Join Date: Aug 2006
Location: Gold Coast
Posts: 374
Well i must be one of the lucky ones..
I only have 10 images there, and they are all still OK. Only damage appears to be seven of the thumbnails are gone, but the original data is still accessible associated with them by clicking on the thumbnail with the sad face...
Let's see how he goes, he may recover data yet...
Reply With Quote
  #6  
Old 17-02-2020, 02:29 PM
raymo
Registered User

raymo is offline
 
Join Date: Sep 2011
Location: margaret river, western australia
Posts: 6,070
All my thumbnails are gone, but as with Greg, data and images are unaffected.
raymo
Reply With Quote
  #7  
Old 17-02-2020, 08:03 PM
skysurfer's Avatar
skysurfer
Dark sky rules !

skysurfer is offline
 
Join Date: Dec 2011
Location: 52N 6E (EU)
Posts: 1,152
Most of mine are also gone.
https://www.astrobin.com/users/skysurfer/


But luckily I have hosted them on a VPS rented by me.


https://www.skysurfer.eu/skypics1.php
Reply With Quote
  #8  
Old 17-02-2020, 08:47 PM
glend (Glen)
Registered User

glend is offline
 
Join Date: Jun 2013
Location: Lake Macquarie
Posts: 7,033
I just checked my account, and it seems I have lost every one of my 158 images in my Gallery, I have nothing left except the text associated with the descriptions of each one. For me this represents years of work, lost. Also I noticed that my account has been dropped down to the free account, from the Premium it used to be; and this is concerning because obviously there us no backup for the account status. I just paid in January for this year, and I will not be paying again. I also will not be wasting my time uploading 158 files and cleaning up his mess.
Luckily, I have all my files backed separetly to my PC hard drive, on a 3 Terabyte drive, including all my final RAW files after completion of processing, and the jpgs that were sent to Astrobin. However, it would be significant work to wade through all the jpg versions I might have for a particular one to fill that blank at Astrobin (kicking myself now for not developing an Astrobin file naming convention), but really I would not be uploading 158 new versions anyway.
I noticed that Astrobin is running dog slow as well, and that is not surprising given the people scrambling to have a look at the damage. Uploading again is probably going to be very dicey in the near term, if your considering that.

As to Salvatore's pitch for more money, which I declined, the timing is incredible. He is sure to lose subscribers.
Make sure you check your or entire gallery, would be my advice, not just recent stuff.
Reply With Quote
  #9  
Old 17-02-2020, 09:14 PM
glend (Glen)
Registered User

glend is offline
 
Join Date: Jun 2013
Location: Lake Macquarie
Posts: 7,033
BTW, my Astrobin account, has defaulted to the free account, which has very limited storage space. So Salvatore you need to fix up the account side of things before people can upload any lost files. I had just paid for a Premium account again in January this year, I expect that to be restored. I hope you have records of account status.
Reply With Quote
  #10  
Old 18-02-2020, 07:02 AM
CalvinKlein's Avatar
CalvinKlein (Kelvin)
Registered User

CalvinKlein is offline
 
Join Date: Aug 2018
Location: Cudgen NSW
Posts: 171
I've been in the I.T. industry 35 years and was doing "cloud" stuff long before the term became popular. I'm somewhat anti-cloud nowadays and quietly pitch my business that way. My observations from over the years are that businesses / services very often do not recover from catastrophic data loss like this, particularly where backups are corrupted or lost.

Even brick and mortar businesses that have total data & backup loss (or have to revert to very old data) have a very high failure rate within a year of getting back on their feet due to migration of customers to the competition whom often become predatory in times of disaster.

I sure hope Salvatore can resurrect the site to something resembling its former glory and maintain enough active clients to make it financially viable. From what I was reading in the Astrobin forum topic on this matter an Amazon tech support guy who subscribes to Astrobin was going to try to leverage his position to make things happen (and also his neighbour who is in the Amazon Web Services team). Salvatore admitted that he may have not read his terms and conditions properly with regards how his data / backups / versioning are retained. Fingers crossed he gets the high level back-end support he will need.
Reply With Quote
  #11  
Old 18-02-2020, 08:02 AM
multiweb's Avatar
multiweb (Marc)
ze frogginator

multiweb is offline
 
Join Date: Oct 2007
Location: Sydney
Posts: 22,060
I dislike cloud hosting because of hacking but AWS should be pretty bullet proof when it comes to redundancy and backup. It's a breeze to do a lot of things that are near impossible or very resource intensive with a more classic setup. If it's just a matter of technicality or SLA confusion because he didn't "tick the right checkbox" in his CP or assumed it was part of the service then I feel sorry for the guy. Having said that I reckon Amazon would keep a copy of everything that's content on their platforms internally so the data would have to be somewhere.

https://aws.amazon.com/ec2/pricing/on-demand/

https://aws.amazon.com/backup/
Reply With Quote
  #12  
Old 18-02-2020, 08:39 AM
CalvinKlein's Avatar
CalvinKlein (Kelvin)
Registered User

CalvinKlein is offline
 
Join Date: Aug 2018
Location: Cudgen NSW
Posts: 171
Quote:
Originally Posted by multiweb View Post
Having said that I reckon Amazon would keep a copy of everything that's content on their platforms internally so the data would have to be somewhere.
Agreed Marc - though I suspect that internal backup data may have a very short lifespan.

This post was from the Amazon worker ...

"Salvatore, I work for Amazon so happy to try to help. even if data was deleted in many cases AWS can recover it. "

"... in many cases ... " suggests its not a sure thing.

Here's the latest from Salvatore ...

"The current information that I have from Amazon's customer support is that the lost images are irrevocably, permanently lost. I'm still trying to escalate the issue as I have a hard time believing it, but it might as well be the truth."

and then this posted 3 minutes ago ...

"Senior engineering staff at Amazon is looking more deeply into the issues to see if they can salvage anything!"

Reply With Quote
  #13  
Old 18-02-2020, 11:14 AM
Greg Bock's Avatar
Greg Bock (Greg Bock)
Registered User

Greg Bock is offline
 
Join Date: Aug 2006
Location: Gold Coast
Posts: 374
Hmmm, looking good for me.
A quick check on my site shows all 10 thumbnails have returned, and all data appears to be OK. In a couple of cases, when I initially clicked on the main image, the sad face was displayed as though the data was lost, but once I clicked refresh, the data appeared OK. Perhaps he's having some success at restoration?
Reply With Quote
  #14  
Old 18-02-2020, 11:34 AM
Atmos's Avatar
Atmos (Colin)
Ultimate Noob

Atmos is offline
 
Join Date: Aug 2011
Location: Melbourne
Posts: 6,980
When it first happened some 70-80% of my images had sad faces. I checked again a few hours ago and only one had a sad face but I just had to manually refresh it.

So I've been one of the lucky ones that doesn't seem to have lost anything.
Reply With Quote
  #15  
Old 18-02-2020, 11:36 AM
raymo
Registered User

raymo is offline
 
Join Date: Sep 2011
Location: margaret river, western australia
Posts: 6,070
All of my images are back.
raymo
Reply With Quote
  #16  
Old 18-02-2020, 11:43 AM
Slawomir's Avatar
Slawomir (Suavi)
Registered User

Slawomir is offline
 
Join Date: Sep 2014
Location: North Queensland
Posts: 3,240
That’s a real shame. Holden yesterday, Astrobin today. All of my images are gone except for a few from 2013, as far as I can tell. I couldn’t be bothered uploading them all again - there were close to a hundred images and heaps of revisions too. A good reminder to get my act together and collate and organise all of my Astro data at home.
Reply With Quote
  #17  
Old 18-02-2020, 12:09 PM
Atmos's Avatar
Atmos (Colin)
Ultimate Noob

Atmos is offline
 
Join Date: Aug 2011
Location: Melbourne
Posts: 6,980
Quote:
Originally Posted by Slawomir View Post
That’s a real shame. Holden yesterday, Astrobin today. All of my images are gone except for a few from 2013, as far as I can tell. I couldn’t be bothered uploading them all again - there were close to a hundred images and heaps of revisions too. A good reminder to get my act together and collate and organise all of my Astro data at home.
I only have a handful of backups so it is a good reminder for me moving forward to keep copies of version I actually process rather than just the original data.
Reply With Quote
  #18  
Old 18-02-2020, 12:23 PM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
I seem to have largely escaped unscathed... worst I've seen so far is a missing sky plot image. I haven't gone and verified all my revisions, but the spot checks I performed suggest I was pretty lucky.

As has been said above, it's a good reminder to back up your data. I always have my "final" processed versions on my hard disk in addition to on Astrobin. I keep other data in various forms (sometimes PI projects, other times raw files, other times calibrated files) on an external hard drive. I'm thinking it would be prudent to get the stuff on the cheap external HDD into the cloud as well.
Reply With Quote
  #19  
Old 18-02-2020, 01:03 PM
Slawomir's Avatar
Slawomir (Suavi)
Registered User

Slawomir is offline
 
Join Date: Sep 2014
Location: North Queensland
Posts: 3,240
Good to hear that not everybody has been significantly affected. I just checked and 104 of my images, which is nearly all, are gone.
I still have them at various places at home, so no data has been lost, but it would take quite some time to rebuild the entire gallery on Astrobin, and I definitely don't have all revisions - but that is not really needed.
I wish Sal all the best and hopefully he will turn this event around and make Astrobin even better than it was before.
Reply With Quote
  #20  
Old 18-02-2020, 04:21 PM
glend (Glen)
Registered User

glend is offline
 
Join Date: Jun 2013
Location: Lake Macquarie
Posts: 7,033
I have gone through everything I had on Astrobin, and tried the steps suggested, but I have lost everything except the basic info comments I entered via text for each image. Uploading 158 images again and trying to match the files with the comments retained is just to hard. I am resigned that it can not be recovered.
I suggest everyone check their Astrobin account status, because my Premium account defaulted back to the free account (10 image limit), when i had just paid my Premium renewal in January. Salvatore probably can't tell who has paid what.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 01:39 AM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Advertisement
Testar
Advertisement
Bintel
Advertisement