Astrobin meltdown - Catastrophic data loss... - Page 2

CalvinKlein (Kelvin) · #21 18-02-2020, 05:35 PM

Quote:

Originally Posted by glend

I have gone through everything I had on Astrobin, and tried the steps suggested, but I have lost everything except the basic info comments I entered via text for each image. Uploading 158 images again and trying to match the files with the comments retained is just to hard. I am resigned that it can not be recovered.
I suggest everyone check their Astrobin account status, because my Premium account defaulted back to the free account (10 image limit), when i had just paid my Premium renewal in January. Salvatore probably can't tell who has paid what.

Glen I'm pretty sure I read earlier today (in one of the early forum pages) that all subscription information is intact or retrievable - it's just all in a bit of a shambles at the moment. It looks like he might have got some high powered Amazon tech support to help him out to recover the lost files. I'm also sure I read NOT to upload your old files just at the moment. I'd wait until at least tomorrow after he formally announces what went wrong and to what degree AWS tech support can recover the data.

Paulyman (Paul) · #22 18-02-2020, 07:23 PM

I just found out about this and have lost everything, including my avatar. While the originals are safe across multiple computers, nas drives and cloud services it will be a pain to reupload. I feel for those of you with 100’s of images lost.

Imme (Jon) · #23 19-02-2020, 06:26 AM

I see the Astrobin owner has released a ‘statement’ on what happened.
Am I right in saying at a very basic level that he pressed delete on nearly 5 terabyte of subscribers pictures because he didn’t know what he was doing???

The term ‘you had one job’ comes to mind.........

Sorry, but if this guy is collecting money to store people’s images then not understanding how the system works is inexcusable in my opinion. Yep, I agree we should all have backup copies of our images but that’s not the point.......he was being paid to provide a service and my expectation is that he would have the knowledge and skills to provide that service on a professional level.

If I pay a mechanic to change my oil I expect them to have the right tools (in the Astrobin case that means hardware), use the right oil (in Astrobin case that means right backup systems) and have the skills to carry out the work (in Astrobin case knowledge of what changing settings will do).

I like Astrobin and will keep using it but not too sure about taking out a new paid membership to provide someone a living for something it appears they are not qualified to do.

Disappointing for the owner and I feel for him.......but in reality if he had followed his own advice posted on the website (Astrobin encourages you to have have backups of all images posted) then there wouldn’t have been an issue.

.......simply saying “I didn’t understand what I signed on for with my amazon agreement’ doesn’t cut it in my opinion.

multiweb (Marc) · #24 19-02-2020, 07:45 AM

Yup. Simple user error. Lesson learnt I bet

The_bluester (Paul) · #25 19-02-2020, 08:55 AM

I seem to be among the very lucky ones, I just rechecked as I had not looked at image revisions, only if the finals were there (I don't have many with revisions anyway) and it seems that apart from the single lost image I found in my staging area I am unscathed. And my account has retained it's proper anniversary date.

CalvinKlein (Kelvin) · #26 19-02-2020, 09:40 AM

Quote:

Originally Posted by Imme

I see the Astrobin owner has released a ‘statement’ on what happened.
Am I right in saying at a very basic level that he pressed delete on nearly 5 terabyte of subscribers pictures because he didn’t know what he was doing???

The term ‘you had one job’ comes to mind.........

Sorry, but if this guy is collecting money to store people’s images then not understanding how the system works is inexcusable in my opinion. Yep, I agree we should all have backup copies of our images but that’s not the point.......he was being paid to provide a service and my expectation is that he would have the knowledge and skills to provide that service on a professional level.

If I pay a mechanic to change my oil I expect them to have the right tools (in the Astrobin case that means hardware), use the right oil (in Astrobin case that means right backup systems) and have the skills to carry out the work (in Astrobin case knowledge of what changing settings will do).

I like Astrobin and will keep using it but not too sure about taking out a new paid membership to provide someone a living for something it appears they are not qualified to do.

Disappointing for the owner and I feel for him.......but in reality if he had followed his own advice posted on the website (Astrobin encourages you to have have backups of all images posted) then there wouldn’t have been an issue.

.......simply saying “I didn’t understand what I signed on for with my amazon agreement’ doesn’t cut it in my opinion.

Yeah I'm pretty blown away by how easily this has happened too. Referring to your analogy I think the one-man-band mechanic forgot to tighten up the oil-drain bolt, your oil has drained from the engine driving along the highway and your engine's then cooked itself. Then the mechanic has told you to scoop up all the oil off the road and put it back into the engine.

I was under the impression Astrobin was some huge project run by a dozen or so programmers from around the world. It sure is one hell of an impressive website though !

I sincerely hope the owner can get the site back to its former glory and possibly improve on it - he seems to have overwhelming support from the Astro-community (though not a peep from any of the astro software / hardware vendors many of whom DO know about this from reading the cloudynights forums.)

It also highlights what I've known for many, many years - that websites and internet-stored data are far from 99.9999% secure. I've seen, and had to deal with, far too many serious website and remote hosting failures.

The_bluester (Paul) · #27 19-02-2020, 09:57 AM

I think rather than a lack of data security it stems from a misunderstanding of the product offer from AWS in the first instance. Depending on how the system is configured I am not actually sure that even if the hosting had the diversity Salvatore thought it did that it would have saved the site from this. If “deliberate” changes were replicated across all copies then it may still have happened.

I can’t comment or n the ease or otherwise of selecting the wrong part of the file store structure for the delete that was done, but can anyone working with large amounts of data say that they have never had that clammy moment when they thought they had just unexpectedly deleted the lot? Regardless of if they actually had or not. The difference here from the small company IT guy accidentally deleting the entire payroll database when he meant to delete a single ex employee is that this has been done in public.

The big issue IMO was not keeping abreast of the change in availability of services such as versioning and snapshots, which may have saved all our bacon here. Beyond saying that, I am keeping my mouth shut as I don’t much like being a demonstration of the Dunning-Kruger effect.

benklerk · #28 19-02-2020, 02:31 PM

I just found out about this and have lost all my pics, including my avatar.

I know it will take me some time to upload everything again . But I have always thought of Astrobin as a place to share your work not a permanent location to store your only copy images.

AndyG (Andy) · #29 19-02-2020, 02:51 PM

For interest, the statement from Salvatore regarding this:
https://docs.google.com/document/d/1...7wy4ScdMU/edit

I'm not an Astronbin user, but I think he's done well under the circumstances. Once I take an AP shot worth sharing, I feel I'd support/use this service.

I also have activities going with Amazon S3, and indeed, the UX could do with some work. There's some rather brutal data functions available with anaemic levels of detail on the dashboard...

Imme (Jon) · #30 19-02-2020, 03:22 PM

Quote:

Originally Posted by AndyG

For interest, the statement from Salvatore regarding this:
https://docs.google.com/document/d/1...7wy4ScdMU/edit

I'm not an Astronbin user, but I think he's done well under the circumstances. Once I take an AP shot worth sharing, I feel I'd support/use this service.

I also have activities going with Amazon S3, and indeed, the UX could do with some work. There's some rather brutal data functions available with anaemic levels of detail on the dashboard...

Understand your point re brutal functions..........but as I've mentioned in a previous post if you pay for professional service then I expect I'm paying a professional to provide the service.

It's no excuse him blaming a poor user interface or the fact he didn't understand what he was doing......which is scary when he has posted his day job is something in the software engineering field (at a high level if I'm not mistaken)

He's done some great work and I will continue to use the sight.....but that doesn't dismiss the fact he has caused a signifiacnt amount of work to a significant amount of people through a stupid mistake.

I truly hope Astrobin remains the goto place for astrophotographers......I like it

The_bluester (Paul) · #31 19-02-2020, 03:35 PM

That is where I would differ . Sub $50 a year for what amounts to data storage and a pretty handy back end that deals with plate solving and annotation plus a forum and image feed is not a "Professional" service in my eyes.

I really only ever used it as a handy place to host images for display elsewhere, with the peripheral benefit of plate solving and the ability to search out equipment. If I wanted or expected it to be a secure backup site I would expect to be paying a lot more to use it, and I would keep my own backups anyway.

I can however say I am writing this comfortable in the knowledge that everything I uploaded I still have on my PC and I could have rebuilt my gallery from my files in an hour or so if I had to. My only broken image I didn't bother with re uploading as it was a nothing image anyway.

etill (Elliot) · #32 19-02-2020, 04:05 PM

Quote:

Originally Posted by The_bluester

I think rather than a lack of data security it stems from a misunderstanding of the product offer from AWS in the first instance. Depending on how the system is configured I am not actually sure that even if the hosting had the diversity Salvatore thought it did that it would have saved the site from this. If “deliberate” changes were replicated across all copies then it may still have happened.

This is exactly right. It's unfortunate for peole that would like the assurance that a provider like this could undo a mistake, but for larger or heavily regulated customers this is a feature, not a weakness. If AWS were able to resotre something that was deleted from a magic backup they keep then they would lose a lot of business.

Our (and by that I mean my employer's) use of public cloud environments is predicated on their ability to prove that our data is under our own control, and that to a large extent the majority of the resources we consume are secured with a comparable level of control to how they would be in our own data centres. The law and various regulations in .au require it (depending on the industry that you are in). These services are designed to keep AWS (or Azure, or GCP) staff themselves from being able to access or control their customers' environments - unless that access is specifically requested or granted. It's an over simplification to state this 100% true in all cases, but its the goal at least.

Consider a regulated entity that had to ensure customer data was destroyed upon request - how could they use a public cloud provider like this if they weren't sure it was actually destroyed after they deleted it?

The other thing I thought was pretty decent is that he's owned the mistake, and been transparent about it from the start. Not every one will do that, it's easy to make something up and blame a provider - although if you tried to claim that an S3 failure caused actual data loss like this, there would have been photos on astrobin of the meteor approaching; outages sure, they happen now and then, but data loss across mutliple avaiability zones would require an absolutely catastrophic failure.

So I'd say technically now that he's been though this and gained the experience, the serivce is better than it was since I doubt he'll make that mistake again. I wouldn't want to lessen the loss people might be feeling or dismiss any anger about it, but he's learned the hard way how the environment works and it will be more resilient in the future because of that.

I only lost two images, and I'm not good at taking space photos, so it will be good practice to have to do the processing again (they'll probably be better this time anyway).

It would be entirely different if I had serious images up there or had spent years building up the collection, even with backups to upload that's a big loss. I really enjoyed browsing through a lot of those peoples images, so I hope they do put them back up.

AndyG (Andy) · #33 19-02-2020, 04:42 PM

Quote:

Originally Posted by etill

The other thing I thought was pretty decent is that he's owned the mistake, and been transparent about it from the start. Not every one will do that, it's easy to make something up and blame a provider

This was the bit that impressed me most. Amazon S3 didn't fail in the slightest. He deleted the files by a script, and versioning (offered by S3 in 2015 but not enabled/paid for) was not enabled. He documented this and apologised.

He may have had a slip of competence, but he's straight as an arrow by my standards.

codemonkey (Lee) · #34 19-02-2020, 07:12 PM

I stand corrected, looks like I have been impacted... I had some thumbnails not loading, but when I regenerated them they worked fine. I'm now finding that if I disable my cache, thumbnails are there, but after I navigate to full size some of my images fail to load and if I regenerate the thumbnails they stop working too.

Slawomir (Suavi) · #35 19-02-2020, 07:28 PM

Quote:

Originally Posted by codemonkey

I stand corrected, looks like I have been impacted... I had some thumbnails not loading, but when I regenerated them they worked fine. I'm now finding that if I disable my cache, thumbnails are there, but after I navigate to full size some of my images fail to load and if I regenerate the thumbnails they stop working too.

Just checked your 10 most recent images and all loaded just fine. Now, if you visit my gallery, nearly all images are gone, plus the most recent ones seem to have been lost completely, including the info..

glend (Glen) · #36 19-02-2020, 07:48 PM

Does anyone know a way to download all the text comments, basic info, gear used, etc, for the lost images? The information is still there, even though the images are gone, including the thumbnails. If I can recover my txt history at least I have that. The txt comments were originally entered from my capture/processing notes, which have not be kept over the years. In cases where I had uploaded say two images of the same object taken through different scopes or cameras, I need to be able to go back through my image files to find the correct image associated. I would like to save the txt info on my backup drive and link it to my image files.

codemonkey (Lee) · #37 19-02-2020, 08:04 PM

Quote:

Originally Posted by Slawomir

Just checked your 10 most recent images and all loaded just fine. Now, if you visit my gallery, nearly all images are gone, plus the most recent ones seem to have been lost completely, including the info..

Yeah, mine seem to be working again now, must have been something transitory in nature that had similar symptoms to the real issue.

Retrograde (Pete) · #38 20-02-2020, 10:07 AM

Sorry to hear that some of you have lost images. I seem to have been lucky and only lost my avatar.

As someone who used to work in the data storage industry, I know the empty feeling in the pit of your stomach when you've made a mistake that potentially destroyed data (so I can sympathise with Salvatore a little bit).

Hopefully there will be no future occurrences and those who have lost images will be able to recover them without too much difficulty.

glend (Glen) · #39 20-02-2020, 11:57 AM

I noticed that there is a "Goodbye Astrobin Community" thread on the Astrobin forum now, started by Andrea A. As was pointed out there, some will go and some will stay. In Andrea's case, she was pretty mad about it, and Salvatore's complicity. While I too will be leaving, once I get my txt info off the system, I realise we all probably expected too much from Salvatore and his low budget one man operation.
Whatever rebirth Astrobin goes through, it will be different, and lessons have been learned by all.
I wish I could thank all the viewers of my Gallery there, particularly those 100 followers who provided great moral support on my journey.

Benjamin (Ben) · #40 20-02-2020, 11:51 PM

I lost everything I had on Astrobin but not much in the way of quality or quantity, and of course I have all my originals on my computer. What I like about Astrobin is not that it stored my images well (I do that anyway) but rather that it has (or had) so many amazing high resolution images to compare my own images to that a quick search would soon reveal obvious or subtle flaws. With recent re-uploads I would say this is probably still the case. Understand anger, annoyance etc. as I also used my gallery as a kind of linked in, online, high resolution photo album, but to me it seems wiser to begin to regather the images that made the community special and functional. I feel like I’m paying Salvatore to facilitate a process rather than to just be a data manger, which is where my forgiveness lies.