Astrobin meltdown - Catastrophic data loss...

[Andy01]

D'oh! :sadeyes: https://www.astrobin.com/users/Andys_Astropix/

Dear all,
it appears that something serious happened on AstroBin today. I've now been investigating and trying to mitigate for 12 hours straight, but unfortunately I am certain that a currently unknown number of you have suffered the loss of some (many, in some cases) original uploads.

The data associated to images, on AstroBin database (which is a different system than the place where the image files are stored), is safe.
I don't know the reason yet, but in the meanwhile I have changed all the passwords used by AstroBin (database, storage, servers, etc). Note that this has nothing to do with your passwords you use to log in to AstroBin: they are encrypted.

I will update with my findings on this forum thread:
https://www.astrobin.com/forum/c/astrobin/annoucements/currently-investigating-a-problem-with-availability-of-thumbnails/

For now if you see the message "We are very, very sorry, but it appears that your image suffered a catastrophic data corruption", then that image is gone. Luckily you don't need to start over and lose the data, the comments, likes etc. You only need to replace the image file on that form, and hopefully you still have it on your computer.

Please visit your gallery after log-in, and the image thumbnails with the sad face are affected. If you are not logged-in, affected images will be hidden.
I can't even begin to explain how much I'm currently in shock, literally, and this is why this message is intentionally brief.
I will continue to investigate, and I’m awaiting help from the Amazon Web Services tech support.

In the meantime, if you are heavily affected, I want to say that I am incredibly sorry, though at this time I don’t know where the fault lies.
After the situation settles and I know more, I will offer compensation for the data loss with free subscription upgrades.

Thank you for your understanding and please keep an eye on that forum post for update.

Thanks!
Salvatore

[The_bluester]

Looks like 50% of my public images and 100% loss in my staging area. I don't think I could be bothered working out which was which and re uploading them, I reckon I will just delete what is lost, some are images quite early in my AP journey and I may not have even bothered to keep the originals.

Edit. I appear to be among the lucky ones, I have only lost one original image, the rest were dead thumbnails. Regardless of that I think it will take a fair while for Astrobin to rebuild trust, particularly as a paid service.

[multiweb]

I quickly browsed through the status report and update thread with the updates and that sounds like déjà vu having been in the same situation years ago. In my case a room containing high voltage electrical exploded. So it was physical damage. Bit extreme I know but the result is the same. So you're stuck between a rock and a hard place. It's not the astrobin guy's fault. He's got data corruption on his HD. He pays somebody to look after the servers (amazon I believe) so they should have redundancy in place for him. Truth is that it never works in the real world. There's always SLA and fine prints. If you don't do your own back up plan then you're left hanging like a piñata with everybody having a shot at you while you're still trying to fix stuff.

[The_bluester]

As hard a sentiment as this sounds, the timing is probably good for Salvatore given the recent thread about him sounding out users about a price rise and making it his primary job and source of income. Imagine if you had upped the price, told the boss where to go and THEN had an issue like this come up.

I am just in the process of uploading a dummy image to the slot where the original has been lost so that I can delete it.

[Greg Bock]

Well i must be one of the lucky ones..
I only have 10 images there, and they are all still OK. Only damage appears to be seven of the thumbnails are gone, but the original data is still accessible associated with them by clicking on the thumbnail with the sad face...
Let's see how he goes, he may recover data yet...

[raymo]

All my thumbnails are gone, but as with Greg, data and images are unaffected.
raymo

[skysurfer]

Most of mine are also gone.
https://www.astrobin.com/users/skysurfer/


But luckily I have hosted them on a VPS rented by me.


https://www.skysurfer.eu/skypics1.php

[glend]

I just checked my account, and it seems I have lost every one of my 158 images in my Gallery, I have nothing left except the text associated with the descriptions of each one. For me this represents years of work, lost. Also I noticed that my account has been dropped down to the free account, from the Premium it used to be; and this is concerning because obviously there us no backup for the account status. I just paid in January for this year, and I will not be paying again. I also will not be wasting my time uploading 158 files and cleaning up his mess.
Luckily, I have all my files backed separetly to my PC hard drive, on a 3 Terabyte drive, including all my final RAW files after completion of processing, and the jpgs that were sent to Astrobin. However, it would be significant work to wade through all the jpg versions I might have for a particular one to fill that blank at Astrobin (kicking myself now for not developing an Astrobin file naming convention), but really I would not be uploading 158 new versions anyway.
I noticed that Astrobin is running dog slow as well, and that is not surprising given the people scrambling to have a look at the damage. Uploading again is probably going to be very dicey in the near term, if your considering that.

As to Salvatore's pitch for more money, which I declined, the timing is incredible. He is sure to lose subscribers.
Make sure you check your or entire gallery, would be my advice, not just recent stuff.

[glend]

BTW, my Astrobin account, has defaulted to the free account, which has very limited storage space. So Salvatore you need to fix up the account side of things before people can upload any lost files. I had just paid for a Premium account again in January this year, I expect that to be restored. I hope you have records of account status.

[CalvinKlein]

I've been in the I.T. industry 35 years and was doing "cloud" stuff long before the term became popular. I'm somewhat anti-cloud nowadays and quietly pitch my business that way. My observations from over the years are that businesses / services very often do not recover from catastrophic data loss like this, particularly where backups are corrupted or lost.

Even brick and mortar businesses that have total data & backup loss (or have to revert to very old data) have a very high failure rate within a year of getting back on their feet due to migration of customers to the competition whom often become predatory in times of disaster.

I sure hope Salvatore can resurrect the site to something resembling its former glory and maintain enough active clients to make it financially viable. From what I was reading in the Astrobin forum topic on this matter an Amazon tech support guy who subscribes to Astrobin was going to try to leverage his position to make things happen (and also his neighbour who is in the Amazon Web Services team). Salvatore admitted that he may have not read his terms and conditions properly with regards how his data / backups / versioning are retained. Fingers crossed he gets the high level back-end support he will need.

[multiweb]

I dislike cloud hosting because of hacking but AWS should be pretty bullet proof when it comes to redundancy and backup. It's a breeze to do a lot of things that are near impossible or very resource intensive with a more classic setup. If it's just a matter of technicality or SLA confusion because he didn't "tick the right checkbox" in his CP or assumed it was part of the service then I feel sorry for the guy. Having said that I reckon Amazon would keep a copy of everything that's content on their platforms internally so the data would have to be somewhere.

https://aws.amazon.com/ec2/pricing/on-demand/

https://aws.amazon.com/backup/

[CalvinKlein]

Agreed Marc - though I suspect that internal backup data may have a very short lifespan.

This post was from the Amazon worker ...

"Salvatore, I work for Amazon so happy to try to help. even if data was deleted in many cases AWS can recover it. "

"... in many cases ... " suggests its not a sure thing.

Here's the latest from Salvatore ...

"The current information that I have from Amazon's customer support is that the lost images are irrevocably, permanently lost. I'm still trying to escalate the issue as I have a hard time believing it, but it might as well be the truth."

and then this posted 3 minutes ago ...

"Senior engineering staff at Amazon is looking more deeply into the issues to see if they can salvage anything!"

:(

[Greg Bock]

Hmmm, looking good for me.
A quick check on my site shows all 10 thumbnails have returned, and all data appears to be OK. In a couple of cases, when I initially clicked on the main image, the sad face was displayed as though the data was lost, but once I clicked refresh, the data appeared OK. Perhaps he's having some success at restoration?

[Atmos]

When it first happened some 70-80% of my images had sad faces. I checked again a few hours ago and only one had a sad face but I just had to manually refresh it.

So I've been one of the lucky ones that doesn't seem to have lost anything.

[raymo]

All of my images are back.:)
raymo

[Slawomir]

That’s a real shame. Holden yesterday, Astrobin today. All of my images are gone except for a few from 2013, as far as I can tell. I couldn’t be bothered uploading them all again - there were close to a hundred images and heaps of revisions too. A good reminder to get my act together and collate and organise all of my Astro data at home.

[Atmos]

I only have a handful of backups so it is a good reminder for me moving forward to keep copies of version I actually process rather than just the original data.

[codemonkey]

I seem to have largely escaped unscathed... worst I've seen so far is a missing sky plot image. I haven't gone and verified all my revisions, but the spot checks I performed suggest I was pretty lucky.

As has been said above, it's a good reminder to back up your data. I always have my "final" processed versions on my hard disk in addition to on Astrobin. I keep other data in various forms (sometimes PI projects, other times raw files, other times calibrated files) on an external hard drive. I'm thinking it would be prudent to get the stuff on the cheap external HDD into the cloud as well.

[Slawomir]

Good to hear that not everybody has been significantly affected. I just checked and 104 of my images, which is nearly all, are gone.
I still have them at various places at home, so no data has been lost, but it would take quite some time to rebuild the entire gallery on Astrobin, and I definitely don't have all revisions - but that is not really needed.
I wish Sal all the best and hopefully he will turn this event around and make Astrobin even better than it was before.

[glend]

I have gone through everything I had on Astrobin, and tried the steps suggested, but I have lost everything except the basic info comments I entered via text for each image. Uploading 158 images again and trying to match the files with the comments retained is just to hard. I am resigned that it can not be recovered.
I suggest everyone check their Astrobin account status, because my Premium account defaulted back to the free account (10 image limit), when i had just paid my Premium renewal in January. Salvatore probably can't tell who has paid what.

[CalvinKlein]

Glen I'm pretty sure I read earlier today (in one of the early forum pages) that all subscription information is intact or retrievable - it's just all in a bit of a shambles at the moment. It looks like he might have got some high powered Amazon tech support to help him out to recover the lost files. I'm also sure I read NOT to upload your old files just at the moment. I'd wait until at least tomorrow after he formally announces what went wrong and to what degree AWS tech support can recover the data.

[Paulyman]

I just found out about this and have lost everything, including my avatar. While the originals are safe across multiple computers, nas drives and cloud services it will be a pain to reupload. I feel for those of you with 100’s of images lost.

[Imme]

I see the Astrobin owner has released a ‘statement’ on what happened.
Am I right in saying at a very basic level that he pressed delete on nearly 5 terabyte of subscribers pictures because he didn’t know what he was doing???

The term ‘you had one job’ comes to mind.........

Sorry, but if this guy is collecting money to store people’s images then not understanding how the system works is inexcusable in my opinion. Yep, I agree we should all have backup copies of our images but that’s not the point.......he was being paid to provide a service and my expectation is that he would have the knowledge and skills to provide that service on a professional level.

If I pay a mechanic to change my oil I expect them to have the right tools (in the Astrobin case that means hardware), use the right oil (in Astrobin case that means right backup systems) and have the skills to carry out the work (in Astrobin case knowledge of what changing settings will do).

I like Astrobin and will keep using it but not too sure about taking out a new paid membership to provide someone a living for something it appears they are not qualified to do.

Disappointing for the owner and I feel for him.......but in reality if he had followed his own advice posted on the website (Astrobin encourages you to have have backups of all images posted) then there wouldn’t have been an issue.

.......simply saying “I didn’t understand what I signed on for with my amazon agreement’ doesn’t cut it in my opinion.

[multiweb]

Yup. Simple user error. Lesson learnt I bet
:)

[The_bluester]

I seem to be among the very lucky ones, I just rechecked as I had not looked at image revisions, only if the finals were there (I don't have many with revisions anyway) and it seems that apart from the single lost image I found in my staging area I am unscathed. And my account has retained it's proper anniversary date.

[CalvinKlein]

Yeah I'm pretty blown away by how easily this has happened too. Referring to your analogy I think the one-man-band mechanic forgot to tighten up the oil-drain bolt, your oil has drained from the engine driving along the highway and your engine's then cooked itself. Then the mechanic has told you to scoop up all the oil off the road and put it back into the engine.

I was under the impression Astrobin was some huge project run by a dozen or so programmers from around the world. It sure is one hell of an impressive website though !

I sincerely hope the owner can get the site back to its former glory and possibly improve on it - he seems to have overwhelming support from the Astro-community (though not a peep from any of the astro software / hardware vendors many of whom DO know about this from reading the cloudynights forums.)

It also highlights what I've known for many, many years - that websites and internet-stored data are far from 99.9999% secure. I've seen, and had to deal with, far too many serious website and remote hosting failures.

[The_bluester]

I think rather than a lack of data security it stems from a misunderstanding of the product offer from AWS in the first instance. Depending on how the system is configured I am not actually sure that even if the hosting had the diversity Salvatore thought it did that it would have saved the site from this. If “deliberate” changes were replicated across all copies then it may still have happened.

I can’t comment or n the ease or otherwise of selecting the wrong part of the file store structure for the delete that was done, but can anyone working with large amounts of data say that they have never had that clammy moment when they thought they had just unexpectedly deleted the lot? Regardless of if they actually had or not. The difference here from the small company IT guy accidentally deleting the entire payroll database when he meant to delete a single ex employee is that this has been done in public.

The big issue IMO was not keeping abreast of the change in availability of services such as versioning and snapshots, which may have saved all our bacon here. Beyond saying that, I am keeping my mouth shut as I don’t much like being a demonstration of the Dunning-Kruger effect.

[benklerk]

I just found out about this and have lost all my pics, including my avatar.



I know it will take me some time to upload everything again . But I have always thought of Astrobin as a place to share your work not a permanent location to store your only copy images.

[AndyG]

For interest, the statement from Salvatore regarding this:
https://docs.google.com/document/d/1PzdYQQpR4LueJUMnbyi162xQLAYkr3MiU_ 7wy4ScdMU/edit

I'm not an Astronbin user, but I think he's done well under the circumstances. Once I take an AP shot worth sharing, I feel I'd support/use this service.

I also have activities going with Amazon S3, and indeed, the UX could do with some work. There's some rather brutal data functions available with anaemic levels of detail on the dashboard...

[Imme]

Understand your point re brutal functions..........but as I've mentioned in a previous post if you pay for professional service then I expect I'm paying a professional to provide the service.

It's no excuse him blaming a poor user interface or the fact he didn't understand what he was doing......which is scary when he has posted his day job is something in the software engineering field (at a high level if I'm not mistaken)

He's done some great work and I will continue to use the sight.....but that doesn't dismiss the fact he has caused a signifiacnt amount of work to a significant amount of people through a stupid mistake.

I truly hope Astrobin remains the goto place for astrophotographers......I like it

[The_bluester]

That is where I would differ . Sub $50 a year for what amounts to data storage and a pretty handy back end that deals with plate solving and annotation plus a forum and image feed is not a "Professional" service in my eyes.

I really only ever used it as a handy place to host images for display elsewhere, with the peripheral benefit of plate solving and the ability to search out equipment. If I wanted or expected it to be a secure backup site I would expect to be paying a lot more to use it, and I would keep my own backups anyway.

I can however say I am writing this comfortable in the knowledge that everything I uploaded I still have on my PC and I could have rebuilt my gallery from my files in an hour or so if I had to. My only broken image I didn't bother with re uploading as it was a nothing image anyway.

[etill]

This is exactly right. It's unfortunate for peole that would like the assurance that a provider like this could undo a mistake, but for larger or heavily regulated customers this is a feature, not a weakness. If AWS were able to resotre something that was deleted from a magic backup they keep then they would lose a lot of business.

Our (and by that I mean my employer's) use of public cloud environments is predicated on their ability to prove that our data is under our own control, and that to a large extent the majority of the resources we consume are secured with a comparable level of control to how they would be in our own data centres. The law and various regulations in .au require it (depending on the industry that you are in). These services are designed to keep AWS (or Azure, or GCP) staff themselves from being able to access or control their customers' environments - unless that access is specifically requested or granted. It's an over simplification to state this 100% true in all cases, but its the goal at least.

Consider a regulated entity that had to ensure customer data was destroyed upon request - how could they use a public cloud provider like this if they weren't sure it was actually destroyed after they deleted it?

The other thing I thought was pretty decent is that he's owned the mistake, and been transparent about it from the start. Not every one will do that, it's easy to make something up and blame a provider - although if you tried to claim that an S3 failure caused actual data loss like this, there would have been photos on astrobin of the meteor approaching; outages sure, they happen now and then, but data loss across mutliple avaiability zones would require an absolutely catastrophic failure.

So I'd say technically now that he's been though this and gained the experience, the serivce is better than it was since I doubt he'll make that mistake again. I wouldn't want to lessen the loss people might be feeling or dismiss any anger about it, but he's learned the hard way how the environment works and it will be more resilient in the future because of that.

I only lost two images, and I'm not good at taking space photos, so it will be good practice to have to do the processing again (they'll probably be better this time anyway).

It would be entirely different if I had serious images up there or had spent years building up the collection, even with backups to upload that's a big loss. I really enjoyed browsing through a lot of those peoples images, so I hope they do put them back up.

[AndyG]

This was the bit that impressed me most. Amazon S3 didn't fail in the slightest. He deleted the files by a script, and versioning (offered by S3 in 2015 but not enabled/paid for) was not enabled. He documented this and apologised.

He may have had a slip of competence, but he's straight as an arrow by my standards.

[codemonkey]

I stand corrected, looks like I have been impacted... I had some thumbnails not loading, but when I regenerated them they worked fine. I'm now finding that if I disable my cache, thumbnails are there, but after I navigate to full size some of my images fail to load and if I regenerate the thumbnails they stop working too.

[Slawomir]

Just checked your 10 most recent images and all loaded just fine. Now, if you visit my gallery, nearly all images are gone, plus the most recent ones seem to have been lost completely, including the info..:abduct:

[glend]

Does anyone know a way to download all the text comments, basic info, gear used, etc, for the lost images? The information is still there, even though the images are gone, including the thumbnails. If I can recover my txt history at least I have that. The txt comments were originally entered from my capture/processing notes, which have not be kept over the years. In cases where I had uploaded say two images of the same object taken through different scopes or cameras, I need to be able to go back through my image files to find the correct image associated. I would like to save the txt info on my backup drive and link it to my image files.

[codemonkey]

:( Yeah, mine seem to be working again now, must have been something transitory in nature that had similar symptoms to the real issue.

[Retrograde]

Sorry to hear that some of you have lost images. I seem to have been lucky and only lost my avatar.

As someone who used to work in the data storage industry, I know the empty feeling in the pit of your stomach when you've made a mistake that potentially destroyed data (so I can sympathise with Salvatore a little bit).

Hopefully there will be no future occurrences and those who have lost images will be able to recover them without too much difficulty.

[glend]

I noticed that there is a "Goodbye Astrobin Community" thread on the Astrobin forum now, started by Andrea A. As was pointed out there, some will go and some will stay. In Andrea's case, she was pretty mad about it, and Salvatore's complicity. While I too will be leaving, once I get my txt info off the system, I realise we all probably expected too much from Salvatore and his low budget one man operation.
Whatever rebirth Astrobin goes through, it will be different, and lessons have been learned by all.
I wish I could thank all the viewers of my Gallery there, particularly those 100 followers who provided great moral support on my journey.

[Benjamin]

I lost everything I had on Astrobin but not much in the way of quality or quantity, and of course I have all my originals on my computer. What I like about Astrobin is not that it stored my images well (I do that anyway) but rather that it has (or had) so many amazing high resolution images to compare my own images to that a quick search would soon reveal obvious or subtle flaws. With recent re-uploads I would say this is probably still the case. Understand anger, annoyance etc. as I also used my gallery as a kind of linked in, online, high resolution photo album, but to me it seems wiser to begin to regather the images that made the community special and functional. I feel like I’m paying Salvatore to facilitate a process rather than to just be a data manger, which is where my forgiveness lies.

[Ukastronomer]

Nothing is catastrophic unless you don't have originals and upload images to the interwebnet! and delete the originals, which I have never done and never will


.

[tim.anderson]

I lost nothing other than my cover photo.

[Paul Haese]

This is the main reason I pay for hosting my own website. I have the entire copy of the site stored in three different places. And; as they say in data storage, "it's not stored unless it is stored in three separate places".

Whilst I am not an Astrobin member, I have several times thought about just putting all my data on to a site like Smugmug. I still have terabytes of data stored in three places and would continue to do so, but I will probably never go to Astrobin now after witnessing this rookie error. I feel sorry for the guy but it was not a great idea to admit you made a rookie mistake out in the open, nor was it a good idea to do something like that without checking first what the implications would be of the deleting data. Especially after recently wanting to up the price of subscriptions.

I think I agree with someone else's analysis; this might well be the end of Astrobin. People do use these places as storage facilities and most likely don't have copies of their own or if they do, it might well be hundreds of images to sort through and starting again will be pain to hard to bear.

Time will tell....

[gregbradley]

Never used Astrobin but I think its a cool site. I was thinking about possibly using it one day.

I do keep copies on external HDD but they can be unreliable (especially if you leave them in a car and get a 40C day).

Hopefully its not too disruptive for users to repair their sites.

I use PBase and it seems pretty reliable but its not an astro community which I see Astrobin has become.

Greg.

[CalvinKlein]

Well I have now been upgraded from the free account to premium (even though my files were automatically recovered somehow).

I had previously decided against going premium as I am currently developing my own website as a sub-domain of my business website, so it costs me no extra money beyond the $19 a month I pay for the business site.

I have been lurking around Astrobin for a long time and admiring others images (and taking screen captures of other people's camera / processing settings for my own education) - only recently did I start posting images to it so I could share the links to full sized images on IceinSpace.

My main concern now is that Salvatore not only has a one-off financial hit for the repair / upgrade / extra short-term capacity of Astrobin's web-hosting but he's now also lost quite a bit of future income by offering quite generous compensation.

I think at some point this year he is going to have put his hand out for donations and/or sponsorship. Given the strong support he has had towards this crisis I'm sure he would get plenty of each.

[CalvinKlein]

I think external hard disks have become significantly less reliable in the last few years. They seem to me to be more susceptible to heat and knocks and bad electrical connections (less gold on the USB connection cables ?? )

All my business customers have at least two alternating offsite USB backups in addition to other backups / archiving (e.g. NAS devices & internet-based storage) - and they are replaced every three years maximum. I still believe portable USB disks are an essential part of any backup strategy despite what the online backup vendors might say. Three of my bigger clients have recovered from large scale ransomware attacks thanks to physically disconnected USB disks !

One thing to be aware of with old disk archives is the concept of "bitrot" where data on hard disks gets silently corrupted and you have no idea its happened. That corrupt data can then be propagated to new disks, or to something like dropbox, and unless you are viewing the photo or video you may not know its corrupted.

For this reason I have my main data repository of "everything" on a small windows server and I run a cheap software application called Fastsum on a regular basis. It initially creates checksums of all the data files on the server, one checksum file per folder.

Then at regular intervals I re-run the software, it recalculates the checksums and compares them to the previous values (creating new checksums for added files). This methodology will highlight any corrupted / changed / deleted data files for you to review. if any files are genuinely corrupted you can restore them from other 'known good' backups such as NAS / USB / internet storage.

Some newer operating systems have this checksum capability built in to the operating system but this manual method using Fastsum is sufficient for my business and personal data and client documentation.

[glend]

I have received an email from Salvatore, advising he is upgrading all account holders to Premium free for one year. I suspect he may have lost the information regarding subscriptions. My account was Premium until the crash, and defaulted back to the basic free 10 image account. I had asked him to re-establish my account via a separate email earlier.

[Slawomir]

That’s great news Glen.

In my case however, since 104 of my images were gone and at least 10 lost all technical info, and because my images are all over the place at home, I thought it will be just easier to delete my Astrobin account, particularly, that just leaving my Astrobin account in a mess also didn’t feel right - so deleting seemed like a better option. For now, I will continue with pbase, as I already had an account there.

[glend]

Well Suavi, I have not decided to continue with Astrobin, as my 158 images are still gone, and I will need to do substantial work to track down exactly which of my many files were actually originally uploaded to Astrobin and in which Gallery slot. The only clue Astrobin seems to give me is date/time stamp when the original file was uploaded.i know that the files I need to find are jpgs so that is a start, so jpgs created around the Astrobin date might be the search criteria. I just dread the task and the rebuild of my Gallery.