site banner
ZorbaTHut  9mo ago (text post)   4707 thread views

Site Rollback to February 3rd

Very sorry about this one, guys.

For reasons that I'm currently unsure of, the database decided to eat itself. I'm giving it something like 40/40/20 "hacker", "postgres bug", or "host glitch".

The site currently has two different backup methods. The manual backup triggers whenever I do a site update; the last site update was on 1/31. The automatic backup is supposed to be daily, and I check it once in a while to make sure it's working. It's been working literally ever since before the main Motte site was launched . . . and it broke on 2/4. Good timing, thanks, system.

In theory, the quality-contribution system captured all reported quality contributions during this time. I'll try to retrieve those. Any opinions on whether I should just go ahead and repost them, or whether I should send them to the person so they can repost them?

I do have a dump of a bunch of text snippets that are all that was left of the database. If you remember some phrases you used I might be able to retrieve parts of lost posts. That said, someone's tried this with a few posts and got 0/2, so absolutely no promises here. Feel free to ask though! If you want to take data recovery more seriously, make a copy of your browser cache, which can then be pored over to find people's posts. I'm not totally sure how important this is, but I bet at least a few people will be sad to lose effortposts they made.

I've fixed the backup issue and set up better monitoring so it will yell at me if it fails again. I've also temporarily increased backup frequency to hourly, just in case there's some serious stability issue right now that I'm not aware of. The good news is that this shouldn't happen again, at least with as much lost data. But that doesn't really fix this one.

Apologies again.

This too shall pass.

17
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

If something is important, always back it up. Browsers can keep text fields filled in memory even after crashing and restarting, but having a text backup on hard drive is better.

Annoyed that the rollback seem to have entirely erased the new alt account I tried to make, BobbyDropTables.

Every now and again losing your precious internet points is good for the soul. Maybe it's too bad there was a backup at all.

Yeah, I don't understand those complaining about losing their posts. Just make them again. Yes a data loss is tragic, FOR THE DEVELOPER/OPS guy who let this shit happen, but it's a forum, for the user it should be a minor inconvenience, if this is a massive bummer to you, reconsider your forum usage habits.

Also can we post comments with Cmd+Enter ?

This website doesn't keep track of each user's total Internet points, though.

It probably still does, there just isn't a display.

It does. See here.

I have an idea for the killer app.

Now I just need someone to code it, finance it, and handle the business side of the operation.

Goddammit I wanted to expand on that Nietzsche post and now it's gone.

Here you go. Just search for your own username on that page.

I have the last few days 2/9 4am EST to 2/10 6pm EST of the feb 5th thread and most 2/12 6am EST to 2/13 5:30pm EST of this week's thread in open and loaded firefox tabs if anybody can tell me the best/easiest way to dump them

Pasted the raw text into a word doc in case microsoft decides now is a good time for an update but this obviously breaks formatting

Edit 2: microsoft did, in fact, decide it was time to update on the way out of hibernation after the battery died so I no longer have the tabs. The raw text was preserved however, and is now in the dump thread.

Feel free to dump raw text here. The expanded character limit is like 500k characters, I think.

Done.

Fffffffffffffff

I wrote a lot of truly excellent posts that are now gone, and that hits hard, but I just want my joke about buying chickens for $20 back. And I know that wasn't captured by the aaqc system because none of you motherfuckers are sophisticated enough to appreciate my genius.

New strategy: report yourself as AAQC, to save the best posts (don't actually do this)

The aaqc system is tied to the report system I think, so I assume most of my best posts are captured. But reporting wasn't the appropriate response to my $20 chickens joke, dying inside was.

check the wayback machine, internet archive, and the raw text dump thread. there's also a guy working on recovering posts from cache files on the discord.

Well, my plan to not enter an email address, memorize my password, and then stay signed in long enough that I forget the password completely was a smashing success. I suppose I’ll probably end up doing it again and becoming Lewis3 the next time this happens.

enter an email address or use a password manager

Easy come, easy go. The only thing I really lose by creating a new account is my saved comments and posts.

WaybackMachine made captures of themotte on the 13th.

So did archive.today.

Also, it is still possible to retrieve pages from Google's cache. I got my profile back that way, with a small effortpost that would otherwise have been lost. I recommend everyone see what they can save that way. But be quick; the cache refreshes periodically.

could you provide instructions on how to get one's profile back? I've saved my cache out, but have no idea how to read the files.

I had some tabs open, but they hibernated themselves and had to reload, so I don't have anything to show for it. I guess I do have the text from Goody-2, but that's not at all important.

My preference is that you just repost the quality contribution nominations, as otherwise some probably wouldn't get posted.

EDIT: Archive.org also has some snapshots, which is probably a lot more effective than trying to cobble things together otherwise.

I've uploaded raw copypasta of the two pages of comments I had handy here

If anyone else has tabs open or wants (very recent) posts, there'd be a good place to look.

If anyone's privately archiving every mottepost, now's the time to say that!

I've fixed the backup issue and set up better monitoring so it will yell at me if it fails again.

Important backups should also send notifications on success. Notification only on failure risks a scenario where both the backup and the notifications fail.

To be even safer, the script that sends the success notification should pull some independent confirmation the backup actually occurred, like the output of ls -l on the directory the database dumps are going to, and should include this in the notification text. Without this, a 'success' email only technically means that a particular point in a script was reached, not that a backup happened.

I've tried that before and what inevitably happens is I just end up ignoring the success notices.

In this case, however, I'm using healthchecks.io to handle this; it'll start pinging me on its own if it doesn't get regular notifications of success. So unless that service goes down, we're good.

To be ever safer, the script that sends the success notification should pull some independent confirmation the backup actually occurred, like the output of ls -l on the directory the database dumps are going to, and should include this in the notification text. Without this, a 'success' email only technically means that a particular point in a script was reached, not that a backup happened.

Ideally, yeah. In this case it's worth noting that it's taking full drive images, so it's, uh, kind of hard to do an ls. I guess I could run it as root and do a whole loopback thing to mount the image but I don't think that's likely to be necessary.

Allow me to significantly bump the “hacker” probability. At 11 or 12 UTC the site was in a state with no threads, no moderation log, and a single admin with an unfamiliar name whose account was created shortly before that.

Nah, that's exactly the expected consequence of the database vanishing - the site resets to Empty, and the first person who makes an account becomes admin.

Which doesn't mean there wasn't a hacker, just that it isn't evidence of there being a hacker.

Also, it means some poor guy picked exactly the wrong moment to make an account and accidentally discovered that they were now the admin of an empty site.

It's a pity then. I thought that it meant for sure that somebody else despises an attempt for the inter-tribe dialog as much as I do, but isn't prevented by a crippling depression from doing something about it.

Could you expand on this at all?

I was curious what happened this morning. Sorry to hear we lost almost two weeks of posts, but I suppose it's not the end of the world. Then again, I didn't lose a wall-o-text.

Thanks for the work you do.

Thanks for the work you do.

Seconded.

You're welcome :)

(still gonna feel bad about it though)

Oops happens.

I've got maybe five-dozen posts from the last couple days still open in my 100+ chrome tabs, including @coffee_enjoyer's "He Gets Us doesn't get it" post and some of the replies.

How does one make a copy of the browser cache? I'd be happy to give it a try if someone's got a tutorial.

Heck. I appreciated that whole sub thread.

indeed. I got your post in the raw dump, btw.

Ahh I was looking forward to replying to your response today! At least I think it was your reply, not 100% sure. I just copy/pasted the post to the CW, if you still have yours you should paste it.

How does one make a copy of the browser cache

Literally take the browser cache folder that gattsuru posted below and make a copy of it to somewhere else

done. any idea how to try to extract post data from it?

If you're on Windows, ChromeCacheView is the easiest way. It will default to the current cache, but clicking the stop button in the bottom left, and then click File-> Select Cache Folder will allow you to select any other folder, and it'll look in that folder and descending folders for cache data.

There's (too many) filter options, but the easiest combo is to enable View -> Use Quick Filter, then type themotte.org into the Quick Filter textbox that shows up. Select the ContentType column header to sort by type, and look for the text/html files. Shift-select the ones you want to export, and then use the right-click Copy Selected Cache Files to... copy them into a directory you can find and read through. They'll then be accessible like normal files.

tried this, and got lots of images and random chunks of html code, but no readable text. a lot of the entries give a 404, or say the content was removed.

If you know how to use a terminal, grep for anything with the string 'motte' in it and then zip those files up and upload them to the dev discord i think is the easiest. It's in a bit of a weird format

specifically 'rg -a -uuu -l motte'

There's a few people on the dev discord talking about it, and it appears to be Actually Kind Of Difficult. Honestly, copy-pasting those posts out would be the easiest solution.

I am desperately in need of sleep and will be tackling the AAQC recovery tomorrow.

Did you see that the wayback machine has a lot of it?

Yeah, there's someone on the dev Discord who did a big dump. I'm currently pinging the mods to make sure there's no strong reason not to post this, and otherwise I'll probably be posting a big chunky JSON archive of everything so people can repost if they want.

(edit: posted)

That would be awesome, because I actually really loved your reply to that post. It was top notch and I would be sad to see it go.

sadly, the page with my reply on it was the one I refreshed.

If you're running Firefox, your browser cache will be located at C:\Users\<YourUserName>\AppData\Local\Mozilla\Firefox\Profiles\<YourProfileName>\cache2. On Windows, the easiest way to access it and filter files is to use MZCacheView.

For Chrome, it's C:\Users\<YourUserName>\AppData\Local\Google\Chrome\User Data\<YourProfileName/Default>\Cache\Cache_Data, and ChromeCacheView works pretty well in Windows.

I haven't tried any Linux or MacOSX equivalents yet; most I've found are Python2-based and likely out-of-date, and while there are reports of getting these apps to work under Wine, I haven't had need before.

In either case, depending on your system settings, cache can get cleaned out surprisingly quickly, so if there are specific posts you're look for, at least copy these directories to new locations sooner rather than later.

Unfortunately, I didn't have TheMotte on my archiving list, so I don't have much in the way of local copies of other people's posts myself. Not sure if anyone else has been more aggressive here.

Test

Ehm... Should i be logged in as you?

That was me. I didn't do anything, I was just logged in as you when I went to the website.

Maybe shut things down until you figure this out? It doesn't seem like a good idea for people to automatically be logged in as head admin.

@ZorbaTHut

Oy. That doesn't surprise me, honestly, I should probably clear logins just in case weird stuff happened.

Offline again, back in a few :V

Can't tell if serious security bug, or Zorba is having a schizophrenic moment.

Like in Life of Pi, I choose to believe it is actually both because that's a much more interesting world to be living in.

I can think of a few ways the site could go kind of weird with regards to login tokens during an incident like that. Easiest way to solve it is just to reset the login signature code.

Now everyone gets to log in again also, which inevitably means I'll get like six emails from people who entered a random password and didn't bother to save it and also didn't set their email so they could recover it.

I don't know if you're all still on friendly terms with Aevann but this [site issue followed by people being logged in as the main admin] was common in the first six months or so of rdrama, there's likely documentation and a path to assistance.

Nah, this was honestly expected - the login system is based on user ID, and if the user IDs get reset and you get a token for a now-available user ID, that token is still valid once the site reboots. No particularly great mystery.

The session tokens identify logged-in user by user ID. This is cryptographically signed serverside. The first person to make an account on the empty site got the server to give them a valid signed session token for UID 9, which persisted after restoration. Anyway, changing the signing key like you did fixes it.

Yeah, it's a little weird because this implies that @loper happened to be the one who made the admin account, and I'm not convinced they're the one who made the admin account because I think they would have mentioned it.

(loper are you the one who made the admin account when the database melted a bunch of hours? if so, that answers that question, I guess)

(also, if so, I'm so curious what you thought was happening)

I might have. I tried to visit the site earlier today and couldn't view anything or log in so I tried creating an account but that didn't change anything. But this was like 10 hours ago. I still wasn't "let in" and I got a 503 service unavailable error until now, at which point I was logged in as you.

The only thing is that is perhaps a bit strange was that the empty site persisted for pretty long and I wasn't particularly quick starting to try different ways to get in. I've created test accounts previously when the site has had issues and this has never happened before.

I think that name was Kaldrop. And if not, what I remember is that a web search for the name should also return something about services for e-commerce sellers.

More comments

What was the name of the account you created? I saw the new admin account, I can't remember its name exactly, but I can tell if something is that name or not.

Huh. That actually might've been you, then, yeah.

This event really made me realize how much time I spend lurking on the Motte. I should do something else with my time.

If you post more instead of lurking you get to trick yourself into thinking you’re Engaging With The Discourse instead of just “wasting time”.

A good way to start Lent.

Your admin invite is on the way.

Post more, obviously.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats