Outage Postmortem - 2021/08/12
Published Aug. 12, 2021, 6:49 p.m.
Unplanned Outage for TranscribersOfReddit
Incident date: 2021/08/12
Summary:
u/TranscribersOfReddit was unresponsive for approximately 12.5 hours, completely stalling all movement from volunteers. General instability from Reddit.com caused a cascading failure where a submission was not created in Blossom appropriately and the edge case code designed to catch this was not set up correctly.
User Impact:
Complete stoppage of work. Nothing was completed during this time.
Incident Response Analysis:
At approximately 3:45am UTC, our team heard from a volunteer that u/TranscribersOfReddit (the bot) was not responding to a message. …
To Infinity And Beyond: Redesigning from the Ground Up
Published June 11, 2021, 4:39 a.m.
Every group likes making incremental changes to fix issues and tech debt... but sometimes the best possible option is to burn it all to the ground and start anew. Seeing as we've just recently finished doing that very thing, this write up will cover:
- the old infrastructure
- why we decided to migrate
- the new infrastructure
- the results of migrating
Buckle in, because this is a long ride!
The Old
The thing about tech debt is that, more often than not, it creeps up on you. You have …
I give myself very good advice, but I very seldom follow it.
