It was day 1 of vacation. Teachers still ran Mac Quattro's or Windows 3.1x systems, Many people were still computer illiterate, and some even go as far as having their pre-teen children talk to us because of how clueless they are. Klara tells Cole it’s good to have space but sometimes … The engineer rolled out of bed and checked his email. Did Reddit achieve the impossible? card. MORE: Borderlands: 5 Saddest Deaths in Series History Source: Reddit Anyways, my boss assured me that two of our partners were going to contact me to walk me through the process, and that I had nothing to worry about.... right. Instead someone just left the cabinet open and informed their supervisor that the problem was solved. save. This brief story fits a "StoriesAboutKeven", "TalesfromCallcenter" and what I think is the best fit "Talesfromtechsuport". After some conversations, I agreed to fly out to this new site to mount some switches, firewalls, and servers to get the network up and running. There was one cabinet full of industrial control systems and network switching gear in that rarely visited area. What's The Most Horrific Thing You've Found On Someone's Computer? card classic compact. Posted by 10 hours … Press question mark to learn the rest of the keyboard shortcuts. Hot New Top. Walk in without masks like fools (they’re required in our state and sign is posted on door). Those don’t even really matter.” He went ahead with the deploy, hastily jamming them into an already overloaded function, eager to get the change out before the 4-day weekend. “Hrm, that can’t be good,” he thought. The counts all matched up, and so it was time to get started. ** We've got a bit of a lull in the queue just now, so kick back, grab … There was a database error. I'm writing this on a PC. With her detective hat on, she got to work on identifying the root issues. I also want to point out that this would be my first time ever mounting this stuff on a rack. u/AutoModerator. That thing had been there since the 1980's and saw many modifications (usually with more stuff being added in). Time was running out, there were only two weeks before the old crash reporting service would stop, It was mid-day in the office, the sun shining brightly in the city. I was confused. He created the diagram and from what I can see everything matches up perfectly so it should be straightforward. “Hey, do you mind not merging my pull requests?” my coworker asks pointedly. “This is easy, right?” he thought, worriedly. You’ll play as Rhys, a Hyperion ‘suit’ with dreams of being the next Handsome … The default termination policy stated to kill the servers that had been with the autoscaling group the longest. Guess what coats the wires and cables in the cabinet? If they failed, it took another 40 minutes to try an alternative fix. This is also not verbatim as it's been 18 or so years ago since it has happened. I’m sure they’re fine,” he thought. Use Old Reddit instead. Tales from the Borderlands is a five part episodic game series from the creators of The Wolf Among Us and The Walking Dead: A Telltale Games Series. “Probably not a big deal though.” He debated ignoring it and going on with his weekend, but something compelled him to check. The young engineer tread through the ominous documentation and applied the necessary changes to the codebase. Hot New Top Rising. Moderators. The engineer rolled out of bed and checked his email. As noted in the hangout, signups are probably broken right now.” “Oh…oh no,” he thought as a scream lodged in his throat. The insulation on the wires in the cabinets were melting or flaking off, allowing bare copper to come in contact with each other. “Wait a minute,” I thought to myself, “Friday, late at night, I was snoring in a sleeping bag in the middle of a national forest, miles away from any cell reception!” A shadow cast across my face. Rising. “You never really do get over ghosts from the past.”. Rising. MOD. Hot New Top. Evidently, I give good phone... u/Sciencekitten. With haste, the young engineer rolled back the Firebase update. 1.5M ratings 277k ratings See, that’s what the app is perfect for. It was a routine change, nothing more. Press question mark to learn the rest of the keyboard shortcuts … While undergoing developer wait-time and context-switching torture, she found satisfaction at seeing the app burn again and again and again. His manager tried to calm him down, encouraging him to fix the problem next Tuesday as long as the impact on users was minimal. I clicked the glowing green mark. That was the day mobile web flatlined for 30 minutes and engineers scrambled to resuscitate the Front Page of the Internet. Join. Tales From Tech Support are written accounts of tech support which occurred in the line of duty. Moderator of r/talesfromtechsupport Archived Comments are locked. The time frame would be early 2002, as Windows XP was only out for about a year at the time. I more or less knew what the finished result is supposed to look like since I've had to troubleshoot switches & servers before, but I never had to set it up from scratch. share. At the time, they were planning to do a major upgrade on one of their main campuses, and wanted to set up a temporary satellite site for employees to work from while the upgrade was taking place. edit subscriptions. New Reddit sucks. “It works on my machine!” she declared, shipping the code to production. Something something replicas. Daily top upvoted Reddit posts. u/Dysiak. “It’s not my fault!” he rejoiced. u/MagicBigfoot. I’ll confirm user signups are fine. The young coder modified the configuration in order to appease the Fabric-ghost-of-crash-reporting-past. TFTS POSTING RULES (MOBILE USERS PLEASE READ!) He heaved a huge sigh of relief. He. xyzzy. Woah, crash reports were at 0%! I get in contact with our first partner, who we'll call Tom. Can I help you find something? They don't necessarily have to be about computers, but they do have to be about helping people with their stuff (or trying to, anyway). “I’ve taken down all Reddit signups for the entire weekend!” he panicked. Tales From the Borderlands Redux is rumored to release in 2020 on PC, PS4, and Xbox One. And it wasn’t…this time. The programmer was ready. It was day 1 of vacation. Use Old Reddit instead. The first oddity she discovered was that the production build of the iOS application was not actually using production configuration at all! r/TalesFromTheFrontDesk: A place where people from the hotel industry can come and share the stories of the things our guests do and say that make … The first message that greeted him: “As noted in the hangout, signups are probably broken right now.” “Oh…oh no,” he thought as a scream lodged in his throat. What else had this hacker done? You know exactly what I'm talking 'bout, Willis. Please review our Wiki Page for more details about sub rules, helpful links, and the Best of TFTS. Tom: Not yet, give Shane a call. What he did not know, and could not know, was that a storm was brewing on the horizon…. Me: Okay Tom, I'll give him a call, thanks! u/Drunken_Black_Belt. Posts Wiki. Good job Procmil! Message the mods. Unfortunately in the aftermath, it was never documented who left the cabinet door open and everyone involved with the mess denied that they were the one who left it opened. And that was I how spent my Thanksgiving week a few years ago, replacing the entire system because everything was assumed to be on the verge of failure. Working tech support for an ISP especially in those days you get all sorts of calls. Back in the early 2000's I worked at a Telco company, in the Internet Support department, as the equivalent as a tier-2 support employee. As her screen filled with error upon error, the engineer sipped her tea rejoicing at her success. “Start tracking more fields in the backend events,” the Jira ticket read. No, we don't have more in the back. Cookies help us deliver our Services. He navigated to the Reddit homepage and proceeded to sign up. Dizzy-Yak. Tom: Okay everything looks good! The new engineer fidgeted in his seat. Something something queue. Reddit iOS’s crash reporting service, Fabric, was getting axed in three weeks. Protect your privacy with 60% off Kaspersky Total Security: #MyPrivacyMatters Reddit power users are an interesting phenomenon. The new servers came up fraught with errors, exceptions spewed across the terminal screen, dashboards bled with red 5xx errors, overtaking the restful green of successful responses. jump to content. What if they have access to our infrastructure? Technology Staff • October 28, 2020 October 28, 2020. Got masks from their trucks and came back in. New Reddit sucks. His manager tried to calm him down, encouraging him to fix. At first, I was very reluctant to go because my significant other was going to undergo surgery during my stay (after this experience I'll never make a sacrifice like this again), but in the end I agreed to go and got to packing. The second issue was that Firebase’s new service required the “dead” API key, otherwise known as the secret password assigned to Reddit from Fabric’s former service! With panic rising throughout his body and mind, he jumped into the outage tracking company Slack channel. 1.0k. Set on the unforgiving world of Pandora after the events seen in Borderlands 2, this is a story full of Borderlands’ trademark humor, following two adventurers on their quest for greatness. In a panic, he reached out to his manager, apologizing profusely for the interruption on a day off. With the diagram pulled up on front of me, I get on the call with him and he starts guiding me through the process. He decided to double-check the code. She was further tasked with replacing it with a new crash reporting service called Firebase. But not all was lost! Over the years that my site has existed tucked between two different airports, we have seen manifested many splendored evidences of intelligent life behaving flawlessly brilliantly on … He just didn’t know what to do. Chilling Tales from Reddit Engineering. u/i_eat_chapstick. After ritualistic testing, she marked the changes ready to ship. By using our Services or clicking I agree, you agree to our use of cookies. The Coincidental Outage. Currently we’re having an outbreak of covid at work and most of the FOH staff is out, so I’m trying to help as much as I can, but I’m exhausted and stressed. popular-all-random-users | AskReddit-funny-gaming-news-pics-movies-explainlikeimfive-worldnews-aww-todayilearned-videos-IAmA-Jokes-science -mildlyinteresting-tifu-LifeProTips-GetMotivated-askscience-television-Showerthoughts … During the troubleshooting, they removed one of the swappable controls module and the plastic housing cracked, with a chunk falling out. Welcome to Tales From Tech Support, the subreddit where we post stories about helping someone with a tech issue. Aerial ballet done by idiots . I send Tom a picture of my work. r/AskReddit - Duration: 10 minutes, 37 seconds. But then the green check marks from the passing CICD build glowed upon my face. He's the networking guy that pre-configured everything and he'll be doing the testing with you. Nothing. “Start tracking more fields in the backend events,” the Jira ticket read. Still, it was itching at me so I checked the Git Log. Posted by. I was confused. It was around 4PM at this time, since the devices took quite a while to arrive. In the build, I saw code running that programmatically committed back to the repo… as my user. There were also 480V cables and breakers inside of that cabinet as well. He just didn’t know what to do. That’s the flow this touches and the only possible user impact. It was using test configurations— configurations that would only work on a developer’s machine. This was too fast to complete a context switch onto another task, yet too slow to watch. And while the circuit boards in the electronics are laminated fiberglass, the components on those boards frequently use plastic shells to protect the silicon chips or other internal components, and are encased in more plastic. With haste, he went to kill all the new servers. A malicious hacker had not impersonated me, our own code had! Although the work week was short, it had felt long  and his bleary eyes needed rest. “That’s right! “400k exceptions per day in the bug tracker!” Oh no, no, this was bad. Use modmail to contact the mods, include a link if possible. He looks at his old house and tells his family that time has passed with a blink of an eye. He tried again: *another* 500 error. Press J to jump to the feed. Our quick-witted engineer prepared a plan to remove the new servers with the ill-conceived configuration, to restore glory to the Front Page of the Internet to the world. Thanks for being patient with me since it's my first time doing it all. Reddit gives you the best of the internet in one place. One of our clients was a large university with multiple campuses spread across the country. Join. By Jim Hefner (u/HungrierJoe) It was a routine change, nothing more. With such care, he executed the change. Time was running out, there were only two weeks before the old crash reporting service would stop working completely. His event errors, while very troubling, weren’t blocking sign-ups. “It can’t break anything either, because it’s just events. Little did it matter, though. u/Enalon. As he frantically scrolled the Slack message history, the clouds began to part: this was unrelated. He looks at the Loop blaring in the distance. It was mid-day in the office, the sun shining brightly in the city. Sounds perfect Wahhhh, I don’t wanna. Had I been hacked? Surprisingly, the Firebase docs were not quite up to date and this was one of its unpredictable kinks. “Hrm, that can’t be good,” he thought. Tales from Reddit — What is a story you have been dying to tell? “Probably not a big deal though.” He debated ignoring it and going on with his weekend, but something compelled him to check. This story takes place while I was working for an MSP late last year. He tried on mobile: *more* 500 errors. What made this entire process FAR more frustrating was that each crash test took 40 minutes to complete. Somehow the 480V cables hadn't lost their insulation yet, which would have been a problem if they had metal-to-metal contact somewhere. With such attention, the meticulous programmer prepared. One subject caught his eye: . Between each test, the engineer would make a small change and wait for the test results to return. During a team meeting on Monday, she checked in on the shiny new crash reporter, Firebase. In a rush and with gusto, the engineer made a change to the termination policy in the Amazon web console that dictated: “Kill the newest instances first,” hoping to rid the world of the diseased servers. Isolated in her apartment, she meticulously tested each configuration, forcing crashes on all versions of the app. my subreddits. Any cables that are in the exposure zone are in armored cables or metal tubing. Sure I can get the manager for you. Closed means CLOSED. I don't have the best grammar, and this is my first posting. Sir, You're Making A Scene. A week later I fly out to the site and start unboxing the devices once I arrive (Everything was sent directly to the site so I didn't have to bring anything with me on the plane apart from some tools). “It can’t break anything either, because it’s just events. I didn’t remember merging his pull request, but then again, it was Monday morning and last week felt draped in a haze. The u/Tales_from_elsewhere community on Reddit. 500 error. But somewhere in this same city was a man, sitting in front of a glowing screen in an office dark as night. “This is easy, right?” he thought, worriedly. Me: Thank you so much the help Tom! That meant that any and all crashes were undetected and unresolved. Another Monday rolled in, except this time, the engineer jumped with glee as 40k crashes came rolling in. card. More than a year down the road, the station starts having erratic behaviors and would randomly alarm out. I … It was time for The Ritual of the Configuration Change Deploy with a set of new Amazon autoscaling group servers, which were forged for this very special occasion of testing the new configuration. Guys in their 20s. He emailed me a diagram of the cabling, and the order in which the devices are supposed to be racked. But somewhere in this same city was a man, sitting in front of a glowing screen in an office dark as night. Unfortunately, the news was much worse: the Firebase update had gone completely wrong and crash reports were no longer being sent. Tales From Tech Support r/ talesfromtechsupport. It’s a shame it ended here because this series could have done with stronger anthological chapters like this. On this particular day the call center was experiencing high call volume so my department joined the regular queue, and the supervises fulfilled my initial roll to boost bodies taking these calls. Sometime ago, it started having problems and one of the technicians used a thermal imaging camera to confirm that it was an overheating problem. Tales from the Loop episode 8, “Home”, ends with Cole now much older with a family. 2.0k. After a decent amount of physical labor, I managed to get everything set up according to the diagram. Indeed, at 11:33pm PST on Friday I had merged his pull request. Press J to jump to the feed. 6 8 1 186. Rubber and plastic. Some of the coolant ends up being aerosolized from the spray and while it is below OHSA's safety limit, it's still enough to slowly degrade the paint inside the facility over time. In a panic, he reached out to his manager, apologizing profusely for the interruption on a day off. 2.0k. Soon, a couple of concerns stood out. pinned by moderators. Welcome to **Tales From Tech Support**, where we share our stories of: * *Incredible Feats* of Networking Heroics; * *Tech Troubleshooting* Under the Direst Circumstances; * *Unsolvable Problems* Cracked by Sheer Genius and/or Pure Luck; * *Moral Support* after Having Dealt with Difficult Clients; * And of course, **Stupid User Stories!