Videos
Development - Where the Plan is Put Into Place
[00:00:00] Josh: Welcome to Horror Stories for Dev shops. My name is Josh and I have Kai here with me. And we're on to our second episode of Developer Horror Stories. So we're gonna share a collection of stories that will make you not fall asleep at night and keep you up late at night and, make you worried about your sanity.
[00:00:23] Kai: Thanks, Josh. Thanks for having me. This is a fun episode. So glad he could be here.
[00:00:27] Josh: Yep. This first one is actually a project that Kai worked on for a little bit, and so he's gonna share the story of this one.
[00:00:35] Kai: Yeah. It's one of those stories where we got to the very, very end of the project.
So specifically, we are coming up a week before launch. We're testing, everything's looking good, and then our client pings us and asks, Hey, why isn't one of these Why isn't one of these items being pulled from, our database? You know, we have a list of items here, why isn't it showing up on our website?
And, we started digging down into it and realized, hey, there's actually some requirements here that we didn't know about. So we dug into it and we looked and we found actually this, the particular item had some business requirements that it only showed up on the weekends.
And we had no idea why or, you know, didn't plan at all for it. And so, we fixed the bug and then we resubmitted. And then, a week later, not even a week later, the next day they came back and said, Hey, looks like this other, it's not showing up either. And we just constantly went back and forth for weeks.
On this. And it finally gotta the point where we're like, Hey, can you just talk to your business and just solidify, get us a list of all the requirements for each one of your, products? And essentially they came back and they said, actually, I don't know if I want to tell you what the requirements are.
I'd actually prefer to see what your guys' code does and see if it automatically picks up those rules.
[00:01:49] Josh: Those bugs. Really.
[00:01:50] Kai: Yeah. Yeah. They call it, the bugs, right?
[00:01:52] Josh: Yeah. When they, they put 'em, As bug tickets and said, these are bugs, right? and we're just gonna see where your product fails and then write 'em up as a bug.
[00:02:01] Kai: Absolutely. And they just wanted to go this back and forth system, and part of it I felt like was, they just wanted to classify it as bugs so they can get free development work rather than actually classify 'em correctly as a task. Right. With requirements.
[00:02:18] Josh: Which, which makes our like score, or if you look at the stakeholder in that company looks at us and like, yeah, you guys are just doing bug after bug after bug, right?
So, yeah. Terrible way to start a relationship there. So, this next story is actually one that I solicited from one of our developers here. And he said he had a story where. His previous company, use a product called Confluence. Now, confluence is kind of like Wikipedia, where developers and engineers can keep all their notes and all of their, you know how to do this and how to do that, in this system.
So Confluence was the system they used and they self-hosted it, meaning they had it run on a server and they, they use it all the time for that documentation. Well, the IT guy. Essentially messed up and they lost a year and a half's worth of recording in there and they could not recover it whatsoever.
So basically think of it as a year and a half of knowledge of people writing in stuff all the time. It's just gone. Just absolutely gone. Oh gosh. So, this next one is the developer we had here at Red Sky Engineering. He was wanting to do load testing on a product that we were just about to release.
[00:03:28] Kai: I remember this.
[00:03:28] Josh: And so he, basically set it up so it would generate random users into their system, but he didn't want it to go into Shopify. All of a sudden, you have these fake customers in a production running Shopify. So he commented that code out. But one thing that we found is that one of the reward systems still logged that as a new user.
About a week later, they push out a notification in the reward system and those 40,000 fake users were in there. And so I, since I'm on the email chain, received over 40,000 emails in a matter of a minute or two, and my phone was just blowing up so much so that I couldn't access my phone. It was just, Bloom.
Bloom. Bloom, new, new email, new email, new email, and so we, we were, fortunately, able to go in and, you know, set those as archived users. But yeah, it was, it was interesting.
[00:04:21] Kai: So there was an employee of ours that shared this with us. He said he had a bug at his, at his previous job where the operator's handheld radios were causing RFI into the hand controllers and inadvertently triggered the abort button.
[00:04:37] Josh: So this is the company that he worked at was a UAV company, so like drones. And like little drones and they had like a hand controller, right?
[00:04:45] Kai: so this required being triggered five times in five seconds. And so the software log showed, an abort was triggered, and the customer swore up and down that they never had in their hand near the button. Many heated meetings were had over this, fault.
Eventually, one of the engineers had the bright idea of blasting the hand controls with RF noise from a hand radio and was able to trigger the abort functionality.
[00:05:09] Josh: Obviously aborts are bad because what happens is that all the motors on the UAV freeze up and stop, and the drone just falls right out of the sky.
[00:05:17] Kai: so essentially you have these drones flying around and these employees trying to figure out what's going on, and then
[00:05:22] Josh: they're, they're blaming the customer. The customers saying, I didn't do it, you know, and, and then once they dug into it, it was RFI interference. Mm-hmm. So here's one that's kind of personal to me. I used to work at a previous company and we made, radar systems that flew on drones, and these radar systems could image. The ground a day or night or through storm clouds and whatnot. And we developed it and we're using it in, the military.
Right. And there came a time when we had to go and deploy it. Basically, I had to fly. to Afghanistan, you know, during, during kind of the heated combat over there and actually deploy and work on it. one of the core pieces of software we had was a 3D mapping program, and that 3D mapping program showed the imagery.
It showed, The, basically the ground maps and all that kind of stuff. And we had tested it a ton, in, in our state, and we had tested it a whole bunch of stuff over here in the USA. And so when I flew with the system over to Afghanistan, one of the bugs that were in there was the, Basically the latitude, longitude.
Now I'm in the Eastern Hemisphere and no longer did the program work and stuff. And so I basically got there and it's like, it ain't working guys. I don't know what's wrong. So I had to call back to the developers and say this, and then they finally chase it down and gave me a patch and all that kind of stuff.
But it was stressful, not, not just because bombs were coming out of the sky, but because our software wasn't working either. So,
[00:06:52] Kai: I have a personal experience. before I even worked here at Red Sky, I worked for a different startup company, you know, developing software still.
So, it was kind of fun though, where we had to come up with the idea. We originated it and then we executed and one of our milestones was to present it to a board of directors meeting. And so I walked through the product. It was, a good enough spot to demo and, and show everyone and. Everyone was super happy, excited.
The CEO was pretty jazzed about what he had. And, immediately after the call, keep in mind that we had another demo with another board member that like next week he said, this is great enough, Kai, like, I'm taking you off. The project completely, and now you're completely devoted to sales.
You're assistant manager or a person that's helping you, who is studying communications, in, in college. Never had any software development experience. We're gonna have her take complete control over this project, and then you're gonna work strictly on sales to see if we can get some people to sign up.
[00:07:52] Josh: So it sounds like you just got a promotion.
[00:07:56] Kai: I mean, keep in mind too, I mean, I've never worked in sales, so Yeah, needless to say, we, we definitely ended up pushing that demo off another couple weeks and learned a lot from the importance of, of software development.
[00:08:09] Josh: So here's another short and sweet one from another employee that we had here. He says I had a developer job interview once where the interviewer fell asleep during it. He did not get the job offer. That actually reminds me, I had the same thing kind of happen in college.
I was selling Cutco knives. If you ever did that, you know that's where you reach out to your friends and your family and you go do this demonstration of these kitchen knives and try to get him to buy it. I got referred to this one lady I didn't know and she was like a nurse or nurse assistant or whatever, so worked really long.
You know late night hours and stuff. And so I was in the middle of the demonstration and I could just see her head just starting in to slowly drop down. That's why you know it's over. It's, yep. And basically, I got to the point where I just stopped talking cuz of her, she was quiet and just basically asleep there, and then she woke her back up and she apologized and bought some knives.
Yeah. That worked out. It did work out for me. So, well here's, here's one that I actually remember this. The story happened at a previous job, so we, it was kinda that same radar company. We actually had a case where we wanted to have our own drones, so we can kind of test our radars instead of flying it in a manned aircraft.
And the developer that was kind of working on it, he had some hardware experience and was trained how to fly it and, and how to operate it. And his friend was there and, and their wife's just kind of like, Came out to like, oh, well let's go check out this, this, it was an Octa rotor, so it actually had eight spinning blades, you know, it was probably four feet wide and you know, three feet tall and all this kind of stuff.
And he was kind of showing him this demonstration and, and they were underneath kind of a tree or, or near a tree. And that tree, unfortunately, was blocking the g p s signals, and so as they, lifted off basically the drone lost its GPS signals. And so it's actually started flying at his, his wife, and his friend's wife.
And so he ran in front of it. And just try to stop it with his hands. . And those blades are like little knives spinning, you know, at yeah, a couple thousand RPM and just kind of cut 'em all up and stuff. And he had to go to the hospital and get stitches and all that kind of stuff, just cuz he was trying to show off for, for his wife, you know.
Is that when you decided to get it out of drones? No.
No, but I'd learned to not be stupid like that.
[00:10:33] Kai: So that's a good one. All right, here's a good one from the internet that we found we wanted to share. So back in 2016, a lady and her team were brought into a huge company scandal. One of their client's in-house developers had leveraged the company's website to siphon revenue from them. The desperate client ran to the lady and asked her to rebuild the website. the client had a very complex website and the lady knew she had to find a group of phenomenal developers to help her accomplish the task.
she ended up finding a group of freelance web developers to estimate and build a new site. The freelance group made an offer with two conditions. One, they must develop the site in Drupal 8, and two, they must be paid by the hour. Rest assured we're all experts in Drupal 8 to the developers, though the lady had concerns about developing the company's website, the newish version of Drupal, she decided to trust that the developers knew what they were doing and signed a contract with them.
A month later, they noticed that they were paying invoice after invoice, but not seeing any results. Concerned. She picked up the phone and called the developers. They assured her, don't worry about it. The project is just a bit complicated. We'll call in a couple more additional freelance developers to help out, even though
she decided to give them another chance. The same persisted eventually. She and her team decided to call in another development shop to audit the work. we fast forward a few months and now the company is headed to arbitration and has to hire an attorney to fight the bills for the hours that weren't worked.
These freelancers essentially turned a terrible situation into a real nightmare. when asked why something like this happened, she said, when you get to the bottom of it, you'll realize that the biggest problem is that freelance shops usually do not have project managers to keep things on track. It simply does not work If the freelancer's interest, if the client does not have a great IT project manager and tight scope.
If the client doesn't have a great IT project manager and tight scope, they should insist of on paying a fixed fee upon the project's completion instead of by the hour. In fact, we found that one of the top reasons why freelance projects fail is the lack of adequate project management.
[00:12:46] Josh: So, yeah, I mean, basically that's one of the things that Red Sky Engineering really insists on is having a project manager that we can actually task them, help pull out their requirements, help make sure things are on track, continually have that communication is, you know, is, is really key and fundamental to running a dev shop.
[00:13:05] Kai: You know, there's a lot of elements here that I, I see over and over again with different dev teams, whether it be freelancers or an actual formalized company. You know, they make promises that they know they'll not be able to make or keep, and a lot of it does come down to management.
[00:13:19] Josh: Absolutely. Okay, so here's kind of a final story that I found on the web called Directory Gone Wild. So we seem to be experiencing some kind of performance challenges with our WordPress site. the website seemed fine on the surface.
So the CEO and his team dug a little deeper after downloading the client's WordPress site, which had been designed and implemented by a freelance developer who had since left the team was completely shocked and speechless in what they found. To our amazement, the site had nearly 900,000 files in it.
This wouldn't necessarily be a problem if they're working on a very complex site, although I disagree it would be a huge problem to have that many files. But this was a nine-page website. So after downloading this beast, they discovered a strange directory structure 2013/01/01/01/01 and then at the end of that 05, 10, 20 15, you know, kind of thing, incrementing up in five-second intervals.
As it turns out, three years before the freelancer had modified the WordPress core to write every single SQL statement to a text file on the server, creating a new directory for each year, month, hour, and second. What's worse was that, when the developer left, he kept it running for three years, though they eventually removed the debug hack and deleted the folder structure.
The freelance developer's failure to write maintainable code costs their client extra time and money fixing a problem that should have never existed. So, Kind of crazy. Always fun to, you know, make sure that you check your code and that's why we kind of have coder reviews. You know, we have our developers submit code and somebody on their peer team has to sign off on it and make sure, and we actually find a lot of those, so, We'll find debug statements that are still left over from when the developer is working on it, or extra temporary files they might be creating.
So it's, it's actually a, a practice that we put in place here at Red Sky Engineering to make sure that the code has to be at least approved by one. We, usually try to look for two, but at least one person before it can actually get merged into a branch.
[00:15:28] Kai: Yeah. to kind of follow that up if we notice some things that come up regularly.
Josh, you guys have trainings every. To kind of walk through performance and how we can improve.
[00:15:37] Josh: Yep. Yep. We're, we're obviously in the continuing learning process. Then, web dev is one of the fastest-changing programmings. Methodologies, that are out there. If you think about it, I mean, in the last 10 years, I mean, how much has our internet world changed?
I mean, from mobile devices and launching with, you know, apple and iPods and iPhones and everything. I mean, basically the whole web, you know, has changed so many times and it continues to change at a rapid pace. So obviously continuing education is A core feature that we look for here.
[00:16:11] Kai: Thanks so much for having me, Josh.
It's been a lot of fun to hear some of these stories.
[00:16:15] Josh: Yeah, it's, it's fun. And we'll keep on doing these horror stories. We hope you enjoyed this and check in next week for more stories and experiences that we have and advice on running a dev shop as well. Thanks.
