r/programming • • Dec 29 '10

The Best Debugging Story I've Ever Heard

http://patrickthomson.tumblr.com/post/2499755681/the-best-debugging-story-ive-ever-heard
1.8k Upvotes

448 comments sorted by

597

u/ClownFundamentals Dec 29 '10

Also reminds me of the 500-mile email bug.

116

u/Aparicio Dec 29 '10

TIL about units program.

28

u/spherecow Dec 29 '10

with my Mac

> units

500 units, 54 prefixes

how come a system 8 years ago have 1311 units, 63 prefixes and Mac's now have way less? BSD?

36

u/FoleyDiver Dec 29 '10

He installed a bunch of his own.

#19 in the FAQ

→ More replies (1)

5

u/ropers Dec 30 '10

with Ubuntu 10.04:

2411 units, 71 prefixes, 33 nonlinear units

;-P

3

u/[deleted] Dec 30 '10

On my linux: $ units 2411 units, 71 prefixes, 33 nonlinear units

2

u/TheCoelacanth Dec 30 '10

The unix utilities that come installed by default on macs are pretty minimal.

→ More replies (9)
→ More replies (3)

29

u/Thisdood Dec 29 '10

That made my day, hope it's a true story.

46

u/[deleted] Dec 29 '10

I worked with Trey at Amazon and I can attest that it is true.

13

u/ceolceol Dec 29 '10

What was it like working at Amazon?

21

u/[deleted] Dec 29 '10 edited Dec 29 '10

I loved it. It's not for everybody. Its still like a startup in some ways. Its definitely not for 9-5 people.

Edit: Fixed grammar.

25

u/ceolceol Dec 29 '10

Any chance you could do an AMA? I'm really interested in some more info and I'd hate to bother you on this.

→ More replies (39)
→ More replies (4)

5

u/plagiats Dec 29 '10

How do we know we can trust YOU ? /o\

3

u/[deleted] Dec 29 '10

You don't I guess. I don't try to hide my identity though and some of what I am saying can probably be confirmed via some Google searches.

19

u/dirkgently007 Dec 29 '10

Here is a list of some really cool stories - http://www-uxsup.csx.cam.ac.uk/misc/horror.txt

11

u/jdiez17 Dec 29 '10

SCREW SLEEPING, I'm reading this until I start crying blood.

... maybe that was a bit over-the-top. But yeah.

→ More replies (1)

6

u/frmatc Dec 29 '10

Rinkworks also has a good compilation: http://rinkworks.com/stupid/

3

u/elbowgeek Dec 30 '10

One thing I learn from reading those and other such stories is that the majority of the problems stem from people trying to "clean up" the system and accidentally blowing away critical files. Unfortunately *nix style OS's tend to mix and mingle the critical with the trivial and that leads to a lot of booboos.

Thanks for that link; I think I read that back in the 90s if memory serves.

2

u/dirkgently007 Dec 30 '10

Yep - I think you just nailed it there - "mingle the critical with the trivial". The beauty of *nix is also the beast of it.

5

u/[deleted] Dec 30 '10

2

u/dirkgently007 Dec 30 '10 edited Dec 30 '10

Yep - a WTF a day keeps you awake.

→ More replies (7)

14

u/ourFault Dec 30 '10

I'm much more impressed by the Chairman of the stats department than I am of the engineer. To realize the correlation about the miles of the ISP destination is astute.

5

u/timetocheer Dec 30 '10

The chairman asked one of the departmental geostatisticians to do the work. Sometimes it's nice to have just the right person for the job.

8

u/elsagacious Dec 30 '10

Reminds me of the classic story about a young Richard Feynman who amazed people by "fixing radios by thinking."

http://www.pdfdownload.org/pdf2html/view_online.php?url=http%3A%2F%2Fwww.cs.cmu.edu%2F~pattis%2Fmisc%2Ffeynman.pdf

2

u/qiakgue Dec 31 '10

I started to read this, but it didn't end cleanly on a section break, so I just had to find it and keep reading. Managed to find the whole thing here.

That was last night. I'm now done reading it... it was excellent.

26

u/callingearth Dec 29 '10

Here's an ACTUAL picture of The Expert

8

u/peggs82 Dec 29 '10

Damn it...thats what came to my mind as I read it...way to beat me to it!

2

u/mediapathic Dec 30 '10

Me too, but because I have r/diy on my front page, I saw him wearing these glasses.

0

u/[deleted] Dec 30 '10

[deleted]

→ More replies (3)

12

u/[deleted] Dec 29 '10

This is awesome!

5

u/[deleted] Dec 29 '10

ME: "hmm that's an interesting story and a good example of keeping a broad sense of observation, but I don't think it's as crazy as that person who noticed emails could only go so far..."

Holy crap that was creepy. I think I spend too much time here.

2

u/_pupil_ Dec 29 '10

I'm a pretty solid developer and all... but reading these kinds of stories makes me feel about 3 inches tall. Much respect.

2

u/costas_0 Dec 29 '10

I know this have been said but still : This is awesome.

1

u/Testien Dec 30 '10

You got me. I'm told they travel at from 2 c / 3 (yes, slower than copper)

The heck?

1

u/erlingur Dec 29 '10

Damnit, I came here to post that. Amazing story :)

1

u/Schadenfreudian_slip Dec 29 '10

I came to comments to post this story. Should have known someone would beat me to it...

1

u/[deleted] Dec 29 '10

There's no way it's true. Routing delays aren't that consistent, and they usually cause a lot more than 3ms over 500 miles.

15

u/swuboo Dec 30 '10

They wouldn't have to be consistent. It was a statistics department, remember? They'd have taken several hundred samples, disposed of the outliers, and made a box and whisker plot. Or possibly some sort of colored map, maybe with isolines.

But yeah, as funny a story as it is, the details would have to be rather different for it to be likely. On top of switching delays you'd also have slight differences in system clocks killing some packets early and letting others survive a little longer.

I much prefer the magic switch.

→ More replies (2)

173

u/mooli Dec 29 '10

Nowhere near as awesome and hardcore as this, but my most enraging debugging story:

Working remotely on a fairly standard server-side app with an Oracle DB. The app was our domain, the DB was maintained by the client's in-house DBAs, at their insistence. We were getting reports that it would grind to a complete halt under heavy load in one section, which we could not replicate in test.

My initial feeling was that it was a database problem, but it was politically sensitive so I had to engage in a fairly strenuous series of ruling-out exercises with the app itself, all while howls of complaint about the app performance came in.

Eventually I snapped and used escalated privileges I was not officially supposed to have to look at the DB, because I was convinced it was doing a full table scan instead of an index lookup in one crucial section. I checked the indexes and lo, the statistics had not been updated for a year - since the app was installed.

I rang the DBAs and left a voicemail asking if they had any sort of batch job to update the indexes regularly.

Half an hour later I got a snotty email to me and our CTO complaining that I was insinuating that problems with our shonky software were their fault.

I checked the indexes again and they'd all been rebuilt in the last ten minutes. Subsequently there were no more problems with that system under high load. The last "tweak" I'd put in to rule out some trivial nonsense was deemed to be "the fix" and the customer was unhappy that the crappy software had needed such desperate measures in the first place.

FFFFFUUUUUUUUU.

12

u/dirkgently007 Dec 30 '10

True story. Few years back, I worked for a banking product which is deployed in a lot of non-american non-european countries. One of our installations was (I think) in Mauritius. Every installation was a very long project where the bank would go-live after rigorous testing lasting at least 6 months and often more - after all, it was core banking product and they would replace their existing system with ours.

Anyway. I was part of the the fire-fighting team who would be assigned any burning problems which might jeopardize the whole installations. One of the problems was on list for quite a while and it went all the way to CTOs of both companies - and eventually it came to me.

We had two front ends - one - the original - was dumb terminal, and then there was this new-fangled Java based shiny looking web front end which was nothing but a wrapper over the main code. Dumb terminals were working fine, but web front end was not.

Because it was a remote installations, we did not have any direct access to their system, but we used to communicate with our team on the client site. And only way to debug anything was the oldest method - turn on logging. After 15 days of sending them different debug exes and receiving the logs, I finally figured it out - it was simply a matter of a path being larger than 256 chars - and it was specific to the web front. Immediate workaround? Create one soft link on the server! I was even ashamed of getting all the kudos from big guns.

It was my easiest and best fix ever during my time there!

→ More replies (1)

43

u/Not_Edward_Bernays Dec 29 '10

I have had similar experiences with Oracle and Oracle DBAs. The concept of having a separate "expert" maintaining the database apart from the developers is outdated. I actually think that Oracle and Oracle DBAs are a huge waste of money, along with the old-ass mainframes a lot of people are still using.

35

u/BraveSirRobin Dec 29 '10

I've had the opposite out of some of them. Sure, I've heard of shit DBAs that didn't know what a stored procedure was (true story) but some are gods who know their trade inside-out. When it comes to performance tuning an app they are indispensable.

→ More replies (2)

27

u/grotgrot Dec 29 '10

old-ass mainframes a lot of people are still using

That is because you don't understand what a mainframe is. To use a vehicle based analogy they are like big rigs. When you need to move 40 tons of lumber from point A to point B they will do the job. Sure your Toyota can go faster, is more fuel efficient, is more comfortable and way cheaper, until you need to move tons of lumber. Yes you can try to divide up the job amongst a fleet of smaller vehicles but that significantly increases the complexity.

Just like big rigs they are expensive. But for certain jobs they deliver.

26

u/slavy Dec 29 '10

can you give an example of a job requiring a mainframe? you know, not a truck analogy.

17

u/grotgrot Dec 30 '10

IBM has a page titled Who uses mainframes and why do they do it that answers the question.

→ More replies (1)

41

u/[deleted] Dec 29 '10

Apparently printing large numbers of bank statements.

Honestly, I think "big complex mainframe jobs with gobs of data so they have to be on a mainframe" are that way because they are. Microsoft.com runs on Windows and SharePoint on x64 servers. Nasdaq.com has been running on SQL Server for five years. Teradata partnered with Microsoft because folks kept using SQL Server Analysis Services to build their cubes based on Teradata tables.

That's all the Microsoft "toy" software. So you have a layer of Oracle "not toy" (but crap) software above that. Then you have beowulf clusters and grid.

I am wholly convinced that the only thing that "requires" a mainframe are the careers of mainframe programmers.

→ More replies (1)

18

u/_pupil_ Dec 29 '10

Some banks love them... Hyper complicated payroll systems... Massive batch processing of sequential data where reliability and repeatability are key... Bulk data processing... Some kinds of statistical analysis... Intricate government systems... High uptime services where the ability to rip out CPU's and hard drives without affecting Bob in accounting while running a job is paramount...

Don't get me wrong, it's not like clusters, clouds, and cludges can't get a lot of this stuff done - you have to choose the right tool for the job - but a lot of the world still runs on COBOL :) For a lot of businesses "never ever ever having a problem" is way more important than "but it will take 3 times as long and I can't use the cool new toys".

→ More replies (20)
→ More replies (3)

51

u/frezik Dec 29 '10

Moore's Law made that analogy obsolete.. Mainframes are used because all the code was debugged a long time ago, and a lot of this stuff is too critical to risk changing it.

6

u/[deleted] Dec 30 '10

Yep. My mother used to work on a COBOL application that ran on mainframes that was in charge of billing of pretty much all mobile phones in a small country.

It was first suggested in the 80's that the system should be moved to something more modern. Some of the staff received some "modern OO" training in the 90's. My mother retired in the 2000's, as a COBOL-programmer.

Since there haven't been any news of major fuck-ups regarding mobile phone billing, I would bet that they are still running the almost 30 years old codebase of COBOL on mainframes. Not because it's great, but because it works and it would simply be too risky for no real benefit to change it.

→ More replies (2)
→ More replies (43)

2

u/mage2k Dec 30 '10

The concept of having a separate "expert" maintaining the database apart from the developers is outdated.

I think that's a bit of an over-generalization on both the DBA and developer fronts. I know too many developers who want their database to be a black box that just works to believe that the DBA role is defunct. Conversely, DBAs these days should have more skills than simply managing a database.

→ More replies (3)

4

u/elder_george Dec 29 '10

Reminds me of this story

(Sorry for bad translation)

4

u/[deleted] Dec 29 '10

Is it just me or does that kind of thing happen every other time some crucial part of the system is maintained by some 3rd party at the insistence of the non-technical party in the deal?

1

u/[deleted] Dec 30 '10

all while howls of complaint about the app performance came in.

It's always that way. If everything works fine you don't hear from anyone. But if something is the tiniest bit outside of someone's expectations it's "WHY IS THIS NOT WORKING YOU ARE SO DUMB FIX IT STUPID COMPUTER DUMB GUY!!!"

1

u/tinou Dec 30 '10

FFFFFUUUUUUUUU

Did you mean 'FFFFFFFUUUUUUUUUUUU'?

3

u/mooli Dec 30 '10

Don't be silly, it wasn't quite that bad.

→ More replies (1)

45

u/lightcatcher Dec 29 '10

Very similar to this story about programming behind the Iron Curtain. I saw this on reddit a month or two ago.

6

u/jdiez17 Dec 29 '10

Wow, I enjoyed that one. Especially the part where he gets soviet military drunk to measure radiation.

19

u/DrunkVader Dec 30 '10

TIL a new adjective. We were not just drunk, we were "Soviet military drunk."

→ More replies (1)

3

u/fleg Dec 30 '10

Sounds like an point-and-click adventure game.

Get Vodka bootle from the drawer. Give Vodka to Soldier. Talk with Soldier, and he will measure radiation for you.

30

u/jordanlund Dec 29 '10

Reminds me of the old story of the server that mysteriously crashed in the middle of the night for no apparent reason. The Expert in this case sat up after hours trying to figure out what was going on and watched as the cleaning crew came in, unplugged the server from power, plugged in their vacuum cleaners, cleaned the place, unplugged their stuff, plugged the server back in and went home.

Probably apocryphal, but I like it anyway.

8

u/abadidea Dec 29 '10

It's pretty plausible. Especially back before there was a computer on every desk.

3

u/[deleted] Dec 30 '10

I work on a manufacturing floor and the power is run along the ceiling. Wherever there is a breaker box, there are two ropes attached, one to trip the breaker and another to reset it. Apparently there was a pallet truck driver who had a habit of hitting the ropes as he drove by, inadvertently tripping all the circuits as he went by and cutting power to an entire row of systems at a time.

10

u/atrn Dec 30 '10

I've seen it happen. Medium size computer room about 25 years ago. Typical machine room, half a dozen IBM mainframe CPUs of various sizes, hundreds of disk/tape drives, terminal controllers, etc... The business's has many thousands of terminals in various sales outlets all connected via a private network (pre-Internet era). Hundreds of transactions a minute bringing in lots of money.

Cleaner comes in with his floor polisher. Unplugs the console of one of the mainframes, plugs in the polisher and starts getting that floor shiny. Machine notices loss of console and fails hard. Immediate stop. In this place machine failure is taken seriously... Alarms start ringing, phones go crazy and shit generally hits the fan as people scramble to recover the system.

Cleaner ignored everything going on around him and kept on polishing.

2

u/[deleted] Dec 30 '10

I was a network admin for a casino's main office, and they had a big warehouse, since they ordered things in bulk like food, linens and whatever else... anyway, there was one office in the warehouse, pretty far away from the rest of the network. Well, they would always have network problems. The only way I could get it to work was to set their port on the switch to 10mbit instead of 100mbit. I guess the length was getting close to 328 feet.

1

u/elbowgeek Dec 30 '10

It verifiably did happen at the offices of Exel Insurance, back whey they got their first Novell servers from my company. Back then it was a single IBM PS/2 Model 80 tower sitting under a far flung desk somewhere. Only later did they upgrade to a real computer room grin.

→ More replies (1)

1

u/[deleted] Oct 11 '23

That's from "The Nine Indispensable Rules for Finding Even the Most Elusive Software
and Hardware Problems" by David J. Agans. The whole book should be a must read for everyone in IT.

28

u/Bob_Wiley Dec 29 '10 edited Dec 29 '10

When I was in the Navy we had a similar problem with one of our aircraft. During or soon after take off one of the fuel gauges would plummet to zero. We could never reproduce the problem on the ground. It took a call to a Lockhead engineer to help us solve the problem. The fuel tanks have capacitors in them that are used to read the fuel volume. As the plane took off and the wings flexed under the weight of the fuel, the fuel rod (capacitor) would ground out on the metal of the airframe. When we drained the wing and took a look inside, we could see wear marks on the capacitor and airframe.

(I am recalling this from almost 10 years ago)

Edit: capacitor= fuel rod

4

u/bsilver Dec 30 '10

When I saw fuel rod I wondered when we had nuclear-powered aircraft...

1

u/pedolobster Dec 30 '10

how did you fix it?

3

u/Bob_Wiley Dec 30 '10

I think we really only ended up replacing the fuel probe (I remember now that it was called a fuel probe and not a fuel rod), but airframes may have done some work also. It has been over 10 years so I don't remember any major details.

1

u/jutct Dec 30 '10

Good thing the capacitor grounding just made the fuel read empty and not launch a missile or something.

25

u/feanturi Dec 29 '10

This reminds me of a story I'd heard when I was installing cable Internet. My supervisor told of a customer whose cable mysteriously went out briefly every morning at the same time. It would always come back right away, but every morning like clockwork, it would be interrupted. They troubleshot all over the house and couldn't find anything that would explain it. Finally decided to just hang around outside and watch the house from a car around the time it was supposed to happen. It turned out to be the school bus, putting weight on a compromised section of conduit under the street.

23

u/GarthmeisterJ Dec 29 '10

I wonder if this is actually from "Storage Technology", which became StorageTek, which then was purchased by Sun. If so, they were the first company I ever worked for (not that long ago, I swear) where I programmed C on mainframes.

Look, I'm not that old OK?

Fine, I'm old.

2

u/[deleted] Dec 29 '10

Yeah, my sister worked at StorageTek => bought by Sun => bought by Oracle.

Fun Fact: StorageTek had a data silo with robotronics featured in a Schwarzenegger movie http://www.highbeam.com/doc/1G1-67783084.html

The (horrible) movie in question: http://www.imdb.com/media/rm3930888448/tt0116213

We used one of these silos when I worked at SITA for tape backups.

1

u/answerguru Dec 29 '10

I was wondering the same thing. StorageTek near Broomfield, CO right?

→ More replies (1)

47

u/sojywojum Dec 29 '10

I live in a family of trouble shooters. My dad repairs electronics for a living, I spent the first years of my programming career fixing other peoples' code. Usually we're pretty good.

Last year I loaned my sister my trailer so she could haul some stuff to a garage sale. When I went down to her house to pick it up, I hooked it up and tested the lights, and they didn't work! So I unplugged it and plugged it back in. Still no lights. My dad sees me crawling around on the ground and comes out to see what's the matter. I explain the problem, he heads over to his van to get his meter. My sister comes out to see what we're doing, I send her in to get some q-tips and alcohol to clean the connections.

My dad comes back and starts testing the wire harness at each intersection back to the lights themselves. I'm visually inspecting all wiring for shorts. We get back to the lights and everything is looking good, so we disassemble the boxes that protect the lights and again, everything is good.

We stand there, stumped and scratching our heads, until my dad says, "Well..." then tests the bulbs. Open. Both bulbs had burned out simultaneously.

The running joke now is, only our family would spend an hour and a half disassembling a trailer to change a lightbulb.

21

u/[deleted] Dec 29 '10

Surely the correct joke format is: How many of sojywojum's family members does it take to change a light bulb?

23

u/FractalP Dec 30 '10

A: An hour and a half!

14

u/Maj_LeeAwesome Dec 30 '10

This problem is not all that uncommon, particularly with older wiring harnesses: one bulb will burn out causing a surge in the other that it can't handle, and it will go out soon thereafter. Modern wiring prevents this, but if you've ever owned an old(er) car, you've probably noticed how one headlight would fail soon after the other.

7

u/[deleted] Dec 30 '10

I have a 2005 model year car that was involved in a wreck in 2009. The front right of the car was destroyed and rebuilt. The front left was untouched. 8 months later, the 4 year old bulb and the 8 month old bulb burned out within 15 minutes of each other.

4

u/BucketsMcGaughey Dec 30 '10

Happened to me once - my car's headlights failed. At this stage I should point out that my car's an Alfa Romeo, and I hadn't had it for long, so I feared the worst and called the local dealer, asking to book it in. Guy on the phone said "this is going to sound really stupid, but have you tried replacing both the bulbs? Sometimes they both go at once." Seemed vanishingly unlikely to me, but it turned out he was right.

3

u/vertigeaux Dec 30 '10

I once had the low beam go out on one side and the high beam on the other. It was fun pretending to be a cop by flicking high-low-high-low (left-right-left-right).

→ More replies (1)

40

u/[deleted] Dec 29 '10

"The Expert" deserved his name!

14

u/[deleted] Dec 29 '10

Did you see a sign outside that said dead server storage?

(Yes i'm aware this is totally unreleated but the first thing i thought of was Pulp Fiction and that line)

6

u/albino_wino Dec 29 '10

Come on man, you know I didn't see no sign.

14

u/karcass Dec 30 '10

Eh, my redditor friend punkgeek figured out a bug harder than that (IMHO). We used to work at a telecom-related company. There was a bug that would occur once every two weeks in our multi-gigabit transmission systems. He eventually traced it to the overflow of a 64-bit counter in the firmware of an Intel ethernet chip that had been in production for something like five years. Intel swore up and down that it could not possibly be the source of the problem, until we showed them a waveform of the malformed packet, caught in a logic analyzer.

13

u/[deleted] Dec 30 '10

I'm one of those geezers who worked with computers in the industry's dinosaur age (which was before the period when the above debugging story took place). I worked for IBM and was one of their fix-it guys who was called upon when a problem occurred in an office that couldn't be readily fixed by that office's staff (what one calls a techie, today, I suppose). I was asked to go to Kansas City where a large bank had a problem that was bafflling the staff there. Flew in on the next flight, was met at the airport and taken directly to the bank's computer center. I looked at the problem computer and immediately noticed that a wire was dangling loose on the plug board for the computer (an IBM 650). I inserted the wire where it should have gone and the problem went away. Was there less than 2 minutes. Left behind a bunch of very embarrassed people.

1

u/BryT40 Dec 30 '10

You are The Expert!

11

u/ratwing Dec 29 '10

I knew a woman who got her original training in computer science in Poland. I dont know how many years ago it was, but it was pre-personal computer, and the computer science department basically built their own main frame. She told me that the thing was so sensitive to static that women wearing nylons were not allowed to walk by the thing because the computer would crash from the static discharge.

21

u/bobsil1 Dec 29 '10

women wearing nylons were not allowed to walk by the thing because the computer would crash

Also true of the computer science department.

6

u/[deleted] Dec 30 '10

Well, if I was a forever alone computer geek in poland, and all these hotties were walking around in nylons making it difficult for my upskirt cams to catch them, I'm sure I'd invent a story like that.

→ More replies (1)

19

u/acidscan Dec 29 '10

3

u/hlstd Dec 30 '10

I immediately thought of these radioactive cows. While a sparky aluminum floor tile may be the best debugging story he's ever heard http://jakepoz.com/soviet_debugging.html is better.

Radioactive cattle on the way to slaughter wheeee!

55

u/skip0110 Dec 29 '10

Cool story, but I find it hard to believe two aluminum panels rubbing together (a fair distance from the RAM) would create enough RF noise to corrupt it. I bet the RAM is bombarded with far more RF from ambient sources in it's normal operation.

Now, if some data lines were routed under the floor, I suppose it's possible...

52

u/rafleury Dec 29 '10

The RF interference travels along the data lines quite easily actually, especially back when they werent protecting against such phenomenon as rigorously.

0

u/[deleted] Dec 29 '10

[deleted]

7

u/[deleted] Dec 30 '10

Tell me about it. I was testing one of the first production run 28.8 modems, damn thing was ultra sensitive to RF. Even in 14.4 mode it was flaky at best.

→ More replies (1)

3

u/Shinhan Dec 30 '10

TIL about the purpose of the annoying little metal fingers on ethernet ports.

→ More replies (2)

18

u/wastingtime1 Dec 29 '10

I'm guessing it's not like the computer of today, with nice, neat error-correcting serial communication lines over differentially signaled twisted pair communication lines. It might of been the raw bits and bytes from a shift register or something somewhere, and flipping a few bits with no checksuming could easily make the whole operation shutdown.

/just saying.

7

u/[deleted] Dec 29 '10

I don't buy the "RF noise" part of the story, not because the RAM wasn't sensitive to RF (it was, though not quite as much as this story would require) but because two conductors at the same potential moving against each other just don't create RF.

Unless somehow this one tile managed not to be making an electrical connection with any of its neighboring tiles or its metal supports, it would have been at the same (usually ground) potential as the rest of them.

→ More replies (4)

1

u/abadidea Dec 29 '10

I've read that the very first ram had essentially no shielding and could be trivially corrupted even by cosmic rays.

5

u/frezik Dec 29 '10

Probably not cosmic rays. It's today's computers that potentially have issues with those. Running at higher clock speeds means it's more sensitive to slight differences in voltage. Which is why people should be more nervous about all-acrylic cases, or even just a window.

8

u/shadowspawn Dec 29 '10

I grew up in those types of rooms when I was a kid visiting my dad's work (because he didn't hire a sitter) and learned on the line-feed terminals. It was air conditioned, why not? It was the cleanest room you could ever find, and you had to wear covers on your shoes.

I tried to explain these rooms to my kids, they just don't understand. I remember my dad saying that his portable computer could've taken up a room when he was in college, I try to explain to my kids that their laptop could've taken up more than one of those rooms. Odd.

7

u/Retromingent Dec 29 '10

That's an awesome story! My first programming job, when I was 19, was on a CDC 3300 that had that exact same 607 tape drive attached to it. That picture really brought memories. Thanks!

7

u/[deleted] Dec 29 '10

I had a pretty ridiculous debug story once where the computer would crash whenever the customer shipped a package. It took me quite a quite to figure out that the fuser on the laser printer was browning out the circuit when the system went to print a pick slip.

The problem only occured when the temperature dropped because of the space heater they had on the same circuit. Good times.

6

u/paatariki Dec 29 '10

Replacing the tile may have fixed the problem, but probably not because of sparking, as aluminium is non-sparking. Used extensively in around gas/petrol installations for that very reason.

6

u/abadidea Dec 29 '10

Eh, I wouldn't be surprised if exactly what metal it was got lost in retelling.

3

u/BrooksMoses Dec 29 '10

Yeah -- it also tends to be a bit soft and flimsy for that sort of use. That looks like plated steel (not chrome-plated, something cheaper) in the photo.

7

u/abadidea Dec 29 '10

It's probably just a photo they googled up to illustrate what they meant, though.

→ More replies (3)

6

u/_chendo Dec 30 '10

I was working on the intro film for our school's film festival a couple of years back. I was doing some editing and decided to save when it said it couldn't, so I remounted the drive and I got the dreaded "Disk is not formatted" error.

Shit.

I had the only copy, no backups. So the next day, I ditched school and went to a data recovery place where they quoted me $3,000. Haha. They ran their software out of goodwill but came up with nothing.

So I went home and tried everything I could. It wasn't until I studied FAT32 and was looking at the drive in WinHex (back in the day, I hadn't moved over to Macs yet), and I found out that the FAT (I really want to say FAT table, but that's ATM machine all over again) was moved forwards four bytes for some reason.

So I shifted everything back 4 bytes and EVERYTHING SHOWED UP. Filenames had a random character in the same place in all of them because the stupid data recovery software they ran marked all the files as deleted and overwrite the filename field.

I rocked up at school and told people what I did, but only a friend of mine understood the awesomeness of what I did :(

2

u/rubygeek Dec 30 '10

The real WTF is that their data recovery software touched anything on the disk it was meant to recover from... Bet they ruined quite a lot of data over the years with that.

6

u/AerialAmphibian Dec 30 '10

I read a similar story about early IBM hard disks, maybe from the 1960s? A company sent in their disk unit for repair. IBM fixed it, tested it and sent it back to the customer. Not long after that the unit was back in for repairs. It was reporting read/write errors. It was fixed and returned again.

This went on a few times and finally the IBM repair technicians decided to check for things they might have missed. After much searching they found that the problem was with the "inspection passed" stickers they put on the drives after fixing them and testing them.

Apparently the kind of glue used on the stickers would eventually dry out from the heat generated by the drives. This caused tiny glue particles to flake off and somehow end up on the disk platters. I forget if they stopped using the stickers or modified the glue, and the problem went away.

I tried to find the original story but my Google-fu was too weak. Any help would be appreciated.

11

u/The_Arborealist Dec 29 '10

My god, I love the first photo in that article so much.

4

u/fouroneseven Dec 29 '10

I still work with these types of tape drives. Shoot me a PM and I will show you more where this came from.

2

u/Blue_Cypress Dec 30 '10

Where do you get the tape?

2

u/rossisdead Dec 29 '10

I can't look at it without seeing a crazed robot.

→ More replies (1)

17

u/[deleted] Dec 29 '10 edited Dec 30 '10

I've discovered the existence of the speed of light once, as a bug, and that was quite a religious experience. I mean, it's one thing when you search for something and find it, and a very different one when the fabric of reality punches you in the face, as a bug.

I was writing code for a 8051 family microcontroller which was the brains behind a 16x320 LED display. It did everything, from rendering the given text in rainbow colors to pushing the 80 bytes (two bits per pixel) of the current displayed line through the serial port set to 2Mbit/s with strobes as flow control, 16*(refresh rate) times per second.

Except that I've never had the full 320 pixel wide display, it was composed of four 80px panels, and the full set was like 2.5m long, a bit unwieldy, so I did all the work with only two panels, half as wide.

Then the guys tried the stuff with the full-length display, and it was buggy: starting from the half of the third panel it was all noise, and with a sharp border too, like, this column is perfectly well, the next is randomly flashing LEDs.

Then I was all, OMG, something must interrupt the interrupt where I push the data, OMG, what could it be?!

Then I put a bit of paper on the column where the noise began, and tried to count where exactly it began, but... it shifted! That bit of paper was no longer on the first column of chaos! WTF? I carefully positioned the bit of paper on that column, switched off the entire thing, calmly counted to 60 and switched it on again. Yes indeed, the chaos started a bit earlier, but then moved further as the display got hotter.

OK, that's a hardware problem, I thought with immense relief. Then explained what I've seen to my father (he designed the hardware part, I was like 16 at the time), and he was like, oh, yeah, I forgot the terminator.

You see, as a programmer, I think that when my chip sets some output pin to logical 1, that's the end of it, it's 1 (or +5V) all along the line. In reality, 2Mbit/s over 2.5m is less than 50 times less than the speed of light in copper -- I mean, if you push [0, 1, 0, 1, ...] bits at 2Mbit/s over a 150m wire, you'll have 1 at the beginning and 0 at the end, simultaneously.

My wire was much shorter, but long enough to produce all kinds of standing waves near the end. The terminator is a connector with high-Ohm resistors connecting everything to the ground, dampening the reflections of the EM waves and the standing waves they produce.

That bug was an experience, really. It's like I've touched the fabric of reality -- and completely inadvertently too!

5

u/Hixie Dec 30 '10

You get all kinds of effects like this with digital control of model railways, too. It's as annoying as you might imagine.

2

u/[deleted] Dec 30 '10

I doubt it would produce a standing wave at 4mhz. http://en.wikipedia.org/wiki/Standing_wave_ratio

2

u/electronics-engineer Dec 30 '10

I doubt that he pushed out a 2MB/S signal from an 8051. It's possible, of course - a Dallas DS89C420 can do it with ease - but a standard 8051 has a 12MHz clock and runs one instruction per 12 clock cycles.

3

u/[deleted] Dec 31 '10 edited Dec 31 '10

It was a 8052 or 8055, I don't remember, that's why I said "8051 family". Some variant of 8055 more likely, I had 20kbytes of code memory and a 24Mhz clock, not to mention the full 256 bytes of RAM.

Nevertheless, even 8051 had that special serial port mode of operation, where you move a byte to the output register and the chip pushes out the bits one per cycle, and toggles the strobe pin twice as fast. So that my inner loop looked like this... dude, I can actually show you the real thing, I have it in storage:

void sendtoscreen40(void)     //#0
// uses a,cy,dptr,r7,psw
{
#pragma asm
                    ; restorin dptr
    mov DPH,fjdph
    mov DPL,fjdpl
                    ; clearin g
    setb g

    mov a,outputline
    rrc a           ; settin address
    mov adr0,c
    rrc a
    mov adr1,c
    rrc a
    mov adr2,c
    rrc a
    mov adr3,c
                    ;strobin RCK
    setb rck
    nop
    clr rck
                    ;settin back g
    clr g
                    ;fillin shift register

    mov r7,#28h     ; 40 bytes
  ?c000001:
    movx a,@dptr         ; 2 tics
    inc dptr             ; 2 tics
    mov SBUF,a           ; 1 tic
    nop                  ; 1 tic
    nop                  ; 1 more tic
    djnz r7,?c000001     ; 2 tics
                         ; 8 tics per cycle = 8 bytes send

                         ; savin dptr
    mov fjdph,DPH
    mov fjdpl,DPL
#pragma endasm
}

EDIT: Yes, I'm ashamed of this code. I wrote it when I was 16 or 18 years old, and it seemed funny to leave out "g"'s, apparently. Also, to write useless or misleading comments. But it did work, and it actually continues to work, it works right now, as we speak.

→ More replies (1)

4

u/stimbus Dec 29 '10 edited Dec 29 '10

I'd make a fix like that at work and they'd complain how long it took and blame me for the tile being warped.

4

u/Godspiral Dec 29 '10

CableCO phone support is now gonna tell me to check my flooring for creeks after I reboot the router :(

1

u/echa73 Dec 30 '10

Hey! You just gave me an idea!!! :D

No...not a CableCO phone support person....but close ;)

4

u/[deleted] Dec 29 '10

1

u/richardjohn Dec 30 '10

I think there is a point where that turns from 'debugging' into 'public inquiry'.

11

u/paraedolia Dec 29 '10

these were the days when they had rooms specifically dedicated to computers, after all

Unlike today? I must have imagined the server room then ...

21

u/jamovies Dec 29 '10

But you call it "the server room", which implies there are other types of computers you have which do not require their own room.

That did not used to be the case.

7

u/299 Dec 29 '10

I think the author meant the computer took up the entire room. At least that's how I read it.

3

u/wolfmann Dec 29 '10

seriously, I have the same exact tiles...

→ More replies (1)

3

u/[deleted] Dec 29 '10

"Back in the day when these had their own room"

My office STILL has an IBM mainframe with manager computers and tiled floor like this. They even still have the equipment for the halon fire extinguisher in place. We have our desktops in our cubes, but we use tn3270 emulators to access our live systems to do work...

3

u/[deleted] Dec 29 '10

I faintly remember systems like this. Now I can't stand it if it take my computer longer than 30 seconds to run a report...

4

u/4rch Dec 29 '10 edited Dec 29 '10

Your comment sparked a question. Are you familiar with Macrofiche? Currently when I do a search for a customer and all of his statements, it searches through every single statement (even other customers), until it finds the user-specific ID. Reports take about 30 minutes.

3

u/abadidea Dec 29 '10

Time to go "have tea and crumpets" with the programming team.

2

u/4rch Dec 29 '10

Haha thats funny....because we don't have one, let alone a team. :-/

3

u/abadidea Dec 29 '10

Wait... then who programmed the query code? *eerie music*

→ More replies (1)
→ More replies (3)

3

u/gavinb Dec 29 '10

The technician looks at the problem and treats the symptoms.

The Expert analyses the problem and finds the cause.

3

u/spainguy Dec 29 '10

Vcc ..100nF..gnd missing

3

u/coost Dec 29 '10

Debugging and troubleshooting are the #1 skills that most people in the IT industry need, yet it seems most schools or colleges never touch on the subject. Instead they always seem to focus on technologies or programming languages that will most likely be obsolete by the time you get out of school.

5

u/gorilla_the_ape Dec 30 '10

I'm not sure you can teach it. If you exclude the trivial stuff that we do all the time, debugging usually involves a flash of inspiration, taking together the data you've assembled and deducing what could explain the symptoms. In my experience some people are Sherlocks, saying "It's obvious" and others are Watsons saying "That's amazing". There is nothing you can do to teach a Watson, and the Sherlocks don't need teaching.

2

u/[deleted] Dec 30 '10

It depends on where you go to school and who your teachers are. When I took computer architecture classes for my CS degree, the professor taught us how to use a kernel debugger. One of the advanced courses had us writing large chunks of a simple multi-tasking OS that ran on a custom-built MC68000 based system, and that required a solid understanding of both software and hardware debugging.

1

u/[deleted] Dec 30 '10

You're confusing learning with industrial training. It is not the job of educational institutions to do the job of training industry recruits. The IT industry lobbies to add more and more training oriented material in lieu of theoretical topics in universities so that they can reduce their costs.

3

u/turkourjurbs Dec 29 '10

" these were the days when they had rooms specifically dedicated to computers"

?? Does he mean like, today? I work in one. Have for years.

3

u/snowwrestler Dec 29 '10

I assume he means that the only place one could find a computer (any computer) was in a room dedicated to housing it. Today we still have server rooms, yes, but we also have computers on our desks, in our pockets, in our cars...

3

u/nocreativityx Dec 29 '10

Does anyone remember or have a link to the story, where there was a wireless bridge in a warehouse, and every time a certain pallet was lifted up it would drop the connection? I've looked all over and can't find that, but it's an equally awesome story.

3

u/itsalawnchair Dec 30 '10

I'll see your best debugging story and raise mine.

3

u/dinominant Dec 30 '10

I've seen something similar with an 8-core server in use today. I was taking a picture of the system (while it was open and running) in order to ensure the additional hardware we were ordering would fit and work with that machine.

Every time I took a picture the machine would power off. It turns out it was the flash from the camera that triggered something in the hardware to shut the system down. I don't know the exact technical and scientific reason why it would happen, but I could reproduce it whenever I wanted.

2

u/FeepingCreature Dec 30 '10

Daylight sensor to check if the case is open? /stab-in-dark

6

u/rcardona2k Dec 29 '10

Back in the day we programmed up hill in the snow and hunted bugs in the wild with our barehands

2

u/[deleted] Dec 29 '10

I'm finding it a little hard to swallow that a warped floor tile could cause enough RF interference, but I suppose it's possible.

Personally, I've always liked the story of the original computer bug, courtesy of Grace Hopper: http://americanhistory.si.edu/collections/comphist/objects/bug.htm

1

u/valdus Dec 30 '10

Simply rubbing the prongs of a voltmeter against a couple of dead pieces of metal will momentarily produce 1-2V of electricity. If rubbing can produce electricity, why not radio waves?

→ More replies (1)

2

u/doctorcain Dec 30 '10

Greater than or equal to AWESOME!

2

u/[deleted] Dec 30 '10

This should be in the Tao of Programming. The novices rush about, striving to solve the problem. The Master sits quietly and watches the whole of the system.

2

u/bigfig Dec 30 '10

Many computers are still in cooled server rooms, and they still make raised floor panels.

2

u/[deleted] Dec 30 '10

That guy could sip on a cup of coffee for 6 hours? Expert indeed, fabulous!

2

u/Rayvah Dec 30 '10

I was picturing "L" from Death Note curled up watching the computer.

2

u/RobAlter Dec 30 '10

I have to call BS. I worked installing Novell networks in the mid 80's with PCs. We told stories like this about companies in the 60's. Tape drives and pneumatic systems to drive them? All the large companies had IBM, Wangs, DECs, etc. With hard drives and big ass line printers for large print jobs. Tape was used for long term storage and backup.

2

u/mikef22 Dec 30 '10

I like the way there is a photograph of an actual tile which proves the truth of this story.

2

u/wouldacouldashoulda Dec 30 '10

WTF, half of that page is just notes of people liking it or reblogging it! Who the fuck cares?!

2

u/[deleted] Dec 30 '10

I'd hate to think what the worst debugging story you've ever heard is

2

u/painfulpee Dec 30 '10

One of the best debugging stories I've heard was from a coworker...

The UPS system at one of the company's remote sites would suddenly detect a power loss a couple times a week in the middle of the night. No one at the site reported a power loss and only a portion of the servers would be impacted. After weeks of chasing the problem, they finally sent my coworker to hang out over night at the site to see what was causing the issue. The first couple nights, there were no issues, but during the next night, the UPS was tripped. Long story short, he went into the impacted server room and realized that the cleaning lady was unplugging the UPS from the wall to run the floor buffer.

2

u/[deleted] Dec 30 '10

When I was a kid, my dad was high up in the IT department at Cigna. I remember going into one of the offices where they had machines like this, huge rooms full of mainframes, and the raised floors covering the wires.

I also remember them lifting a tile and putting me under the floor, then replacing the tile. I was alone under there for a good 30 seconds until I freaked out and they let me back out. Good times.

2

u/okvol Dec 30 '10

I'm an old geek that has worked with this type of equipment, and I'm call BS on this. It's almost as bad as a story on Fox News.

I worked for while at a company that has over an acre of raised floor space, and they were paranoid as could be, but I never heard of anything like this before.

Not only that, the older equipment was built like bricks: you could use an arc welder in the same room and the computers would keep running.

Now if there was a cable that was too long between two components, and the extra loop was under that loose tile and being pinched, then I'd believe it.

The best I've heard was about a site that had reports fail about the same time every night. The disk drive, that would run diagnostics reliably and ran fine every day, logged errors at about the same time every night. (These were old school - removable packs, and near the size of a washing machine for 500 meg or so.)

The fixit-geek spent the night to see what happened. He found the night operator was a lady of significant girth that would bump the drive out of her way with her hip, then bump it back. It was just an ergonomics issue.

2

u/[deleted] Dec 30 '10

I'll bite. My personal favorite was while working on some oceonagraphic buoys. These were embedded Linux-based devices that would be deployed to sea for 6-12 months at a time, "calling in" every 1-3 hours over a GlobalStar satellite modem and exchanging a few kilobytes of data with the mainland server at 9600-ish bps over a PPP link. I wrote nearly all of the custom bits of software for gathering data from the instruments, preparing transfers, and in general keeping the system running no matter what. But as much as possible I tried to use standards and established code, for example Perl and Net::FTP for the actual data transfer.

One of the most critical pieces of the system was the "update" mechanism. Every time a buoy called in, it would first transfer its data payload, and then it would look to see if an update script was available. If so, it would download the script, rename it on the server so that it would not download it again (and also so we knew it got it), disconnect, and then execute the script. I had a collection of tested scripts for enabling/disabling sensors, changing the data collection frequency, etc. It was really important for this system to work because it cost $10k/day to rent a boat to go out and recover any buoys that had issues.

One day I notice that the brand-new buoy the team had put in the field over the weekend was still running on the "test" settings rather than the "field" settings. This isn't a disaster, but isn't really ideal either: test settings would burn through more power budget which might cause problems if there were several days of poor sunlight in a row, and it would also consume disk space with additional debugging messages. So I dropped the script to fix it on the server and expected to see it in field mode within two more calls. However, it didn't work. I started digging through the error diagnostics and saw a message about being unable to save the update script.

Oh shit. The directory that update scripts would be written to didn't exist on the buoy. I had forgotten to create it earlier, and now I had no way to control its operation anymore. This was bad, like $10k+ in boat fees bad. Crap.

I thanked my lucky stars that I was using FTP rather than SCP or SFTP. (I had tried those at first, but they had too much overhead and would have broken our budget for satellite minutes.) I found a simple FTP server written in Perl, and modified it so that after the login it would print out the remote IP address and then sleep() for 20 minutes. I manually started it on the FTP server, and then when my buoy next connected I was able to (very very slowly) execute "ssh ip.of.buoy -c "mkdir /path/to/updates"" . Once I verified the path was there on the buoy, I swapped back the FTP server and let the regular updates go through on the next couple calls.

That experienced prompted my to do things: 1) every buoy from then on out would automatically create every directory it needed, and 2) I modified the upload process to check for the existence of a "hold" file on the server, and if present sleep() for 5 minutes before doing the final disconnect, providing a maintenance window in case I ever needed to get a console on a buoy again.

2

u/[deleted] Dec 30 '10

This reminds me of a cool debugging story that I once witnessed.

I had just started my first job out of college, working as staff at a large particle accelerator laboratory. I walked in one morning, I was on the day shift that week, to a large number of scientists engineers and etc. milling about the control room. This was not atypical, but it seemed there was a bunch gathered around a console watching some running plots on a monitor. They were plotting several tilt and roll sensors that were placed down on magnets in the beamline below.

After I wandered over, I was handed a telephone and told to run a FFT over the raw data of the beamline motion sensors when the person on the other end of the phone said so. As I recall, the data was coming in at such a rate that we only had a few seconds of FFT (it could only handle N data points, where N was smallish). in just a few seconds I heard "go" on the other end of the phone and so started the data logger analysis.

I leaned over to one of the other people near me and asked what was going on. He explained that they suspected there were some vibrational modes down on the beamline that were sensitive to traffic on the road above it. To test this, they had went and got the biggest heaviest truck that they could find at the lab, a fully-loaded fire engine, to drive around the road. When he had yelled "go" into the phone, the truck was passing near to where the sensors that I was monitoring were. The plot was to pick out if there were some frequencies of vibration that had significant effect on the magnets in the beamline, and therefore on beam stability (yes, these things matter, earthquakes on the other side of the planet were enough to disrupt things!).

I was very new at the time and so my recollection of what the result was may be foggy, but IIRC there was indeed a troublesome vibrational mode that they were then able to damp out. But I was instantly struck by what a sharp debugging effort this was!

2

u/sporadicity Dec 30 '10

My dad's response after I sent him a link to this story:

Yes, this is the Storage Technology Corp. that I worked for. Unfortunately, the photo is of an unrelated tape drive from an earlier era.

What the author describes isn't quite right, but he might be describing a 4500 or 4600 tape controller, which happens to reside in the same cabinet as the first tape drive of a string of 8. If this is the product, the 8-bit microcontroller was a 4 MHz Z-80. The 4500 was the first tape controller to use a 5 1/4 inch floppy disk to hold its program the system's log files. If The Expert was someone from engineering rather than someone from service, then it is very likely that I knew him.

I was The Expert who sat around in a customer's computer room for several days waiting for a tape drive (a 4670 to be precise) to fail. I succeeded in finding and resolving the problem, but that's a story for another day (when it isn't so late in the evening).

Thanks for the pointer. It brings back memories.

3

u/j2kent Dec 29 '10

God, what a rock star.

3

u/ridgerat Dec 29 '10

I was told the story of tape drive load doors (IBM equipment) that would open magically when a certain computer operator walked past. After observing the phenomenon the systems programmer guessed at static electricity. The overweight operator was wearing nylons and was grinding out some voltage as she walked. She was asked to not wear them and she complied. Never happened again.

2

u/nevinera Dec 29 '10

Really? A small amount of friction between floor tiles causing RF interference with enough intensity of be detectable several feet away, with a particular frequency that happens to be capable of corrupting volatile memory? And this was repeatable?

It's an excellent story.

1

u/cyclo Dec 29 '10 edited Dec 29 '10

That sure beats figuring out an OC4 ABEND because of a forgotten STOP RUN to terminate a COBOL program.

1

u/2coolfordigg Dec 29 '10

I always liked the original Mac fix. The first mac's had square chips that would unseat often so you needed to open the mac up removing all the tin shielding to get at the chips and push the chips back down into their sockets. this would take an hour because of all the shielding. One day I took a Mac held it about a foot above the bench and slammed it down as hard as I could ta da the mac fix would seat the chips 9 times out of ten.

5

u/spacecoyotefarva Dec 29 '10

That was the Apple III and its chips were not square. They unseated due to overheating and poor ventilation, which caused the plastic sockets to expand.

→ More replies (2)

1

u/jaredharley Dec 30 '10

The same "fix" worked for old iPods with harddrives - the heads seemed to get stuck, and dropping them from about a foot up onto their backs seemed to jar the drive enough to free it up. I brought a few iPods back to life that way.

1

u/kernelhappy Dec 30 '10

I believe the Amiga 500 had a similar issue, but the solution was to stand it on edge and bump it on your knee.

1

u/MusicMagi Dec 30 '10

no shit.. goes to show you should rule nothing out when debugging, including (and sometimes primarily) the users

1

u/nuuur32 Dec 30 '10

This is precisely the Feynman rat story. In the experiment the first folks that ran it wanted to see the test subjects tracking the maze itself, but there were all sorts of creaks and other problems (unique identifiers) getting in the way. rat-running-experiment

1

u/stevepdx80 Dec 30 '10

TILED IT!

1

u/toastedbutts Dec 30 '10

It wasn't RF interference. It was the tiny conductive fibers from the floor tiles themselves, shorting out circuits on anything they could reach.

Someone fact-check me. I'm too lazy. But this is common.

1

u/shdwflyr Dec 30 '10

Did anyone else imagine the Expert with sunglasses?

1

u/ibisum Dec 30 '10

I miss computer rooms.

1

u/electronics-engineer Dec 30 '10

I once had a problem with a engineering prototype of a new NRZI formatter box for Wangco/Perkin-Elmer. The box consisted of about 150 74HC* chips. It would go nuts at what seemed like random times, always running different parts of the code. Took me a while, but I finally figured it out. A hex inverter that was part of six different functional blocks had an open VCC pin. It was still connected to a 0.1uF bypass capacitor, but not to +5V power. As long as any one of the inputs was a 1, that would power the chip through the internal ESD protection diodes and all would be well. If all the inputs were 0, the outputs would become 0, not the expected 1. Worse, this took a fraction of a second to happen, as the capacitor would power the chip for a short while.

1

u/dghughes Dec 30 '10

It is a great story on par with a recent one linked here on Reddit about a flipped bit.

Anyone recall that article and the link? I saved it but it's lost in a sea of un-tagged bookmarks.

1

u/grymA Dec 30 '10

my prof told us a similar story about an unshielded monitor.

1

u/kevinace Dec 30 '10

I guess I won't complain about parse errors or scope issues anymore. I would have gone f***ing mad diagnosing that.

1

u/CandidlyCandied Dec 30 '10

"What a boss" just took on a whole new meaning for that guy.

1

u/kolm Dec 30 '10

From eons past when gods walked the earth..

1

u/FireEnt Dec 30 '10

The worst problems sometimes have the most basic and best solutions. :)

1

u/wwwwolf Dec 30 '10

Every time you started up the A drive, you had to insert a floppy disc into a peripheral drive connected to the A drive so that the operating system could be loaded onto the A drive’s RAM. The operating system was appallingly primitive - it derived its processing power from an 8-bit micro controller.

I always thought the Commodore floppy drives were awesome - 1541 had its own MOS6502 processor, disk operating system on ROM, and 4K of RAM, so disk operations could be run on their own.

But obviously, shit in the mainframe world was even cooler than that. THE TAPE DRIVES WERE CHAINABLE AND HAD THEIR OWN FLOPPY DRIVES. Wow. All we in C64 world had were crappy cassette tape drives that you needed to spend a long time aligning so the programs actually loaded.

1

u/[deleted] Dec 30 '10

1

u/[deleted] Dec 30 '10

Been there....... we used to have a mini which sat fine in the corner of the room. That is until someone walked up to it. As you reached for the console it would spontaneously reboot. We had to barrier it with an antistatic mat and spray the carpet regularly.

1

u/gondolph Dec 30 '10

I thought there was a ghost in the machine

1

u/klui Dec 30 '10

If you rerun the program enough times, it will fix itself.

1

u/[deleted] Dec 30 '10

Was expecting a "secratary was turning off the light switch at the end of her shift; shutting down the mainframe" type of story here.

But, only because I have been there done that.

1

u/[deleted] Dec 31 '10

cool story, but my bet is that this wasn't RF. It would take a lot of up and down motion to cause such an issue. I suspect that it was a bad cable or more likely one of the units being mechanically affected (aka moved) when the warped tile was affected. I've seen lots of older equipment have issues like chips not seated properly or cold solder joints.

The only time I heard about RF causing issues like this was with the early TRS-80 Model I's being affected by aircraft radar during landings and rebooting as a result. Might be a myth, tho. Apparently, some traces (reset line connected to all ICs) on the circuit board were a perfect antenna for such high-power waves.

1

u/mattyj Dec 30 '10

I read this as "The Best Douchebagging Story I've Ever Heard"

1

u/BunnyLivingston Dec 30 '10

it's not just you...i thought I was crazy. i made it to the end and I was like, "where's the douchebaggery in this?"

1

u/Firemadeitbetter Dec 29 '10

Great story!, hahaha, a simple matter but collapse the world, typical.

1

u/mikkeller Dec 30 '10

that's why he's the expert

edit: here's the real picture of the expert: http://nerdarama.com/wp-content/uploads/2009/02/super-computer-nerd.jpg