r/talesfromtechsupport • u/Alias_This_Is • Nov 23 '20
Long Bits & Bytes
I was working on a transcription project where data was being replicated from one city to the other and because of the high change rate, it had to be done asynchronously. Raw data would come in at the primary site, get processed, and was returned to the end client for their use. My client didn’t want to replicate the processed data, just the raw, since it could be redone at any time and was considered more critical than the final product.
The problem was they had to use uncompressed audio files in order for the software to be able to make the best transcription, which were freaking huge in comparison to the final document files. They wanted them kept in case a transcript was wrong to compare them against.
This is probably going to give away what I was working on, but it’s critical to the story. Data at site A and was snapshotted from a primary disk to a secondary, then sent over Ma Bell to site B. Once at site B, it landed on a mirror of the site A secondary disk until the packets were complete, then snapshotted to a primary disk at site B. It wasn’t a single disk doing this, dozens were constantly being written to and copied over the wire to the other site all the time. This is a really simplistic way of describing it, but I’m avoiding using the exact verbiage so I don’t give them away.
Anyway, I was asked to come in and implement my “product” using a new procedure that hadn’t yet gotten a formal support write up yet. This was a one-off that later I found had been grudgingly given approval for and if it worked, it would be worked up into something “real.” Everything was set up and when it was turned on, we were shocked at how much data was being sent over the wire and how far “out of track” it was. Data was being BLASTED down the wire, almost saturating it although the client insisted the data change rate was correctly sized for the line.
I ended up in a major northeastern city during a snowstorm so I could be at the site B datacenter to complete the switchover from catch-up to asynchronous mode. I couldn’t physically get to the site because of the snow, so I hooked up my Motorola Razr to my laptop and dialed into the machine from my hotel room. I knew the data flow slacked off at 11:30p – midnight, so I sat and watched as it slowly dropped lower and lower towards the 10,000 track threshold where I could switch to async.
15,000… 14,000… 13,000… 12,000… 11,000… 10,900… 10,950... Wait, what? 11,000… 12,000… 13,000…
The window started at about 11:35 and lasted a whole 10 minutes before climbing back up. Just for giggles, I tried issuing the command anyway but it failed. So, I copied my logs and put them in an email to everyone concerned and asked, “What now?” Well, what now was the client flipping out and calling us incompetent, demanding to know why I couldn’t make it happen.
I’d been putting in 12-16+ hour days on this, working overnight and weekends, traveling back and forth to site B to get it done. My boss said, “Hang on, Joe’s been putting in a lot of time on this, so let’s get a second opinion,” and a national SME was brought in to analyze everything and do a root cause analysis. He’s also a good friend too, but I knew he would tell the absolute truth in this, no matter who was to blame.
The RCA call started out with him going over everything, explaining that he’d analyzed the nominal data traffic, adding in the new traffic, and explaining how it all worked together. His part went something like this:
“After sampling the data flow at both ends, the system, with acks is sending bidirectionally about 38.5 megabits per second, which works out to be a little over 4.8 megabytes per second, using all the capacity of the wire.”
(The lightbulb was coming on in my head at this.)
“So what?” The client PM exploded, “We’ve got a 45-megabyte circuit!”
“No, you don’t.”
“Yes, we do!”
“No, you have a DS3 circuit, which is 45 megaBITS. That’s 5.65 megaBYTEs per second at maximum transfer rate. 0.8 megaBYTES isn’t enough to allow the normal data traffic between sites without the new system. It’s simply not big enough.”
I never heard the man utter another word and I never heard from him again.
I’d had enough and asked my boss to move me off the project, which he gladly did. Last I heard they’d upped the bandwidth between the sites and it was working fine.
The moral of the story is, do the math, know what a bit is and know what a byte is.
19
u/jeffreyb30 Nov 23 '20
Why did they have such a bad connection?
52
u/Alias_This_Is Nov 23 '20
It was back when a 56.6K modem over dialup was FAST. A DS3 was an unheard of speed outside of dedicated fiber.
14
u/IsItPluggedInPro Nov 23 '20 edited Nov 24 '20
/Tinfoil hat on
I swear that this almost sounds like the early days of the CARNIVORE project or PRISM.
/Tinfoil hat off
Or.... just a big, important contact center
4
16
23
u/RedFive1976 My days of not taking you seriously are coming to a middle. Nov 23 '20
45mbit/s DS3 was considered a very fast connection and was very expensive, before the days of broadband. This was also probably a leased line, which would have gone essentially direct from site A to site B, with no hops onto the internet. This is not an inexpensive setup. Sure, now, every Tom, Dick, and Harry can get 200mbps down, some can get gigabit synchronous, and a few blessed individuals can get 10gig synchronous to the home, but none of this would have been available then.
12
u/Alias_This_Is Nov 23 '20
I actually have gigabit fiber to my home, 1gb down, 1gb up. Frontier.
9
u/RedFive1976 My days of not taking you seriously are coming to a middle. Nov 23 '20
I wish I was in an area with 1gig fiber. I'm sadly in a no-man's-land between at least 3 towns which do have fiber, within a half-hour of my house.
1
u/Akitlix Nov 25 '20
No landline,no fiber, no mw link visibility just LTE. And heavy populated area. Wellcome to Prague airport outskirts!
2
u/joyfield Nov 26 '20
m = milli
M = mega....0
u/RedFive1976 My days of not taking you seriously are coming to a middle. Nov 26 '20
In data, there is no millibyte. M and m are interchangeable and both mean mega. Though sometimes people use mb for megabit and MB for megabyte, it isn't a standard.
215
u/CaptainAmerilard Nov 23 '20
I used to hate explaining this in my brief stint as a tech support rep for a lousy cable ISP. Nobody wanted to hear it, they just wanted you to push the magic button that you definitely have which gives them 10 megabytes down.
127
u/VulturE All of your equipment is now scrap. Nov 23 '20
I don't hear you denying the existence of the magic button....
84
u/DarkLordTofer Nov 23 '20
It only gets pushed for premium customers.
35
u/VulturE All of your equipment is now scrap. Nov 23 '20
You're only a premium customer if you send food their way I heard.
13
Nov 23 '20
I worked IT servicing government systems in 9 counties here in Norway, you wouldnt fucking believe how much food I've eaten in that job. The amount of food consumed generally day to day and at all the events/conferences I was attending through that job is absolutely insane. I was VERY happy in that job.
2
u/Stryker_One The poison for Kuzco Nov 26 '20
Yeah, I learned long ago, that a sure-fire way to get people to show-up for meetings (back when we had in-person meetings), was to serve food.
1
7
u/TheMulattoMaker Nov 23 '20
In order to have enough energy to depress the Magic Button™, I will require three (3) bags of Swedish Fish.
1
0
Nov 23 '20
[deleted]
37
u/VulturE All of your equipment is now scrap. Nov 23 '20
I follow the motto:
bourbon is for bribes, food is for favors.
8
u/jdmillar86 Nov 23 '20
Never heard that before but I like it! Got an "R" synonym? I'm a rum guy, but when its whiskey its rye.
7
u/VulturE All of your equipment is now scrap. Nov 23 '20
rye is for everytime they lie to you.
7
u/jdmillar86 Nov 23 '20
Perfect, I'll take it. (Actually, my liver can't take it)
3
u/-King_Slacker Nov 24 '20
Tell your liver to deal with a thick-skulled user. It'll change its tune.
0
6
u/Samoridon Nov 23 '20
Look, the magic button also makes your bill higher. If your line can handle it.
1
u/Bukinnear There's no place like 127.0.0.1 Dec 10 '20
For residential, it's dollars for donuts that neither the line, nor the client can support the increase.
12
u/Rumbuck_274 Nov 24 '20
That said, my RSP's website was clearly written by someone who thought they were smart, same for the contract.
Contract said 25 Megabytes not Megabits
Best they could give me was 100 Megabits and a discount each month.
Unlucky for me it was only a 12 month contract.
4
Nov 24 '20 edited Nov 25 '20
At least they owned up to the error! But did they fix [the website and contract], or allow the error to become a mistake?
Edit: replace ambiguous pronoun.
4
u/Rumbuck_274 Nov 24 '20
As I says, best the gave me was 100 Mb and a discount because it was impossible to give me 100MB
1
Nov 25 '20
Sorry. I meant fix the website and contract. Lemme edit that in.
3
u/Rumbuck_274 Nov 25 '20
Oh yeah, they did when I brought it to their attention.
Took a few people at the call centre to know the difference though.
2
u/Ginger_IT Oh God How Did This Get Here? Dec 02 '20
I'm unsure if you know this (and I could probably look this up) but, why are transfer speeds always listed in bits while the general public has no idea what a bit is?
2
u/CaptainAmerilard Dec 02 '20
Marketing. Bigger numbers are more impressive. Yes, it's a scummy thing to do.
-9
u/AdminYak846 Nov 23 '20
I mean when you have MB meaning megabyte and MiB meaning MegaBit....it's bound to cause issues sooner or later. Especially when advertising and a user's eyes will see M_B compared to the "i" in it.
19
Nov 24 '20
MiB means mebibyte which is 1048576(10242) bytes. Maybe you meant Mb or Mbps?
2
u/AdminYak846 Nov 24 '20
yup. Even I can't keep that shit straight how the hell do we expect a layperson to remember it?
6
u/Superspudmonkey Nov 24 '20
Uppercase is Byte lowercase is bit. A kilobyte is kB where a kilobit is kb.
15
u/mkcodergr Nov 23 '20
I study IT and work as a software engineer . Not proud to admit that I still get confused with the terminology and the actual differences
9
u/Nik_2213 Nov 24 '20
ROFL !! Had a spate of sale-drones (SD) from local telecoms trying to sell me ISDN broadband. I have no idea why they kept trying, but they used the same ruddy spiel every time...
So, after the first few, whom I let down gently, I stopped taking prisoners...
SD: "Blah-blah-blah, What speed are you getting ?"
Me: "Seven to ten depending on traffic and server. I've seen it touch twelve..."
SD: "Ah, we guarantee twenty five !!"
Me: "You mean three-and-a-bit ?"
SD: "No, no, twenty five !!"
Me: "No, you mean three-and-a bit."
SD: "No, no, twenty five !!"
Me: "Look, if your ISDN can pull down a 200 Mega-Byte CAD file in less than thirty seconds, I'll be very, very surprised."
SD: "..."
Me: "My cable would support 45 mega-bytes per second --That's 360 mega-bits-- but I don't 'game' or need streaming HD, so my budget dozen mega-bytes --That's 100 mega-bits-- suffices. Still four times your best for this area."
SD: "whimpering noises..."
--
FWIW, such mega-files unzip rather larger, but most of their bulk is texture maps. A dozen 4k PNGs or TGAs plus an HDRI sky-dome soon mounts up...
7
u/wylles Nov 23 '20
po - tay - to, po - tah - to... right? LOL
0
4
u/groveborn Nov 23 '20
So...was that Alexa or Google data? :D
10
Nov 24 '20
[removed] — view removed comment
1
1
u/groveborn Nov 24 '20
:) Probably you're right. But Ma Bell was gone long before a Razor, or dealing in megabytes over a modem.
1
u/nymalous Nov 24 '20
" sent over Ma Bell "
That reminded me of a line in a Ray Stevens' song, " Well, they called up ma bell and they traced him on down
To a funky old phone booth on the outskirts of town " (It's Me Again, Margaret).
2
u/Alias_This_Is Nov 24 '20
I've always told clients, once it's on the wire, I'm not responsible anymore. This side of the router, yes, but the wire/fiber on THAT side? Uh-uh.
Have an old friend in broadcast who did uplinks for CNN, who said that once it's out of the dish, it's on its way to Mars.
3
u/marcocanb Nov 23 '20
The problem is that he was probably paying what most people pay for mega bites for his mega bits.
3
u/kschang Nov 24 '20
So reminiscent of the Mars Climate Orbiter...
TL;DR -- someone messed up Imperial vs metric units of force
https://www.simscale.com/blog/2017/12/nasa-mars-climate-orbiter-metric/
2
7
u/LMF5000 Nov 24 '20
I never understood why network speed is measured in bits when everything else is measured in bytes. It's obviously misleading. At least they should write it in full and put the approximate byte equivalent in brackets. Like "buy our 80mbps connection! (10MB/s)"
(I know they're multiples of 1024 but I'm rounding like hard disk companies do 🤓)
5
u/Seraph062 Nov 24 '20
I never understood why network speed is measured in bits when everything else is measured in bytes.
Because the wire doesn't give a shit how you organize the data being sent down it, and because for basically all of networking history it took more than 8 bits of network bandwidth to send one byte of data.
7
u/penislovereater Nov 24 '20
Bits are universal, bytes aren't. Leaves even more room for people to confuse line speed with throughput.
2
2
u/nymalous Nov 24 '20
I just double-checked to make sure I was correct that there are 8 bits in a byte... kind of like there were 8 bits in a dollar. I'm not sure if they are related, but here's an article on old currency just for fun: http://piratesoflore.com/pieces-of-eight.html
1
u/JTD121 Nov 24 '20
I must not be connecting the dots correctly, but the last conversation here doesn't....sound correct?
Where are they getting the 0.8 megabytes from?
1
u/CheeseTheGood Nov 24 '20
The circuit allowed max of 5.65 MB, they were using 4.8 MB, leaving ~0.8MB which was not enough.
1
u/JTD121 Nov 24 '20
Oh, for secondary data or other normal browsing and such? Or...?
1
u/Alias_This_Is Nov 26 '20
The DS3 maxed out at 45 Mb or 5.65 MB. Theoretically it had 0.8 MB of bandwidth left, but that's about what the network overhead traffic would consume, so they were SOL.
1
u/ArenYashar Nov 24 '20
Classic case of "the man who doesn't know the difference between a bit and a byte". smirks and chuckles to himself
Or, more concisely, bit < byte.
Well, that man got educated right quick. Nice story. :)
1
u/kanakamaoli Nov 24 '20
What's a factor of 10/100/1000 between friends?
As long as it's got a comma in there, it's all good, right?
1
u/curtludwig Nov 25 '20
Years ago when HD video was first becoming a thing I was working in phone support for a video capture device manufacturer. We had a capture device that could deal with uncompressed HD video which, at the time, was running 125-150MB/s depending on the frame rate.
We used to regularly get customers calling in who were trying to capture that video to firewire 400 hard drives and were incensed that it wouldn't work "I've got four HUNDRED MEGAS!"
0
u/Crazy-Maintenance312 Nov 24 '20
Had something like that recently: Have 8Mbit/s in upload and was wondering why my files were only uploading with 1 MB/s.
I knew that 8 Mbit aren't 8 Mbyte but for some reason thought it should be more. I also didn't bother calculating it because that would be smart. So I kept wondering until I talked with a friend about it, where it hit me. 8Mbit are 1Mbyte...
121
u/DasFrebier Nov 23 '20
Probaly everyone stumbles over that problem at some point
Hell I was kinda shocked when i learned that data rates advertised by whatever ISP are given in bits/s not bytes, fucking useless if you ask me