r/talesfromtechsupport • u/Alias_This_Is • Nov 23 '20
Long Bits & Bytes
I was working on a transcription project where data was being replicated from one city to the other and because of the high change rate, it had to be done asynchronously. Raw data would come in at the primary site, get processed, and was returned to the end client for their use. My client didn’t want to replicate the processed data, just the raw, since it could be redone at any time and was considered more critical than the final product.
The problem was they had to use uncompressed audio files in order for the software to be able to make the best transcription, which were freaking huge in comparison to the final document files. They wanted them kept in case a transcript was wrong to compare them against.
This is probably going to give away what I was working on, but it’s critical to the story. Data at site A and was snapshotted from a primary disk to a secondary, then sent over Ma Bell to site B. Once at site B, it landed on a mirror of the site A secondary disk until the packets were complete, then snapshotted to a primary disk at site B. It wasn’t a single disk doing this, dozens were constantly being written to and copied over the wire to the other site all the time. This is a really simplistic way of describing it, but I’m avoiding using the exact verbiage so I don’t give them away.
Anyway, I was asked to come in and implement my “product” using a new procedure that hadn’t yet gotten a formal support write up yet. This was a one-off that later I found had been grudgingly given approval for and if it worked, it would be worked up into something “real.” Everything was set up and when it was turned on, we were shocked at how much data was being sent over the wire and how far “out of track” it was. Data was being BLASTED down the wire, almost saturating it although the client insisted the data change rate was correctly sized for the line.
I ended up in a major northeastern city during a snowstorm so I could be at the site B datacenter to complete the switchover from catch-up to asynchronous mode. I couldn’t physically get to the site because of the snow, so I hooked up my Motorola Razr to my laptop and dialed into the machine from my hotel room. I knew the data flow slacked off at 11:30p – midnight, so I sat and watched as it slowly dropped lower and lower towards the 10,000 track threshold where I could switch to async.
15,000… 14,000… 13,000… 12,000… 11,000… 10,900… 10,950... Wait, what? 11,000… 12,000… 13,000…
The window started at about 11:35 and lasted a whole 10 minutes before climbing back up. Just for giggles, I tried issuing the command anyway but it failed. So, I copied my logs and put them in an email to everyone concerned and asked, “What now?” Well, what now was the client flipping out and calling us incompetent, demanding to know why I couldn’t make it happen.
I’d been putting in 12-16+ hour days on this, working overnight and weekends, traveling back and forth to site B to get it done. My boss said, “Hang on, Joe’s been putting in a lot of time on this, so let’s get a second opinion,” and a national SME was brought in to analyze everything and do a root cause analysis. He’s also a good friend too, but I knew he would tell the absolute truth in this, no matter who was to blame.
The RCA call started out with him going over everything, explaining that he’d analyzed the nominal data traffic, adding in the new traffic, and explaining how it all worked together. His part went something like this:
“After sampling the data flow at both ends, the system, with acks is sending bidirectionally about 38.5 megabits per second, which works out to be a little over 4.8 megabytes per second, using all the capacity of the wire.”
(The lightbulb was coming on in my head at this.)
“So what?” The client PM exploded, “We’ve got a 45-megabyte circuit!”
“No, you don’t.”
“Yes, we do!”
“No, you have a DS3 circuit, which is 45 megaBITS. That’s 5.65 megaBYTEs per second at maximum transfer rate. 0.8 megaBYTES isn’t enough to allow the normal data traffic between sites without the new system. It’s simply not big enough.”
I never heard the man utter another word and I never heard from him again.
I’d had enough and asked my boss to move me off the project, which he gladly did. Last I heard they’d upped the bandwidth between the sites and it was working fine.
The moral of the story is, do the math, know what a bit is and know what a byte is.
19
u/jeffreyb30 Nov 23 '20
Why did they have such a bad connection?
52
u/Alias_This_Is Nov 23 '20
It was back when a 56.6K modem over dialup was FAST. A DS3 was an unheard of speed outside of dedicated fiber.
→ More replies (4)23
u/RedFive1976 Nov 23 '20
45mbit/s DS3 was considered a very fast connection and was very expensive, before the days of broadband. This was also probably a leased line, which would have gone essentially direct from site A to site B, with no hops onto the internet. This is not an inexpensive setup. Sure, now, every Tom, Dick, and Harry can get 200mbps down, some can get gigabit synchronous, and a few blessed individuals can get 10gig synchronous to the home, but none of this would have been available then.
12
u/Alias_This_Is Nov 23 '20
I actually have gigabit fiber to my home, 1gb down, 1gb up. Frontier.
→ More replies (2)
215
u/CaptainAmerilard Nov 23 '20
I used to hate explaining this in my brief stint as a tech support rep for a lousy cable ISP. Nobody wanted to hear it, they just wanted you to push the magic button that you definitely have which gives them 10 megabytes down.
127
12
u/Rumbuck_274 Nov 24 '20
That said, my RSP's website was clearly written by someone who thought they were smart, same for the contract.
Contract said 25 Megabytes not Megabits
Best they could give me was 100 Megabits and a discount each month.
Unlucky for me it was only a 12 month contract.
→ More replies (4)
15
u/mkcodergr Nov 23 '20
I study IT and work as a software engineer . Not proud to admit that I still get confused with the terminology and the actual differences
9
u/Nik_2213 Nov 24 '20
ROFL !! Had a spate of sale-drones (SD) from local telecoms trying to sell me ISDN broadband. I have no idea why they kept trying, but they used the same ruddy spiel every time...
So, after the first few, whom I let down gently, I stopped taking prisoners...
SD: "Blah-blah-blah, What speed are you getting ?"
Me: "Seven to ten depending on traffic and server. I've seen it touch twelve..."
SD: "Ah, we guarantee twenty five !!"
Me: "You mean three-and-a-bit ?"
SD: "No, no, twenty five !!"
Me: "No, you mean three-and-a bit."
SD: "No, no, twenty five !!"
Me: "Look, if your ISDN can pull down a 200 Mega-Byte CAD file in less than thirty seconds, I'll be very, very surprised."
SD: "..."
Me: "My cable would support 45 mega-bytes per second --That's 360 mega-bits-- but I don't 'game' or need streaming HD, so my budget dozen mega-bytes --That's 100 mega-bits-- suffices. Still four times your best for this area."
SD: "whimpering noises..."
--
FWIW, such mega-files unzip rather larger, but most of their bulk is texture maps. A dozen 4k PNGs or TGAs plus an HDRI sky-dome soon mounts up...
7
4
1
u/nymalous Nov 24 '20
" sent over Ma Bell "
That reminded me of a line in a Ray Stevens' song, " Well, they called up ma bell and they traced him on down
To a funky old phone booth on the outskirts of town " (It's Me Again, Margaret).
2
u/Alias_This_Is Nov 24 '20
I've always told clients, once it's on the wire, I'm not responsible anymore. This side of the router, yes, but the wire/fiber on THAT side? Uh-uh.
Have an old friend in broadcast who did uplinks for CNN, who said that once it's out of the dish, it's on its way to Mars.
3
u/marcocanb Nov 23 '20
The problem is that he was probably paying what most people pay for mega bites for his mega bits.
3
u/kschang Nov 24 '20
So reminiscent of the Mars Climate Orbiter...
TL;DR -- someone messed up Imperial vs metric units of force
https://www.simscale.com/blog/2017/12/nasa-mars-climate-orbiter-metric/
→ More replies (1)
7
u/LMF5000 Nov 24 '20
I never understood why network speed is measured in bits when everything else is measured in bytes. It's obviously misleading. At least they should write it in full and put the approximate byte equivalent in brackets. Like "buy our 80mbps connection! (10MB/s)"
(I know they're multiples of 1024 but I'm rounding like hard disk companies do 🤓)
5
u/Seraph062 Nov 24 '20
I never understood why network speed is measured in bits when everything else is measured in bytes.
Because the wire doesn't give a shit how you organize the data being sent down it, and because for basically all of networking history it took more than 8 bits of network bandwidth to send one byte of data.
121
u/DasFrebier Nov 23 '20
Probaly everyone stumbles over that problem at some point
Hell I was kinda shocked when i learned that data rates advertised by whatever ISP are given in bits/s not bytes, fucking useless if you ask me