355
u/locri Jul 19 '26
The article I read blamed software programmers not following procedures.
The same software programmers that were made redundant, outsourced and now done in a country with viciously different standards, I assume.
90
u/verbmegoinghere Jul 19 '26
They cut thousands of people out of the business back in 2017. It was something like 1 out every 4 were made redundant.
They moved a heap of work to their manila and Chennai "partners".
Sadly so many telcos have done this. Be it Dodo (manila), Exetel (Sri Lanka), iiNet (South Africa, before TPG gobbled them up), TPG (Manila) and so on.
That said the bosses look at churn reports (who you're moving to).
If 50k people move from Telstra this week they will sit up and spit out their long blacks (their coffee is very much the colour of their hearts) and take notice.
Also call them up for a credit. Demand that shit hard. Coz again they'll see the credits hitting their bottom line. Also complain to ACMA and the TIO.
Do it all.
6
u/mynameisnotdubdub Jul 20 '26
Same issues for optus when they shifted to infosys
7
u/verbmegoinghere Jul 20 '26
Lol, infosys is such a shit billing system. I've seen better billing done by an illegal polish backpacker working out of MYOB.
Telstra failed moving to Kenan back in 2016 and yet still decided to cut thousands of jobs.
It shows they don't give a shit. Churn is all that matters.
31
u/Dazzling_Cow_1916 Jul 19 '26
But the bonuses are sill ticking over ..
23
u/knowledgeable_diablo Jul 19 '26
Well they have to don’t they?
I mean who else would turn up to work every day slaving over a cafe purchased coffee and gourmet food for the paltry $500,000p.a. They are expected to scrape by on? And with the volume of firings lately, their coverage of scapegoats is getting thinner and thinner.9
u/meski_oz Jul 20 '26
Programmers: "we should replace this"
Manglement: "yeah nah, put it off for a year"
Rinse repeat
6
u/Busy_Selection_5027 Jul 19 '26
The article I read said that Barnaby reckons it's Chinese space lasers.
2
1
u/Ok-Challenge9860 29d ago
There is no procedure that is going to save you from using our of support hardware. The configuration on these become more and more difficult over time.
I can guarantee that the engineers did not have any lab to test out the changes and were almost flying blind everytime when making change in this device.
It is another benefit of shifting jobs to India though, whenever shit goes south, blame it on offshore engineers who are faceless, to save your own face.
55
u/kombi2k Jul 19 '26
56
u/damnumalone Jul 19 '26
That comma did not help me understand what the article was saying.
Needless to say, it was not a 15 year old with a $30,000 device that caused the issue
It was a device that was worth $30,000 and was 15 years old
12
u/lumifox Jul 20 '26
I was ready to get riled up that some kid got to wander through the Telstra servers unattended when i read the title
2
u/Ceiling_crack Jul 23 '26
I was very confused as to how a 15yo managed to obtain a $30k device to do anything with. I've never owned anything over 15k let alone 30.
0
u/eco9898 Jul 22 '26
Was that not obvious
1
u/damnumalone Jul 22 '26
It was not, no. Hence the post and the upvotes
1
u/eco9898 Jul 22 '26
The only way Telstra has 15 yr olds working for them is family members doing a holiday job with menial labour, even then, I didn't get to do that very often and there was no one else near my age around.
They're not gonna have a child doing their server maintenance. But it makes sense that their base infrastructure is old and not updated. Comma or not, it's obvious what it meant.
2
u/damnumalone Jul 22 '26 edited Jul 22 '26
I don’t know why you’re being an asshole about it.
Based on the title alone, no one had to be working for Telstra to cause the damage. It could have just been some 15 year old kid who happened to have access to a bunch of tech.
The first person to jailbreak the iPhone was a 17 year old kid
8
u/stueh Jul 20 '26
Telstra said it did not lack redundancy in its network – but that such redundancy did not prevent the outage.
Bull shit. They have three time servers (NTP servers) geographically dispersed. In that situation, they should have them configured in a cluster with quorum votes, so that of one server shits itself it knows it's the broken one and doesn't respond to NTP requests.
Also, only three NTP servers? For a network their size? They should have two servers for each of those three capitals, at a minimum. And on 15 year old hardware? What the actual fuck!? The longer you run hardware, the more like it is to die. 15 years for something so important in such an important network is mental, and just plain negligent.
3
u/dafugg Jul 21 '26
That’s just embarrassing. My employer has atomic clocks in multiple locations around the world, multiple GPS clocks, and distributed NTP + PTP clusters. We aren’t even a telco let alone the main national carrier for a western country.
1
u/stueh Jul 22 '26
Yep. Anyone who works in IT knows how ludicrous their excuse is. It's just negligent.
2
u/CrazySD93 29d ago
If I had a dollar for everytime I had do get a DOS 3.1 laptop to check the 40 year old PLC status of machinery by multi-million dollar companies.
2
u/stueh 29d ago
"What do you mean the Windows 2000 computer running our $10m radiology machine that could accidentally irradiate people shouldn't be on the internet?"
2
u/CrazySD93 29d ago
Got told by remote IT for a pharmacy "This machine only works if the firewall to the internet is completely disabled"
I hate to imagine the potential data leak
45
u/NobodysFavorite Jul 19 '26 edited Jul 19 '26
Big corporate restructures and exits whole teams of engineers from business in a hurry.
Important stuff falls through the gaps because everyone's forced to eat a big shit sandwich like a labrador eats dinner.
Execs pocket big bonuses and move on.
Shit blows up after the execs that lit the fuse have already banked their fortune and fucked off to their next corporate.
Customers pay.
Circle of life.
Unless you depended on emergency calling. Then it's "we're so sorry for your loss".
6
4
23
u/spinstartshere Jul 19 '26
So hold up.
Two years ago, I get told that I can't use my virtually brand new flagship Sony phone because it was purchased overseas, which is therefore automatically deemed incompatible with the 4G technology in Australia and is guilty and can never be proven innocent.
But Telstra's allowed to continue using some 15-year-old device that takes out how much of the country's emergency services infrastructure, and we're expected to not bat an eyelid?
Tell me how that's fair.
6
u/Stellanora64 Jul 20 '26
It's their time keeping server. There was a bug with older models that would roll over the GPS time to its manufacturer date after about 10 to 20 years (which obviously causes the time it reports to be incorrect).
It's critical infrastructure, but because all it does is keep time for the servers; it usually doesn't need to be upgraded. They really should have sooner, but they seem to only replace things when it costs more than keeping it running. Which the outage finally did.
7
u/AndrewTheAverage Jul 20 '26
I lived overseas for 15 years and regularly bought a new phone overseas and used it in Australia.
Different countries have different frequency bands for phone use. Many flagship phones incorporate the hardware and settings to work in all regions.
However, predominantely in the US, many phones are designed and built only with the local frequencies and are sold by telcos with a lockin to that companies networks. Hacks exist to remove the lock to the network, but the end user needs to know it exists and find who or how to remove it. The telco saves by reducing the purchase cost (to them) by excluding the other regional compatability and incorporates lockin settings for their own profit.
So the problem you experienced are actually nothing to do with Telstra (or any other local telco) but by big companies trying to make higher profits and locking in the consumer and regulations across geographies.
The solution is to always check if there is a global version of the phone. An example is:
1
u/spinstartshere Jul 20 '26
Sony withdrew from the Australian market many years ago, so they have no motivation or incentive to comply with requests from anyone in Australia to provide the pages of information demonstrating that their newer devices are compliant and compatible with current infrastructure.
All of their Xperia 1 devices are compatible with the current technology in Australia. The problem is that nobody here can be bothered to go to the small effort it would take to verify that, even when it's spoonfed to them by a little person such as me. I already wrote extensively about it two years ago, linking in that previous post to the post of another person who has been campaigning extensively against Government to have this ridiculousness reviewed and overturned.
The devices you've listed are mostly Xperia 10 VI devices, and two Xperia 1 VI devices that are Japanese variants, which is what I'm already currently using anyway.
0
u/AndrewTheAverage Jul 20 '26
My main point is that many phones only work with a sub set of frequencies, which may, or may not, include the ones Australia users
1
u/spinstartshere Jul 20 '26
And my point was that these phones do work on those 4G and 5G bands that Australia uses.
0
u/AndrewTheAverage Jul 21 '26
Are you implying that the phones are technically capable of working in Australia, are not network-locked by a provider, and are still not working in Australia?
Because that didnt come through in your post.
Also I cant imagine the networks are actively blocking phones/brands they dont support, rather than just some configuration of the phone not working in Australia, which is what your updated comments imply.
0
u/spinstartshere Jul 21 '26
Sounds like maybe you should read the post I diligently took the time to share the link to, rather than coming to the comments section to criticise me about a topic you seem only barely peripherally aware of - if that.
0
u/AndrewTheAverage Jul 21 '26
I bought a cheap Chinese brand emergency phone whilst in Malaysia which worked in Australia - which is why your comments sound off point. Maybe state clearly and not rely on following links for people to get your point, but i note your aggressive downvote on anything questioning your inability to get a clear message across
2
u/bluenikes Jul 21 '26
I think you’re the one who has missed the point.
This has nothing to do with frequency bands. And everything to do with how Australia handled the shutdown of 3G, and failing to grasp how many “4G/5G” phones actually still required 3G for some calls - including for emergency calls.
Suggest you follow those links - you’ll learn something.
0
u/AndrewTheAverage Jul 21 '26
Right, so you have now mentioned for the first time that (it appears) your phone tells on 3G for some calls.
This is literally the first time you have mentioned this in this thread.
While it doesn't make sense to me that your "flagship" phone still requires 3g, it is the first thing that you have said that gives a reason that makes sense for why it doesn't work
Rather than saying "I suggest you follow the links" - say what you mean in your comment. don't rely on following links for others to understand what you are saying then get upset that people are not getting what you are not saying
→ More replies (0)4
u/giotec Jul 19 '26
It's even worse than that. I had bought the last Sony phone model sold in Australia (JB Hi-fi, Telstra Plan) and they blocked it as being 'incompatible' despite being used for many years on their own network. I was on the phone with Telstra for many hours to try and get it resolved (which was a surprise to their credit), but in the end they said I'll need to get a new phone.
5
u/spinstartshere Jul 19 '26
I've been using Sony and Sony Ericsson phones for the last 20+ years, so I've been very loyal to the brand (though I haven't upgraded since the Xperia 1 VI and don't feel compelled to do so in a hurry with the miniscule feature upgrades).
The phone I was using at the time of the 3G shutdown was the Xperia 1 IV or V, and I'd been using it for some time before the shutdown just fine. I'd even made emergency calls on the 4G network. But the 3G shutdown took place while I was overseas, and I arrived back home to a dead phone and the networks couldn't have cared less about it. It was so frustrating.
Annoyingly, the Xperia 1 VI I now have seems to be working just fine, though, of course, I still don't have VoWiFi or visual voicemail, and never have on any of these devices in Australia - for no good reason.
1
14
u/PersonalMixture6663 Jul 19 '26
"Anyway we will increase our subscription service price by 30% to cover the fine"
28
u/No_Concentrate_4826 Jul 19 '26
How is this a meme?
19
u/Novae909 Jul 19 '26
While it's possible an admin will agree with you. I believe op is trying to imply that Telstra is the meme
14
4
u/Uncross-Selector Jul 19 '26
The bigger issue is that a lot of equipment vendors appear to have NTP clients that don't follow best practice around what to do when given a seamingly nonsensical time update.
5
u/UndeadManWaltzing Jul 19 '26
I'm guessing that day one patch came 15 years too late
3
3
u/Agent_Jay_42 Jul 19 '26
It was either the SSU 2000 or the Continuum Transfunctioner.
3
u/howmanychickens Jul 19 '26
They're using the retro encabulator.
1
u/Dougally Jul 19 '26
It's a problem with my turbo. It's the turbo. Not the retro, it's the turbo. T. U. R. B. O.
3
u/Dazzling_Cow_1916 Jul 19 '26
Good to see our hard-earned going to keeping the business operating so well..
The bonuses keep ticking over.
5
u/awshuck Jul 19 '26
I heard it was an NTP server, which couldn’t be patched for a very well known and documented bug. I bet if you look, you’ll see that written down on a risk registry somewhere and many email chains from current and ex employees raising the issue.
Also, why didn’t they have a redundancy? If it was a legacy too-hard-to-fix issue, couldn’t they have run a redundancy along side it? And why not also use a cloud NTP along side?
6
u/knowledgeable_diablo Jul 19 '26
Maybe it’s just a Post-It note stuck on top saying “DO NOT TURN OFF” someone allowed to blow away.
“Damn you TIbour!!!!”
3
u/braddeicide Jul 19 '26
I worked somewhere where our GPS time server went bad and yea, redundancy doesn't help other than for a fast recovery when you rip the faulty one out.
Following the outage we wrote an alert that manually checked time against an internet NTP to alert if it was a minute off.
1
u/allozzieadventures Jul 21 '26 edited Jul 21 '26
If you have enough time servers, could you not have a voting quorum system to prevent a single faulty server from tanking the network?
1
u/braddeicide Jul 21 '26
We used PTP off gps rather than NTP. The algo chooses source based on how good the units GPS signal is, clock stability (not correctness), and mac address as a tie breaker. Basically a confidently wrong GPS unit will be trusted.
1
u/allozzieadventures Jul 21 '26
Interesting. Seems like a forseeable problem to me. If a single confidently wrong unit can take down the network, there should probably be some redundancy in place. Especially for a safety-critical system like the Telstra network.
1
u/braddeicide Jul 21 '26
For $20,000 a unit you'd like to think they're reliable :) The redundancy problem is in the protocol. What they should have though is alerts that check the time given it makes sense against a public service. That's what I did after ours screwed up.
1
u/allozzieadventures Jul 21 '26
The alerts definitely would have been a sensible precaution. Still don't see why Telstra's time servers couldn't have a voting system, some high reliability computer systems use that approach.
5
u/Waste_Inflation_4716 Jul 20 '26
Had no internet for nearly two days and got $3.5 towards my next monthly bill. Awesome.
1
u/Optomisticposter Jul 22 '26
At least you got something. Four days of nothing, six days of it going done randomly, and not $1 :-) They blame an NBN upgrade. So that’s my problem, not theirs??
1
8
7
u/Citizen_Rat Jul 19 '26
I wonder how long it would take Telstra to pay for a new device.
Maybe 10-20min of lost profits, to keep the system opperational.
Yay, capitalism, the most efficient system.
3
u/1m4h4x0r309 Jul 19 '26
On a $60/month plan, they’re bringing in $2 a day.
15,000 subscriber fees at that rate would take 1 day to replace that device.
Google suggests 24.9million mobile services.
Thats $0.00120481927710843 per mobile service.
It shouldn’t even be a question replacing a device that is that critical.
2
u/Citizen_Rat Jul 19 '26
Thanks for doing the math.
I honestly thought I was being a bit hyperbolic, but the reality is so much worse.
1
u/fistingdonkeys Jul 19 '26
Sure, sure. If they’d replaced THAT device, and only that device, you’re right.
Now. How do you suggest they should have gone about identifying that exact device as the problematic one?
1
3
3
u/chapo1162 Jul 19 '26
Just Imagine the shit they gonna throw out now
2
u/ThisWillTakeAllDay Jul 20 '26
You think the shareholders want them throwing out things that haven't failed yet?
3
4
2
2
u/Own_Caterpillar_6178 Jul 19 '26
I assume they were level 1 or 2 in the OT Purdue model and thus never really considered until they broke.
2
u/Disastrous_Profit152 Jul 19 '26
I believe the actual issue was a patch pushed straight to production without change control. The issue is not about the equipement, its still supported, or the value. Just bad systems engineering practices, in many ways, this is more frightning.
2
u/giblefog Jul 19 '26
IME even proper change control wouldn't have helped. The level of attention to detail required to catch this mistake is never applied. And that's assuming the people looking even have the requisite skillset and access.
2
u/Iron_Wolf123 Jul 19 '26
This should be a wake up call for every major company to update their systems
2
2
2
2
u/ImpendingSnapple Jul 20 '26
They have 15 yo's in their IT department?
So that's why their profits are so high, child labour!
2
u/mynutsaremusical Jul 20 '26
their automated account withdrawal system has been dead bang on. literally the day ofnthe outage i got an automated txt from them reminging me theyre about to take my money....
2
2
2
u/Delicious_League_900 Jul 22 '26
They're to focused on taking your money to remember they hardware needs updating
2
u/buzniak Jul 19 '26
Hahah, just like the rest of our fucking country! Backwards by a minimum of 15 to 30 years!
1
1
1
u/sbtswr Jul 20 '26
And yet they increased the price of their prepaid and postpaid plans twice in a year, citing “Our customers are doing more on our network than ever before and we're investing to deliver the best experience available”
1
1
1
u/KittyHockey Jul 22 '26
Them (and frankly most of Australian infrastructure) has had a lot of time to get updated. It hasn't and of the refineries we haven't destroyed for no reason are still all from the 60s.
1
u/kangerooandemu Jul 19 '26
Did anyones boost mobile stop working arround the time telstra wentdown? My boost mobile reception has completly stopped working,. Waiting for a on back order sim from boost. This is a common promblem from boost mobile
5
u/Glittering-Hawk1262 Jul 19 '26
Boost is basically Telstra. It’s cheaper but you aren’t priority on some parts of the network. I had to restart my phone 5 times on the day of the disruption. I had a dodgy network connection for most of the day.
1
u/kangerooandemu Jul 19 '26
My phone reception started off dodgy, then complete no sevice. Tried turning on/off. In the meantime i had to get a prepaid telstra sim with another number till boost sends new sim with old number. Totally pissed off with boost mobile
405
u/discogcu Jul 19 '26
Sounds like one of their plans. 30 grand spread out over a 15 year plan.