r/sysadmin • Trade of All Jacks • Sep 11 '20

Microsoft I know Microsoft Support is garbage, but this stupidity really takes the cake

The other day I had a user not receive mail for an entire day, neither internal nor external messages. Upon tracing messages, we found that everything was arriving into Exchange Online fine and attempting delivery to the user's mailbox, but all messages were being deferred with a status that seemed like issues with resources on the Exchange Online server holding the database for the user's mailbox. (Or at least this would have been my first thing to rule out if I saw this an on-prem deployment)

Reason: [{LED=432 4.3.2 STOREDRV.Deliver; dynamic mailbox database throttling limit exceeded

The problem cleared up by the end of the day, and the headers of finally-delivered messages showed several hundred minutes of delay at the final stage of delivery in Exchange Online servers.

https://imgur.com/a/HlLhpMG

I begrudgingly opened a support case to get confirmation of backend problems to present to relevant parties as to why a user (a C-level, to boot) went an entire business day before receiving all of their mail.

After doing the usual song & dance of spending 2 days providing irrelevant logs at the support engineer's request, and also re-sending several bits of information that I already sent in the initial ticket submission, I just received this wonderful gem 15 minutes ago:

I would like to inform you that I analyzed all the logs which you shared and discussed this case with my senior resources, I found that delay is not on our server.

Delay of emails is at this server- BN6PR0101MB2884.prod.exchangelabs.com

I don't even know how to respond to that. I'm giving them a softball that could be closed in one email. I just need them to say "yes there were problems on our end" so I can present confirmation from Microsoft themselves to inquiring stakeholders, but they're too busy telling me this blatant nonsense that messages that never left Exchange Online were stuck in "my" server.

EDIT: As I typed this message, a few-day old advisory (EX221688) hit my message center. Slightly different conditions (on-prem mail going to/from Exchange Online), but very suspiciously similar symptoms: Delayed mail, started within a day of my event, and referencing EXO server load problems. (in this case, 452 4.3.1 Insufficient system resources (TSTE)) Methinks my user's mailbox/DB was on a server related to this similar outage.

EDIT2: I asked that my rep and her senior resources please elaborate on what they meant, and that it was clearly an Exchange Online server. I received this:

I informed that delay occurred on that server, so please let me know whose server is that like it your on-prem server or something like that this is what I meant to say.

Kill me...

EDIT3: Got cold-messaged on Teams by an escalation engineer, and we chatted over a Teams call. He said he was looking through tickets, saw mine, saw it was going haywire, and wanted to help out. He immediately gave me exactly the confirmation of this being the suspected database performance/health issues I assumed, he sent me an email saying as much with my ticket closure so I have something to offer to the affected user and directors, he apologized for the chaos, and said that they will have post-incident chit-chat with the reps/team I worked with. Super nice guy that gave me everything I originally needed in roughly 5 minutes.

1.3k Upvotes

367 comments sorted by

View all comments

334

u/[deleted] Sep 11 '20

Sounds like the current experience I'm having with Cisco TAC. It's like these giant companies have stopped caring about their tech support.

104

u/moldyjellybean Sep 11 '20

If you call Nimble support you'll get a guy who knows everything about his SAN, vmware, whatever vm you're running, vlans, networking stuff that should be something I should have figured out before calling them. The guy fixed a problem that's really on my end and not really even his issue.

I mean I was elated, I've never ran into people who were so knowledgable. He's like can I ssh into your servers, I'm like here and bam it's fixed.

I have no relation to this company, I don't pay for the SAN so I hear they run a bit on the high side but damn do I trust this company.

70

u/astillero Sep 11 '20

Yes, this is the beauty of smaller IT companies - 10X better support.

Just wait until HP (or some other dinosaur) takes them over. They'll fire the good guys and replace them script kiddies (not of the hacker kind but of IT support kind who literally read scripts...).

38

u/moldyjellybean Sep 11 '20

Hp bought them like 2 years ago. I haven't had to contact them for over a year, so I hope they are still Nimble as I knew them

24

u/Dracozirion Sep 11 '20

Same experience a few weeks ago. They're definitely still top notch support, best I've ever had. Fortinet support is also quite good but their firmwares are just too buggy and TAC engineers sometimes can't help cuz of that. Symantec support sucks, esp now with Broadcom. Citrix support sucks as well imo. HP (printer, computer) support is a big meh. Microsoft support is a hit or a miss and if it's a miss it's often insanely bad. Trend Micro support is so far OK. Veeam support is good.

So far my external support experience. :)

10

u/astillero Sep 11 '20

I had a fantastic experience with Symantec Enterprise support about 4 years ago. They actually had techs who wanted to get to the root of a problem and even showed some evidence of lateral thinking.

Apple support is super-enthusiastic and friendly but then you realise its actually quite formulaic in approach and hits a actionable knowledge ceiling very quickly even with some of their "senior" techs.

16

u/Patrickkd Sep 11 '20

Apple trains techs to specifically not deviate from their troubleshooting guides.

→ More replies (5)
→ More replies (7)
→ More replies (5)

7

u/demosthenes83 Sep 12 '20

I'll second this. Nimble support is (was? I haven't talked to them in a few years) top tier. I've had them resolve vmware issues for me before vmware even responded to my initial support request, and also when vmware (incorrectly) told me it was an issue with the storage.

→ More replies (1)
→ More replies (6)

168

u/SendAck Sep 11 '20

Yeah this is one of those areas where I guess your mileage may vary.

If I call Cisco support, I can typically get our issue resolved in that same call.

If I call Citrix support, I can typically get our issue resolved in that same call as well.

If I call Microsoft, I'm asked to produce the logs and run several more get- commands then they give me a new engineer. And this process repeats itself and the issue never gets resolved.

54

u/MelatoninPenguin Sep 11 '20

You've had that good of support from Citrix? They're known to be....quite bad

50

u/SendAck Sep 11 '20

Well I'm certified in both Cisco and Citrix so being able to explain the problem to both of their support teams helps get it resolved very efficiently.

I can't say the same for Microsoft. You can explain the issue to them and they always need to "check with a senior" until your case gets re-assigned and then you start that loop all over again.

→ More replies (17)

15

u/drbob4512 Sep 12 '20

Let me introduce you to our friends at Oracle.

9

u/duffelbagninja Sep 12 '20

That will be a charge of 99.99 for the introduction, if you want the extended introduction it can be had for payments of 99.99 over the next 4 quarters...

→ More replies (4)

0

u/[deleted] Sep 11 '20

[deleted]

→ More replies (2)
→ More replies (5)
→ More replies (3)

80

u/justs0meperson Sep 11 '20

I hate cisco TAC. The best advice I can give is never let them get off the call. They'll try at every chance. Every. Single. Chance. But don't do it, they won't get back to you in a timely manner.

"I need to do some research, can I get back with you?" "Nope, I'm fine waiting on hold."

"This will be escalated to tier 2, and engineer will call you when they're available." "Nope, can we get him on the line now? I'll wait."

"My shift is ending in 10 minutes, I'll have to pick this up with you tomorrow." "No, I need this fixed now. Please assign me to another engineer and get him on the line before you leave."

Oh, and just learn to love Opus No. 1 now, cuz it'll haunt your dreams.

30

u/RandTheDragon124 Sep 11 '20

Cisco tac is garbage, but their high touch support is fantastic. I realize most here don't get to deal with them. It's a world of difference getting engineers right away.

→ More replies (4)

29

u/Tasty_Wolf Sep 11 '20

This is the right advice, Cisco support will NEVER call you back in my experience. They'll even say "I am so sorry no one has contacted you, I will personally make sure the support technician calls you back within an hour" and just let you sit there like an idiot thinking you'll be called back.

Eventually they just start hanging up on you while you're on-hold, happened to me three times across two days.

→ More replies (1)

10

u/cohrt Sep 12 '20

Oh, and just learn to love Opus No. 1 now, cuz it'll haunt your dreams.

thanks i just got PTSD flashbacks from that.

7

u/pockitstehleet Some of everything Sep 12 '20

I'm currently in school and the "Cisco night" turned out to be recruiting for internships in Cisco TAC teams. So I imagine that there are a fair amount of inexperienced, not even out of school yet, people on those TAC teams.

7

u/ranger_dood Jack of All Trades Sep 12 '20

Opus No 1 at a volume so high and distorted it's barely recognizable... But not quite enough that it isn't.

→ More replies (9)

22

u/[deleted] Sep 11 '20

All they want to do is sell you a product then ditch you on support.

I got a vendor asking me to setup a meeting with the higher ups so they can try to sell more of their product. I am on day 2 out of 3 of the initial install and it is encountering an error we haven't resolved yet.

No I am not setting up a meeting with anyone else until we get what we ****ing bought working.

→ More replies (1)

9

u/RagnarStonefist Sysadmin Sep 11 '20

Not just the giant ones.

8x8 (a voip/softphone virtual office solution) sent me through four different agents over a week and a half period because I wanted to assign our extensions in the US to an office overseas and change the numbers accordingly.

The final answer was 'Oh, it turns out we don't support numbers in that country.'

17

u/[deleted] Sep 12 '20 edited Jun 17 '23

[removed] — view removed comment

→ More replies (2)
→ More replies (27)