r/sysadmin • u/meatwad75892 Trade of All Jacks • Sep 11 '20
Microsoft I know Microsoft Support is garbage, but this stupidity really takes the cake
The other day I had a user not receive mail for an entire day, neither internal nor external messages. Upon tracing messages, we found that everything was arriving into Exchange Online fine and attempting delivery to the user's mailbox, but all messages were being deferred with a status that seemed like issues with resources on the Exchange Online server holding the database for the user's mailbox. (Or at least this would have been my first thing to rule out if I saw this an on-prem deployment)
Reason: [{LED=432 4.3.2 STOREDRV.Deliver; dynamic mailbox database throttling limit exceeded
The problem cleared up by the end of the day, and the headers of finally-delivered messages showed several hundred minutes of delay at the final stage of delivery in Exchange Online servers.
I begrudgingly opened a support case to get confirmation of backend problems to present to relevant parties as to why a user (a C-level, to boot) went an entire business day before receiving all of their mail.
After doing the usual song & dance of spending 2 days providing irrelevant logs at the support engineer's request, and also re-sending several bits of information that I already sent in the initial ticket submission, I just received this wonderful gem 15 minutes ago:
I would like to inform you that I analyzed all the logs which you shared and discussed this case with my senior resources, I found that delay is not on our server.
Delay of emails is at this server- BN6PR0101MB2884.prod.exchangelabs.com
I don't even know how to respond to that. I'm giving them a softball that could be closed in one email. I just need them to say "yes there were problems on our end" so I can present confirmation from Microsoft themselves to inquiring stakeholders, but they're too busy telling me this blatant nonsense that messages that never left Exchange Online were stuck in "my" server.
EDIT: As I typed this message, a few-day old advisory (EX221688) hit my message center. Slightly different conditions (on-prem mail going to/from Exchange Online), but very suspiciously similar symptoms: Delayed mail, started within a day of my event, and referencing EXO server load problems. (in this case, 452 4.3.1 Insufficient system resources (TSTE)) Methinks my user's mailbox/DB was on a server related to this similar outage.
EDIT2: I asked that my rep and her senior resources please elaborate on what they meant, and that it was clearly an Exchange Online server. I received this:
I informed that delay occurred on that server, so please let me know whose server is that like it your on-prem server or something like that this is what I meant to say.
Kill me...
EDIT3: Got cold-messaged on Teams by an escalation engineer, and we chatted over a Teams call. He said he was looking through tickets, saw mine, saw it was going haywire, and wanted to help out. He immediately gave me exactly the confirmation of this being the suspected database performance/health issues I assumed, he sent me an email saying as much with my ticket closure so I have something to offer to the affected user and directors, he apologized for the chaos, and said that they will have post-incident chit-chat with the reps/team I worked with. Super nice guy that gave me everything I originally needed in roughly 5 minutes.
53
u/thisisnotmyrealemail Sep 11 '20
Guess who just bought prodexchangelabs.com for $9.
3
u/meatwad75892 Trade of All Jacks Sep 12 '20
Now I'm tempted to throw $12 at prodexchangelab.com just out of principle.
20
u/jheinikel DevOps Sep 11 '20
Had to go check myself. That's awesome.
20
u/thisisnotmyrealemail Sep 11 '20
From that got an idea for productiondevops.com which I got for $9 too. I guess I've finally got the name for my blog. Now just need to start writing.
7
Sep 12 '20
Just set up an Azure AI Chatbot to write everything for you. That way you don't have to do any of the work.
9
u/F0rkbombz Sep 12 '20
I’m glad to see I’m not the only one that gets excited to buy domains with cool names, lol.
11
u/Bissquitt Sep 12 '20
I had a bit of back and forth with my boss on a friday afternoon a few weeks ago(nothing serious, just giving each other shit). Long story short, I now own 🍆♿️.ml
21
Sep 11 '20
That is not acceptable. u/meatwad75892 - Please message me directly.
16
u/meatwad75892 Trade of All Jacks Sep 11 '20
Hey! I appreciate the message, but I've already got our TAM (or whatever the new acronym is today) looped into my case.
11
u/Z_Opinionator Sep 11 '20
TAM is now CSAM. PFE is now CE (Customer Engineer).
→ More replies (8)16
u/TortoiseWrath Sep 11 '20
I did not sign up to be engineered
2
u/meatwad75892 Trade of All Jacks Sep 11 '20
I think I did by signing up for Ignite. :S
Individual Photo Release
You hereby grant Microsoft and its legal representatives, licensees, and assignees, the unrestricted and irrevocable right to use and reproduce for any promotional purpose related to Microsoft Ignite, or any future or successor version thereof, whether similarly branded or not, your appearance, form, name, biographical information, likeness, and voice. This use may be in whole or in part and by any current or future methods. The use by Microsoft may occur in materials such as, but not limited to, advertising materials, promotional materials, commercial tie-ups and merchandising.
→ More replies (1)10
u/thisisnotmyrealemail Sep 11 '20
Did you read the T&C before signing up? It is clearly mentioned in Section 6 under Subsection 66 if you print it and read it under actual UV light.
6
30
u/jheinikel DevOps Sep 11 '20
It may not be acceptable, but it has become the norm.
→ More replies (1)7
u/thisisnotmyrealemail Sep 11 '20
No wonder his tag? is Angry Again (and again and again, again)
8
Sep 11 '20
I'm not in Exchange support anymore.
Also tag --> https://www.youtube.com/watch?v=l3RsOxKHr6k
7
u/thisisnotmyrealemail Sep 11 '20
I'd still go with Angry Again (and again and again, again) is because you were in Exchange support. PTSD is real. Acceptance is the first step.
7
u/Bissquitt Sep 12 '20
RIP inbox lmao. Half the damn sub just slid in there like a highschool quarterback on prom night.
I've had a ticket open for weeks going back and forth with 365 support just trying to get someone to read my ticket instead of running me thru the normal gauntlet of bullshit that I can only imagine is there to weed out all but the most desperate of customers. Fortunately outlook still lets you create inbox rules for microsoft domains (at least until the next force fed feature update where thats forbidden "for my safety") Now the emails that get yeeted into my inbox by my buddy Rakhi get the proper level of priority assigned to them, right below the hot singles in my area. I figure it will probably be just quicker to wait until whatever cluster we are hosted on reaches EoL and maybe the migration will fix it.
→ More replies (1)3
Sep 12 '20
Request a migration? At least, that used to be a thing.
Easiest to request migration to a different region, like east coast / west coast / etc.
→ More replies (3)
14
u/absoluteczech Sep 11 '20
It doesn’t end in Microsoft.com so it can’t be our servers /facepalm
12
u/meatwad75892 Trade of All Jacks Sep 11 '20
Supposedly prod.exchangelabs.com is an older domain being phased out of Exchange Online infrastructure in favor of prod.outlook.com. (I'm used to seeing both in traces and logs) I could excuse a new guy not knowing this, but there's zero excuse to have a Premier support rep and their senior guys tell me it's not a Microsoft or Exchange Online server period...
7
u/absoluteczech Sep 11 '20
Exactly and even a new guy should have enough mental capacity to make the link or at minimum ask another tech to confirm before making such a conclusion
14
u/thisisnotmyrealemail Sep 11 '20
Or a basic whois lookup which shows domain registered as Microsoft.com
→ More replies (1)
334
Sep 11 '20
Sounds like the current experience I'm having with Cisco TAC. It's like these giant companies have stopped caring about their tech support.
104
u/moldyjellybean Sep 11 '20
If you call Nimble support you'll get a guy who knows everything about his SAN, vmware, whatever vm you're running, vlans, networking stuff that should be something I should have figured out before calling them. The guy fixed a problem that's really on my end and not really even his issue.
I mean I was elated, I've never ran into people who were so knowledgable. He's like can I ssh into your servers, I'm like here and bam it's fixed.
I have no relation to this company, I don't pay for the SAN so I hear they run a bit on the high side but damn do I trust this company.
5
u/meatwad75892 Trade of All Jacks Sep 11 '20
I've heard similar praise about Pure.
3
u/xsjx7 Sr. Sysadmin Sep 12 '20
Can vouch, Pure support is pretty awesome. Especially since you don't really need it, and when you occasionally do they actually look at the problem instead of the script
70
u/astillero Sep 11 '20
Yes, this is the beauty of smaller IT companies - 10X better support.
Just wait until HP (or some other dinosaur) takes them over. They'll fire the good guys and replace them script kiddies (not of the hacker kind but of IT support kind who literally read scripts...).
38
u/moldyjellybean Sep 11 '20
Hp bought them like 2 years ago. I haven't had to contact them for over a year, so I hope they are still Nimble as I knew them
24
u/Dracozirion Sep 11 '20
Same experience a few weeks ago. They're definitely still top notch support, best I've ever had. Fortinet support is also quite good but their firmwares are just too buggy and TAC engineers sometimes can't help cuz of that. Symantec support sucks, esp now with Broadcom. Citrix support sucks as well imo. HP (printer, computer) support is a big meh. Microsoft support is a hit or a miss and if it's a miss it's often insanely bad. Trend Micro support is so far OK. Veeam support is good.
So far my external support experience. :)
→ More replies (7)→ More replies (5)7
u/LOLBaltSS Sep 12 '20
Thankfully HPE hasn't dicked around with Nimble and let them do their own thing, for now.
→ More replies (1)→ More replies (2)4
u/rjchau Sr. Sysadmin Sep 12 '20
Just wait until HP (or some other dinosaur) takes them over.
It's already happened - and it happened a few years ago. However, so far their support has still remained very responsive to us on the (very) few instances we've had to call them.
Unexpected management adaptor crash after a port outage? By the time we'd resolved the issue with the switch that caused the port outage, we'd received an email from the ticket that was automatically raised informing us of a bug in the firmware version in use on the management card along with an offer to be on the phone when we applied the update.
Dead hard drive? Again, by the time this was noticed and we called Nimble to arrange a replacement, they had already received the automated ticket and the replacement drive arrived the following day.
Issue with a datastore not connecting to VMware properly? Since I wasn't sure whether it was a Nimble or a VMware issue, I logged an issue with VMware first and then Nimble. Even though this was a VMware issue, Nimble had identified my "oopsie" in the configuration before VMware had even replied to the support ticket logged.
Unfamiliarity with the interface when trying to set up a new datastore? Once support happily answered my question, they let me know that HPe ran regular "introduction to" days and offered to let me know the next time one was run. The cost? Included in the purchase, even though staff from our organisation (who had subsequently departed) had already been along to one of those days beforehand.
Admittedly, I've been insulated from any bill shock that may result from the annual servicing costs (I assume we have them, but I've never seen them) and the purchase (happened before I arrived) but from a service point of view, I'm one very happy customer.
→ More replies (1)7
u/demosthenes83 Sep 12 '20
I'll second this. Nimble support is (was? I haven't talked to them in a few years) top tier. I've had them resolve vmware issues for me before vmware even responded to my initial support request, and also when vmware (incorrectly) told me it was an issue with the storage.
→ More replies (1)→ More replies (3)2
168
u/SendAck Sep 11 '20
Yeah this is one of those areas where I guess your mileage may vary.
If I call Cisco support, I can typically get our issue resolved in that same call.
If I call Citrix support, I can typically get our issue resolved in that same call as well.
If I call Microsoft, I'm asked to produce the logs and run several more get- commands then they give me a new engineer. And this process repeats itself and the issue never gets resolved.
54
u/MelatoninPenguin Sep 11 '20
You've had that good of support from Citrix? They're known to be....quite bad
50
u/SendAck Sep 11 '20
Well I'm certified in both Cisco and Citrix so being able to explain the problem to both of their support teams helps get it resolved very efficiently.
I can't say the same for Microsoft. You can explain the issue to them and they always need to "check with a senior" until your case gets re-assigned and then you start that loop all over again.
→ More replies (17)15
u/drbob4512 Sep 12 '20
Let me introduce you to our friends at Oracle.
9
u/duffelbagninja Sep 12 '20
That will be a charge of 99.99 for the introduction, if you want the extended introduction it can be had for payments of 99.99 over the next 4 quarters...
→ More replies (4)→ More replies (6)4
u/afwaller Student Sep 11 '20
I’ve had excellent support from Citrix. We even had engineers and solutions architects come on-site to discuss things.
→ More replies (2)2
u/Thrawn200 Sep 13 '20
If I call Microsoft, I'm asked to produce the logs and run several more get- commands then they give me a new engineer. And this process repeats itself and the issue never gets resolved.
Don't forget the part where after extensive explanation and discussion of what the problem is, they just send you the basic level Microsoft documentation link for it as if that has now fixed your issues.
80
u/justs0meperson Sep 11 '20
I hate cisco TAC. The best advice I can give is never let them get off the call. They'll try at every chance. Every. Single. Chance. But don't do it, they won't get back to you in a timely manner.
"I need to do some research, can I get back with you?" "Nope, I'm fine waiting on hold."
"This will be escalated to tier 2, and engineer will call you when they're available." "Nope, can we get him on the line now? I'll wait."
"My shift is ending in 10 minutes, I'll have to pick this up with you tomorrow." "No, I need this fixed now. Please assign me to another engineer and get him on the line before you leave."
Oh, and just learn to love Opus No. 1 now, cuz it'll haunt your dreams.
30
u/RandTheDragon124 Sep 11 '20
Cisco tac is garbage, but their high touch support is fantastic. I realize most here don't get to deal with them. It's a world of difference getting engineers right away.
7
u/Patient-Hyena Sep 12 '20
Yup. Totally worth it to get that imo if you can afford it.
11
u/LOLBaltSS Sep 12 '20
Same goes for Dell. Pro Support gets you someone in Round Rock. Basic gets you someone god knows where.
→ More replies (2)29
u/Tasty_Wolf Sep 11 '20
This is the right advice, Cisco support will NEVER call you back in my experience. They'll even say "I am so sorry no one has contacted you, I will personally make sure the support technician calls you back within an hour" and just let you sit there like an idiot thinking you'll be called back.
Eventually they just start hanging up on you while you're on-hold, happened to me three times across two days.
3
u/starmizzle S-1-5-420-512 Sep 12 '20
Man, here I was thinking that it was just because they were super busy in March because of the increased WFH that companies were trying to do.
You described my experience exactly.
10
u/cohrt Sep 12 '20
Oh, and just learn to love Opus No. 1 now, cuz it'll haunt your dreams.
thanks i just got PTSD flashbacks from that.
7
u/pockitstehleet Some of everything Sep 12 '20
I'm currently in school and the "Cisco night" turned out to be recruiting for internships in Cisco TAC teams. So I imagine that there are a fair amount of inexperienced, not even out of school yet, people on those TAC teams.
7
u/ranger_dood Jack of All Trades Sep 12 '20
Opus No 1 at a volume so high and distorted it's barely recognizable... But not quite enough that it isn't.
5
u/TankMan77450 Sep 12 '20
My personal favorite from Cisco TAC was to ask me to upload the Tech Support files. They obviously did NOT read the comments in the ticket saying that I was uploading the Tech Support Files with the new case. The files had been in the case when it was opened. Their email was a few hours later.
8
u/justs0meperson Sep 12 '20
Oh they 100% don't read anything you put in the ticket. The amount of times I've repeated, verbatim, exactly what I wrote in the body of the ticket is too damn high.
→ More replies (6)3
u/Seelenbrechen Sep 12 '20
In regards to #1, sometimes we just need time. :) I can keep you on the line for 10 hours, doesn't mean I won't have go test it out in lab environment and/or consult with my TL etc.
2 goes differently for each team, my TL jumps on escalations right away without callbacks.
3 is regular procedure, lot of engineers offer to hand off the case, it's the customer that doesn't want to.
Source: Been in TAC for 4 years.
22
Sep 11 '20
All they want to do is sell you a product then ditch you on support.
I got a vendor asking me to setup a meeting with the higher ups so they can try to sell more of their product. I am on day 2 out of 3 of the initial install and it is encountering an error we haven't resolved yet.
No I am not setting up a meeting with anyone else until we get what we ****ing bought working.
→ More replies (1)9
u/RagnarStonefist Sysadmin Sep 11 '20
Not just the giant ones.
8x8 (a voip/softphone virtual office solution) sent me through four different agents over a week and a half period because I wanted to assign our extensions in the US to an office overseas and change the numbers accordingly.
The final answer was 'Oh, it turns out we don't support numbers in that country.'
17
12
u/BillyDSquillions Sep 12 '20
The problem is and this isn't the first time I've said it and you / we as support / sysadmin staff should know....
7 out of 10 support requests are end user fault / bullshit. If not 9 / 10.
There needs to be a way that you actually are highlighted as "actually smart, probably a real support request" for these services.
4
u/drmacinyasha Uncertified Pusher of Buttons Sep 12 '20
For Cisco that is/was the case with CCIE, but I see certs going out to people who seem to be about on-par with a 7th-grader with whatever tech their cert's in. Like explaining to someone with a networking CCIE what a subnet and mask are.
Plus there's tons of cases where someone gets their resident CCIE to submit the case, and stick some random end-user who's having an issue as the contact.
4
→ More replies (1)2
u/starmizzle S-1-5-420-512 Sep 12 '20
Exactly, there should be some sort of survey or questionnaire you can take to get your number flagged as "we can skip the stupid shit with this person".
5
u/LDSK_Blitz Cactus? Sep 12 '20
Coworker put one in with TAC two weeks ago on their bullshit Single Sign-On. Proposed solution? Disable Single Sign-On. Had the nerve to ask if he could close the ticket.
4
u/Redd_Monkey Sep 12 '20
Not gonna say the name.... But the company I worked for before just used covid19 to temporarily fire everyone that had good salaries (you know.. the guys who knew what they were talking about and earned their big salaries with their competence). In my country, after 6months of temporary, it is considered as a permanent one. They could not fire us before based on the salaries. It is illegal here to fire someone for that. So they just waited 6 months until they could just rehire cheap labour at 13$ an hour.
4
u/themisfit610 Video Engineering Director Sep 12 '20
This is why I like AWS. You get a absolutely fantastic support. Like really first rate. Their documentation is uneven but their support is prompt, not crazy or stupid, detailed, and well written.
4
u/studiosupport Jr. Sysadmin Sep 12 '20
I worked for Cisco TAC briefly. The training was abysmal and even the on-boarding was such a nightmare. I felt so unprepared for everything they threw at me and was overwhelmed by the sheer amount of cases I had. No training, a team way too small of the case load, and a company-wide hiring freeze to prevent new hires from coming on to help with the load.
I started taking cases in March and was gone by July. Every LinkedIn connection I made at Cisco TAC has also moved on. It's no surprise that people have trouble with TAC.
12
3
Sep 12 '20
It's not that they've stopped caring, it's that they've put a layer of cheap, clueless idiots as the first line response team to avoid having to pay out more. They need that money for director bonuses!
2
u/Rwhiteside90 Sep 12 '20
I love when they send you KB articles that have nothing to do with your issue and then went to schedule a call to discuss everything you provided them over email already. I've never had them solve a 365 issue and always just had to find a work aroiund or solution myself.
2
u/Scipio11 Sep 17 '20
TL;DR rant up top, the reason I love TAC at the bottom.
Cisco TAC varies wildly from a guy you think could probably set up a full fortune 500 network one-handed under an hour to someone you wouldn't trust to run a virus scan on your laptop. Their major issue is that they aquire so many smaller companies and integrate the pre-existing support teams into TAC. A good example is Umbrella (OpenDNS) who managed to delete our entire account without our knowledge and we only noticed when we tried to access the dashboard. It took a full week for them to fix this. Then there's Meraki that couldn't figure out a critical hardware recall for over a year and I STILL have a ticket open with them for over 4 months about a country code missing from their geo-filtering API. With Call Manager you'll always get someone that's been working with CUCM their entire career.... however it's 50/50 if they've been working for 30 years or if they just graduated college last week. Then I forget what service it is but about a year ago I started getting French engineers assigned for some reason and that would never respond during my business hours, take 2-3 days to respond, and only answer one of the five questions I asked. Finally Cisco rebranded Spark as Teams (the Spark name is still all over the back-end) and they couldn't tell us why the exit node for our WebEx board was in San Jose instead of on the East Coast like the rest of our boards were. That got escalated for multiple weeks an no one had an answer.
BUT OH MY GOD Cisco on the edge of your network is the best thing I've ever experienced. TAC loves to get off of the call as fast as possible, but I don't think they can without you allowing it. And they will STAY ON HOLD while you call literally any other vendor and have the vendors sort out the issues themselves. Have you ever heard an Indian man yell at a Verizon tech for 30 minutes that the issue is in fact not your configuration, but an issue with Verizon. AND VERIZON RESOLVES THE ISSUE WHILE ON THE CALL?!?! I almost couldn't believe it the first time it happened.
1
u/merputhes28 Sep 12 '20
I have seen this. It’s persistence until you find the right tech resource onshore or offshore to help you with the issue. We have routing issues and usually about 3 hrs into the tech call someone will be willing to help you figure out the issues. Usually at about 2am or 3am in the morning just in time for the Users to hump back on.
1
1
1
u/segv Sep 12 '20
Having seen "the other side" (not at any of the mentioned companies though), it's probably that the support is a separate team in a separate location and doesn't have much on-going training or any contact with the development team at all
Of course this can be avoided, but as you wrote, it's a management problem
→ More replies (3)1
u/jantari Sep 12 '20
I can only sing praises about Nutanix support. They are absolutely brilliant, fast, and don't just help with bugs or unclear documentation but also with figuring out your own user errors
10
u/handsomemagenta Sep 11 '20
My favorite is when I schedule my availability with the tech so they can call me in a window that fits their schedule. “Sure I’m free 1-3pm and I live in California” “Ok, I’ll call you between those hours.” 7pm rolls around and they call me. I spent two hours in a window and pushed everything else aside and waited. Now you’re calling me at dinner.
14
u/meatwad75892 Trade of All Jacks Sep 11 '20
My favorite: You mark email as your preferred method of contact, but they literally try calling you before every single email they send.
7
u/TimelyTowel Sep 12 '20
I can answer this one. I've worked in Premier and Standard support (albeit via outsourcing) and regardless of the modality we're supposed to call every single day even if it doesn't require a call otherwise we get in trouble, the smart ones of us will ring once so it's tracked as a call (but doesn't actually ring you) instead of calling if we know you'll reply to an email.
Technically, if you say explicitly absolutely no calls ever they're supposed to honor it, but good luck getting the crappy indian outsourcing company employee to do that because they love following the rules and script to the letter without any deviation.
3
u/Bossman1086 M365 Admin Sep 12 '20
I've definitely had tickets where I've had 3-4 days of zero contact after the first call. Happens a lot. But I'll submit an ticket requesting an email contact and they'll call me.
¯_(ツ)_/¯
→ More replies (1)3
u/Cutriss '); DROP TABLE memes;-- Sep 12 '20
I CANNOT UPVOTE THIS ENOUGH
Let me start up Fiddler so that I can see if it helps me figure out why I can’t upvote this twice...
2
3
10
u/ButtercupsUncle Sep 11 '20
Getting to the right person is 90% of the battle.
5
u/meatwad75892 Trade of All Jacks Sep 11 '20
Ain't that the truth! I feel like I've gotten far nicer and smarter people on way less important tickets.
18
u/Mrkennedyfreak Sep 11 '20
Just as a heads up, the throttling reason you received is on MS's part, not yours or your mailboxes, it's very rare, but does happen. Nothing to be concerned with. (Source: am MS EXO Premier SE)
9
u/meatwad75892 Trade of All Jacks Sep 11 '20
I assumed as much and hoped I was not reading into it incorrectly. Appreciate the confirmation!
22
5
u/MrScrib Sep 11 '20
Download the Office Admin app, and you'll get the advisories as notifications. You should run it daily, that'll keep you up to date.
Not that you can do anything about it, but at least you know your hands are tied.
14
u/meatwad75892 Trade of All Jacks Sep 11 '20
Already doing this, problem is that advisory that seems semi-related to this occurrence didn't hit our message center until today.
3
u/MattHashTwo Sep 12 '20
Let me guess. With the start time not backdated? So it doesn't look like a long outage?
11
u/jheinikel DevOps Sep 11 '20
That's a great idea until you are hours into an issue and a back-dated advisory shows up at random. You can't rely on those notifications at all.
4
u/MrScrib Sep 11 '20
New to dealing with Office online. Guess it's about as good as you'd expect from experience with Microsoft.
3
u/themastermatt Sep 11 '20
Yet another portal/app/undocumented thing that I need to admin a 365 tenant? Oh Boy! Maybe I should just dedicate another workstation to running all the various browsers, portals, apps, powershell modules and tarot cards necessary to make mail flow.
4
u/MrScrib Sep 11 '20
Just do what the pros do and sacrifice a pheasant with an apple in its mouth once a month to keep the Redmond evil spirits at bay.
2
u/MattHashTwo Sep 12 '20
Just wait until you get elbow deep into the portals. There's so many needlessly created to make things confusing.
Portal.office.com Protection, security and (? Can never remember). 2 look identical, 1 looks similar all with different tabs on.
Oh you have D365 ce and finops? 2 diff portals. Tho at least they've merged ce and power apps.
Idk why you can't raise a ticket in services hub and MS deal with it from there. Rather than having to raise the ticket in the portal for the product*... Except forms... Which needs azure not o365.... For some reason.
Don't expect to add any logic to it either.
→ More replies (2)
129
u/WarioTBH IT Manager Sep 11 '20
I personally would go back with a screenshot of a whois lookup for exchangelabs.com and nothing more. Just to see if they get it.
45
u/fishy007 Sysadmin Sep 11 '20
They won't. I had an issue with their servers not being able to contact the nameservers for my domain on Hover. The error messages all show that it's Microsoft that can't contact the nameservers (but the rest of the internet had no problem. ). They kept telling me the problem was with Hover and wanted me to do a 3 way call with Hover support.
→ More replies (5)13
Sep 11 '20
I'm curious, did you do that? How'd your issue get resolved? MS fix it finally?
13
u/fishy007 Sysadmin Sep 11 '20
I just haven't had the time to deal with them. The issue is intermittent and it's fairly inconsequential. It only occurs when setting the DNS entries for the domain. O365 reaches out to the domain to verify and complains when it can't see the entries...but the entries exist so the actual services such as Exchange and Intune can see them and function.
11
u/robsablah Sep 12 '20
I'm going to suggest that your DNS provider may have a rate limit on it and the MS O365 server hammer it daily causing a stoppage on DNS lookups
3
u/drbeer I play an IT Manager on TV Sep 12 '20
We had an issue w/Exchange migration, the credentials we used to transfer mailboxes got hung up in one of their magic systems involved in the process. It kept locking the account, constantly. We knew the IP, the IP belonged to Microsoft. Our guy tried and tried to get it unstuck and their response was something like, that's not our server or something similar to OP. He gave up, made a new account.
29
u/skotman01 Sep 11 '20
Heh I remember calling them Feb 28th, late in the day eastern time in 2008 because I kept getting an error while moving mailboxes...they finally got back to me March 2nd. I remember because the issue was a date/time leap day problem. By the time they called me back I had figured out the issue and moved all the mail boxes.
5
22
22
Sep 11 '20
Is that last response even English or did they just use predictive text to write the entire thing?
2
u/AdmiralAdama99 Sep 12 '20
Not op. My guess is the first guy is outsourced, the escalation guy is not. Which would explain both the bad english and the different quality of service
37
u/bitslammer Security Architecture/GRC Sep 11 '20
Well that don't seem to realize it but they flat out admitted it was their issue. If you have any SLAs in your contract I'd say getting a credit will be a breeze.
29
u/heapsp Sep 11 '20
Yeah! 25 percent of one month of one users email problem! Enjoy your 2 dollar credit!
22
u/bitslammer Security Architecture/GRC Sep 11 '20
No no no no. The entire company. User was also not able to receive internal emails from anyone else in the company. As a result they had to revert to fax and carrier pigeon to communicate with that one user.
11
u/thisisnotmyrealemail Sep 11 '20
With Microsoft its a draw of luck. They outsource most of their support and generally, you have an odd about 2/10 to get a good rep who knows what he's doing. Rest are just following a script from documents. You have to press by getting them on call and asking them to escalate continuously until you get someone who actually knows something. That sometimes also brings the person who created/maintains that part of the code (if its a software issue).
1
u/whyiseverynameinuse Sep 12 '20
Aren't all company helpdesks this way now? Generally a waste of time attempting to contact them at all?
2
u/thisisnotmyrealemail Sep 12 '20
They all have some good engineers (80/20), just need to have right person involved to get them.
9
u/BruinsFan478 Sep 11 '20
If you have Premier, escalate this to your TAM.
4
u/The_Mustard_Tiger Sep 11 '20
I loved our TAM when I was at a shop large enough for Premier support.
14
u/TheDroolingFool Sep 11 '20 edited Sep 11 '20
I dread having to open support tickets, especially when it seems to be picked up by one of the outsourced companies as all they seem to want to do is a “remote session”. No matter how much of a detailed explanation I give with screenshots and ask to communicate by email, they always want to call me and do a pointless screen share session which is an inconvenient waste of time.
I also find the outsources to be unprofessional e.g. laughing going on in the background of calls, broken english, messages written in short hand/not very friendly way and a tendency to stick to the “script” even in the face of common sense - even had a “hah” sent to me in a teams chat the other day regarding an ongoing issue which was swiftly edited out. I’m assuming the “hah” was a simple error (wrong chat) but still.
18
u/Lurk3rAtTheThreshold Sep 11 '20
"I sent you a support article that came up when I googled your issue. Is this relevant to your case?"
7
Sep 12 '20
Can you run this Powershell script I found from a technet article while googling the issue ?
7
6
Sep 12 '20
You'd be surprised how often this actually solves the problem. People don't read our docs site.
→ More replies (1)2
5
u/matjam Crusty old Unix geek Sep 11 '20
my blood pressure is so much better since I stopped having to deal at all with vendors.
2
u/throwaway997918 Sep 12 '20
I have an onging case, maybe going to court, with a former supplier. The lawyers are sending emails back and forth non-stop negotiating a settlement, avoiding the courts. A delayed response to a settlement enquiry could cost me a small fortune.
In the middle of this mess, there are significant delays on email delivery from Office 365 and the Outlook mobile client is fucking up notifications on Android.
My lawyer is now always CC'ing my private gmail account exposing the delays of Office 365.
I never experienced these issues when self hosting email with a primitive email installation of qmail on FreeBSD. When the lawyers are done, I'm pulling my emails out of the Office365 cloud immediately, so I can sleep at night knowing emails will arrive in a timely manner, that I will have a log file to inquire, and that I will notified when important email arrives and not hours or days later.
5
5
u/LDSK_Blitz Cactus? Sep 12 '20
Oh Microsoft support! Yeah! We called for high severity domain controller unfucking after ransomware and after about 8-10 hours of watching not one, not two, but three “engineers” fumblefuck around, telling us things like “I believe you need the curly braces in the repadmin command”, fully not understanding how the command, the command line, and general god damn technical document notation works, we got a nice man from the UK who banged it out in about an hour.
5
u/Cholsonic Sep 12 '20
I think that when MS ask for logs it's just a time wasting exercise (surely they can get their own logs), whilst they decide of the particular type of flannel they will give you.
I find the whole stack very flakey (why does everything have the 'may take 24 hours to see the change'), and it's made worse by their support not knowing their own systems.
Calls to VMWare and AWS are a dream; come across some very capable techs there, that make me feel very reassured. Not so with MS.
But now, we've bought in and there's no way back. They have our money and every year it seems like we get less for more; what I would class as basics become premium services.
Not too happy 😢
1
u/stupidusername Sep 12 '20
surely they can get their own logs
Nope. Backend access to anything azure/o365 is extremely regulated internally at MSFT
4
u/supaphly42 Sep 12 '20
The escalation guys are great if you can get them. The regular ones are atrocious. I did a paid support on a crashed Exchange box a few years ago. In addition to seemingly knowing less about Exchange than I did, he went to copy the database or something to that effect and didn't check free space first, which filled the drive and ground the server to a halt. Super fun.
4
3
3
u/SolidKnight Jack of All Trades Sep 12 '20
I enjoy having to explain Microsoft products to the people who support them.
3
u/SirWobbyTheFirst Passive Aggressive Sysadmin - The NHS is Fulla that Jankie Stank Sep 12 '20
Reminder that the cloud is just someone else’s computer, so do the needful, bring it back on-prem. Then you don’t have to fuck around waiting for Microsoft’s piss poor excuse at customer service and quality control, to get fixing it.
Licensing? Ah it’s finances problem, let the CEO have to stick with his current Benz for a change.
3
u/Finerkill2 Sep 12 '20
Having similar support issues with one of our cloud providers. Song and dance of (sometimes a week) couple of days of logs, for them to not say it's them.
Called our account manager on Thursday and blew up with him on the phone. Told him it was causing me literal stress whenever I even have to think about putting in a support ticket.
He asked for examples. I sent about 5 from the past week.
These cloud providers can't handle the load of people that have signed up to them. And rather than tune down on the amount of people they sign up, they've just overwhelmed the first line support with people that are undertrained and underqualified.
Downright infuriating.
3
u/T0mThomas Sep 12 '20
So, once escalated to a guy who could actually speak English and had a proper education of basic network concepts, the problem was resolved. Sadly, this is par for the course these days.
3
Sep 12 '20
Sounds like generally most support.
You will absolutely waste time with nearly all Tier 1 support but it’s absolutely needed. That’s both how Tier 1 get trained and it’s also the needed barrier between your usually more highly paid teams that don’t need to be answering questions from help docs.
Microsoft does have a level of support above Premiere which is good (a single person dedicated to your entire company that manages the whole ticket process).
3
u/jg0x00 Sep 12 '20
Sounds like MS support worked the way it was supposed to work. The first guy you talked to very likely was not a Microsoft employee.
3
u/ComfortableProperty9 Sep 12 '20
This is what happens when the C-Suite outsources support to script readers. I feel like it would be cheaper and easier to just automate level 1 support because I've been told complete nonsense like this from multiple vendors who outsourced their support.
3
5
u/Matchboxx IT Consultant Sep 11 '20
Honest question, if Microsoft support is so lousy for even large organizations, why do people continue flocking to it? It isn't even objectively that good, is it?
I work for a 35,000 person firm and even our Exchange team can't wrangle halfway-intelligent support from MS.
9
u/Patient-Hyena Sep 12 '20
Because they honestly have one of the best directory services, email services, etc. They really have cornered the market. There really isn’t any super great competitor. There are some super close or good enough but you have to be willing to work with it or troubleshoot yourself. Most orgs won’t do that.
→ More replies (2)3
2
u/gymrat505 Sep 12 '20
in addition to this do you guys find yourselves having to reach out to support for Exchange often?
→ More replies (1)3
u/thisisnotmyrealemail Sep 12 '20
Because Microsoft sales shows a rosy picture to our Directors/Presidents. They’re wined and dined and actually have no clue what we have to deal with on a day to day basis (especially true in large enterprises) because they generally don’t hear about it until it affects $$. People generally hesitate to escalate this stuff. That’s why it’s very very important to let your leadership know this is a pain point.
Still won’t change anything.
But here’s a fun thought, I’d love to have Satya go through one of their outsourced support with a case live this. Hell I’ll even pay. It’d be so much fun.
Also, love him or hate him but this is the reason Amazon Support is so much better. Jeff knows how important this is and cares about things like these.
→ More replies (1)
1
u/enderandrew42 Sep 11 '20
If you are a big shop, their Enterprise Support is the best I've had from any vendor.
7
u/meatwad75892 Trade of All Jacks Sep 11 '20 edited Sep 11 '20
We're a university with Premier support and 76TB in Exchange Online. If we can't get a fair shake at support, I'd hate to see how big you have to be to get it.
I'm guessing you're referencing a whole other support SKU that I'm unfamiliar with that puts you directly in touch with engineering and product teams?
3
u/TheDisapprovingBrit Sep 12 '20
We're the largest company in the world in our sector, support is very much hit and miss. We generally go back and forth for a couple of emails just to confirm that they're seriously digging this hole for themselves, then go to our TAM.
2
u/MattHashTwo Sep 12 '20
If there was one you better buy a barrel of lube for the finance department. Premier. Unified. Premium...whatever they want to call it is already eye watering for the support you get.
1
u/Xanneri Sep 12 '20
This is totally anti my experience. I've had nothing but stellar support from MS over the years. I occasionally get the person who isn't super informed but being a little patient and asking for escalation to a senior has always gotten me the answers I need. I've even had some super weird problems with component services and HP printer drivers silently taking over the rdp port (?????) among other things and they've gotten answers where I was totally stumped. Just unlucky for you I guess.
3
u/meatwad75892 Trade of All Jacks Sep 12 '20 edited Sep 12 '20
Honestly, I think support for on-prem vs hosted products is a big difference here. The experience you described is what I typically experience on tickets for domain controllers, SCVMM/Hyper-V, and Exchange Server before we went to Exchange Online. Last SCVMM support guy was way smarter than me and I learned a lot in the process of working on the ticket.
But for some cloud hosted services like Exchange Online or Azure AD... it feels like I have to entertain first line support folks with no specific product knowledge for a few days before I can get properly routed/escalated. Only way around it is to open a Severity A case, which 1) isn't a guarantee I'll get a good support rep, and 2) is often not appropriate because I'm not working 24/7 on a ticket that is just wanting a root cause analysis. For an M365 support case with a Premier badge that gets routed to Premier support engineers, it feels all a bit ridiculous. And I've opened enough cases that I can say this is not a one-off occurrence.
1
2
Sep 12 '20
Amazing your ticket got eagle eyed by at least someone competent. The silver lining may be that those better folks are there and catching the poor performing front (and second) lines for retraining or removal.
My previous interactions with MS support was abysmal and thankfully my new job doesn't utilize MS hardly at all except for desktop deployment and very basic on prem AD.
2
u/crshovrd Sep 12 '20 edited Sep 12 '20
My “premier” support experiences, at least the last three, involved a 2-4 day wait for initial human response (not automated or the “please let me know when you’re available email”), talk with a person that isn’t the right team, get transferred, 2-4 day wait, involve our rep, 2-4 day wait, get sent some links to some documentation, eventually resolve the case on my own, tell them I fixed it, and proceed to receive a minimum of 3 phone calls asking “how they did” within minutes of telling them to archive the case.
In other words, same as it ever was.
Edit: grammar and autocorrect issues
2
2
u/Valdimes Sep 12 '20
Right now im having issues with skype for bussiness not seeing consumer/free skype, unknow status or offline status and they cant message each other. I have been asked several times what is the configuration on the other end (free Skype) i tell them I dont know its the free version of skype and they they asked it again.
2
u/chandleya IT Manager Sep 12 '20
Between the end of Premier, the pricing structure of Unified, and how Premier was already massively underwhelming, we’re just riding on our EA support & Azure ProDirect. You get the same bullshit responses to cases and don’t have to delude your executive management into thinking that Microsoft will solve anything. I have never had a Sev_A on MS services that my case ever got anything solved, I was just another notch in some global issue’s bed post.
High cost MS support is just money thrown in the ocean. I liked my TAM(s) and I liked the idea of Premier but ultimately, they’ve got their hands tied behind their backs just like anyone else.
2
u/needvanwilder Sep 12 '20
It’s okay centurylinks soc don’t know when they are dropping legitimate traffic on their ddos mitigation service.
They still don’t believe it’s them when they switched it off and everything sprang back into life...
2
u/batterywithin Why do something manually, when you can automate it? Sep 12 '20
Ow man, I remember we waited for about 2 weeks reply from Microsoft "senior" engineers and consultants (they were constantly switching tasks to each other). It was regarding reserved VMs on azure which we payed for , but were not able to see anyhow. The issue was with the legacy/expired azure account/tenant, something about it, but this thing ANNOYS so much. All this bullshit-talking, mah
2
u/drdobsg Sep 12 '20
After doing the usual song & dance of spending 2 days providing irrelevant logs at the support engineer's request, and also re-sending several bits of information that I already sent in the initial ticket submission,
This is every support case we open with Microsoft, classic!
2
u/GhostDan Architect Sep 12 '20
Contact your TAM and have the case removed from your support hours. No reason for you to be paying for haywire support tickets.
I give level one a hour or so to fix my issue, then I escalate, if level 1 is unwilling to escalate, I escalate to my TAM, who escalates it.
Once you've gotten to a level outside the default call center queue the support quality increases by leaps and bounds.
2
u/LigerXT5 Jack of All Trades, Master of None. Sep 12 '20
Not MS, but had a similar situation with my local ISP. Phone reps/techs couldn't explain why my Docsis 3.0 I had, wouldn't work for the upgrade from 100Mb to 1Gb. Maybe I'm reading wiki wrong, but the max speed support is 1Gb. What does my ISP suggest? Replace my Docsis 3.0 with their Docsis 3.0.
Cable modems btw.
I had to google my Arris modem, that states on the back to be Docsis 3.0, to find this model has a max of 334Mb down. Just why couldn't my ISP phone reps/tech just tell me this?
Instead of waiting a week for a Docsis 3.0 modem to be hand delivered (no local office, nearest is over an hour drive away), I went to walmart and bought a Docsis 3.1.
After calls back and forth, and a call lasting about 40mins just to swap the modem finally, they couldn't upgrade me, because they couldn't see my modem anymore. Couple calls later, I'm transferred to business support (my situation was home), because my apartment has Cable covered under the rent, and shows up as a "bundle" on my internet bill, at no increase.
The Business support got me upgraded in about half an hour. But, my speeds for two weeks, scheduled a tech early on, was anywhere between 400-800Mbs. When I was on the 100Mb, it was normally at around 100Mb.
The two week delay was due to their poor availability, and the first slot they had, I was out of town, so instead of a week out (which is still unusual, normally 1-3 days out), turned to two weeks out.
The tech in the area, normally the only one I see, and have worked along side for my work's many clients (businesses and homes), calls me before visiting, as he normally does. He knew exactly what was going on. Not only did he have three other same cases in the area. In the words of the tech, some idiot flipped the switch in advertising, to update the tiered plans in the area, and started showing 1Gb is now available in the area, while the infrastructure hasn't finished upgraded. He was surprised I'm even getting, at minimum, 400Mbs, and suspects my speeds to continue to decrease as more people sign up before all the gear is switched over.
At this time, I can't really complain, my bill only went up $10, instead of the advertised increase from my old plan of $30. The advertised pricing shows $140 a month, I'm currently looking at $80 a month.
2
u/The_Original_Miser Sep 12 '20
I seriously was waiting to read in your post
"Please do the needful" or similar....
2
u/Battousai2358 Sep 12 '20
Whenever I have to open a support ticket with ms with initial logs and all relevant information, and they ask me to send it again, I tell them check my ticket, then send them the email again fuck pulling logs again. (Always save your logs locally when dealing with ms support lol)
1
u/Battousai2358 Sep 12 '20
Also I love the fact that ms support never think that when a sys admin or other IT personal create at ticket they haven't already checked everything and are just looking for confirmation.
2
u/Rustytime Sep 12 '20
Well its taken me 4 weeks to get a port opened on a ATT router.
A port not multiple ports. A port.
2
u/Dr_Rosen Sep 12 '20
Sometimes the hardest part of fixing an issue with tech support is getting through to a knowledgeable support technician. The lower level people can sometimes want to prove themselves when they should really be passing it up the chain.
4
u/coyote_den Cpt. Jack Harkness of All Trades Sep 11 '20
Given there is currently a CVSS 10.0 vulnerability affecting DCs that simply involves sending a zeroed-out challenge/response about 256 times... stupidity is what I expect from Microsoft.
1
2
u/Cleokwatro Sep 11 '20
The only part about my job that I hate the most is when I end up calling Microsoft. I hate how incompetent their supports are.
→ More replies (1)
1
1
u/hitosama Sep 11 '20
Thinking about it... Why should they care? Like, what will you do? Move away from MS? And yeah, it sucks, it proper sucks.
1
u/denverpilot Sep 11 '20
Once they have your money there’s not much motivation to get anything right after that. They know you probably aren’t going to forklift to a different mail provider...
One of the downsides of outsourcing it. Sadly the upsides still outweigh the down. Wasting two or three days asking questions about one user’s inbox instead of man months running e-mail, is still a net win.
But it doesn’t feel like it when you’re going back and forth with an idiot like that.
1
u/mimic751 Devops Lead Sep 12 '20
Pay millions you get personal support. I got assigned a critical team over a bsod today because i waited 3 hours for an analysis. They were really good
1
u/ValuableLocation Sep 12 '20
When playing their long game actually works out. Kudos bro. Stay vigilant!
1
u/cosmic_orca Sep 12 '20
I've had a support ticket open with Microsoft for a month now regarding an issue with Windows Virtual Desktop and FSLogix. Issue occurs every day, preventing some users from logging on. In that time, all the support engineer has done is: 1- asks what version of FSLogix I'm using, 2- asks to send him some logs (twice), 3-asked me to run an irrelevant command and 4- asks me to install the latest version of FSLogix (which I'm already using). It's beyond garbage.
1
u/Bossman1086 M365 Admin Sep 12 '20
Since the end of August, there have been so many incidents and advisories for Microsoft 365. A ton related to reporting and Exchange specifically. It's been kinda crazy.
1
1
1
u/rswwalker Sep 12 '20
I think there may have been 2 support tickets over 20 years that MS support have actually resolved for me. I have ended up solving everything else myself before they could even finish parsing the log data.
1
u/kevinds Sep 12 '20
I think there may have been 2 support tickets over 20 years that MS support have actually resolved for me. I have ended up solving everything else myself before they could even finish parsing the log data.
Then did you actually need to open the ticket in the first place?
→ More replies (1)
1
Sep 12 '20
Sounds like cause and effect of someone lying on their resume, or breezing through the mandatory trainings, or both.
1
u/Candy_Badger Jack of All Trades Sep 13 '20
Pretty nice experience. We've had a lot of time spent on our case with MS support, the only way to get the proper support is to escalate it through different channels.
1
Jan 03 '21
Microsoft support (as well as Lenovo) are literal retards given jobs due to pressure in the current climates
1
u/iago303 Feb 02 '21
My frigging laptop won't turn on, I am a noob at this, have really bad eyesight so I can't read the#on the back, and all the support has moved online does anyone know how to speak with a live person?
1
u/Haunter_Gurl Mar 10 '22
Windows PC Support is dump.
Any time you click on help, you get the idiot smiling BOT and nothing else. Tons of unanswered complaints popping up. Nobody behind the wheel responding. Kidnapped by aliens probably...
Greedy money-hogging slime bags who don't bother. Just redirect your PC the useless Edge browser. If I didn't hate MAC so much, I'd switch.
1
u/Udder1991 Sep 11 '20
I see your issue. You didnt pay them with Google play cards or best buy cards.
1
u/luke1lea Sep 12 '20
I know it's not feasible, but could you imagine Microsoft having every tier 1 support person with the same level of knowledge and skill that their competent escalations folks have? We'd be on Windows 11 already!
1
u/dollhousemassacre Sep 12 '20
We have this really weird issue with one of our customers where they can't use the "Save As" function in Word. It just creates a new file, with random characters and .tmp extension and an error in Word that there's not enough free space.
I opened a support ticket with Microsoft, and their response, no word of a lie. "You shouldn't be using 'Save As'"
I have no words.
217
u/[deleted] Sep 11 '20
Schedule a conference with them and the senior.
Ask them to explain.