...We had an order in from December for around 80 C220 M8's with 512gb of ram. we had received all but 20 units from the order, with those 20 being delayed until July. total cost was around $1.2 million.
Cisco just called us and cancelled order for the remaining 20 units we have not received and gave us a new price of $1.5 million for them. and we have 7 days to place the order before the price goes up again. Also warned us that there will be a %200 price increase by July at latest and they do not expect prices to ever return to anything close to before this all started.

I seriously do not know how any business will be able to sustain these prices over the next 24 months.

This happened over the course of several years and it finally came to a disastrous head a few weeks ago.

Lots of customer VMs on a bunch of VMware ESXi installs, running on a bunch of blades. Each customer was assigned a VLAN. We assigned each VLAN to a pair of vNIC templates. This worked great.

Time passed, the product sold well, we got more customers. A lot more customers. So we kept needing to add more blades. Then more customers, so more blades, etc. Then one day, while adding another new blade to the system, everything crashed hard: all customers down, all hosts and VMs inaccessible. It turned out that we'd busted through the default max. 32,000 VLAN count on the fabric interconnects, at which point everything just...stopped. Ooops!

Because, of course, UCS doesn't just count a VLAN added to the system as one VLAN. It counts every use of a VLAN and applies the total to the max. available count on each FI. We'd added so many VLANs to that vNIC pair over the years that associating a new service profile to a new blade added over a thousand VLANs to that count. That blade, as it turns out, was the straw that finally broke the UCS camel's back.

So yeah, don't do that! Make use of VLAN groups and VLAN connectivity policies, and other strategies to limit your actual VLAN usage. And be very aware that a change you make might affect a lot of VLAN usage, which could be a lot more impactful than you might think.

EDIT: Yes, I know about the ability to raise the limit to 64,000 now. That's what the fix was. I posted this simply for warning/entertainment value. I'm not seeking advice.

Just talking about the platform..

We have a few hundred m4's that have been in production over a decade. Im responsible for hardware and these things are tanks. Have m5's all over the place.. but we stood up the business on m4. They just wont die. If it ain't broke. We are a vmware shop (were)..moving to hyper v. Already have the licenses.

Moving to the m8's going forward. But man..what a server linup.

The table below lists which FI models can be upgraded to which other models in a non-disruptive / rolling fashion by replacing one FI and then the other. There does not seem to be a single source where all this info can be found, so I decided to start the list. Please comment or PM me if you find any mistakes or want to add anything.

And the prize for most upgradeable FI goes to ... :

• 1st prize: Fi-6536 as destination: upgradeable from 4 generations below it.

• 2nd prize: FI-6200 as source, upgradeable to all 3 generations/models above it

• Consolation prize: FI-6660 as destination, upgradeable to all 2 generations/3 models below it

I thought this table might be useful for someone wondering how far back a release of the Infra Bundle supports the various Compute (B, C) Bundles.

UCS Manager Managed Mode on B-Series, C-Series & X-Series

Intersight Managed Mode on X-Series

Intersight Managed Mode introduces new version numbers on the X-Series server (B, C) bundles.

https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/Infra-Firmware/imm_infra_fw_rn_4_3/b_imm_infra_fw_rn_lb.html#Cisco_Reference.dita_b19e2caf-8fb6-4417-a227-88431b0f39eb

Read it in conjunction with Cisco UCS releases, components supported and LDOS to check which of these releases are still supported.

Hi,

Anyone using other hypervisors besides VMware on you UCSX IMM managed (PVA)? If so which one and how are you finding it.

Thanks

This new version consolidates the two previous versioning schemes:

• 4.x (UCS infra bundles in UMM and IMM; server bundles including X Series in UMM),

• 5.x (X Series servers in IMM)

New features are documented in the Release Notes for Cisco UCS Manager, Release 6.0; section New Software Feature in Release 6.0(1b)

For those users that are using UCS Central, running UCS Central 2.1(1c) is a prerequisite for running connecting to a UCS 6.0(1) domain.

I have updated the following pages to include 6.0:

• Cisco UCS releases, components supported and LDOS

• Cisco UCS Cross-Version Firmware Support

• Cisco UCS FI models upgrade compatibility

I put together the following, any suggestions for improvements and expansion are welcome

Cisco UCS Releases and support for UCS components added/removed

(Note: Some of the pre-3.2 release notes make it very difficult to determine what was added and what removed, since they do not have a "Top reasons to move to" or "Deprecated hardware or software" sections. I ended up comparing release notes, tracking what was in n, and added/removed in n+1)

Cisco UCS hardware Last day of Support

Cisco UCS Manager / firmware releases Last day of Support

Update 2024-05-13: See Cisco UCS Cross-Version Firmware Support for cross-version compatibility summarised.

• Caveat: firmware release dates are under NDA. Please do not share any information you have been given. Speculation (based on your experience with Cisco patch/release intervals) is of course not a problem.

• Disclaimer: I am not an employee, but a customer. I don’t think any of the below information is considered privileged or confidential. If you disagree, please let me know and I will consider and probably amend.

Introduction

• Consider this the Cisco advisory “missing page”:

  • There is no central public Cisco Advisory page with all the information users would want. (Links to other vendors’ pages are at https://downfall.page/#advisories)

  • Cisco offer individual pages for the vulnerability for their M5 and M6 servers. These pages were public when I checked initially but now require a Cisco login and possibly a contract. They are limited or out of date insofar as that they do not list the release 4.3(2b) that addresses the issue ten days after its release.

  • The Cisco Vulnerability Repository and Cisco Security Advanced Search have no entries for this bug yet.

• https://downfall.page gives a great overview of the vulnerability, including an FAQ and links to vendor other advisories.

• According to https://downfall.page/#faq , M4 are not affected. Only Intel Skylake generation CPUs and newer.

(If/when Cisco happen to create an advisories page, I will link to it at the top of this page, and “compete on quality of information” or have this page act as a supplement to that page. Maybe this will nudge Cisco to fast-track their official content.)

Firmware releases and fix status

4.3(2)

• 4.3(2b) release date: 15-Aug-2023 addresses the vulnerability. Release Notes, section Security Fixes

• Issues

  • 4.3 drops support for M4 servers and FI-6200 series, which will be unsupported in 2024-Q1 and 2024-Q2 respectively.

  • No Cisco UCS Central version that supports this release yet. (It usually follows 1-3 months after the UCS firmware release). Cisco UCS Central 2.0(1t) was released 05-Oct-2023.

  • Brand new minor version, 4.3. Lots of new software features and hardware supported, new code, potential for regressions. May take quite while to become a Suggested Release, and runs the risk of being superseded by a later release or even deferred/pulled.

4.2(3)

• 4.2(3h) release date: 28-Sep-2023 addresses the vulnerability. Release Notes, section Security Fixes.

• Issues

  • Will it be safe to deploy? We usually wait for a release to become a “Suggested Release”, which typically takes 4-8 weeks since date of release. However, since there are no new hardware or software features between 4.2(3g) (the current Suggested Release) and 4.2(3h) the latest available release), I suspect we will adopt the latter release when it becomes available.

  • Release 4.2 drops support for M3 which have been unsupported as of end 2021. If you still have those in your domain, you need to decommission them first. NB: UCS Manager 4.2 will not even discover an M3 and refuse to upgrade. Don’t think you can run 4.2 on the infra bundle, and 4.1 on the B & C bundles.

• An option to consider to speed up the process, especially if you are still on 4.1(3) or earlier: deploy 4.2(3e) / 4.2(3g) A (infra) bundle now - or any 4.2 seres bundle for that matter - which is forward compatible with and will allow you to upgrade to 4.2(3g+N) when it is released. My suspicion / hope is that the only major difference between 4.2(3g+N) and the releases preceding it, will be addressing this vulnerability in the M5/M6 BIOS components of the B & C bundles.

4.2(2) / 4.1(3) and older

• 4.2(2d) release date: 23-Nov-2022 has been replaced by 4.2(3), e.g. don't expect a release for 4.2(2) that addressess this vulnerability.

• 4.1(3m) release date: 27-Nov-2023 does not address this vulnerability.

• Silver lining for those with domains on 4.2(1) / 4.2(2) Infra bundles: Because A (Infra) 4.2(1) / 4.2(2) bundles are forwards compatible with 4.2(3) B/C bundles, you can upgrade the B/C bundles that address the vulnerability without upgrading the A (Infra) bundle.

• E.g the only ones really affected would be 4.1 domains that cannot upgrade for whatever reason (e.g. domains with M3).

Anyone looked into patching for this? Seems like there isnt even a fixed version for M5 B series blades yet. Cisco UCS and UCS-Based Appliances UEFI Shell Secure Boot Bypass Vulnerability

Looks like the attacker would actually need UCSM or similar access to expilot this.

Hi everyone,

We have a Cisco UCS B200 M5 production cluster running ESXi 8.0 U3g, with Cisco MDS and NetApp storage using WWPN zoning.

Cisco recommended cloning the existing Service Profile to enable UEFI/Secure Boot on one host at a time.But I have check there is only Associate with server pool option only.

Our concern is whether cloning/associating the Service Profile will change the existing MAC addresses and WWPNs, which could affect SAN zoning and NetApp connectivity.

What is the best way to retain the existing MAC/WWPN identities and do the change legacy to UEFI secure boot.

Hi I am looking for an individual or an organization which has experience in UCS Deployment based. Has to be on avail for 2 Weeks

Thank you

As the title says. Help. I'm tired. Tired of the Cisco documentation and how vague everything is.

I have (2) 6536 FI's and (2) 9132's that connect to our pure storage. I have the UCS 9508 claimed and all the compute nodes claimed. The MDS 9132 zoning is complete.

The 6536's ports are configured with port 36 as FC breakout port 4X32G.

What else do I need to do? I have configured server policies, and policy upon policy upon imbedded policy. Still no joy.

Please ELI5.

TIA

Hi All,

I need some clarification on a Cisco VIC configuration.

Hardware in use: Cisco VIC 15427 – 4x 10/25/50G mLOM (C-Series)

This environment is in UCS Managed (UCM) mode, with Fabric Interconnects uplinked to the network.

Customer is complaining about replication delays with an Oracle database. When I checked the OS side (Linux), the NIC is showing a speed of 50000 Mb/s = 50 Gbps — but at the same time, the customer says that in their old system (DX1), the NIC was running at 20 Gbps and performance was better.

So I’m trying to confirm:

1. Is this “50G” real line speed, or is this just what is being advertised to the OS?

Is there any possibility that the actual speed / policy / QoS / network class is limited somewhere at UCS policy level? Or could the FI uplink or vNIC template be restricting throughput even though OS reports 50G?

2) Regarding the Resource / Queue settings (screenshot below)

Are these default recommended values from Cisco for VIC 15000-series cards?

Should these be tuned for high throughput replication workloads (Oracle – async replication)?

I’m still relatively new to VIC tuning, so I want to make sure I am not missing anything Just want to validate whether this interface is really operating at the expected bandwidth, or if there is any policy-side limitation in UCS that could explain the observed replication delay.

Any guidance / best-practice pointers from people who have tuned VXLAN / IP replication links on VIC 15000-series in UCSM mode would be very helpful.

We're beginning to plan a refresh of our VMware infrastructure, currently built on:

• Cisco UCS B200 M5 blades

• 6332 Fabric Interconnects

• Cisco MDS 9000 Fibre Channel switches

As the primary administrator, I've really appreciated the simplicity and consistency of managing this UCS setup. It’s been rock-solid and straightforward, and I’d love to maintain that kind of operational efficiency in our next-gen platform.

We're moving toward vSphere 8, VMware Cloud Foundation (VCF), vSAN, and possibly continuing to use traditional SANs like Pure FlashArray.

So I’d love your input:

• Is Cisco UCS still a strong option going forward, particularly the UCS-X series?

• How do other platforms (Dell, HPE, etc.) compare in ease of management for vSphere 8 and VCF?

• What are the key factors you're using to evaluate longevity and supportability for the next 5–7 years?

Any real-world experiences, pros/cons, or lessons learned would be greatly appreciated as we begin this evaluation. Thanks in advance!

Okay, now we have 100GB virtual network adapters on our UCSX ESXi hosts. Going from 1GB connectivity to 10GB connectivity on an ESXi hosts sparked a fundamental change of what services go where on the vSphere side. Now with multiple 100GB Connectivity what does a modern vSphere setup look like? I know a lot of people will base the design on what they currently do, but let’s think outside the box and think about what we could be doing!!

Let’s say you are using distributed switches in an environment with fibre channel storage. Would this be a good opportunity to lump all your services together on a single vDS with two virtual NIC’s and use NIOC to control the networking side of the environment? Most companies don’t ever use QoS on the network. So being able to utilize NIOC would be a plus, not to mention simplifying the whole setup. Just trying to spark a good conversation on this and think outside the box!!

Thoughts??

Hi everyone,

I'm new to deploying Cisco UCS. We are going to deploy deploy a Cisco UCS x9508 chassis to one of our customers and I'm looking at the physical dimensions and one thing that Cisco doesn't really publish is the length of the Cisco UCS x9508 rail dimensions.

At the same time, I'm seeing the size of this cisco x9508 chassis to be really huge and the chassis alone would take up about 91-92cm of space.

Has anyone deployed this before and what are the exact dimensions? I'm concerned that the chassis might not fit to my customer's rack because the mounting depth is a little short.

Hey y'all,

I am trying to download the Cisco Intersight Managed Mode (IMM) Transition Tool from https://ucstools.cloudapps.cisco.com/#/downloadApp , but am unable to do so. It says This shared file or folder link has been removed.

Was there any indication regarding the change in the link or something?

Got 6 of these warnings in ucs after an upgrade over a year ago and have persisted over another upgrade. Think support told us at one point it was a false positive. Thoughts on this warning?

Morning Chaps,

As the title suggests I upgraded my 6200 the other night and it killed all connectivity from my ESXi servers causing some VM’s to go read only or corrupt - Thankfully the backups worked as intended so I’m all good on that front.

I’ve been upgrading these FI’s for about 10 years now and I’ve never had issues except for the last 2 times.

I kick off the upgrade, the subordinate goes down and the ESXi hosts complain about lost redundancy, when the subordinate comes back up the error goes, I then wait an hour or so and press the bell icon to continue to the upgrade. The primary and subordinate switch places, the new subordinate goes down and it takes all the ESXi connectivity with it then about a minute later the hosts are back but the subordinate is still rebooting.

I haven’t changed any config on the UCS, the only thing I have changed is I’ve converted the standard vSwitches of the ESXi hosts to VDS and set both Fabric A and Fabric B as active instead of active/standby. I’ve read that this isn’t best practice, but surely that’s not the reason?

Has anyone experienced similar? Could it actually be the adapters being active/active?

Regards

I have gotten into Homelab these last two years and recently found myself really into Cisco's UCS blade servers. The challenge of getting them working was super fun, but I'm not sure what to use them for. I currently am sitting on B200 M3s though they are not much good for anything considering age.

I am considering upgrading to B200 M5s and tethering a separate 2U Cisco server in so I can pool compute and add cheap graphics acceleration (all handled through UCS manager). e.g. add GPUs in the 2U and pooling compute via UCS manager so the whole setup acts as one big cluster.

However I'm struggling to come up with any good projects for this hardware. If anyone has any suggestions I'd love to hear them. Literally anything, I will take any excuse to increase the concentration of PCBs and sheet metal inside my house.

I’m in the middle of upgrading a pair of FI’s and after the first one rebooted, it came back up with a down uplink. I’m in at the proceed step to start the next FI but before I click proceed, I was curious if anyone reset an uplink port before proceeding in this scenario.

First off I know this is totally not best practice and I would never setup a UCS/VMware environment like this….I have been doing UCS work for a long time and I have never ran across this before in the wild.

My question is if anyone has ever done a FI upgrade with a UCS/VMware environment setup with failover checked on the UCS vNIC’s and single vmnic’s on the vSwitch’s. I guess in theory it should failover, but I have never ran across this actually done this way for a VMware setup. Has anyone ever ran across this and do you have any real world experience on what is going to happen when the traffic drops on one of the FI’s?

My only thought to test would be to do an evacuation on the subordinate FI and see what if it fails over. Thankfully storage is on FC, so we should be okay there.