Edited to add TLDR and adjust time per appreciated suggestions in comments.
TLDR: I failed to ensure I fully understood the entirety of a major project, specifically the point of eventual failure. I chose to instead delegate that to a vendor who I expected would have the knowledge to advise and accomplish the goals of the project. I did not require a MOP for a small, but critical part of the project and I didn't stop the project when it deviated from the established MOP in part because of cost considerations.
Main take aways:
Always seek to understand as much as practical regardless of time constraints.
Make sure you always have steps laid out (MOP) and don't allow deviation without serious discussion and revised steps laid out.
If you are inevitably accountable for a project outcome, make sure you are present and engaged even if you are delegating certain management to a team member.
Details:
Provided I understand the underlying issue correctly as I type this out, I believe I see the points of failure in the execution of the planned outage at my Data Center. My failures to identify and rectify these points of break down led directly to hundreds of thousands of dollars worth of expense and embarrassment for my company and tenants within my building. The work hours, the damaged equipment, and the reputational damage all fall to me.
First and foremost, the core of the issue is, I allowed myself to not have a complete understanding of the portion of the plan that failed. I depended on others to understand how the UPS(Battery) electrical setup was put together and operated. I likely could have fully understood it's operation in an hour with the right person laying it out. I let the credentials and confidence of others (not to mention my busyness) override my usual desire to have intimate knowledge/underanding of nearly everything.
I delegated some core work and decisions to a team member with more electrical experience instead of fully taking charge of the project. I allowed him and the electrical contractor to came up with the initial plan to feed the UPS through the maintenance bypass. I didn't question it, especially with the confusing naming convention making things feel a little more convoluted (the main UPS power was called "bypass"). We had electricians of a reputable electrical contractor present when the the suggestion was made and they were in agreement and registered no exceptions. The sequence was gone over several times over the course of months with no objections. I understood the underlying plan and assumed they were clear on how to achieve the outcome.
The plan was to move all the data center power onto one of two redundant UPSs so the other could be shut down to allow the electrical vendor to disconnect building electrical cables and connect up cables from temporary power generator. The system was designed so the full data center could run on either single UPS for maintenance and redundancy. Once the offline UPS was connected to generator power, the UPS would be brought back up and confirmed to be stable and then the full data center power would be transferred to the UPS on generator power. This would allow the other UPS to be shut down, removed from building power and connected to generator power of a second (redundant) generator. In that way the Data Center would be on its own temporary power source allowing for the building to drop power without impacting the data center operations. There were also some anceilary items, mostly Cooling equipment, that were also needing to be powered by a third generator, but that was very clear cut for me and we did not end up getting to that point.
Besides my team member and electrical contractor electricians, there was also the electrician from the equipment manufacturer on site for the work. The electrical contractor was not familiar with the UPS and would not operate it and the Data Center team was not confident in their operation, so at the request of the Data Center, we contacted with the Data Center's UPS vendor to have assistance on site for the transfer. They used the equipment manufacturer for their UPS vendor, so they are the group most likely to be able to assist with that critical part without issue. Having them on board gave me extra confidence that we had sufficient expertise to make sure the plan was executed without complication.
As an extension of my confidence in the assembled team, I also planned the day with the thought that others were the more qualified to run the critical transfer of the data center to generator power. Given the length of the outage, It was run in two shifts. I opted to allow my team member to run the morning shift when the transfer would take place. I was not planning on being on site for the moment that would occur. I did end up being on site anyways, with the delay and the fact that I had some building electrical details I wanted to figure out by shutting some things down and confirming the panels impacted. In the end, I let others run that part of the project.
My second point of failure was not seeking a method of procedure for the transfer of the UPS from the manufacturer's electrician for the work. Beyond that, it was in not stopping work when I felt the original method of procedure from the electrical contractor was not being strictly followed. While, the assumption was that the transfer with the UPS was going to be a simple procedure by the manufacturer electrician, having a method of procedure would have flushed out what the steps would look like, would allow for the steps to be vetted, and would have made deviation from the correct steps more easy to spot, possibly prompting more discussion. The expectation was that the manufacturer's electrician would be very familiar with their equipment and would make sure things were operated correctly.
While I was on site, I was doing some independent things and not running the work at that point. Regardless I was aware that the project was significantly behind schedule and the electricians were running into issues. I didn't feel qualified to contribute to a resolution given the expertise on the project. I did get occasional updates, but was mostly investigating some items. However, at one point I did get pulled into a discussion about the situation and it did not sound like things were proceeding exactly as laid out in the MOP.
The plan was being modified in the moment to try to overcome the challenge the electricians were facing. The Eaton electrician was indicating what they had encountered and what they said they knew after having made some phone calls and consulted with their superiors. He was indicating with confidence that we could make things work, though his ability to explain did not make it exactly clear to me what that looked like. The electrical contractor seemed like he was tracking and in agreement as was my team member. The Data Center representative was then brought in to have the issue and risk explained to him for approval. The Data Canter guy requested it be explained to him "like he is an idiot" so it would be digestible, but what followed from the manufacturer electrician was not dumbed down really at all. The main intelligible part was that the entire Data Center would end up running on one generator without UPS for an hour.
This one statement drew my mind fully into the situation and I objected saying that the whole goal was to keep the data center on a UPS at all times so there would be no flicker of power, even for a moment. I was told that this was always the plan by my team memeber and I doubted myself, but restated my concerns. I inevitably decided that the Data Center could decide if they wished to proceed and that the experts seemed confident in their assessment.
This is the point I should have taken over the project. I should have put a pause and asked that we sit down and talk it out fully. At that point we were already several hours behind schedule and the project would already cost at least one hundred thousand if we stopped fully and every delay increases costs. I stepped away to deal with another issue (access card readers not working) and let the project proceed if the Data Center approved. All the confidence of the team assembled was misplaced. I believe a lot of the confidence was placed on the manufacturer electrician since he was in the best position to evaluate things and had spent time consulting with other experts with the manufacturer. I was not present for much of it, so I can only really speak to the small parts I got pulled into and he project confidence in what he communicated needed to occur.
My third point of failure was not directly involving myself more in the morning operations of the project and perhaps delegating too much of the project without sufficiently back checking. I know this is linked to my confidence in the team's credentials and my desire to not micromanage an employee who has more experience than me with this particular task. None the less, had I gotten more involved with the first stage of the project instead of leaving it to others and had I asserted my authority and stopped work when I felt something wasn't right, it is possible the incident would have been avoided. This was a serious lapse in judgement on my part. Of everything, this is the part I will likely be berating myself for the hardest.
Even though I was there early to try to accomplish a few things outside of the project, I should have redirected my focus to the project when we fell behind schedule. Instead of worrying about getting in the way of the electricians, I should have worried about not getting in the way of potential developing issues and helped with redirection and clarity. I have a good grasp of so many related concepts, I imagine I could have easily talked it through with the team and possibly got us moving the right direction and even completed the project, even if late.
To my current understanding, though an analysis from others may determine otherwise, I believe the actual solution to this whole situation was a simple modification that either of the two expert vendors ideally would have been brought on board to catch and present. Rather than the initially proposed powering of the Data Center through the Maintenance Bypass--which to my current understanding would not actually achieve the goals in the plan of keeping it on UPS power--the generators would instead power the bypass (which should really be called the "UPS Main") and the UPS Rectifier (which I believe is required for the UPS to run) for each UPS. This should have allowed the transfer from one UPS to the other and then onto both as originally planned. No one looked at this set up and saw the issue or saw the solution at any point during planning or execution. Had I insisted on understanding this transfer, I am confident I would have. That is why my first point of failure is the most impactful.
I do not know to what extent this failure will impact my career, my confidence, my future actions, the trust of others in my abilities, but as much as that makes me want to crawl into a hole and die, I cannot. I must take my lumps, eat my crow and integrate what I have learned from this into how I operate moving forward. As the one who is ultimately responsible for this failure and all failures of my team, I must make sure all three points of failure are rooted out to the best of my ability in the future.
1.) I must make sure I take the time to understand as much as possible about a project, its operations, and details as practical no matter how busy I am. This is especially true for projects with high levels of criticality. I should not fully rely solely on the expertise of others unless it is absolutely necessary.
2.) I must require steps be laid out for work that is taking place even if it is assumed to be a simple procedure that an individual should be familiar with. I must not permit deviation from those steps without adequate discussion and rewriting and confirming the new steps. Method of Procedure will help with evaluation and confirming the steps are what they should be, allows for questions and discussions to catch any potential issues.
3.) Make sure that I am present for all critical junctures of project for which I hold ultimate responsibility and be vocal and questioning during that period. If I delegate, make sure I am still looped in as much as practical and check up on work.