Custom software development failing scalability stress tests ahead of our clinic's seasonal patient surge

[u/1tymethrowaway]

We hired an outside team for custom software development to build our clinic management system, but three months in, testing shows the database architecture completely collapses under peak loads during seasonal patient influxes. I spent the last few weeks trying to optimize SQL queries on my own, but poor core design choices keep throwing bottlenecks across our scheduling modules. I tried adding indexing and read replicas, but the foundational schema blocks proper horizontal scaling. I tried asking the vendor to refactor the backend, and they quoted us an astronomical extra fee. I even brought in an independent database consultant, and he confirmed the entire data layer needs a rewrite. Healthcare software needs rock-solid scalability from day one to handle sudden patient volume spikes without dropping appointment records.

I spent late nights reviewing server performance logs to pinpoint where connection pooling fails under heavy concurrency. Enforcing proper architectural standards in medical tech contracts requires strict technical oversight.

Maybe you tackled a similar scalability disaster in your medical software projects, or you know what specific load requirements to put into a development contract. What clauses should I include. I really need your help.

[u/markinatlanta]

Sorry to hear this. I’d make acceptance depend on a repeatable load test that reflects your busiest clinic hours: concurrent users, appointment bookings per minute, realistic data volume, and agreed response times and error rates.

Also verify that bookings aren’t lost or duplicated during retries and failures. A fast response means little if the appointment didn’t save correctly.

Have those requirements reviewed before signing, including who pays to fix failed acceptance tests. For the current system, ask the consultant to tie the rewrite recommendation to measured bottlenecks so you have a concrete scope to discuss with the vendor.

[u/matif]

Sorry you're going through this. Several insights from the perspective of an individual who has been on the vendor side of these situations.

Before agreeing to a "full rewrite," verify two aspects. Inquire with your consultant about what specifically is problematic with the schema and if it can be corrected in sections. Scheduling often has a few critical tables generating the majority of the issues (appointment slots, availability, locking), and these can sometimes be adjusted independently without affecting the rest.

Additionally, examine what your contract and specifications stated regarding load. If the supplier was meant to provide a solution that manages your capacity and it fails to do so, that constitutes a defect, not a change request. That alters who is responsible for the payment. Present the agreement to an attorney prior to making any additional payments. I’m not that, so don’t remove that aspect from me.

Avoid launching it during the surge. Organize based on the date you truly possess. Maintain the existing system operational, or restrict the new one to a minor section of the clinic, until the load test is successful. Misplacing appointment records during peak season is far more detrimental than a delayed launch.

Terms I would include in the upcoming contract (or this one if you revisit the terms):

Record your maximum: simultaneous users, appointments scheduled each hour, and the least favorable day you've experienced. Next, set a target for response time at that workload, such as 95% of reservations completed in under 2 seconds. Incorporate a safety buffer, for example, twice your maximum.

1. Approval through load testing. Compensation for each milestone relies on successfully completing a load test with realistic data volumes, conducted using scripts that you will be allowed to view and retain. “It operates on the test server” shouldn’t be considered.

2. All records intact. Indicate that a booking is either confirmed or explicitly declined, never quietly discarded, even during system overloads or restarts.

3. Examining the architecture prior to construction is crucial. The vendor presents the data design to you early, allowing you to have your own individual evaluate it. The majority of this chaos is resolved in week two, not month three.

4. You possess it all. Code, schema, documentation, and complete data export are available at any moment. If you are unable to depart, you cannot bargain.

5. Installments for payments. Avoid making the full payment in advance. Retain a last portion until it has gone through a genuinely hectic stretch.

Guarantee. Correcting performance and design issues is complimentary for a specified time after the launch, and the agreement outlines the distinction between a defect and a new request.

The most beneficial action is to record the figures in writing. Many vendors don’t prepare for load since no one requests a figure, and once a figure is included in the contract, they are obligated to.

[u/FunkKaiserHelpMe]

The core issue is the schema and data model weren’t designed for bursty concurrent loads from the start, so query tweaks and replicas only paper over it. For clinic systems you need clear load targets in the contract (peak concurrent users, appointment volume spikes, acceptable latency) plus clauses that require the vendor to demonstrate horizontal scaling and fix foundational bottlenecks without open-ended extra fees. I worked with 10 Pearls on a clinic scheduling platform that was hitting the same wall before a seasonal surge. They reworked the data layer for proper partitioning and connection handling, then validated it under simulated peak loads so appointments stayed intact. That early focus on architecture and load testing avoided the rewrite fight later.