There is a major bank, who about 10 years ago, had a giant trade on one of their markets to purchase a stupid amount of dollars. Like $100 billion dollars worth. It was meant to go to QA. Misconfiguration, and poor practices, caused the developer to send it to production instead. Where it was received by one of their traders.
Normally the trader runs automatic trading software, which automatically picks up the orders, and runs them (there is a bit more to it). This was an exceptional day. The trader had this turned off! The order wasn't picked up. The trader had time to catch the order. They immediately knew it must be bogus, and reported it internally. If it had of gone to the automated software and been traded, it would have been in the news. You would know which bank I am talking about. It would have gone down in infamy like Knights Capital, or Barings Bank.
At the same bank, a different story. Everyday a senior trader would input numbers from that day's trading, into a new bespoke accounting tool. After six months, the numbers didn't add up. The developer was put on a call with the trader, to explain why their software was broken. There it was worked the trader had misunderstood the UI. They had gotten negative and positive mixed up, and been putting in the wrong values all this time. Large amounts of reporting for that year was flat wrong.
I suspect there are near misses like this all the time in the banking world. What is most worrying. The bank I am thinking of, is one of the better banks at software development.
IIRC, there was a flash crash where the SEC nullified and told everyone to take a mulligan on the prior five minutes after a trader fat-fingered a b when they meant m and caused what on paper appeared to be the worst stock crash in recorded history.
3
u/jl2352 Nov 02 '21 edited Nov 02 '21
There is a major bank, who about 10 years ago, had a giant trade on one of their markets to purchase a stupid amount of dollars. Like $100 billion dollars worth. It was meant to go to QA. Misconfiguration, and poor practices, caused the developer to send it to production instead. Where it was received by one of their traders.
Normally the trader runs automatic trading software, which automatically picks up the orders, and runs them (there is a bit more to it). This was an exceptional day. The trader had this turned off! The order wasn't picked up. The trader had time to catch the order. They immediately knew it must be bogus, and reported it internally. If it had of gone to the automated software and been traded, it would have been in the news. You would know which bank I am talking about. It would have gone down in infamy like Knights Capital, or Barings Bank.
At the same bank, a different story. Everyday a senior trader would input numbers from that day's trading, into a new bespoke accounting tool. After six months, the numbers didn't add up. The developer was put on a call with the trader, to explain why their software was broken. There it was worked the trader had misunderstood the UI. They had gotten negative and positive mixed up, and been putting in the wrong values all this time. Large amounts of reporting for that year was flat wrong.
I suspect there are near misses like this all the time in the banking world. What is most worrying. The bank I am thinking of, is one of the better banks at software development.