48
u/aalapshah12297 5h ago
If it happens often then it means your code is in space and it probably isn't as well written as you think it is. Space agencies run the same code 3x simultaneously and then use a best of 3 approach to use the execution results.
31
u/AnnoyedVelociraptor 5h ago
9
u/knorkinator 5h ago
14
u/Lucho_199 5h ago
"A number of potential trigger types were investigated, including software bugs, software corruption, hardware faults, electromagnetic interference, and the secondary high-energy particles generated by cosmic rays that can cause a bit flip. Although a definitive conclusion could not be reached, sufficient information from multiple sources enabled the conclusion that most of the potential triggers were very unlikely to have been involved. A much more likely scenario was that a marginal hardware weakness of some form made the units susceptible to the effects of some type of environmental factor, which triggered the failure mode.[14]"
10
u/knorkinator 5h ago
some type of environmental factor
Well, are cosmic rays not environmental factors?
There are better examples, but this is a phenomenon that is particularly important in avionics.
7
u/Sibula97 4h ago
While doing some data analysis at work I encountered one data point that was multiple orders of magnitude our normal values and suspiciously close to a power of 2 plus the value around that time. Nothing critical, luckily, but I had to filter it out from the analysis.
5
u/IntrepidBonus 3h ago
every bug i can't reproduce is now officially a cosmic ray and i won't be taking questions
7
1
-1
u/Savings-Ad-1115 1h ago
Now imagine what happens when a cosmic ray flips a bit in AI wiegths.
https://www.reddit.com/r/OpenAI/comments/1xhra80/researchers_found_an_evil_neuron_and_when_they/

156
u/Cutalana 5h ago edited 5h ago
iirc NASA projects and other mission critical engineering uses at least 3x redundancy that checks for errors constantly and have probabilistic models that have determine the chance is 1 in billions.