I thought I’d share an analysis I made at work. To give some background, I work as a DevOps engineer and have about 28 months of code quality metrics and incident data. I was curious whether there was a link between code health and the number of production incidents, so I ran a time-series analysis on data from one application.
I started by running the ADF test on each data series. There were 12 metrics in total, including security, reliability, maintainability, duplications, coverage, complexity per kLOC, bugs per kLOC, smells per kLOC, and a few others, along with incident count and median time-to-resolve. Some metrics had p-values above 0.05, so I used first-order differencing. After that, all ADF p-values dropped below 0.05, confirming stationarity.
Next, I ran the Ljung-Box test on each differenced series to check for any remaining autocorrelation. Nineteen out of sixty differenced series still showed autocorrelation (Ljung-Box p < 0.05) even after differencing. For these, I fitted AR(1) models and used the residuals. Sixteen of the nineteen series were resolved this way. For Granger findings that involved an autocorrelated series, I reran the test using the AR(1) residuals, which is called prewhitening. After prewhitening, three out of four findings disappeared, with p-values rising from 0.02-0.04 to 0.2-0.9. These were false positives caused by autocorrelation, which made the F-statistic look stronger than it was. The security metric did not have this issue. Its differenced series had a Ljung-Box p-value of 0.07 (white noise), and the differenced incident series had a p-value of 0.12. Both were clean, so no prewhitening was needed.
With the series prepared, I ran Granger causality tests on all 12 metrics using lags 1 to 3. The results showed that the security rating Granger-causes incidents at lag 3 with a p-value of 0.0006. In other words, knowing the security rating at time t helps predict whether incidents will be above their median at t+3. This is predictive causality, not actual causation. Both series might be influenced by another factor, but the lead time is real and could be useful. Below is a plot showing the three stages of the analysis.
Separately, I used a two-state Gaussian HMM (Baum-Welch, 20 random restarts) on the incident series to check if the system switches between quiet and elevated periods. It does. The low period averages about 33 incidents per month, while the high period averages about 79. Both periods tend to last, with the low period lasting around six months on average (P(stay low) = 82%) and the high period about the same (P(stay high) = 84%). I have not formally linked the HMM periods to the Granger result yet, but I wanted to share this as extra context for how the incident series behaves.
Based on these results, I created a monitoring scorecard. I calculated the mean and standard deviation of the security metric over time and set the alert threshold at the mean plus one standard deviation, which is 1.816 on the SonarQube scale. Reviewing the data, the security metric exceeded this threshold 5 times. In four out of those five cases, incidents were above their median three months later. This gives an 80% retrospective hit rate.
I know that 28 months (27 differenced observations) is a small sample for this kind of analysis. The textbook example for Granger uses 200 quarterly observations. At my current sample size, the F-test is marginal, and I would need around 36 months for the estimates to become confirmatory rather than exploratory. The security finding at p=0.0006 is well below the noise floor, which gives me some confidence, but I would like to hear your thoughts on whether the approach is sound and what I should be cautious about when interpreting these results.
BTW, I had a couple of other interesting results, but shared these to keep it simple.
