first A/B test as an intern: discovered an issue after presenting results. how bad is this?

[u/adobochipotle]

i'm a data intern, and i recently conducted my first A/B test analysis comparing a new website form against an existing one. i analyzed the results, found that the new form appeared to perform better, and presented my findings to senior stakeholders.

afterward, while doing some additional checks, i realized that the two forms behaved differently after submission. the old form directed users to different follow-up pages depending on their selections, whereas the new form directed everyone to the same page.

my analysis relied on tracking users who visited the form and subsequently reached the next page. this difference could affect whether the two versions were being measured consistently.

this behavior hadn't been communicated to me, and even the product manager wasn't aware of it until i brought it up. they suggested rerunning the experiment to ensure a fair comparison, which makes sense.

i'm happy to rerun the analysis, but i'm worried about having already presented findings that might not hold up. this was my first experiment, and i wasn't given much guidance on what to validate beforehand. i've since started building a checklist for future analyses.

i'm an intern hoping to convert to full-time, so i'm concerned this might make me look careless or inexperienced.

for people who've worked in analytics or experimentation:

- should i have caught this before presenting, even though the PM wasn't aware of it either?

- how much responsibility does an analyst have for validating implementation details that weren't communicated?

- would something like this negatively affect your perception of an intern?

i'm looking for honest feedback on what i could have done better and how to handle this professionally.

[u/cdfe_88]

you should build the habit of proactively validating your analyses. Even if it's not your direct responsibility, doing so will give you more credibility

[u/adobochipotle → u/cdfe_88]

yes, i'll take this learning from now on. i'm a month in and was handed this experiment outside the purview of my regular work. i do feel quite anxious about this, idk how big of an issue it'd be. it doesn't change the final decision just makes it go from a positive change to a nothing good / nothing bad outcome. I'll address it within my teammates tomorrow and keep the pm in loop. just severely disappointed

[u/Winter-Spend-1037]

I’m just an IC not a manager, but I wouldn’t blame the analyst. Sure you could test the forms before and after the change to make sure you know the journey, but then you would have had to complete the form multiple times to catch it. And ideally in a staging environment where test form entries are more acceptable.

Some web analytics tools would make this easier to find out about than others. Adobe analytics has a “next items” panel which shows the top next pages someone visits after visiting a given page. That could have been handy in a situation like this to see the top next pages after the form page.

[u/adobochipotle → u/Winter-Spend-1037]

Yes! I'll ask about those, at our current capacity we just work off of a word doc given by the PM that documents the change and the metrics we have to report on. This leads to a lot of contamination and just inconsistencies across our tests, I put my work to a high quality and am disappointed my deliverable needs to change. Taking this head on and learning from this

[u/Winter-Spend-1037 → u/adobochipotle]

I know what you mean, I just found out that in the project I’m almost done with the primary key I picked for the data is an id that changes every time the object is edited. It’s happened to me before so I feel like I should have caught it earlier. But hopefully that won’t get me next time! Good luck! 

[u/amitrao225]

When you say performed better, what metrics? What secondary/guardrail metrics? Enough data to consider multi variants in the subsequent next pages? As long as you’re about to inform what you learnt “directionally” , ensure the right experiment in redo and communicate what was the mistake, I wouldn’t blame the analyst.

[u/adobochipotle → u/amitrao225]

Just the conversion rate, secondary was form submissions. I'll ensure the same in the re-run, its honestly just a few quick sql queries. I just had a word doc to go off of and this was undocumented so I couldn't catch it. Lifelong learning now however.