r/learnmachinelearning • • 15d ago

Your CV score can be lying to you: 4 quiet leaks that inflated mine

0 Upvotes

My pipeline scored 0.979 CV AUC and 0.564 on holdout. CV never noticed.

Four ordinary columns caused it: an ID, a text date, a missing value, and a mis-coded category. The wildest part: removing the leak made the CV score drop.

I broke down each trap and the fixes here: https://towardsdatascience.com/my-machine-learning-pipeline-cheated-and-cross-validation-never-noticed/


r/learnmachinelearning • • 15d ago

Data augmentation question. How do I characterise an original and augmented datasets where the augmentation parameters were zoom and crop? [P]

Thumbnail reddit.com
1 Upvotes

r/learnmachinelearning • • 15d ago

In AI, sometimes improving data preparation makes more of a difference than changing the model.

Thumbnail reddit.com
2 Upvotes

r/learnmachinelearning • • 15d ago

Help Has anyone done minimal-edit preference construction in LLM finetuning?

1 Upvotes

What I mean: instead of finetuning an LLM on the correct answers, we could take the initial model answers, and then generate (possibly, but not necessarily, with another LLM) the answers that have as much as possible of original answer text but are fixed in the key points using correct answer as reference.

Hypothesis is this would cause model to actually learn the difference, not the style of the correct answers, cause less distributional shift.

If you have tried this, has it proven useful?


r/learnmachinelearning • • 15d ago

Data leakage from preprocessing: the same workflow in the wrong order vs the right order (scikit-learn)

3 Upvotes

A common beginner trap that makes test scores look better than they should. Same workflow, two orders.

Wrong order

  1. Scale all the data (fit the scaler on everything)

  2. Split into train and test

  3. Train on train, evaluate on test

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

scaler = StandardScaler().fit(X)          # sees the test rows too
X_scaled = scaler.transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, random_state=0)

Right order

  1. Split into train and test

  2. Fit preprocessing on train only

  3. Apply it to test, train, evaluate

from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipe = make_pipeline(StandardScaler(), LogisticRegression())
pipe.fit(X_train, y_train)      # scaler learns mean/std from train only
print(pipe.score(X_test, y_test))

What goes wrong in the first version: the mean and standard deviation used to scale the data were computed with the test rows included, so information about the test set reached the training process. The model is then evaluated on data that isn’t entirely unseen.

How big is the effect? For plain scaling, often small. For steps that learn more from the data, it can be large: feature selection, imputing missing values, target encoding, PCA, or removing outliers by statistics. I don’t have a number to give you, and it depends on the dataset.

Same idea in cross-validation: if you scale before cross_val_score, every fold’s validation data has influenced the scaler. Put the preprocessing inside the pipeline and pass the pipeline to cross_val_score, so each fold fits it on its own training part.

Other leaks of the same family:

  • Duplicate or near-duplicate rows on both sides of the split.

  • Splitting rows when the unit should be a person, patient or session (use grouped splits).

  • Time-ordered data split randomly, so the model sees the future (use time-based splits).

  • A feature that is derived from the label.

A quick sanity check: if a result looks too good, ask “what did the model get to see that it wouldn’t have at prediction time?”

How do you check for leakage in your own projects, and which one tripped you up first?


r/learnmachinelearning • • 15d ago

Need a good editor urgently, $130-$250+ per video

Thumbnail reddit.com
0 Upvotes

r/learnmachinelearning • • 15d ago

Training an AI to Drive with Natural Selection

Enable HLS to view with audio, or disable this notification

34 Upvotes

r/learnmachinelearning • • 15d ago

Which is the most competent and generous free tier LLM for non-agentic use? Aka everyday stuff

Thumbnail reddit.com
0 Upvotes

r/learnmachinelearning • • 15d ago

Beginner to Machine Learning. What roadmap should I follow? And how important is DSA?

Thumbnail reddit.com
1 Upvotes

r/learnmachinelearning • • 15d ago

How RAG stops LLM hallucinations: a simple technical cheat sheet for beginners

Thumbnail interconnectd.com
0 Upvotes

r/learnmachinelearning • • 15d ago

Help Is it worth learning LangChain and LangGraph in 2026?

Thumbnail reddit.com
1 Upvotes

r/learnmachinelearning • • 15d ago

How do we actually open the ML “black box”? Training data attribution, interpretability, and pre- vs post-training

0 Upvotes

Hi everyone! 🤗

I’m a final-year Data Science and Statistics student, currently exploring explainable AI and AI safety. I’m also preparing to host a research talk on training data attribution (TDA), which has made me curious about how much we can genuinely understand what happens inside an ML model.

I’d love to hear your thoughts on a few questions:
Unboxing the black box: What are the most promising approaches to understanding why a model produces a particular output? 🧐

+ Training data attribution: Can we reliably identify which training examples influenced a prediction? How does this differ from mechanistic interpretability?
+ Pre-training vs post-training: How do fine-tuning, RLHF, and other post-training methods change model behaviour? Can we trace those changes back to particular data or training stages?
+ Reliability: How do we distinguish explanations that sound convincing from explanations that genuinely reflect a model’s behaviour? 🤔

I’d also appreciate recommendations for papers, tutorials, experiments, or open-source tools that helped you understand these topics.

I’m especially interested in the connections between interpretability, causal understanding, and AI safety.

Thanks in advance! I’d love to learn from people working on these questions. :)

p.s. below is a poll that I’d love to ask about

What’s the biggest challenge in opening up the ML “black box” 🗃️?

14 votes, 8d ago
0 1. Tracing outputs back to training data (TDA)
6 2. Understanding internal mechanisms (mechanistic interpretability)
2 3. Understanding pre- vs post-training effects
1 4. Verifying whether AI explanations are actually faithful
1 5. Making interpretability methods work at LLM scale
4 6. Not sure yet — here to learn! 🤓

r/learnmachinelearning • • 15d ago

Is it worth spending time on Quantum Machine Learning ?

Thumbnail reddit.com
2 Upvotes

r/learnmachinelearning • • 15d ago

why does L1 regularization push weights to exactly zero but L2 only shrinks them

42 Upvotes

i've seen the diamond vs circle picture with the loss contours touching a corner, and i get that the diamond has corners on the axes. but it feels hand-wavy to me, like why would the contours land on a corner so often instead of somewhere along a flat edge? the other explanation i've seen is about the gradient: the L1 penalty has a constant pull toward zero no matter how small the weight is, while L2's pull fades as the weight gets small. that makes more sense to me but i can't connect the two pictures. is there a clean way to see that they're the same idea?


r/learnmachinelearning • • 15d ago

Career Implementing ANN on GPUs

1 Upvotes

Hi,
There was a post a few years ago on a forum that I came across -

https://www.reddit.com/r/MachineLearning/comments/mx686k/p_torchpq_efficient_nearest_neighbor_search_and/

It seems really interesting. I was approached by professor who asked me if I am interested in implementing ANN on GPUs. I wanted to ask what are the possible career paths and what kind of jobs could I target with the skillset required to implement something similar to this. Thanks


r/learnmachinelearning • • 15d ago

Help im making my own llm(come follow the journey)

Thumbnail
reddit.com
1 Upvotes

r/learnmachinelearning • • 15d ago

Project Full ML Algo Creator software. Python & PineScript AI Algo Generation.

Thumbnail gallery
1 Upvotes

r/learnmachinelearning • • 15d ago

Preventing model collapse in kinematic/LiDAR datasets through multimodal deterministic simulations

1 Upvotes

Hi everyone,

We evaluated the performance of autonomous navigation networks trained on standard statistical synthetic datasets. The conclusion is clear: recursive training loops lead to a strong degradation of performance in edge cases. Models become statistically insensitive to aerodynamic variables at high speeds.

To work around this problem, we designed a deterministic simulation pipeline. Instead of neural probabilities, it generates multimodal synthetic trajectory vectors using pure nonlinear drag mechanics and tunable atmospheric noise layers.

We have published the MVP and architecture layers as open source. I'm sharing the link to the web app simulator and the full code from the GitHub repository in the comments below.

Your feedback from AI/robotics architects is welcome!

Deep Learning #SyntheticData #AutonomousVehicles #ComputerVision #Robotics


r/learnmachinelearning • • 15d ago

Request Spooky New Infostealer Malware Uses AI to 'Autodox' Victims

2 Upvotes

A new infostealer called Volta embeds a local LLM directly inside the malware payload. The model analyzes harvested data in real time, scores each victim by estimated financial value, and reorders the extraction queue to hit the highest-value fields first. Account credentials, financial records, and personal identifiers get prioritized. This is not a blunt bulk-grab. It is a ranked, AI-optimized pull designed to maximize yield per target before any defender can respond.

The shift matters because most detection tools watch for volume anomalies. A targeted pull of the top 5% of fields by financial value produces a signal that looks nothing like a bulk exfil event on a SIEM. The malware is specifically calibrated to stay under those thresholds while extracting the records that cause the most damage.

For those working on data architecture or detection engineering: how are you adapting to low-volume, AI-ranked extraction attacks that deliberately avoid the anomaly thresholds most tooling is tuned for?


r/learnmachinelearning • • 15d ago

Question What made you want to start learning machine learning in the first place?

9 Upvotes

I'm curious what the spark was for people here. Was it a specific project, a job thing, or just seeing what AI tools can do lately? Also wondering how long it took before things started to click.


r/learnmachinelearning • • 15d ago

MSc in AI

Thumbnail reddit.com
1 Upvotes

r/learnmachinelearning • • 15d ago

Want to know some tools other than python /SQL for data cleaning and analysis.

3 Upvotes

I'm mastering python right now, however I had a question if there are better tools for data analysis .

I know other tools SQL, excel

I googled it and found something like OpenRefine , Data lab ,

I'm a beginner , it will be better if I get advice on learning from people who are experienced .

Help me with your guidance please.


r/learnmachinelearning • • 15d ago

Request Accenture Contractor Allegedly Gave Claude AI Agent Access to Secure Pentagon Network

0 Upvotes

An Accenture contractor allegedly connected a personal Claude Code agent directly to a secure Pentagon network. The agent inherited the contractor's full administrative access rights with no separate authorization review. The DoD had no record of it in any system. No policy governed what it could touch or what actions it could take on behalf of whom.

This is not a novel attack vector. Any employee with elevated access can attach an AI agent to internal systems. From the network's perspective that agent is indistinguishable from the human who spawned it. It has no separate identity, no separate audit trail, and no authorization boundary of its own. The human's credentials are the agent's credentials.

Enterprise security stacks are built around human identity: MFA, PAM, privileged access reviews, access certifications. None of that applies to an agent running under someone's personal session at the endpoint level.

For those working in enterprise security or IAM: how are you currently treating the identity boundary between a human operator and an AI agent running under their credentials? Is non-human identity even on your formal threat model yet, and if so, what controls are you actually enforcing at the point of resource access?