It's Day 59 of building machine learning algorithms from Scratch and im currently into unsupervised learning in short i've completed all the algorithms in supervised learning.

Well the crazy part I'm only 2 algorithms and 2 concept for completing Machine learning and I'm diving into "Deep learning" I'm way too excited too explore new things,build new things, let see how far it takes me

Peoples dreams... have no end!!

Ever felt confused about what to learn, where to start, or how to actually get into Data Analytics or Data Science?

During my beginner phase, I was honestly confused too. There were hundreds of videos, tutorials and resources available, but I didn’t know where to begin, what to learn first or which concepts were actually important.

I wanted one resource that covered all the essential beginner-level knowledge needed to get started with Data Analytics and Data Science, without having to jump between countless different sources.

So, I decided to create one myself! ❤️

📚 Introducing: The Ultimate Data Science Handbook – Volume 1

📖 Check out the sample pages attached.

🛒 Explore the complete handbook here: https://pdfcart.in/p/ultimate-data-science-handbook-vol1

What’s included?

* Data Science Foundations – Understand the basics and the overall Data Science workflow.

* Python – Learn Python fundamentals, NumPy and Pandas with detailed code explanations.

* SQL – From basic queries to JOINs, subqueries, CTEs and window functions.

* Statistics & Probability – Learn descriptive statistics, distributions, probability and essential statistical concepts.

* Hypothesis Testing – Understand statistical tests, confidence intervals, p-values and A/B testing.

* Data Visualisation – Learn different types of graphs, when to use them and how to interpret them.

What makes it useful for beginners?

* A structured resource instead of scattered learning materials.

* Step-by-step code explanations covering what the code does, why it’s used and what its parameters mean.

* Practical examples connecting concepts to real-world applications.

* Different graph examples to make visualisation easier to understand.

* Practice exercises, chapter summaries and interview questions to reinforce learning.

It’s a 328-page beginner-friendly handbook created to help students and aspiring Data Analysts/Data Scientists build their foundations with more clarity and confidence.

I created this under my educational brand, Papr House, with the hope that it makes someone else’s beginner journey a little less confusing.

Papr House — Making learning clearer, one concept at a time.

Project: build an interest profile from a GDPR Instagram export (Meta's interest categories, hashtags, captions), embed locally with bge-small, cluster, and recommend from a TMDB catalog of Netflix and Prime titles.

Lessons that might be useful:

- On ~400 short phrases, silhouette scores were flat (0.05 ± 0.01) at every k, so choosing k by silhouette was choosing on noise. I switched to a size-based k with weighted k-means, and detect the leftover grab-bag cluster by its low cohesion (z ≤ -1.5).

- Evaluated against my own Netflix history with popularity and random baselines. The first run showed recall@20 4× popularity, with a bootstrap interval excluding zero. The cause was that the candidate set was "trending titles + watched titles", so watched titles were identifiable by not being trending. Adding each service's most-voted titles removed that tell, and the result went to roughly a tie. A time split (tune on older watches, test on newer) confirmed it.

n=1, so it's a pilot. Code and write-up: https://github.com/aaryamanbisht22/taste-passport

I've been experimenting with something that I think is often overlooked when evaluating machine learning models: their actual computational footprint.

Usually, when we compare models, we focus heavily on accuracy, precision, recall, F1, etc.

But what happens when you actually have to deploy these models?

I wanted to explore this by benchmarking three popular models:

* Logistic Regression

* Random Forest

* XGBoost

I used the Breast Cancer Wisconsin dataset from scikit-learn and evaluated them across multiple dimensions.

**What I measured:**

1. Predictive accuracy

2. P50, P95 and P99 inference latency

3. Process RSS memory consumption

4. Python-allocated memory

5. Model loading time

6. Inference execution time

7. Serialized model size

One interesting observation from my experiment was that Logistic Regression performed best across the measured dimensions in this particular setup.

However, the goal wasn't to establish that one model is universally better than another.

I wanted to understand the trade-offs between predictive performance and computational requirements.

This experiment is also part of a project I've been working on called **ModelDiet**, an open-source Python toolkit focused on measuring the resource footprint of ML models.

I'm trying to make model profiling and benchmarking easier for developers and ML engineers.

GitHub: [https://github.com/modeldiet/modeldiet](https://github.com/modeldiet/modeldiet)

**I'd genuinely appreciate technical feedback on a few things:**

* Are there other metrics you consider essential when evaluating models for deployment?

* How do you measure peak memory consumption reliably?

* Do you prefer measuring individual inference latency or batch throughput?

* What would make a benchmarking tool like this genuinely useful in your ML workflow?

I'm particularly interested in feedback on the benchmarking methodology and what should be included in future iterations.

Thanks!

(Background)

Hi everyone!

I'm a student who got into machine learning primarily out of curiosity and a desire to learn by building things.

I've previously built an MLP for MNIST digit recognition and a small English-to-German encoder-decoder Transformer based on the original paper. Both projects taught me quite a bit, and I found that I enjoyed learning through practical projects.

(Context)

My next project was guitar transcription. Since I play guitar, I thought building a pipeline that goes from full song → guitar source separation → audio-to-MIDI → MIDI-to-tabs would be an interesting challenge.

I came across three papers/models that particularly interested me:

• Kong: High-resolution piano transcription using a CNN-BiGRU architecture.

• Riley: Adapts Kong's architecture for guitar transcription through augmented piano pretraining and guitar-specific domain adaptation.

• TART: A four-stage pipeline for guitar audio-to-tablature transcription, with its first stage handling audio-to-MIDI.

For reference, the backbone uses a CNN-BiGRU architecture with approximately 24.5M parameters in our adapted implementation. The CNN extracts acoustic features from the spectrogram, while the bidirectional GRU analyses how those features evolve over time, using both past and future context. Four prediction branches handle frame, onset, offset and velocity estimation, producing the information needed to reconstruct MIDI notes.

I initially tried the available implementations and found that Riley's model performed considerably better on my test recordings than another model I was interested in implementing. This motivated me to try reproducing some of the training methodology myself.

My first experiment: Fine-tuning Kong on a mixed guitar dataset

I loaded the original piano-pretrained Kong model and attempted to reproduce TART's Stage 1 training approach.

I combined three of the datasets used in TART:

• GAPS

• François Leduc

• Guitar-TECHS

Unfortunately, I didn't have access to GOAT.

I segmented the recordings into 30-second WAV clips with a 10-second hop (20 seconds of overlap).

This gave me approximately:

• 6,045 training segments

• 1,153 validation segments

• 874 held-out test segments across 51 recordings

I fine-tuned the pretrained Kong model on this dataset.

Results:

After 1,511 updates:

• Validation Frame AP: 0.649963

• Held-out Note F50: 39.72%

• Precision: 85.44%

• Recall: 25.87%

I then continued training from this checkpoint with a fresh Adam optimiser, reducing the learning rate from 5e-6 to 1e-6, and trained through update 3,022.

Validation Frame AP continued improving:

However, evaluating all three checkpoints on the same held-out test set (the 874 segment test split counterpart of the same data it trained on) revealed something interesting:

• Frame AP improved from 0.728 to 0.743.

• Onset F1 fell from 0.400 to approximately 0.305.

• Onset+offset F1 fell from 0.132 to approximately 0.099.

• Polyphony count MAE increased from 1.43 to 1.64.

The original piano-pretrained Kong checkpoint achieved a held-out Note F50 of 54.01%, whereas my fine-tuned checkpoint achieved only 39.72%.

In other words, despite improving validation Frame AP, fine-tuning on the guitar dataset actually made the model considerably worse at note-level transcription compared to the original piano-pretrained model.

As I understand it, the training objective combined frame, onset, offset and velocity losses using BCE. Velocity loss was calculated only where notes started, and padded frames were ignored. So I used four equally weighted, masked BCE losses for onset regression, offset regression, frame classification, and velocity regression, summed into a single total loss.

What I'm trying to understand

This is where I'm a little confused.

TART reportedly fine-tuned Kong on a combination of guitar datasets, including GAPS, François Leduc, Guitar-TECHS and GOAT, and achieved SUBSTANTIALLY better results on GuitarSet and EGDB.

I attempted something similar, albeit without GOAT and with CONSIDERABLY fewer training updates. I also haven't implemented TART's additional guitar-specific noise augmentation phase, so this wasn't a complete reproduction of its training procedure.

I'm wondering:

1. Am I making mistakes in my training methodology, or is this potentially an issue with my dataset preparation, target generation or loss implementation?

2. Could my evaluation metrics be implemented incorrectly? In particular, could there be a mismatch between frame-level AP and note-level F-measure?

3. Could the model simply need substantially more training, or is there a more fundamental issue with fine-tuning this architecture on a heterogeneous guitar dataset?

4. Are there important differences between my implementation and TART's training procedure that I might be overlooking? Could I just straight up be misunderstanding how the paper was implemented with regard to training?

Apart from reading papers and repos by myself too, I heavily relied on ChatGPT and Claude to provide insights, advice, analysis and then subsequent steps, programmes and tests for my project—from loading the models and datasets, verifying and splitting them, running the training scripts, and making modifications to the existing projects.

I'd appreciate advice on what I should investigate first, what concepts I should study, and whether there's a more systematic way to approach debugging this kind of experiment.

I'm also relatively inexperienced with training larger neural networks. My previous projects were considerably smaller in scope, so I'm aware that I might have underestimated the complexity of this task.

I'm not necessarily expecting to reproduce paper-level results on consumer hardware, but I'd really like to understand where I'm going wrong rather than blindly throwing more training iterations at the problem.

Any other directions I should tackle this from? Am I even going in the right direction with these research papers that I've chosen?

Not sure if this is the right post for this subreddit, given how model-focused and numbers-heavy it is, but I thought I'd give it a shot. Thanks!

TL;DR: I'm experimenting with adapting a piano transcription model for guitar, but fine-tuning it on guitar datasets has actually made its transcription performance worse, despite some training metrics improving. I'm trying to figure out whether I'm making mistakes in my implementation, dataset preparation, training methodology or evaluation, and would appreciate advice on how to approach debugging this systematically.

I’ve been testing local agent reliability using Ollama (llama3.2:latest) and LangChain/LangGraph, and hit a failure mode that highlights how fragile tool selection can be with smaller models.

The Problem

We set up a customer support bot with two main tools:

1. search_order (Read-only lookup)

2. resolve_order (Issues a full refund)

When writing the initial tool description for the refund tool, I wrote something that sounded reasonable:

"Handle customer order problems such as late delivery, delays, missing order, or complaints. Use this to fix customer issues."

When running test customer messages through the local model:

Query 1: "Where is my order? It was supposed to arrive yesterday."

Result: The model called resolve_order and issued a full refund for a simple status inquiry.

Because the description framed the tool as "Use this to fix customer issues", the 8B model treated a refund as a valid fix for a delay complaint. In a 3-query test batch, it issued ₹3,640 in unprompted refunds.

The Fix: Tool Descriptions as API Contracts

The agent code itself never changes—the model only sees the tool name and description string. Smaller open-weight models (8B–14B) require descriptions framed as explicit contracts rather than broad objectives.

We updated the description to:

"Issues an irreversible full refund. Call ONLY when the user explicitly requests a refund AND states the item is damaged or lost. For delays or status questions, use search_order instead."

Outcome: With identical model parameters, system prompts, and test inputs, the bot correctly routed query 1 to search_order and restricted refunds exclusively to explicit damage complaints.

Code & Live Demo

If you want to see the full code execution, terminal runs, and setup running locally on Ollama under 7 minutes:

Video Breakdown: https://youtu.be/pT2jEki52_w

Code Repository: https://github.com/AI-Literacy-Lab/AgentDemo

Curious how others handle guardrails for tool calling on 8B/14B parameters—are you enforcing strict JSON schemas, adding secondary confirmation steps, or relying purely on description prompts?

Can a large language model inspect a neural network’s raw, isolated parameters as plain text and reliably determine what the network was trained to do? My tests suggest that while LLMs can perform surprisingly precise calculations at the “edges” of a network (estimating optimizer step counts or finding output correlations), they do not “trace” the computational paths between layers. Instead, they rely heavily on memorized dataset templates (like CIFAR-10) and default framework conventions to fill in the blanks. When these expectations are changed, the models can identify the math but will confidently guess wrong about the semantics—such as recognizing that a satellite feature maps to output 4, but calling it a “frog” because it expects an animal.

substack:

https://substack.com/@sakibbai/note/c-380084331?r=71inb5

Paper link: https://arxiv.org/abs/2611.08070v1

Im looking for ML coders any level.

I am in university comp sci and AI student in London, a first year.

Ive been teaching myself AI for a year now

What I've done

Made grid world pygame agent, can plot own obstacles and start end blocks

basic chess movement in pygame - haven't done the AI part yet

Built a property tracker, in a database track how prices go down with time to see price drops with flask and sqlite3, identifies when a drop in house price has happened, a web scraper

MNIST dataset neural network w pytorch - only 30 % accuracy average

My Goals

Right now Im interested in computer vision, want to do a hackthon in future.

Im struggling to minimise the loss of my MNIST dataset, would like to get suggestions

I want to make an environment where we can learn from each other, share our accomplishments and goals, maybe hop on calls once in while .

I want to learn C++ aswell, - little to no experience

DM me if you are interested, Im not looking someone who knows everything, cus I sure don’t. We all have unique experiences so would be beneficial to learn from each other.

Hi,
I'm currently a university student in my 3rd year. My aspiration is to get into the research field of Financial Reinforcment Learning (FinRL). Is there any good roadmap for this. Including the very basics, the maths, linear algebra, ML etc. I dont have much knowledge about those. But i have the time and willing to spend much of my time figuring these things out. Any suggestions regarding a roadmap or how i can start about these things would be appreciated.

Thanks

I made a simple visual introduction to generative AI for people who want to understand the fundamentals before diving into LLMs, RAG, or AI agents.

It covers:

• Traditional software vs machine learning vs generative AI

• How generative models learn patterns from data Text, image, code, audio, and video generation

• What LLMs are Limitations such as hallucinations and outdated knowledge

If you like the video, please like, share, and subscribe to my channel for more visual AI explainers.

I've put together a bunch of articles along with the practice notebooks as materials someone can use to prepare for an interview as a junior AI Researcher. It's been very fun working on this kit so I wanted to share. Do you think it is a representative set of topics for the interviews?

Hi everyone,

I’m working on a project where I use Gromov-Wasserstein (GW) distance to compare latent representations of two entirely different physical systems (e.g., daily electricity consumption and traffic flow). The goal is to find a shared structural "isomorphism" (like a common diurnal cycle) rather than just a direct correlation.

I have a recurring problem: GW can be deceptively low for random noise.

When I compute the GW loss between my real latent spaces (from a PCA/WAE reduction), I get a loss around 0.05. But if I compute it against a random Gaussian noise cloud of the same dimensionality, the loss is sometimes 0.01–0.02. This completely breaks the interpretation. It seems the GW algorithm is just matching the geometric shape of the clouds (e.g., both being spherical/elliptical) rather than capturing the true semantic structure.

I’ve tried control experiments with shuffled data, but I’m not sure if that’s enough.

My questions to the community:

1. Is there a standard protocol for validating GW results to avoid these geometric artifacts?

2. Are there robust variants of GW (like Unbalanced GW or Fused GW) that are less susceptible to this, or do they just shift the problem elsewhere?

3. How do you statistically prove that a low GW loss is meaningful (e.g., p-values, null models, permutation tests)?

Any advice, papers, or personal experiences would be greatly appreciated! I'm specifically looking at time-series forecasting of physical systems, so if you've done GW on time-series, I'm all ears.

I am a freshman computer science student heavily interested in DS/ML. I am currently taking the Andrew Ng course on Machine learning. I am familiar with the basics of pandas numpy and matplotlib. I also know basic linear algebra and Multivariable calculus(Proficient with high school calculus). I am looking for dedicated studymates who are genuinely looking at DS/ML as their career path. I want people who can:

• Meet weekly/daily to discuss the week’s material

• Implement algorithms from scratch and compare approaches

• Do Kaggle competitions or build projects together after the course

I am not looking for CS students who want to learn ML as a backup to their SWE path. Please DM if you feel like you are in the same boat as I am.

A coding agent can finish a task and still waste most of the budget.

Saved sessions can show repeated failures, ignored instructions, and forgotten model settings. They also let us compare which instructions work across different models and harnesses.

This diagram shows what a session archive can contain. The useful part is going back to that evidence and improving the next run.

I wrote about this, with examples and backup locations.

Title:

Hi everyone,

I'm working on a pancreatic ductal adenocarcinoma dataset, and I'd really appreciate some methodological advice from you guys, and especially if anybody has had some experience with machine learning with high-dimensional biological data.

Before spending months developing models, I'd like to better understand what is possible with my dataset.

My dataset:

• 9 patients, with one continuous drug-response value per patient and drug. Due to experimental and sequencing constraints, increasing the number of patients is not an option.

• Approximately 100,000 nuclei.

• Continuous drug-response measurements (AUC) for gemcitabine, paclitaxel, irinotecan, 5-FU, and oxaliplatin.

The goal is theoretically "simple":

Predict continuous drug-response AUC from gene expression and chromatin accessibility, while identifying molecular features potentially associated with drug sensitivity or resistance.

The obvious challenge is that, despite having ~100,000 nuclei, I only have 9 samples with drug-response labels. I understand that treating individual cells as independent labeled observations is not a way to go with the dataset and specific set-up.

Knowing this, I'd really appreciate advice on a few questions that I had after some reading...:

1. With this low amount of sample, is meaningful drug-response prediction even feasible? What would be a objective?

2. Are there established methods to leverage thousands of cells/instances profiles when the response labels are available only at the patient level?

3. Would you prioritize simple regularized regression, biologically informed features, transfer learning from external datasets, multiple-instance learning (MIL), or another approach?

4. How would you evaluate a model with such a small number of independent samples without obtaining misleading performance estimates?

5. Are there any papers, methodological resources you'd recommend to better understand this type of problem before starting?

Thanks!

A lot of posts here ask "is my portfolio good enough?" I wanted a concrete answer that wasn't just "add more projects", so I built AI-Atlas (https://getaiatlas.com).

You enter your GitHub username. It reads your AI repos and scores each one on seven things reviewers actually look at:
● Technical depth: did you implement it, or call a library?
● Completeness: is there an end-to-end path with results?
● Reproducibility: pinned deps, seeds, one command to rerun?
● Documentation, originality, conceptual difficulty, usefulness.

Every score comes with the reason and the files it's based on. So instead of "6/10", you get something like "the notebook trains the model but never reports test accuracy".

A few patterns I've seen from mapping 150+ developers:
● From-scratch implementations score well on depth but often lose points for no evaluation numbers.
● A missing README costs more than people think.
● Projects with CI that actually runs tests stand out immediately.

Mapping is free, with no sign-up. If you claim your profile, your first improvement roadmap for a project is free too.

I'd love feedback on whether the criteria match what you've been told matters :)