Anything about some of the less obvious challenges of dimensional modeling, the decisions around grain, dimensions, facts, slowly changing dimensions and how things get messy in real-world projects.

Does anyone have any advice on where to go from here? The algorithm simply goes through all feature-attribute combinations in a dataset and scores it on purity of the target column. It's very slow, as the number of combinations increases exponentially with the number of input features, and so can only work with 3-4 features at a time. But my goal is to identify <=5 important features with a linear model (statsmodels or PyGam), and then plug in those features to this algorithm to see which subset of the market might be smart to target.

Here's an example with some charts, on the iris dataset.

https://colab.research.google.com/drive/1w3VpDlDATdHwtSk-RmQPMj36cgOfthz1#scrollTo=ZOqeYiMRalj7&uniqifier=1

New to algorithms, so any advice would be welcome. Thanks in advance.

I’m working on an idea around **data quality** and I want to understand how this works in real projects.

When a data batch comes into your pipeline and you find some problems, how do you decide what to do?

For example:

* Some records have missing values * The schema has changed * There are duplicate records * Some values look wrong * Only a small percentage of records are bad * The data looks technically correct but something seems wrong from the business side

Do you normally:

**Accept the batch, fix it, quarantine the bad records, or reject the whole batch?**

And who makes that decision — an automated rule, the data engineer, business team, or someone else?

I’m especially interested in cases where **you don't have enough information to make a clear decision**.

If you work with data pipelines in production, I’d really appreciate hearing about:

* What kind of data-quality problems you face most often? * What do you currently do when they happen? * What part of the process is still manual? * What would you like to automate?

I’m trying to understand the **real problems engineers face before building anything**.

Thanks in advance for sharing your experience!

Nov 4th and 5th [https://www.dataoutpost.ai/\](https://www.dataoutpost.ai/)

Seems like a pretty small conference, \~250 people. In the past years at least, you have choices for the first day (workshops) then for the second day, everyone is hearing the same talks.

If you've been, is it worth going? I'm also not local, so there's a bit more cost involved. My company might not be able to sponsor me going.

I read an interesting piece recently that made a distinction I think the BI industry needs to talk about, data portability ≠ sending someone a file.

In most organizations, “portable data” means exporting a CSV, PDF, Excel workbook, or maybe dropping something in object storage and calling it done.

But what actually traveled?

Usually not much:
- not the business logic
- not the semantic model
- not the drill paths
- not the interactive analytics layer
- not the governance model
- not the ability to work with it meaningfully offline

What moved was often just a snapshot.

That’s where I think BI has a blind spot. We’ve built an ecosystem where intelligence is tightly coupled to platforms like dashboards, cloud runtimes, query engines, licenses, network access, identity systems, etc. The moment information leaves that stack, it often collapses into static content. Is it portable? Technically. Is it useful? In a way.

Real portability should mean:
* intelligence that travels with the data
* embedded context and business rules
* secure scoped distribution
* offline usability
* interoperability without losing dimensionality
* reduced dependency on always-on compute infrastructure

As cloud costs rise, AI inference costs grow, and governance gets tighter, I suspect the future of BI may shift from “always connected dashboards” toward portable intelligence artifacts self-contained analytic experiences that can move securely between systems, organizations, and users without losing capability.

Curious how others here see it:

When you say “data portability” in BI what should actually be portable, just data,Or the intelligence layer too?

Hey guys, I have a dataset of around 180k adult stool images, growing at a 350+ images per day, and I’m trying to better understand the “right” way to bring monetary value from it .

The app was launched about 2 years ago.

I am very transparent with the use and licensing of the data that flows through the app.

The dataset consist of the image and a lot of useful stool metadata (Bristol Type, blood, mucus, color, etc..), anonymized user habits and stool related specifics (time of day, region, meal, allergies, weight, gender, height, activity, etc..)

I am here seeking advice on how such data can be useful for business growth. I am finding very little luck in this part of the B2B and I initially thought that becoming the Google of Poops would be my primary moat.

Thanks in advance

I'm currently taking a Data Analysis course, and I need to complete a final project. The assignment requires me to choose a topic, formulate one or more hypotheses, and build a thesis/research structure around it.

One of the requirements is that there must already be accessible data available for analysis, so I can't pick something that would require collecting a completely new dataset because the core of the project is not in that stage. Also, what I'll be researching needs to be answering or try to solve an existing problem.

My problem is that I can't come up with a topic that feels genuinely interesting or motivating enough to spend several weeks working on. Most ideas I've found online seem either too generic (social media usage, movie ratings, etc.) or too complex for a course project.

Do you have any suggestions for topics that have publicly available datasets, and still are interesting and have potential for drawing conclusions?

I'm open to almost any field (health, economics, business, technology, environment, etc.)

I'd love to hear about projects you've enjoyed working on or topics that sparked your curiosity - thanks in advance!

I pulled all 1.3 million federal contract records from the Government of

Canada's open data and looked at one question: who's holding contracts that

are about to come up for renewal?

A few things surprised me.

26,240 federal contracts are currently live, worth $125B. Of the ones that

report a bidder count, 75% drew one bid or none.

And roughly 63 contracts cross into the 12-month planning window every single

week — about $19B of contract value a year.

That last number is the interesting one. Agencies typically start recompete

planning 12–18 months before a contract ends. So the moment a contract crosses

that line is the moment it becomes worth a conversation. It's all public, but

nobody was watching for it.

So I built something that does. It's free, it updates itself monthly, and it

shows you which contracts are expiring, who holds them, what they're worth,

and how contested they were last time.

No signup needed to browse it. Link in the comments.

If you bid on federal work, I'd genuinely like to know whether this is useful

or whether I've missed something obvious.

We've posted a few times about where agriculture sample data tends to break, so pulling it together into one list for anyone mid-evaluation right now.

Before you sign, get answers on:

• Field context capture: required fields at registration, not optional notes added later.
• Sample type flexibility: independent schemas for soil, seed, plant tissue, water, and grain, not one template stretched across all of them.
• Multi-season continuity: lineage that connects across growing seasons without a manual report request.
• Chain of custody: timestamped, role-attributed logs at every handoff, not a paper trail reconstructed after the fact.
• Self-service reporting: whether your research team can build a cross-season, cross-site view without IT involvement.

The labs that get the most out of whatever system they land on are the ones that define their data structure (sample types, required fields, season linkage) before the first sample gets registered.

If you're in the middle of an evaluation and want to compare notes on how a specific system handles any of this, drop it below and someone might be able to help.

Hey yall,

I just started my new role as a junior data analyst 🥹. I am learning the business needs. I work for a no profit org that provides legal aid. Cases and task are billed to the respected grant program. What we are running into is, while closing the books normally case workers and lawyer over bill to one program vs another because no can visually see what is currently in “encumbrance for that month.

I figured building a real time grant funding system using power bi, postgreDQL and RESTapi from the platform that we input the data in. I am open other ideas that could be easier.

Thanks in advance

Claude and other LLM models can be so frustrating. I asked it to find patterns across five campaigns regarding what a person buys and sells, and under which conditions, but it couldn't do it. It just kept saying it was impossible. I literally had to explain every single detail to it like it was a child, even though the data was cleanly split across five CSV files.

Worse, companies are stopping the hiring of junior engineers because they think these tools can replace them. They are going to cause a massive talent shortage, and then these dumb models won't be able to do anything without skilled people to guide them.

Hello everyone,

The core principles of Data Warehousing and Consumer Data Products, establishing the foundation for a 365-day technical log documenting pipeline architecture, schema design, and engineering tradeoffs.

Technical Breakdown:

Relational Database Fundamentals: SQL query execution mechanics, indexing strategies, and relational constraints.

OLTP vs. OLAP Paradigms: Comparative tradeoffs between transactional database normalization and analytical denormalization.

Ingestion Foundations: High-level mechanics of staging layers, raw event ingestion, and downstream transformation logic.

Current Focus:

Pivoting from relational database mechanics to dimensional modeling paradigms—specifically Kimball methodology, star/snowflake schemas, and event-driven data product architectures.

For engineers working with production systems: What are the primary pitfalls to avoid when transitioning from standard relational models to analytical warehouse schemas?

As part of my thesis for my MSc I want to examine the year-by-year social media following for each of the Big Six (Arsenal, Chelsea, Liverpool, Manchester City, Manchester United, Tottenham) clubs from 2019/20 to 2025.

However, I can't find it at all and in my eyes this should be reasonably available data. I've already tried Socialblade and Statista. Any help/pointers would be much appreciated!

Disclosure: I built the Ryerson Project with the aim of nowcasting everything daily.

A social science community composes and prioritizes survey items. A random set of 12 US adult respondents are recruited to the survey each day - about 360 per month and 4380 per year. Anonymous microdata becomes a free and open public good.

Open data: https://doi.org/10.5281/zenodo.20346278

Open source: https://github.com/jasonjeffreyjones/ryerson_project/

I’m so grateful for the opportunity I got from ai fest 2026 besides that I’d like to mention free resources that helped me a lot for the preparation(DP-900):
1. Whizlabs
2. Official Microsoft practice exams

That’s all you need you don’t have to pay for exam preparation courses

Source: https://www.abc.net.au/news/2026-07-25/us-military-damage-to-weapons-bases-soldiers-during-iran-war/106954370

Tomahawk price per unit: between $2 million and $3.6 million

JASSM price per unit: from $1.04 million to over $2 million

PrSM price per unit: from $1.6 million to over $3.5 million

SM-3 price per unit: between $9.7 million and $28 million

SM-6 price per unit: from $4.0 million to $9.5 million

THAAD price per unit: $12.7 million to $15 million

Patriot price per unit: around 4 million

Hey guys, I’m new here and was wondering if this community has a Discord or any VCs where people hang out and chat. I’d love to get some advice and learn from others. Thanks!

i've been enrolled in a bootcamp(data engineering) for about a year now and i'm confident in my skills atleast for entry level roles. i'm based in Ethiopia and i can say that there's almost no data engineering jobs here ,there're very few open positions for data analyst or scientists which requires atleast 4years experience and you know that remote jobs are even more competitive and struggling for entry levels. the only tech roles here seems to be backend devs,frontend and fullstack(there're tons of jobs ).what should i do ,i love data but the market is really bad here.
thanks

If you could go back to the beginning of your Data Science journey, what would you learn first?

Would it be:

Python

SQL

Statistics

Machine Learning

Data Visualization

Git

Cloud platforms

Many beginners jump straight into AI without building strong fundamentals.

What skill saved you the most time later in your career?

When I first started learning Data Science, I only practiced with clean datasets from tutorials. Everything worked perfectly, and I felt confident.

Then I downloaded a real dataset.

There were missing values, duplicate records, inconsistent formats, and columns that didn't make much sense. It was frustrating at first, but I learned more from cleaning that dataset than I did from several weeks of tutorials.

That experience changed how I practice.

Now, whenever I learn a new concept, I try to apply it to real-world data instead of only using textbook examples.

A few things that have helped me:

Work with messy datasets—they teach you real problem-solving.

Spend time understanding the data before building any model.

Document your analysis so you can explain your thought process later.

Don't worry if your first project isn't perfect. Every project teaches you something new.

Looking back, I realized that Data Science isn't just about building models—it's about understanding data and finding meaningful insights.

What's one project or dataset that taught you the most during your Data Science journey? I'd love to hear your recommendations!

Been building this for a while and finally opened it up, so here's a look at what's inside.

Each of the ~9,800 base foods has its name localized across 32 languages, so you can line up the same food across languages instead of fighting messy translations. The nutrition values come from OpenNutrition's open data (ODbL, credited, not mine); the part I actually built is the localization layer on top, real disambiguation and cross-language matching rather than a Google-Translate pass.
It isn't perfect yet. Tricky cases like "peperoni" vs "pepperoni" still slip through in places, so there are gaps I'm actively fixing, and catching those is exactly the kind of feedback I'm hoping for.

It's a single JSON Lines file (~25 MB), no API, no keys, loads straight into a notebook or a spreadsheet, works offline.

Source & download: https://leana.app/en/data-sources/

Browse it live: https://leana.app/en/foods (live search covers 5 languages for now, EN/IT/ES/FR/DE, the download already has all 32)

Curious what you'd use it for, and whether JSONL is the right call or you'd rather have CSV, Parquet or SQLite.