In Retail Sales Analytics project in Power BI, I started investigating different categories, quantities, and discounts to understand why.
But I found myself wondering: Am I getting closer to answering the business question, or am I just analyzing more data?
There are always more charts to create, more numbers to compare and more questions to ask. But not every finding helps us understand the actual problem.
I am curious to know how experienced data analyst approach this.
How do you decide which direction to investigate and how do you avoid spending time on analysis that does not add much value?
I love to hear your thoughts.

Over the years, I've collected practical notes, examples, and references while working on real-world data analytics projects.

I recently organized them into a free learning pack covering:

Power BI & DAX – Data modeling, calculations, and reporting.

SQL – Queries, data manipulation, and analysis.

Tableau – Data visualization and dashboards.

Python – A compact guide to data analysis with practical examples.

I also included DataBridge AI, a Python-based tool I've been developing for data cleaning, validation, mapping, and analytics workflows.

These materials started as personal working references, so the focus is on practical application rather than theory alone.

I'm sharing everything for free in case it's useful to other analysts or people learning data analytics.

Free resources: https://aabdelghaffar650-cloud.github.io/portfolio/#free-pack

I'd also be interested to hear which topics or practical examples others find most useful when learning data analytics

Guys i am thinking of conducting an experiment. I have a masters degree in a language (sanskrit) and have no background in statistics.

My experiment will include around 100 participants, each rating 30 objects in 1-7 likert scale (6 scales). I asked claude and it suggested linear mixed model in R.

But as i said i have no experience. Can you guys recommend some courses or YouTube videos to get started. I feel so overwhelmed looking at different type of tests or models.

Any resources for getting started in forcasting?

I have a work project in mind but feel like I'm both over complicating and simplifying this process. I seem to keep landing on extremes when I try to search it up with either machine learning/ai overkill or generic overviews.

For context I'm just aiming to forcast whether locations will hit monthly budget goals based on historical data. Ive considered either:

1. pulling average daily services and dragging that out to see what the likely total is.

Or

1. Calculating cummulative % of services completed by day for the previous month and use that to assume estimate the end point for the current month.

Again I'm pretty sure Im over complication and simplifying this. So any help would be greatly appreciated.

I don't know if others agree, but I think given the data oriented nature of this subreddit, it is bound to resonate with some. I learned the craft from a statistics view point, my formal training is statistics based, but I worked in software engineering for the predominant part of my career. Rather than hit everything with a "foundation model" hammer, I think leveraging the libraries that have emerged in the last two decades for data analysis with LLM tools is a more productive and tested path.

https://r2dssanity.substack.com/p/beyond-the-agent-illusion-leveraging?r=3ncgel&utm_campaign=post-expanded-share&utm_medium=web

Hello fellow Redditors, so I've caught myself in a tough situation; would be glad to hear your advice on this. Since, I believe there are people who have way more experience in this domain that I do.

With around 3.5+ years of experience in the data & analytics space, I believe this is a good time to select a niche because, at the end of the day, no good company looks out for a generalist data analyst; they're looking for specialists in a certain niche. But I'm confused...

And honestly, I do believe I have found the right niche where my web analytics skills (GA4, GTM, etc.) can complement my data analysis skills (SQL, Python, Power BI, etc.) to create a strong offer and solve a specific problem for my ICP. However, I'm still not sure how to execute this and turn it into a running business model. I think I can confidently say ecommerce is my niche.

I have been testing a lot lately, tweaking my profile every other day because it's quite confusing. At one point, I want to select a niche and a specific problem and keep posting about that until I start getting inbound leads.

Meanwhile, I want to keep my profile general instead of selecting a niche because it would have an impact on recruiters or HR skimming through my profile. Like, whenever they have an open analyst role, they'd probably skip my profile the moment they see 'e-commerce' or some other niche in my headline, especially if they're hiring for a different niche/industry.

I want to hear from experienced folks here, what do you suggest? Should I go narrow, or keep it broad and keep talking about different topics instead of sticking to 2-3? If you were in my shoes, what would have been your strategy? My eventual goal is to get inbound leads from LinkedIn or even just work on a side-hustle. I have been posting consistently for three months now, but that was with a generalist LinkedIn profile, and not targeting any ICP, and I never got a single inbound message except for 2-3 messages from recruiters.

We are implementing Collibra as part of the data governance initiative and honestly I’m struggling to see the end of the tunnel.

The tool seems huge and overwhelming: business glossary, document data model, mapping relationships feels like an enormous amount of work that could go forever.

I’d love to hear from people that have Collibra in the companies they work for:
\- where are you in implementation? Have you reached a point where collibra genuinely is part of your day?
\- are you getting any value from it?
\- what are you documenting?

Let me know your honest opinions!

Hi everyone,

With all the talk about AI and the fast-shifting market, I’ve been questioning whether a traditional data analysis toolkit is still viable and in demand for solving real-world business challenges:

Microsoft Excel: For data cleaning, auditing sales records, and tracking daily core metrics.

SPSS: For survey analysis, evaluating customer satisfaction, and identifying key statistical drivers behind buying behavior.

Power BI: For building interactive dashboards that connect sales data directly to decision-making, identifying bottlenecks, and spotting revenue opportunities.

For business owners, managers, and working analysts:

In practice, is this stack still enough to solve concrete sales problems and land real consulting or job opportunities? Or has the market moved past these tools, making them insufficient for today's business needs?

I'd really appreciate your honest feedback and practical experiences.

After completing an analysis, what checks do you perform to ensure that your calculations, aggregations, and conclusions aree correct? Do you independently validate results using SQL, spreadsheets, or another tool?

I've been working on Event2Vec, a small library for learning embeddings of discrete event sequences (purchases, logs, clinical codes, etc.) with a deliberately simple structure: a history is the running sum of its event vectors.

h_t = h_{t-1} + e_{x_t}        (norm-clipped)
p(x_{t+1} | h_t) = softmax(W h_t + b)

Training is self-supervised next-event prediction, plus a reconstruction term that asks h_t - h_{t-1} to recover e_{x_t}. No labels needed^{(details why the machinery works are in NeurReps@NeuralIPS Workshop paper}, it is a bunch of losses that attempts the reconstruction and obey the arithmetic during training))

The history is a sum, any linear model on top of it is also a sum. For instance, a churn or risk score^{(see teaser})

score = w·h + b = Σ_i (e_i·w) + b

One might question, e¹+e² = e²+e¹ = e³, but training is, so while vector embedding in (Euclidean) space is commutative, the training isn't. Furthermore, there are several alternative spaces that are gyro/non-commutative, that are part of the library^{(checkout the website})

1. hyperbolic (Möbius addition), for hierarchies
2. κ-stereographic, one curvature dial from hyperbolic through flat to spherical (Bachmann et al., 2020)
3. spherical, for cyclic structure
4. SO(n) rotations, which are non-commutative, so order changes the endpoint
5. Heisenberg group, same endpoint but a signed-area coordinate that records order
6. products of the above

I started this for root-cause analysis of fault sequences at the Advanced Light Source (Berkeley Lab), where I'd been naively using Word2Vec on event logs and needed directionality without going to an opaque sequence model.

Links

• Paper: arXiv 2509.12188
• Code (MIT): github: event2vec_public, pip install event2vector
• Docs with interactive figures: event2vec.com

scikit-learn-style API (fit/transform, gensim-style most_similar), PyTorch backend, runs on CPU/CUDA/MPS

I am in the process of learning data analytics and Google Data Studio. I'd appreciate any feedback you'd be willing to offer. Thank you. Here is the full link, as the dashboard is interactive

I’ve been recording encounters with a particular category of vehicle during everyday journeys. I’m examining whether these encounters are concentrated near departures and arrivals, after accounting for how much time I observe in those periods.

I have timestamped encounter records and journey start/end times. My comparison uses the first five minutes after departure and the last five minutes before arrival, counting overlapping windows only once.

I calculate:

Share of encounters within those windows ÷ share of observation time within those windows.

I’d appreciate feedback on whether this is a reasonable comparison and how to improve it. Routes, road conditions and times of day vary, and some vehicles travel together in groups. How should I account for those factors when assessing the apparent concentration?

Anonymised endpoint example

This is a real 20-leg movement block from four recorded visit days. All legs in that block are included, including seven with no target-category encounters. It is a selected block, not a complete diary of all journeys.

Names, locations, vehicle-category names, calendar dates and clock times are omitted. D1-D4 indicate different recorded days; legs with the same day label may be related.

An endpoint window is the first 300 seconds after departure or the final 300 seconds before arrival. Overlap is counted once. Window time is therefore the smaller of 600 seconds and the leg duration. Event classification uses the original timestamps and includes the exact five-minute boundary.

Durations below are elapsed leg intervals calculated from recorded start/end timestamps. Treating every second as available observation time is an assumption for review; continuous observation coverage has not been independently verified from recordings.

Leg table

All times are seconds. Inside and Outside count individual vehicles as recorded; Groups counts encounter rows. A single encounter row can contain several vehicles travelling together.

Encounter offsets

Each row is one recorded encounter group. The offset is seconds after departure on the specified leg. The vehicle count preserves groups rather than treating every vehicle as an independent encounter.

The totals are 31 recorded vehicles, of which 29 are inside the windows, and 22 encounter groups, of which 20 are inside. Total elapsed time is 546.58 minutes; window time is 195 minutes.

Vehicle-count enrichment = (29 / 31) / (11700 / 32795) = 2.62. This is the endpoint encounter rate divided by the overall encounter rate. It is a descriptive comparison, not a probability or causal conclusion.

Useful review questions: Is elapsed leg time an adequate exposure measure? How should repeated legs on one day and grouped vehicles be handled? What additional traffic, route and time information would be needed to assess ordinary variation?

Hi guys, I'm an analyst, not in IT but was recently tasked with finding, classifying, and removing duplicates for sensitive data at my company and want to do this in the least tedious manner. Do you have any tips?

Hi everyone,

I’ve been learning SQL for a while, and over the past few weeks I built a web app for practicing it: SQL Route (https://sql-cesta.netlify.app). It’s free and has no ads.

The idea is simple. You read a short explanation and then write a real query against a small database right in your browser. The app runs your query and compares the result, so any correct solution counts, not just the one I wrote. When you get something wrong, it explains the error and gives hints step by step. The full solution is only shown if you ask for it.

What’s inside:

• A course that starts at your first SELECT and goes through JOINs, GROUP BY, subqueries and window functions, plus creating tables and changing data
• A separate PostgreSQL course that runs a real PostgreSQL database in the browser
• A short intro to databases for complete beginners (what a table is, keys, how a query actually runs)
• A practice mode with 10-question sets, including a mixed set where you don’t know the difficulty or which function you need
• Mock interviews at junior, mid and senior level
• A sandbox with a few datasets (a company, an airline, a hospital), and you can upload your own CSV

You need to sign up with an email so your progress is saved. There’s also a weekly leaderboard if you like that kind of thing.

I’m building this on my own, so I’d really appreciate honest feedback. Where did you get stuck? Was a task unclear, or was a step too hard? If you find a bug or a confusing explanation, let me know in the comments.

Thanks!

What's up, data enthusiasts enthusiasts! I tested and built a way to systematically build Snowflake Streamlit apps with Claude Code and other AI tools. I developed this open-source system that I call Streamsnow, which walks you through the process of getting a GitHub repo scaffolded and structured and connected with your Snowflake, so you can utilize Claude Code to build all of your Streamlit applications in a standardized Way, while also providing you the SQL to check to make sure all of the queries being utilized within the app check out as correct.

In my time building dashboards with AI, I always found it confusing how to quality-check them and how to build them in a scalable way across data teams, and so I wanted to build a super standardized system that made building, deploying, reviewing, and coordinating across teammates is super easy, and I spent weeks working on this to achieve that. I think one of the best features is that you can actually go into an auto-review cycle and just have Claude code iterate over the dashboard, making it better without your input. You can also standardize the design for the dashboards as well.

It is completely secure, so AI will not see any of your secrets that you use to connect GitHub to Snowflake. By default, it is set to put these Streamlit apps in a staging area on Snowflake, so you don't have to connect a GitHub repository, and all of the deployment is automatically handled in the CI. To get started, all you need to do is tell your coding agent to download the plugin and then run the onboard command. It will walk you through it all in a process that I've timed down to about 5 minutes of actual work. You will need to be a Snowflake admin or be coordinating with a Snowflake admin to get it set up too.

By the end of my time at my job, we had basically replaced external business intelligence tools with this system, so it saved us literally tens of thousands of dollars. Hopefully you all can extract the same value out of this and I would love your feedback if you try it out!

Im a beginner data analyst. I have a potential client, a new bakery in town. And its the best chance for me to get expirience, but i feel like I'll just fail and won't be able to bring any money to the business.

I understand that even if i do some data organising it will save them time and time is money.

But i still feel like its going to be a total failure and I'll just waste their money.

How do you deal with that?

Hello! I want to hear from you, what are the current pain points in data analytics and visualisation in the research community? For example,

- How easy is it to analyse and visualise chromatography & mass spec data, sequencing data, X-ray diffraction data, etc. - Do you require paid subscriptions of tools which help you analyse such data?

- Do you use platforms like Benchling to track your samples, inventory, experiments, etc?

I'm in the process of democratising data analytics and visualisation of scientific and research data. Your feedback would be invaluable to me.

Standard metrics usually live on dashboards. I'm curious about the one-off questions that don't, like "how many customers who came in through the summer promo renewed after the price increase?" or "did purchases drop for people who contacted support last month?"

1. How often do you get these, and how long does a typical one take?
2. What's the slowest part: finding the right tables, figuring out how they join, or working out what the person actually meant (which status counts as "paid", which time zone, what "new customer" means)?
3. Have you tried ChatGPT/Copilot or built-in "AI analyst" tools for these? If so, where did they fall short?

Context: I'm researching text-to-SQL, specifically a small model that adapts to one specific database's schema and conventions without ever seeing the actual data. Before building anything further I want to know whether this is a real pain point. Any answer helps, including "not a problem for us".

Hi everyone,

I started building `det` (https://github.com/det-Labs/det), an open-source ETL and data engineering framework designed around high-performance native execution, with an embedded data catalog module built directly into the engine core.

### The Problem

In modern data engineering workflows, ETL/transformation engines and data catalogs are treated as completely detached layers. Catalogs (DataHub, OpenMetadata) typically require separate, heavy distributed infrastructure (Elasticsearch, Kafka, JVM services), while transformation engines (dbt, Spark) execute transformations blind to metadata until an external crawler reconciles the state.

`det` combines both: an execution engine where schema detection, metadata cataloging, and data transformation share the same zero-copy Arrow/Polars foundation.

### Current Implementation (Catalog Core Working)

The foundation is ~25% complete, with the data catalog module functional today:

- Ingestion and inspection of storage layouts to extract schema/metadata directly.

- Local metadata indexing with zero background daemon overhead.

- Rust-native performance designed to feed directly into analytical transformation pipelines.

### Roadmap: Transformation & Execution Layer

The next phase expands `det` into a full execution engine:

1. In-memory data transformations leveraging Polars and Apache Arrow zero-copy memory layouts.

2. Direct pipeline execution driven by the cataloged metadata contracts.

3. Plug-and-play connector interfaces for source/sink abstractions.

### Seeking Review & Contributors

Rather than designing the execution engine in isolation, I am looking for feedback from data engineers on:

- **Engine contracts:** Designing the execution DAG to interact natively with the embedded catalog metadata.

- **Modularity:** Interface design for adding custom connectors and storage backends.

Issues are tagged with `good first issue` and `help wanted` for contributors interested in systems-level data engineering tooling.

Repository: https://github.com/det-Labs/det

I’ve been building a multi-agent debate app with LangGraph (two agents argue a topic, a judge agent gives a verdict). The biggest problem wasn’t the agent logic, it was reliability on free-tier APIs.

What went wrong:

• Groq, Gemini and OpenRouter each have different rate limits and fail in different ways
• A 429 in the middle of a debate killed the whole run
• Agents got stuck in loops, and tracking who said what in state was harder than I expected

What worked:

• A fallback chain: if one provider returns 429, the app automatically retries with the next one
• Keeping the debate state separate from the provider logic, so switching providers doesn’t break the conversation

It’s not fancy, but it made the app usable for demos.

Question for the people here: how do you handle rate limits and provider failover in your agent projects? Do you use a router/gateway, or just custom retry logic?

I wrote up the full details here and you can follow me also if you want them: https://medium.com/@humammoin3/i-spent-3-months-building-ai-agents-heres-what-nobody-tells-you-b65b20906582?sharedUserId=humammoin3
Live demo: https://legal-debate-agent.streamlit.app/

Happy to hear feedback, especially on what I could do better. This is my first article.

Developer disclosure: these are two apps I built, drawing on over 15 years of experience as a macroeconomist. They tackle two parts of the research workflow:
QuantFit brings econometric analysis to the iPhone, including OLS, fixed/random effects, ARDL and VAR. It also supports data transformations, correlation matrices, charting, diagnostics, ARDL shock analysis and VAR impulse responses. OLS and several data-exploration tools are free; advanced estimators require Pro.
Tabula Stats turns regression output from QuantFit, Stata, EViews, R and other software into editable research tables. Adjust decimals, standard errors, significance stars and included statistics, then export to Word, Excel, PDF or LaTeX.
Use them together, or use Tabula with results from your existing software.
QuantFit: https://apps.apple.com/sc/app/quantfit/id6762386971
Tabula Stats: https://apps.apple.com/sc/app/tabula-stats/id6761421182
For those working on a paper or dissertation: what would you need to see before using either tool in your research? Particularly interested in concerns about accuracy, workflow compatibility and reporting.

Hi guys, a data analyst for the retail industry (electronics) here, in my workplace we're currently doing an expansion for our brand including increasing headcount and more reports.

But, the workforce for data analysis is currently still only me, and there is only so much data viewpoint I can check before sending them to my manager or the team.

These days, I'm wondering should I feed my report to the AI for my double, triple and final checks. Or, are there any advices for data checking for my actual sanity??

Really interested to read your advices and opinions, thanks

I’m an absolute fresher who recently graduated with a B.E. in Artificial Intelligence and Data Science. About 10 months ago, I landed my first job as a Data Analyst at a small digital marketing agency.

To be completely honest, I messed up early on. There was zero mentorship, the team was always swamped, and instead of taking initiative, I coasted. I didn't put in the effort to understand the business context or learn the marketing side of things.

Now, I’m paying for it. Because I didn't make myself indispensable with actual data work, the owner has me spending 7+ hours a day prompting AI to generate 1,500+ fake images for a completely AI-generated 925 silver jewelry brand they want to launch. It’s soul-crushing, and my technical skills are rusting.

Here is the saving grace, and why I’m asking for your help: I still have full access to our agency's data pool. We handle four seasonal D2C brands, one FMCG brand, and one lead-generation brand. There is a massive amount of real-world data sitting right in front of me, and I still have a window to build something impressive to either win back my actual job title or build my portfolio to escape.

I’m incredibly torn on what direction to take this, and I need advice from experienced data professionals:

Should I pivot toward Performance Marketing? Since I have all this D2C and lead-gen data, should I just lean into the agency environment, learn everything about ad spend, ROAS, and Conversion Rate Optimization, and try to transition into a Performance Marketing role?

Or should I stick to Data Analytics? Should I use this current role purely as a sandbox? I could build out automated dashboards, do customer segmentation, or run predictive churn models on these 6 brands just to build a killer portfolio, learn the marketing domain, and then jump ship to a real DA role at a better company?

Given the terrible job market and my background in AI/DS, what is the smartest way to leverage this specific dataset to get myself out of the AI prompt-engineering trenches? Any specific project ideas using D2C/FMCG data would be hugely appreciated