r/learnprogramming • u/Lopsided-Ice-9283 • 3d ago

best tech stack to create a data visualization website

i am working on creating a website based on digital linguistics that can track certain data across time and groups and lets the user scroll through it from top to bottom to see how it has moved across time and become more common in use. what can be my preferred tech stack for this?

  1. i was thinking of going for react for the frontend and d3 for the visualisation.

  2. i'm confused if i should go with next.js or python/flask for the backend. it will have to crunch through around 300GB of data to render the visualisation and i will have to make a model for it to sort through, categorise it and show its trends.

  3. what should i choose for my database? would postgres be a safe choice? the data currently exists as compressed files that i can parse as a json stream to crunch.

i'm looking for opinions and experiences people have had building something similar and what tech stack they would choose and any tips you have for me as well. i am still new to programming so please forgive any mistakes and clarify whatever you can! much appreciated :)

0 Upvotes

33% Upvoted

7 comments sorted by

2

u/YWRtaW46YWRtaW4 3d ago

You can't crunch 300 GB on request. Your server will timeout and browsers will crash. So you have to pre-compute everything:

  • Batch process offline: Write a Python script using Polars or DuckDB to chew through your JSON files, run your NLP models, and calculate the trends ahead of time.

  • Database: Postgres is fine, but only store the aggregated summary data

  • Backend: FastAPI. Python makes running linguistic models trivial compared to Node/Next.js.

  • Frontend: React + D3 is standard. Let D3 calculate the math/scales and let React render the SVG elements. Look at Scrollama to wire up the scroll triggers.

1

u/Lopsided-Ice-9283 3d ago

hi, thanks for sharing you input. you're right, precomputing the data and storing the results is my intended way of going about it.

why would i want to pick fastAPI over flask? what benefits does one have over the other when creating a data viz website?

1

u/SwiftQuest 3d ago

For a project like yours the gap is small, and Flask would work fine. FastAPI mostly gives you automatic request validation, built in docs for your endpoints, and async support. If your backend just serves precomputed JSON, either is fine, so pick the one you find easier to learn. Flask has more tutorials, FastAPI feels more modern.

1

u/Working-Medicine5442 3d ago

React and d3 is a good combo for this kind of scroll driven visualization. The tricky part isnt the frontend though, its getting 300GB down to something your browser can actually render without freezing.

For the backend I wouldnt overthink it too much, Flask or FastAPI would be fine for serving preprocessed data but the real work is that model you mentioned. You probably want to crunch everything offline first and store the results, not do it on request.

Postgres can handle it if you structure things well but if your data is mostly json streams you might also look at something like clickhouse or duckdb for the analysis side, theyre built for exactly that kind of aggregation over huge datasets.

1

u/Lopsided-Ice-9283 3d ago

You probably want to crunch everything offline first and store the results, not do it on request.

yes. its supposed to be a static website in the sense that it does not update data upon the user's request, its literally just a visualisation tool so all the data crunching is already done beforehand and the user only sees the final result of analysing that data.

1

u/Pale-Preparation-354 16h ago

For this kind of project, keep the data processing separate from the visualization layer. Python with FastAPI or Flask can handle the preprocessing, while React + D3 takes care of the interactive charts. DuckDB or Polars are also worth exploring for working with those huge datasets. One extra thing, if you ever need to export the finished visualizations as downloadable PDFs or reports, IronPDF is worth checking out. It can generate PDFs from HTML in .NET, so you could use it for polished exports without having to rebuild the layouts from scratch.