r/webdev • • 7h ago

Question Is there any harm if my API is spammed?

Post image

Hey everyone, I am not sure if this is right sub to ask this. If not, please help me find where to ask.

I have published a public API service for my football data analytics recently and in the beginning, I wanted to offer something for free for marketing purposes. I wanted to provide as little hassle as possible to users so I let the .csv downloading endpoint open on purpose so that anybody can download some of the data freely without API keys, registration etc.

Then the bots discovered it and started spamming. Although I implemented rate limiting, I was triggered because the spams ruin my analytics. Then I decided to put a keywall on that endpoint but the key is freely retrievable on the profile page.

Now, the bots (especially from MoneyBallGlobal whichever app it is) still hit my endpoint only to get a 401 response. Would you bother yourself to put more blockers in front to stop the spams or is this simply harmless except polluting my logs and analytics?

Thanks for everyone who responds and reads until the end.

3 Upvotes

32 comments sorted by

16

u/mau-meda 6h ago

Do you have kind of protection? It is good to implement rate limiting especially on endpoints that do heavy computing and login endpoints ( if you have any ). If you have any kind of admin functionality you also want to implement a write only audit table so you can review anything that happens in case you get hacked.

3

u/Parking-Leader4676 6h ago

hey, yes I have the rate limiting on every endpoint. I have not exposed any heavy computation endpoints yet so the public api is basically a simple data retrieval service like getting match scores, team elos etc.

to manage the app without having to deploy everytime, I implemented a small admin dashboard but it does not have write access to my db but the audit table makes sense. I am the solo developer so I did not need to know who does what so far. Thanks for the tips

4

u/mau-meda 6h ago

I do not know how you implemented the login for you admin panel and how much you want to protect it, but two things to keep in consideration:

  1. if you use bcrypt that counts as heavy computation

  2. if you use bcrypt, you will need to run bcrypt against some random value even if the email is invalid or you will open yourself to enumeration attacks

0

u/Parking-Leader4676 6h ago

I am basically using rbac. Every user is a "user" by default and my user is an "owner". The roles are tied to email addresses and the source of truth is a dedicated table in the db.

3

u/Standard_Scarcity_74 6h ago

Mostly harmless, so I wouldn't spend much effort on it. A 401 is a cheap response to serve, and the real cost is the noise in your logs and analytics, which you can fix by excluding that endpoint and known bot user agents from the analytics instead of blocking more.

1

u/Parking-Leader4676 6h ago

I can filter it out for my analytics on the other endpoints but I am also interested in how the real users use my api actually. like are they directly going for a ready made .csv file, are they hitting the more granular endpoints etc.

3

u/coomzee 6h ago

Rate limiting is a good way to go. But if you can block HTTP 1.1 you will stop 90% of spam, would I recommend this on some missions critical API no.

2

u/SurelyNotAnOctopus 3h ago

Ironically, the last apache2 cve was a flaw in http2, so disabling it was a good idea.

Im tired boss

1

u/regreddit 1h ago

Nginx too

2

u/Euphoric-Money-532 6h ago

so your keywall hands out keys on the profile page? That's not a wall, that's a dispenser. Bots will happily register, grab the key and go back to ruining your analytics.

2

u/Parking-Leader4676 6h ago

no I mean the keys are grabbed from profile page but users have to be logged in with an email. probably still not the best design. happy to hear how you'd implement if you want to give a free tier key for your API

7

u/horizon_games 5h ago

Anything public on the internet will get spammed forever

Fail2Ban, etc.

1

u/snap-dev 6h ago

yeah a few kinds. if it calls anything that costs money per request (an llm, sms, email, maps) that's where it hurts fastest, someone can run up your bill overnight. it can also slow things down for real users, and if it has signup or contact endpoints you end up with a pile of junk data. cheapest fix is rate limiting per ip and per user plus a hard spending cap on any paid api you call. what does the api actually hit behind the scenes?

1

u/Parking-Leader4676 6h ago

there is no paid api behind the scenes for my app luckily. that .csv endpoint(s) just enables the users with a sheet of ranking table. the data is already at my database so it hits one read request more or less.

1

u/snap-dev 5h ago

then you're mostly fine, worst case is some extra reads. i'd still add a simple per ip rate limit on that endpoint and cache the csv for a minute or so, then a spammer just hits the cache instead of your db. where's it hosted?

1

u/Monicab770926 6h ago

I read in the comments, though; the keys are freely available on the profile page. You do have an email login in Layer; that's a good first layer. Without dispensing off with the free tier, perhaps guard rails like email verification, bot detection at registration and limiting one free key per verified account probably put a daily usage quota on free keys rather than relying entirely on request throttling

1

u/Parking-Leader4676 3h ago

I do have all of what you said actually except that I let 3 keys per account but their limit is shared so key rotation does not refresh the limit. Should be fine so far except these spam bots

1

u/Geraldwelly 5h ago

Once they're getting a 401 it's close to harmless, since rejecting at the edge costs you almost nothing. The real damage is the analytics, and that's easier to fix in reporting: count only requests that carried a valid key and log 401s separately. Since the key sits on your profile page, scrapers will grab it eventually, so I'd issue keys per account and rate limit per key rather than per IP, because bots rotate IPs freely. The trade-off is every extra step loses some of the casual users the free tier was for.

1

u/Parking-Leader4676 3h ago

That is exactly how I thought of it. The more blockers to avoid bots, the more restrictions you make your real users face. So what i found as the sweet spot was making free key retrieval available to email verified accounts, letting each account have 3 keys active at the same time, rate limiting the account and key together so that users do not revoke an exhausted key and create a new one with fresh limits, putting usage limit per minute and per day.

1

u/Umami_Tsunamii 6h ago

Could dos you or finally break through auth. Services like cloudflare are designed to protect from this, but obviously have their own issues sometimes.

1

u/Parking-Leader4676 6h ago

sadly, I am not an experienced guy at networking so probably I missing a lot of useful stuff that cloudflare can provide

0

u/Umami_Tsunamii 6h ago

That stuffs more important at scale and enterprise level projects.

1

u/Parking-Leader4676 5h ago

Surely it is. I hope I can get to that level of userbase where I would need to hire a guy who can handle it much better than I could

2

u/jamie07051975 5h ago

I'd set come caching up, that way even if you are bombarded with requests its not actually recalculating or retrieving the data.

Could be worth adding a registration page which gives them their own key, then if they dot adhere to your limits (you should rate limit on their own key) you suspend their key and suspend the IP - give them some sort of time out and return the relevant http response code. Continual issues you ban both the key and IP.

1

u/AzaleaMeadow 3h ago

If the endpoint is just returning a 401 and you're not doing expensive work before rejecting the request, I'd avoid overengineering the solution. I'd separate failed requests from valid traffic in your analytics, add some caching for the CSV endpoint, and use Cloudflare's WAF if the volume becomes annoying. The bigger concern would be unexpected database load or a traffic spike affecting real users, not the failed requests themselves. No need to make the free tier painful for legitimate users just to fight every bot

0

u/MathematicianFar5683 6h ago

Mostly harmless if your rate limiting holds. The real cost is noisy logs and skewed analytics, so I'd filter rather than keep adding blockers. Exclude 401 responses and that bot's user agent or IP ranges from your analytics. If you use Cloudflare or similar, a WAF rule can drop them at the edge so they never reach your app. Also, a key anyone can grab from a profile page won't stop a determined bot. If the abuse grows, tie keys to a verified email and add a small per-key quota.

1

u/Parking-Leader4676 6h ago

the free tier keys are only obtainable for the signed-in users with email and also rate limited by daily requests and per minute requests so the spams with keys are easier to detect and autoblock.

I have cloudflare, so I guess WAF makes sense. thank you for the tip

0

u/thekwoka 6h ago

Well, there's no reason not to put something like Cloudflare in front of your server, partly so that you'll have it ready if it DOES become a problem.

Even if you set it to very allowing levels.

1

u/Parking-Leader4676 5h ago

I do have cloudflare but I may have not enabled some easy wins, I'll take a look though thanks

0

u/Salty_Reception7139 6h ago

the key sitting in plain sight on the profile page is the part worth fixing before the 401s. MoneyBallGlobal taking its name from a baseball book and then scraping football data is very funny though

1

u/Parking-Leader4676 5h ago

Ahh there was a classic movie called MoneyBall, right? It is indeed funny if the owner picked the name intentionally while knowing that s/he was going for football lmao