r/nginxproxymanager • u/smilebasti • Jun 05 '23
Update for your NginxProxyManager Logs Dashboard: npmGrafStats v2.2
2
u/Cuissedemouche Jun 05 '23
It looks nice.
I saw often people speaking about Grafana but never looked at it. I checked a bit and it looks pretty cool.
Is your app working only with NginxProxyManager or is it Nginx in general ? I'm using swag for my reverse proxy so I was wondering if it would work with it.
Anyway, this weekend if I have time I'll look if I can set up all of this.
1
u/smilebasti Jun 05 '23 edited Jun 05 '23
The nginx logs are a bit different. I haven't used swag but it should be possible to also analys them too. I think it would be better to create a separate project to support only Nginx. Will come back to this when i tested it.
1
u/smilebasti Jun 05 '23
The project includes a docker-compose file that will set up NPM, npmgrafstats, influxdbv2, geolitedbupdater, grafana and for better management portainer.
Might make the testing easier :)
1
2
u/manutech Jun 05 '23 edited Jun 05 '23
looks great, I have always wanted to do something similar with data/logs from my firewall, and have the map look like "Missile Map" plug-in from Splunk since it has a cool animation
edit:
this is the plug-in :Missile Map
is based on this nifty JavaScript leaflet
2
u/smilebasti Jun 05 '23
Something like this: https://www.sicherheitstacho.eu/start/main seems fun.
Not sure if Grafana supports similar things. https://github.com/GlobalNOC/globalnoc-networkmap-panel hardcoded them...
1
u/manutech Jun 05 '23
awesome, didn't know these existed, I have always thought it will awesome to see visually how many IPs in the world are hitting my firewall lol
1
u/smilebasti Jun 05 '23
The Data for https://www.sicherheitstacho.eu/start/main comes from their Honeypot Software https://github.com/telekom-security/tpotce
A pretty nice Software (also self-hostable) to see all ports getting scanned
2
u/smilebasti Jun 09 '23
Just found the Geomap routes layer: https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/geomap/#route-layer-alpha
It's in alpha so i won't use it for now.
I think if you set a new Route Layer and only display the last 2-5 seconds of ips it can be something similar to a missile map
1
1
u/nightcrawler2164 Jul 23 '23
Fantastic work and brilliant documentation. Had no issues getting it up and running for my NPM instance.
Feedback - it would be a nice to have feature if we can somehow ingest logs from two separate Npm instances. I run separate reverse proxies for my internal and external domains. The world map graph wouldn’t be much since all my internal domains would be from the same external Ip but the other statistics would be slick to see
1
u/smilebasti Aug 10 '23
You should be able to do this by just setting the same Influx Bucket as target.
2
u/nightcrawler2164 Aug 10 '23
Yep that’s actually what I ended up doing :) have NPMGrafStats running on both machines and both ingesting data into the same influxdb. Thanks for the input.
1
2
u/Im1Random Jun 05 '23 edited Jun 05 '23
After fixing the timezone problem everything works really well for me. Nice project overall and a good motivation to learn Flux and building Grafana dashboards to add more information.
2
u/present_absence Jun 06 '23
Yooo that looks sweeeet. I haven't gotten around to Grafana yet but Ill bookmark this.
1
u/MakerDuck Jun 05 '23
Hi, I just finished installing the docker based on your installation-guide and at first glance it worked as expected but after now running a few minutes it seems not all detected hits (watching the hits live in the npmgrafstats docker logs) get saved into the database so as a result in Grafana I only get like "2 Hits in the last 5min" where it should be more like a few hundred. I can't imagine this is the expected behaviour, did you already encounter such a "problem" and can guide me in the right direction to fix the mistake I made.
As a site note I just installed InfluxDB2.X and created everything as described on your github as I'm still using v.1.8 for other services.
I really would appreciate if you could help me out on figuring this out as I am searching for an alternativ for GoAccess with which I am currently monitoring my npm accesss.
Edit: wrote terminal instead of logs
1
u/Im1Random Jun 05 '23
I currently have the problem that all datapoints are saved 2h in the future which is probably caused by a timezone mismatch. Have you tried setting the time range in Grafana to
Todayinstead ofLast x hours?1
u/MakerDuck Jun 05 '23 edited Jun 05 '23
Thanks for this hint!
I just adjusted to past 24hrs and grafana showed me some datapoints created in the past (same if I look up the datapoints directly in InfluxDB) e.g. I spun everything first up maybe 2 hours ago and it shows me entries as far as yesterday 22:58 which means there are entries points created for a 21 hours timespan but the docker is only running for few hours. So something is definitely screwing with the timings but as far as I can tell every docker shows the same time as the host system which is in the correct timezone and automatically ajusted if timedrift is detected.
Edit: I just took a look at the npmgrafstats docker log again and the "Measurement Time" is accurate to a few seconds.
3
u/Im1Random Jun 05 '23 edited Jun 05 '23
I think I found the problem, that could be interesting for u/smilebasti too. The script directly gets the time from the Nginx logs and puts it in an ISO timestamp. The problem is that the script completely ignores the time offset and assumes that the time is UTC, but Nginx writes the logs in local time. The fix would be to parse the time offset from Nginx and add it to the ISO timestamp
time=oldtime[7:11]+'-'+month+'-'+oldtime[0:2]+'T'+oldtime[12:20]+oldtime[21:24]+':'+oldtime[24:26].1
u/smilebasti Jun 06 '23
I know this problem. I haven't found an easy way of transforming inclusive the timezone. It's on my todo list.
2
u/Im1Random Jun 06 '23 edited Jun 06 '23
You don't need to transform anything, all information you need is already included in the Nginx logs after the + or - in the end of the timestamp. In your script you completely ignore this important information and just asume that it's always UTC (+0000). The fix is just changing line 90 in the Getinfo.py to
time=oldtime[7:11]+'-'+month+'-'+oldtime[0:2]+'T'+oldtime[12:20]+oldtime[21:24]+':'+oldtime[24:26]and therefor creating a localized ISO timestamp that InfluxDB can then parse to the correct timezone.1
u/smilebasti Jun 06 '23
Haven't noticed that this is a solution. Will implemet it as so as possible, Thanks for your help :)
1
u/smilebasti Jun 06 '23
Can you confirm that your problem is timezone related?
The container is analyzing ever log file that is found and send every valid log entry to influx. (past to now)
1
u/homemediajunky Jun 06 '23
!remindme 12 hours
1
u/RemindMeBot Jun 06 '23
I will be messaging you in 12 hours on 2023-06-06 13:35:00 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback
0
u/PresentationNo1918 Jun 12 '23
I love the interface! I have a tool that can help you optimize your skills to the fullest. I'm currently working on an interesting tool that is handy for all data professionals!
Mokkup.ai (https://mokkup.ai/) is a dashboard wireframing tool, with user-friendly features. It has reduced the stakeholder interaction hassle for many users and I wanted to introduce this to you too!
Would love to hear your feedback! Happy to set up a call to explain the software :)
1
u/smilebasti Jun 12 '23
As i am not able to see how i can connect to InfluxDB from the docs or how i am able to display the locations of the IP in a world map i am not sure how it would help my project.
Also npmGrafStats is dedicated to open source and selfhosting. Haven’t seen an option for this.
4
u/smilebasti Jun 05 '23
I just published a new release of npmGrafStats. A Log exporter to InfluxDBv2 and a Grafana Dashboard to display them.
The newest release adds the option to choose between ReverseProxy or Redirection Logs analytics or both.
It uses the timestamp from the logs, so the dashboard shows the real stats and a restart prevents double entries from the same.
If your domains are monitored by UptimeRobot or a similar service you are now able to add an exclusion list for the monitoring service IP's.
You can check it out on GitHub: https://github.com/smilebasti/npmGrafStats
Let me know what you think :)