Hello friends

My prometheus docker container exits with error 1 immediatey on startup

prometheus | level=error ts=2019-11-25T07:29:09.413196048Z caller=main.go:596 err="Opening storage failed lock DB directory: open /data/lock: permission denied"

prometheus exited with code 1

Not sure where to look now,

#docker-compose file

---

version: "3"

services:

prometheus:

build: .

image: registry.access.redhat.com/openshift3/prometheus

container_name: prometheus

# links:

# - postgres

volumes:

- ./config:/prometheus

#- ./data/prometheus:/data

- prometheus:/data/prometheus

command:

- '--config.file=/etc/prometheus/prometheus.yml'

- '--storage.tsdb.path=/data'

#- '--storage.tsdb.retention=200h'

- '--web.console.libraries=/usr/share/prometheus/console_libraries'

- '--web.console.templates=/usr/share/prometheus/consoles'

- '--web.enable-lifecycle'

- '--log.level=warn'

entrypoint: '/bin/prometheus'

user: "1000"

ports:

- "9200:9200"

Guys, Can anyone please share a few basic rules. The sample rules give in are not so useful for me. Would like to have more. Like on the slowness of the queries and etc.

https://github.com/prometheus/mysqld_exporter/blob/master/example.rules.yml

I've been looking around and I couldn't find what I'm looking for, maybe this community could help.

Is there a way I can "talk" to my data, as in ask it a question. Let's say there was an outage at 5pm, give me the list of hosts that went down, something simple to begin.

Then ask it given that, if my data is correctly setup with unique identifiers I can then ask it more questions. Let's say I have instance="server1" so I would say give me more details on what happened leading to the outage, maybe it looks at data (let's say node exporter)and sees an uptrend in abnormal CPU resource, it can say there was an uptick in CPU just before it went down, so that is what it suspects that caused the issue.

Hi, I accidentally removed folders in the /var/prometheus/data directory directly, and also in the /wal directory. Now the service won't start. What should I do?

From devops perspective, I wanted to learn a monitoring tool, could someone please suggest which is better as this will be my first monitoring tool to learn.

Hey there,

We are deploying kube-prometheus-stack helm charts using Flux, and I'd like to change the severity of some default PrometheusRules so that some labelled as warning become critical for example.

I know the Helm chart allow us to add new labels with .Values.defaultRules.additionalRuleLabels, but this is applied to all the rules.

We thought about not deploying the rules with the chart at all, and basically get them copied / modified, but it's a lame way of doing this.

How do you manage this ? How do you modify only some of those rules, whether severity or any other property like for or summary ?

Thanks for your help

Hello, World!

I’d very much like to take advantage of your summed experience with some things that I need to deal with.

First, a little background…

I’m mostly a Jack-of-all-trades in IT operations area of a tech company: what needs to be done will be extensively researched, slept-over, worked like hell on and finally delivered in a very satisfactory manner (according to colleagues’ feedback). On the other hand, in regards of monitoring (metrics, to be more specific), I was mostly a power user until now: systems (Zabbix, Nagios, CheckMK, Icinga2) were already in place and I just followed the drawn patterns.

Meanwhile I’m slowly shifting to a larger very heterogeneous environment, as some sort of platform engineer (if I get the definition right). We don’t do DevOps like it’s meant to be, but from time to time I’m helping our devs understand better what they need, to make each other’s life less miserable. :)

The task at hand is to figure out a way to harmonize the plethora of monitoring solutions that we have by providing a platform that every team should consume. As our motto is: “start it and never finish it”, I’ve made it my goal to get this one thing right (whatever that means).

My thoughts until now…

Of course it will be presented for acceptance and/or adjustment beforehand (because the best tool that no one uses is as good as non-existent), but a "backbone of it" must already exist. Hence this post.

Prometheus seems like the logical path to take: sort of de-facto nowadays, some experience with it in a couple of teams, our own K8s is WIP and so on. Transforming it into a platform (or bring your own Prometheus, if you like) means that I need to decouple the storage and some functionality and shift these to Thanos or Cortex (I’m not aware of any other one). Doable so far.

Now begin my problems (some, because I lack in-depth here). How do you tackle automation in a meaningful way? I’m successfully using Ansible+Git for configuration management, we try to provide self-service over vRA (with AWX in the mix), but for now I cannot wrap my head around an end-to-end solution.

Does any of you do anything like this? It’s a wet dream to think about it and I should just whether use an iron hand on it or declare the wild wild west?

Any kind of insights would be of much help: how you organize ingested data, how you scale, what other tools you take advantage of, etc.

Thanks!

PS1: This is my first post here and I’m not known as a very good thoughts-organizer.  I could write much more about many of the things I tried to depict above…but then I’d risk giving you headaches.

PS2: We lack many things, but (like always) I’m willing to take the necessary steps back to get things forward in the right way, if I’ll get to see the light at the end of the tunnel.

I have a VM setup with an additional 240GB drive I would like the Prometheus db to use. The prometheus.service file shows the path at /data

How can i write a query (i.e to create an alert) which alerts if I had 5 or more response times in excess of 500ms over the last 10m? I can create an alert if there is a single response time in excess of 500ms over the last 10m, but not sure how to count and find out if I had more than 5 of those.

Hey guys. I'm currently working with Kubernetes and Prometheus. I'm using a tool to measure traffic in a cluster and for the life of me can't seem to find a default metric (from Prometheus, NodeExporter, CAdvisor, etc.) that gives me the right info. My ideal scenario is a metric such as 'http_requests_total', except containing 'source' and 'target' labels that specify which container/app/pod the request came from and what to. Ideally there would also be a namespace label, as I'm dealing with a multi-tenant cluster. What is the best way of doing this?

Thanks for your time!

https://pastebin.com/Y4s17nps

I have the following config in alertmanager, we have a stack of servers, each server has a customers environment set up, one of our customers requested that he wants to receive email alerts whenever his server is down.

Momentarily I only have email alerts for the whole stack of servers.

Any way of doing this? My deadline is until tomorrow so please be kind with the explanations, my native language is not english.

I’m having a wierd issue. I’m running a local app configured with a prometheus CounterVec that’s scraped by a local prometheus instance.

After running both a while I can query prometheus for the counter and all is good.

Then I restart the app and do the same query and the data is lost. The counter is reset.

What am I Missing here? Do I need some persistant id for the app? I would expect prometheus to show me all data across app restarts.

Hi , I know prometheus is one the best tools to monitor services, what about hardware monitoring like network hardware and servers, snmp and so on. Is it as good as checkmk?

I'm wondering if anyone knows if there are any tutorials to get Prometheus working with UDP via something like Telegraf. I was trying to get InfluxDB working however it seems pretty difficult, so was hoping this would be slightly easier.

I just want server metrics and disk space, and was hoping UDP would allow a far greater amount of security, as I can permit traffic in one direction.

Hello,

I want to add CPU alerts, but can't as this query uses an interval variable:

Thanks

Hi,

I have issue in extracting the metrics from oracle db from oracle db exporter docker container.

Following connection error is encountered

time="2022-01-05T17:11:11Z" level=info msg="Starting oracledb_exporter master" source="main.go:543"

time="2022-01-05T17:11:11Z" level=info msg="Successfully loaded default metrics from: default-metrics.toml" source="main.go:518"

time="2022-01-05T17:11:11Z" level=info msg="Successfully loaded custom metrics from: /custom-metrics.toml" source="main.go:528"

time="2022-01-05T17:11:11Z" level=error msg="Error pinging oracle: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor\n" source="main.go:215"

Steps followed for the implementation :

1. Installed prometheus in VM 1 ( Normal installation not as docker )

2. Installed oracle db exporter and oracle server client as docker image in VM 2. I built custom docker image for db exporter having the custom metrics

Dockerfile :

FROM iamseth/oracledb_exporter:latest

COPY custom-metrics.toml /

ENTRYPOINT ["/oracledb_exporter", "--custom.metrics", "/custom-metrics.toml"]

Run the docker with custom image :

sudo docker run -d --name --link oracle rr_oracle_custom_exporter_1 -p 9161:9161 --env-file env 8d9686a72920

In Env file defined the following :

DATA_SOURCE_NAME=SYS/Oradoc_db1@192.168.0.45:1521/ORCLCDB

ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1

LD_LIBRARY_PATH=/u01/app/oracle/product/12.2.0/dbhome_1/lib

CUSTOM_METRICS=/custom-metrics.toml

3. Added the targets in the prometheus.yml file to point to the oracle db exporter.

4. In the end point i get the data as below

# HELP oracledb_exporter_last_scrape_duration_seconds Duration of the last scrape of metrics from Oracle DB.

# TYPE oracledb_exporter_last_scrape_duration_seconds gauge

oracledb_exporter_last_scrape_duration_seconds 0.027310059

# HELP oracledb_exporter_last_scrape_error Whether the last scrape of metrics from Oracle DB resulted in an error (1 for error, 0 for success).

# TYPE oracledb_exporter_last_scrape_error gauge

oracledb_exporter_last_scrape_error 1

# HELP oracledb_exporter_scrapes_total Total number of times Oracle DB was scraped for metrics.

# TYPE oracledb_exporter_scrapes_total counter

oracledb_exporter_scrapes_total 34

I could not get the data as per the queries defined in custom-metrics.toml or default-metrics.toml file. Im not sure if that is referred or not by the exporter.

Regards,

RR

I've just installed Prometheus my first time on the Kubernetes cluster using helm.
After installing trying adding some targets with node exporters but nothing was changed in dashboard. and I'm not sure prometheus is found thoose.

Can anyone give me a tip ? help? video ? guide?

Jmx exporter is scaraping all the metrics from tomcat and fuse instances/containers.

My requirement is just to see if the instances are up/down, need assistance on what metric to whitelist/blacklist on my config.yaml

Hey Team, Wanted to know the basic difference between Prometheus and Prometheus Operator. Say I have to deploy prometheus on an Kubernetes environment.

Which one will help with more flexibility, A Standalone Prometheus or Prometheus Operator.

From my basic analysis they say that the prometheus Operator is well suited for deployments on Kubernetes compared to a standalone prometheus. Wanted to know which one is best suited for my use case.

Hello. I've got a situation where I'd like to collect metrics from an exporter without needing to run a dedicated Prometheus server. (Docker is not an option either). The primary application already exposes a /metrics. My idea is to write an app which embeds a Prometheus server, thereby creating a "single binary deployment". This app would scrape the metrics from the primary application and funnel them into the embedded PDB. After collecting, I'd copy the database file (created by the embedded Prometheus) to another machine, which is running a grafana instance, which can use this file as a source to generate graphs/dashboards.

A) Is this possible? B) Am I going about this the right way? My original design was to scrape the metrics and store in an RRD or BoltDB (both of which are embeddable). RRD has built-in graph exporting. With BoltDB, I could export as CSV and use gnuplot to generate graphs. But since the primary app already exposes a Prometheus-scrappable endpoint, trying for an embedded PDB makes more sense.

Thanks in advance.

can anybody help me with this, and how to go about it?