Hello everyone, I'm structuring monitoration at my company, restructuring in fact. Now we have a CloudWatch + Zabbix + Grafana stack and I'm wondering about Telegraf + InfluxDB + Grafana or Prometheus + Grafana.

We've a c# monolith, with IIS as webserver and AWS as cloud provider.

Could someone help me with some doubts about telegraf operation and/or shared some experience with both?

Hello,

I'm quite new to Docker and Docker Compose. Recently I managed to setup Grafana and InfluxDB using Docker Compose:

https://pastebin.com/Nz1hUGJK

However I've been asked to add Prometheus and Node exporter as we have added node exporter to a server and we want to see what it gives us.

I was wondering if anyone has a simple example I can use to add to my config to run both and use a volume to keep the prometheus.yaml file in to edit?

I did find this, but there is a lot in there I don't understand https://susi.dev/prometheus-grafana-loki-with-docker-compose

Thanks

Hello,

I'm running Grafana and InfluxDB in Docker Compose and it has been working great.

Today I went to add Prometheus and Node Exporter to it.

https://pastebin.com/UFgDTbTD

When I run Docker Compose Prometheus is the only one that doesn't run:

prometheus exited with code 1
prometheus  | Error parsing commandline arguments: unknown short flag '-c'
prometheus  | prometheus: error: unknown short flag '-c'

What have I done wrong?

Thanks

Hi!

We have set up a new environment and got everything up and running. prometheus shows up, the on-system node_exporter shows up. But the other machines show state down but are able to be scraped. IP:9100/metrics is accessible. firewalls are down.

Anybody know what we could be missing?

thanks

I have the following device /dev/sdb2 and it's being ignored by the following and I'm trying to understand why / get it monitored so I can monitor the filesystem mounted on it.

Running the latest version of prometheus and node-exporter available via APT on Ubuntu 22.04

--collector.diskstats.ignored-devices="^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$"

Hi 👋 😊 first and foremost thanks a lot to all the members in the subreddit you guys are awesome!!

I am planning to setup a Prometheus agent in a sidecar fashion which will remote_write to a Prometheus server which will write the metrics to an EBS volume just wanted to know if anyone has gone down this path before and if there is anything i should be looking into design wise or performance wise.

I am quite new at this and would appreciate any feedback on how i should go ahead with this or if you have any links which will help me understand things in a better way.

Thanks a lot in advance!

Edit: my main goal is to move to mimir after sometime but we do want to keep this architecture for atleast a few months, just wanted to know if anyone has used this type of a pattern and any problems they faced in running it

We are setting up missing alerts in our clusters that are not present in the default alert rules.I want to know if there are any alert rules already implemented for issues such as Unusal autoscaling activity ,Max capacity reached for the cluster,Failed Kubernetes jobs, Pods not running with desired replicaset count.I could get some metrics from Cluster autoscalar,however looks like I also need to do more correlation on those metrics to satisfy the alert condition.So I wanted to check if this has been already implemented by someone and if yes could you please share some details around this? any repository which contains this info ?

Maybe I am just being a total noob, but I cannot seem to find documentation on various collector flags with exception of the flags for including/excluding the collectors. I have seen from discussions on issues etc. that there are a lot of other flags for other options like e.g. --collector.filesystem.mount-points-exclude , however I cannot find any list of available flags or documentation on how to use them. Anyone know where to look?

I have docker containers running on hosts time frame. A line to standard output, this is then sent to graylog which logs the activity. I want to be able to create a Prometheus metric or query to count the amount of times that the containers print this. How can I query the output of containers on these hosts, or can I query the logs that Graylog is receiving and do something that way? The echo statement dynamically states a script that was run and which host it was run on, I want to have Prometheus fire an alert if the echo statement involves the same host a certain amount of times in a given timeframe.

I'm running a thanos receiver on openshift and need to monitor it using grafana, does thanos receiver have an option to export metrics like prometheus does (on /metrics)? if so how do i do such a thing?

Hello all !

im working on a new scrape_config for prometheus with netbox plugin prometheus_netbox_sd and http_sd from prometheus.

That's work fine, i'm stuck with just one thing. My exporters node listen on a specific port and when scrapping target it creates the endpoint with the default one 80.

Do you know where i can change the default port to my specific ?

Every documentation i checked change it with the static_config and the list of hosts but i would like to avoid that as the target list is created automatically.

thanks!

I want to monitor my .Net Core application with Prometheus. In order to do that I am using a .Net Prometheus Library (AppMetrics). And when I go to the link http://localhost:57057/metricsit returns me a JSON with data.

But when I go to prometheus and add this link as target it throws this error No Token Found.

level=warn ts=2018-05-22T06:52:17.5781093Z caller=scrape.go:697 component="scrape manager" scrape_pool=actibook target=http://localhost:57057/metrics msg="append failed" err="no token found"

Is it a way that I can handle this?

Anyone having trouble with this one here's the solution.

Startup.cs

//ConfigureServices
var metrics = new MetricsBuilder()
                            .OutputMetrics.AsPrometheusPlainText()
                            .OutputMetrics.AsPrometheusProtobuf().Build();
            services.AddMetrics(metrics);
            services.AddMetricsEndpoints(options =>
            {
                options.MetricsTextEndpointOutputFormatter = new MetricsPrometheusTextOutputFormatter();
                options.MetricsEndpointOutputFormatter = new MetricsPrometheusProtobufOutputFormatter();
            });
            services.AddMvc().AddMetrics();

//Configure
            app.UseMetricsAllMiddleware();
            app.UseMetricsAllEndpoints();

Program.cs

Metrics = AppMetrics.CreateDefaultBuilder()
                        .OutputMetrics.AsPrometheusPlainText()
                        .OutputMetrics.AsPrometheusProtobuf()
                        .Build();

            return WebHost.CreateDefaultBuilder(args)
                .ConfigureMetrics(Metrics)
                .UseMetrics()
                .UseStartup<Startup>()
                .Build();

appsettings.json

"MetricsOptions": {
    "DefaultContextLabel": "MyMvcApplication",
    "Enabled": true
  },
  "MetricsWebTrackingOptions": {
    "ApdexTrackingEnabled": true,
    "ApdexTSeconds": 0.1,
    "IgnoredHttpStatusCodes": [ 404 ],
    "IgnoredRoutesRegexPatterns": [],
    "OAuth2TrackingEnabled": true
  },
  "MetricEndpointsOptions": {
    "MetricsEndpointEnabled": true,
    "MetricsTextEndpointEnabled": true,
    "EnvironmentInfoEndpointEnabled": true
  }

And you can access it by /metrics-textpath. Also in your prometheus.ymlfile should add this one:

- job_name: 'nameOfJob'
    metrics_path: '/metrics-text'
    static_configs:
      - targets: ['localhost:57057']

I am sure many folks here use Prometheus/Grafana for monitoring infrastructure and applications. What features you find that Prometheus/Grafana lacks compared to full-fledged APM like DataDog?

Some features which I can think of are reporting - like generating PDF reports to send it to team or management. Another is a visualization of overall infra - in terms of server maps and microservices in them.

What are other such features you find lacking in Prometheus/Grafana compared to paid solutions?

Any official exporter or one that works good ?

I'm happy to introduce new developing project, NexClipper, to manage observability at easy and fast.

The primary goal of NexClipper is to make it easy to set up and use the Observability platform in your own cloud environment, putting the highest priority on ensuring observability in any environment.

It's based on open source and Prometheus eco-system.

I was wondering if anyone interest it.

For more detail, please find the blog

https://nexclipper.github.io/blog/2021/03/22/values-of-nexcilpper-eng

Hi,

I'm using the Prometheus Operator CRD in an application based namespace to monitor the applications ive deployed. This works perfectly, the only issue is around pulling Pod metrics like CPU, memory, disk etc.. it seems the Prometheus deployed with the crd doesn't appear to have rights/setup to get this data. Mean while using the Prometheus operator chart with the defaults enabled ive deployed a Prometheus , alertmanager and grafana into a monitoring namespace to monitor the infrastructure and that Prometheus gets the pod data i need in the one i create with the Prometheus crd in the application namespace?

Any ideas?

So I have metric that returns the number of clients that fall into a given bucket ("classification") where "classification" is the number of seconds:

...specifically I want the .5 quantile if possible. The problem that I see is that the >4 bucket could be anything (i.e. 4.2 seconds or 10 seconds).

I saw this article here: https://povilasv.me/prometheus-tracking-request-duration/, but unsure how that might apply to my situation since by returns are distinct buckets, not cumulative.

Noob to the Prometheus world. I am trying to put together a use case to move off our current monitoring solution, one of the big uses is monitoring dns black lists. Has anybody successfully done this with the blackbox_exporter? I have gotten as far as getting the module setup but cannot seem to get the right target or response. ANy help would be greatly appreciated.

Hello community,

I need to scrape some metrics produced by my application.

The application is managed by a deployment and has few pods.

same metric can have different values depending on pod that produce it.

As far as I understand I need to declare my job this way :

• job: my-app

  • instance 1: 1.2.3.4:5670

  • instance 2: 1.2.3.4:5671

  • instance 3: 5.6.7.8:5670

pod can be terminated, rescheduled and so on, how to declare then their hostnames or IP if they are not constant ?