documentation is not clear on how you connect to the gateway from a push client

[JohnBasrai]

I started the container with the following command line

docker run --name prom-agg-gateway -d -p 8888:8888 -p 80:80 ghcr.io/zapier/prom-aggregation-gateway

This maps the two ports it uses by default to the host. Docker logs say:

$ docker logs prom-agg-gateway
2023/10/16 16:35:25 lifecycle server listening at :8888
2023/10/16 16:35:25 api server listening at :80

The readme says to use one of the stand standard clients. I tried the python one.

https://github.com/prometheus/client_python

This page has the following example of how to push metrics.

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
registry = CollectorRegistry()
g = Gauge('job_last_success_unixtime', 'Last time a batch job successfully finished', registry=registry)
g.set_to_current_time()
push_to_gateway('localhost:9091', job='batchA', registry=registry)

I tired adjusting this client code to use each of the two ports that appeared in the log for prom-aggregation-gateway (80 & 8888) and both of them get the same 404 error.

urllib.error.HTTPError: HTTP Error 404: Not Found

For reference here is the help message I get from the container:

$ docker exec -it prom-agg-gateway /prom-aggregation-gateway --help
prometheus aggregation gateway

Usage:
  prom-aggregation-gateway [flags]
  prom-aggregation-gateway [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  start       starts up the server
  version     Show version information

Flags:
      --AuthUsers strings        List of allowed auth users and their passwords comma separated
                                  Example: "user1=pass1,user2=pass2"
      --apiListen string         Listen for API requests on this host/port. (default ":80")
      --cors string              The 'Access-Control-Allow-Origin' value to be returned. (default "*")
  -h, --help                     help for prom-aggregation-gateway
      --lifecycleListen string   Listen for lifecycle requests (health, metrics) on this host/port (default ":8888")

Use "prom-aggregation-gateway [command] --help" for more information about a command.

There is no mention here or in the readme on what the two ports are but if I were to take a guess I think the second one above --lifecycleListen string is the one I should be using since it mentions metrics.

Please help.

[djeebus]

The docs make a quick mention here to send metrics to /metrics/, with the implication that http://localhost/, but we could do a better job of calling out exactly where to send the metrics. You can also add labels in that url by sending metrics to /metrics/label1/value1/label2/value2/, etc.

The "lifecycle server" only exists to give kubernetes a place to send probes to, and expose internal metrics. The metrics exposed by the lifecycle server describe the health and activity of the gateway itself, as opposed to metrics that are forwarded by and aggregated for the gateway's clients.

[JohnBasrai]

Thanks for the reply,
Here is what I tried and what I see in response

$ docker run -d -p9091:9091 prom/pushgateway
$ echo '
http_requests_total{method="post",code="200"} 1027
http_errors_total{method="post",code="500"} 6
'
+ curl --data-binary @- http://localhost:9091/metrics/domain/sometest.com/instance/nginx-1
404 page not found

Thanks again.

[djeebus]

You're using the prometheus pushgateway there, not prom-aggregation-gateway. Theirs requires that your path includes a job parameter first:

[~]$ echo '  
http_requests_total{method="post",code="200"} 1027
http_errors_total{method="post",code="500"} 6     
'| curl --data-binary @- http://localhost:9091/metrics/job/some-job/domain/sometest.com/instance/nginx-1 -v
*   Trying 127.0.0.1:9091...
* Connected to localhost (127.0.0.1) port 9091 (#0)
> POST /metrics/job/some-job/domain/sometest.com/instance/nginx-1 HTTP/1.1
> Host: localhost:9091
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 99
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 200 OK
< Date: Tue, 17 Oct 2023 17:48:36 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact

prom-aggregation-gateway, on the other hand, allows you to add the /job/$job label pair anywhere in the url, or omit it altogether.