
We are getting to grips with alerting so from time to time need to clear out old alerts which we did by calling the HTTP API, to remove the pseudo time series where the alerts were stored, e.g.:
DELETE https://prometheus/api/v1/series?match[]={__name__="ALERTS"}
We have recently upgraded our Prometheus server from 1.8 to 2.2.1.
Calling this endpoint now gives
{ "status": "error", "errorType": "internal", "error": "not implemented" }
I have done some research and found a solution in various locations, which I will summarise in an answer below in case it's of use to my fellow StackOverflowers
2 Answers
Firstly the admin API is not enabled by default in Prometheus 2. This must be made active by starting the server with the option
--web.enable-admin-api
There is a new endpoint in v2 at
https://prometheus/api/v2/admin/tsdb/delete_series
This takes a POST
specifying the search criteria, e.g. for a time series with the name ALERTS
where the alert name is MyTestAlert
, post the following application/json
to the delete_series
endpoint, from the tool of choice (Tested with Postman 6 on Mac)
{ "matchers": [{ "type": "EQ", "name": "__name__", "value": "ALERTS" }, { "type": "EQ", "name": "alertname", "value": "MyTestAlert" }] }
For completeness and to free the disk space where the alerts were persisted, POST
an empty payload to
https://prometheus/api/v2/admin/tsdb/clean_tombstones
Answer aggregated from:
- https://prometheus.io/docs/prometheus/latest/querying/api/
- https://github.com/prometheus/prometheus/issues/3584
- https://groups.google.com/forum/#!msg/prometheus-users/ToMKsb9fYp8/az6afuX3CgAJ
Curl variant:
# delete alerts series curl -g -XPOST 'http://prometheus:9090/api/v1/admin/tsdb/delete_series?match[]=ALERTS' # delete data from disk curl -XPOST http://prometheus:9090/api/v1/admin/tsdb/clean_tombstones
1ncG1vNJzZmirpJawrLvVnqmfpJ%2Bse6S7zGiorp2jqbawutJoa3JwZW6Ad3yOoaawZZSkeqp5w56jnqyVYq5uwMimnGarlae2pr%2BMn6mopV2lv7C5xK2fnq2jYsNzedKpnJyhlp6worjLsmSaZaOav6qx0mamn2WRobKzwA%3D%3D