We are getting to grips with alerting so from time to time need to clear out old alerts which we did by calling the HTTP API, to remove the pseudo time series where the alerts were stored, e.g.:
DELETE https://prometheus/api/v1/series?match[]={__name__="ALERTS"} We have recently upgraded our Prometheus server from 1.8 to 2.2.1.
Calling this endpoint now gives
{ "status": "error", "errorType": "internal", "error": "not implemented" } I have done some research and found a solution in various locations, which I will summarise in an answer below in case it's of use to my fellow StackOverflowers
2 Answers
Firstly the admin API is not enabled by default in Prometheus 2. This must be made active by starting the server with the option
--web.enable-admin-api There is a new endpoint in v2 at
https://prometheus/api/v2/admin/tsdb/delete_series This takes a POST specifying the search criteria, e.g. for a time series with the name ALERTS where the alert name is MyTestAlert, post the following application/json to the delete_series endpoint, from the tool of choice (Tested with Postman 6 on Mac)
{ "matchers": [{ "type": "EQ", "name": "__name__", "value": "ALERTS" }, { "type": "EQ", "name": "alertname", "value": "MyTestAlert" }] } For completeness and to free the disk space where the alerts were persisted, POST an empty payload to
https://prometheus/api/v2/admin/tsdb/clean_tombstones Answer aggregated from:
- https://prometheus.io/docs/prometheus/latest/querying/api/
- https://github.com/prometheus/prometheus/issues/3584
- https://groups.google.com/forum/#!msg/prometheus-users/ToMKsb9fYp8/az6afuX3CgAJ
Curl variant:
# delete alerts series curl -g -XPOST 'http://prometheus:9090/api/v1/admin/tsdb/delete_series?match[]=ALERTS' # delete data from disk curl -XPOST http://prometheus:9090/api/v1/admin/tsdb/clean_tombstones ncG1vNJzZmirpJawrLvVnqmfpJ%2Bse6S7zGiorp2jqbawutJoa3JwZW6Ad3yOoaawZZSkeqp5w56jnqyVYq5uwMimnGarlae2pr%2BMn6mopV2lv7C5xK2fnq2jYsNzedKpnJyhlp6worjLsmSaZaOav6qx0mamn2WRobKzwA%3D%3D