Bulk delete elasticsearch

I had similar problem, after filling elasticsearch with 77 millions of unwanted documents in last couple of days. Setting timeout in query is your friend. As mentioned here. Curl has parameter to increase too (-m 3600)

curl --request DELETE \
  --url 'http://127.0.0.1:9200/nadhled/tree/_query?timeout=60m' \
  --header 'content-type: application/json' \
  -m 3600 \
  --data '{"query":{
            "filtered":{
              "filter":{
                "range":{
                  "timestamp":{
                    "lt":1564826247
                   },
                  "timestamp":{
                    "gt":1564527660
                  }
                }
              }
            }
          }
        }'

I know this is not your bulk delete, but I've found this page during my research so I post it here. Hope it helps you too.

In latest Elasticsearch(5.2), you could use _delete_by_query

curl -XPOST "http://localhost:9200/index/type/_delete_by_query" -d'
{
    "query":{
        "match_all":{}
    }
}'

The delete-by-query API is new and should still be considered experimental. The API may change in ways that are not backwards compatible

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

ES 8.11, 2024-01

I don't know what the situation was in 2016, but maybe you could consider doing a bulk delete. What is being described in the other answers at this writing is a _delete_by_query, not a bulk delete. Study that page to understand how bulk operations work.

The downside of this is that it might be quite complicated to determine the _ids of all the LuceneDocuments (index documents) you need to delete. Typically you might have to run a _search query to find these _ids on the basis of your query. You must have these _ids to do a bulk delete.

Someone might say: "ah, but if you have to run a _search and then sort out the _ids and then do a bulk delete, surely that's going to be terribly, terribly slow". I'm not at all sure that that's right: I believe a _delete_by_query turns out to be a very lengthy operation, as the OP has found, when it is meant to apply it to a large number of records, ... whereas a _search might return results very quickly indeed. Parsing the results, and simultaneously creating the bulk delete string, could also be very quick¹. And then ES executes these bulk operations incredibly fast.

You also have to make a bulk string conforming to the strict string format required. A bulk delete operation is simpler than a bulk index operation: it just looks like this:

{ "delete" : { "_id" : "234" } }
{ "delete" : { "_id" : "235" } }
{ "delete" : { "_id" : "236" } }
...

(This is the "application/x-ndjson" format you have to use).

¹ The choice of computer language might even come into play here, particularly if you can parallelise the parsing of the results from the _search. If I had to do this in a Python project I would seriously think about using a Rust module. But Python's limitations in terms of execution times shouldn't be exaggerated.

I would suggest that you should rather create a new index and reindex the documents you want to keep

But if you wanna use delete by query you should use this,

curl -XDELETE 'http://xxxxxx:9200/app/syslog/_query'

{
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        }
      ]
    }
  }
}

but then you'll be left with mapping.