Searching for Alerts

ANTARES has indexed its alert database using ElasticSearch so that the community can perform quick searches.

The search module of the ANTARES client provides functionality for searching the ANTARES ElasticSearch database.

Writing ElasticSearch queries is beyond the scope of this documentation but visit the instructions on the ANTARES Advanced Search page for a short introduction.

Let’s say that we are interested in finding all of the alerts with:

  • A right ascension of approximately 110 degrees (+- 0.1 degree)

  • A ZTF real/bogus score greater than 0.9

We represent this query in Python as follows:

query = {
    "query": {
        "bool": {
            "must": [
                {
                    "range": {
                        "ra": {
                            "gte": 109.9,
                            "lte": 110.1,
                        }
                    }
                },
                {
                    "range": {
                        "properties.ztf_rb": {
                            "gte": 0.9,
                        }
                    }
                }
             ]
        }
    }
}

And can search through the ANTARES database for matching alerts:

from antares_client.search import search
result_set = search(query)

We can also download the result set as a compressed/uncompressed (gzipped) .json or .csv file:

from antares_client.search import download
result_set = download(query, "results.csv", format="csv", decompress=True)
result_set = download(query, "results.csv.gz", format="csv", decompress=False)
result_set = download(query, "results.json", format="json", decompress=True)
result_set = download(query, "results.json.gz", format="json", decompress=False)

Retrieving these result sets can take a long time, depending on their size. Both the download and search functions accept a progress_callback function as a keyword argument. This function is called when the query is submitted to the server and frequently as the query is processed and a response prepared. This callback is always called with a search.JobStatus as its first argument; it should also handle arbitrary keyword arguments, even if you don’t do anything with them. A simple example to print the status of the job and to demonstrate what keyword arguments are passed to the callback at different stages of the job’s life:

def progress_callback(status, **kwargs):
    print("Search status: {}; Received kwargs: {}".format(status, kwargs))

result_set = search(query, progress_callback=progress_callback)

Query Syntax

Queries can have a complex and deep structure. Most queries will be nested within a bool structure, this allows multiple conditions to exist together. Let’s look at the conditional structures:

Must

All documents must match the clause in order to be returned. Think of this as an analog to AND. Notice that you can have multiple conditions, these are placed within a list using square brackets ([]).

{
  "query":{
    "bool":{
      "must":[
         {
           "match":{
               "properties.passband.keyword": "g"
             }
         },
         {
           "range":{
             "properties.ztf_magdiff":{
               "gte": 0.25
             }
           }
         }
      ]
    }
  }
}

Should

Any documents that match one or more criteria are returned. should is not exclusive, think of this as the analog to OR. These can also be placed in a list.

{
  "query":{
    "bool":{
      "should":[
         {
           "range":{
             "properties.ztf_srmag1":{
               "gte": "16.01"
             }
           }
         },
         {
           "range":{
             "properties.ztf_srmag1":{
               "lte": "14.99"
             }
           }
         }
      ]
    }
  }
}

Must Not

must_not is the logical NOT operator.

{
  "query":{
    "bool":{
      "must_not":[
         {
           "match":{
               "properties.passband": "g"
             }
         },{
           "match":{
             "properties.passband": "R"
           }
         }
      ]
    }
  }
}

Ranges

Ranges can have gt, lt, gte, lte (greater-than, less-than, greater-or-equal, less-or-equal respectively) conditions.

{
  "query":{
    "bool":{
      "should":[
         {
           "range":{
             "properties.ztf_srmag1":{
               "lt": "17.01"
               "gte": "16.01"
             }
           }
         },
         {
           "range":{
             "properties.ztf_srmag1":{
               "lte": "14.99"
             }
           }
         }
      ]
    }
  }
}

Set Membership

You can search for alerts that have properties in a given set of values with the terms property.

{
  "query": {
    "bool": {
      "filter": {
        "terms": {
          "locus_id": [
            2042517,
            2085365,
            2471567,
            2627841,
            2761143,
            2797326,
            2822419,
            2896237
          ]
        }
      }
    }
  }
}

Compound Queries

You can combine these different conditional clauses to write advanced queries. For example:

{
  "query":{
    "bool":{
      "must_not":[
         {
           "match":{
               "properties.passband": "g"
             }
         },
         {
           "range":{
             "dec":{
               "gte":20.23,
               "lte":28.00
             }
           }
         }
      ],
      "must":[
        {
          "range":{
            "ingest_time":{
              "gte": 1551398400,
              "lt": 1554076800
            }
          }
        }
      ],
      "should":[
         {
           "range":{
             "ra":{
               "lte": 66.13
             }
           }
         }
      ]
    }
  }
}