.. include:: ../global.rst.inc

Searching for Data
==================

The ``search`` module provides functionality for querying the ANTARES Database. The
search index is powered by ElasticSearch and we provide a handful of helpers for common
search operations.

Lookup By ID
------------

You can lookup objects by their ANTARES IDs or their ZTF Object IDs:

.. code:: python

   from antares_client.search import get_by_id, get_by_ztf_object_id

   # Lookup by ANTARES ID
   locus = get_by_id("ANT2020j7wo4")

   # Lookup by ZTF Object ID
   locus = get_by_ztf_object_id("ZTF20aafqubg")

Cone Searches
-------------

You can also find objects in particular locations of the sky:

.. code:: python

   from antares_client.search import cone_search
   from astropy.coordinates import Angle, SkyCoord

   center = SkyCoord("20h48m25.1805s 29d45m4.8361s")
   radius = Angle("1s")

   for locus in cone_search(center, radius):
       pass

Advanced Searches
-----------------

We also expose an API for writing queries directly against our ElasticSearch database.
These queries are complex but very powerful.

Let's say that we are interested in finding all loci with:

* Between 50 and 100 magnitude measurements
* Tagged as a nuclear transient

We represent this query in Python as follows:

.. code:: python

   query = {
       "query": {
           "bool": {
               "filter": [
                   {
                       "range": {
                           "properties.num_mag_values": {
                               "gte": 50,
                               "lte": 100,
                           } 
                       }
                   },
                   {
                        "term": {
                            "tags": "nuclear_transient"
                        }
                   }
                ]
           }
       }
   }

And can search through the ANTARES database for matching objects:

.. code:: python

   from antares_client.search import search
   first_result = next(search(query))

The return value of the ``search`` function is an iterator over loci in the result set.
This means that the result set is not immediately available in memory unless you did
something like ``result_set = list(search(query))``. Because result sets can be so large,
we recommend against doing so. Prefer, instead, operations on the iterable like:

.. code:: python

   for locus in search(query):
       do_something(locus)

Query Helpers
-------------

We plan to provide a number of tools to simplify writing queries in the future. In the meantime,
you can use the Python ``elasticsearch_dsl`` library to remove some of the boilerplate associated
with structuring ElasticSearch queries.

If you've run ``pip install elasticsearch-dsl``, you could also accomplish the previous example with:

.. code:: python

   from antares_client.search import search
   from elasicsearch_dsl import Search

   query = (
       Search()
       .filter("range", **{"properties.num_mag_values": {"gte": 50, "lte": 100}})
       .filter("term", tags="nuclear_transient")
       .to_dict()
   )
   first_result = next(search(query))

Query Syntax
------------

Queries can have a complex and deep structure. Most queries will be nested
within a `bool` structure, this allows multiple conditions to exist together.
Let's look at the conditional structures:

Must
~~~~

All documents must match the clause in order to be returned. Think of this as an
analog to *AND*. Notice that you can have multiple conditions, these are placed
within a list using square brackets (`[]`).

.. code:: json

   {
     "query":{
       "bool":{
         "must":[
            {
              "match":{
                  "properties.passband.keyword": "g"
                }
            },
            {
              "range":{
                "properties.ztf_magdiff":{
                  "gte": 0.25
                }
              }
            }
         ]
       }
     }
   }
              
Should
~~~~~~
                
Any documents that match one or more criteria are returned. `should` is not
exclusive, think of this as the analog to *OR*. These can also be placed
in a list.

.. code:: json

   {
     "query":{
       "bool":{
         "should":[
            {
              "range":{
                "properties.ztf_srmag1":{
                  "gte": "16.01"
                }
              }
            },
            {
              "range":{
                "properties.ztf_srmag1":{
                  "lte": "14.99"
                }
              }
            }
         ]
       }
     }
   }
              
Must Not
~~~~~~~~

`must_not` is the logical *NOT* operator.

.. code:: json

   {
     "query":{
       "bool":{
         "must_not":[
            {
              "match":{
                  "properties.passband": "g"
                }
            },{
              "match":{
                "properties.passband": "R"
              }
            }
         ]
       }
     }
   }
              
Ranges
~~~~~~
                
Ranges can have `gt`, `lt`, `gte`, `lte` (greater-than, less-than,
greater-or-equal, less-or-equal respectively) conditions.

.. code:: json

   {
     "query":{
       "bool":{
         "should":[
            {
              "range":{
                "properties.ztf_srmag1":{
                  "lt": "17.01"
                  "gte": "16.01"
                }
              }
            },
            {
              "range":{
                "properties.ztf_srmag1":{
                  "lte": "14.99"
                }
              }
            }
         ]
       }
     }
   }

Set Membership
~~~~~~~~~~~~~~
               
You can search for alerts that have properties in a given set of values with
the `terms` property.

.. code:: json

   {
     "query": {
       "bool": {
         "filter": {
           "terms": {
             "locus_id": [
               2042517,
               2085365,
               2471567,
               2627841,
               2761143,
               2797326,
               2822419,
               2896237
             ]
           }
         }
       }
     }
   }

Compound Queries
~~~~~~~~~~~~~~~~

You can combine these different conditional clauses to write advanced
queries. For example:

.. code:: json

   {
     "query":{
       "bool":{
         "must_not":[
            {
              "match":{
                  "properties.passband": "g"
                }
            },
            {
              "range":{
                "dec":{
                  "gte":20.23,
                  "lte":28.00
                }
              }
            }
         ],
         "must":[
           {
             "range":{
               "ingest_time":{
                 "gte": 1551398400,
                 "lt": 1554076800
               }
             }
           }
         ],
         "should":[
            {
              "range":{
                "ra":{
                  "lte": 66.13
                }
              }
            }
         ]
       }
     }
   }