Uploading and using Data Files

Some filters require access to data files, such as statistical models.

ANTARES supports this by storing such files as binary blobs in a database table. These data files can then be loaded into filters when the filter’s setup() function is called.

Uploading files into ANTARES

Files are stored under a unique key, which you create. The key must be formatted as specified below.

Use the function dk.upload_file to upload your file into the Devkit DB.

upload_file(key, file_path)

Store a datafile in ANTARES for use by filters.

The key must be formatted like:

<author>_<file-name>_<version>.<extension>

eg:

stubens_myFile_v1.txt
wolf_nnmodel_v4.pickle

Files are stored as binary strings, not ASCII. All file types are supported. The Filter which uses the file is responsible for parsing it.

Parameters
  • key – key under which to store the file.

  • file_path – path of file.

Returns

the key, for confirmation.

Example

First, place your file (eg. myFile1.txt) in the directory in DataLab in which you are writing your filter.

Then, in a DataLab Jupyter notebook:

import antares.devkit as dk
dk.init()

dk.upload_file(key='cstubens_myFile_v1.txt', file_path='./myFile.txt')

You may now test your filter using dk.run_filter.

Accessing Files from Filters

In the setup() function of your filter, file data are available as bytes objects.

Here is an example of a filter which requires a file, transforms that file into a Python object or datastructure, and then uses that object when the filter runs:

import antares.devkit as dk


class MyFilter(dk.Filter):
    ...
    REQUIRES_FILES = [
        'cstubens_myFile.txt'
    ]

    def setup(self):
        """
        ANTARES will call this function once at the beginning of each night
        when filters are loaded.
        """
        # ANTARES will load all files in `REQUIRED_FILES` into a dict
        # on the filter object `self.files`. Values in this dictionary
        # will be byte strings of class `bytes`.
        # You can then access them like:
        file_data = self.files['cstubens_myFile.txt']

        # Construct a Python object or datastructure from the raw `file_data`.
        # TODO: your code here
        # Then, you can store it on the filter instance for use in `run()`:
        self.my_object = my_object

    def run(self, locus):
        """
        ANTARES will call this function in real-time for each incoming Alert.
        """
        # Here you can use `self.my_object` in your processing of the alert.
        # TODO: your code here

Filter Submission

BEFORE you submit your filter to ANTARES, you must contact us to request that we copy your datafiles from the DevKit database into the production database. Please provide the file key value(s) which you uploaded the files to.