Teragrep Archive Datasource

Table of Contents

Limitations

Archive access is currently enabled on all languages that belong to the Spark group. Currently only the Spark MicroBatch interface is implemented.

Trigger will keep processing indefinitely, but it is currently the only trigger that will process all the data. Trigger.once will process only the amount of objects specified as partitions, so it will likely process less objects than you want.

These limitations do not apply to DPL. Other language workarounds can be produced.

Archive Datasource is actively being developed to overcome these obstacles. If you require some of the missing features, please submit an issue at Teragrep’s Github.

Archive Query

Archive Query is made in the XML format, which has the capability of representing complex query structures. You can make a query in the Archive Datasource with the query option.

This DPL query:

index=f17

…​is translated to the following:

"""<index value="f17" operation="EQUALS"/>"""

This DPL query:

index=f17 earliest="01/01/1970:00:00:00" latest="04/27/2021:08:55:54"

…​is translated to the following:

"""<AND><AND><index value="f17" operation="EQUALS"/><earliest value="0" operation="GE"/></AND><latest value="1619513754" operation="LE"/></AND>"""