Live File Processor Agent File

Overview

Live File Processor Agent's data are assigned an EBATCH number when loaded into the database. Thus the data's source file is known, and if the data are bad, it can be removed from the database using the EDD Rollback option in EDP.

Note: This functionality requires the Live Schema (schema.LIVE.xme) for EQuIS 7.20.2 Build or later, which adds the EBATCH columns to the DT_LOGGER tables.

Setting	Type	Description
ebatchEddFile	Composite String	The ST_EDD_BATCH.EDD_FILE value. Default Value: {folderWatcherPath}/{fileNameWithExtension}
ebatchEddType	Composite String	The ST_EDD_BATCH.EDD_TYPE value. Default Value: Live {processor}
ebatchReuse	Boolean	If True and an ST_EDD_BATCH table row exists where EDD_FILE = ebatchEddFile (and ROLLBACK_DATE is null), then this row is used; else, a new ST_EDD_BATCH table row is added. If False then a new ST_EDD_BATCH table row is added (each time the file changes, regardless of if an ST_EDD_BATCH table row exists where EDD_FILE = ebatchEddFile). Default Value: True

Setting

Type

Description

ebatchEddFile

Composite String

The ST_EDD_BATCH.EDD_FILE value.

Default Value: {folderWatcherPath}/{fileNameWithExtension}

ebatchEddType

Composite String

The ST_EDD_BATCH.EDD_TYPE value.

Default Value: Live {processor}

ebatchReuse

Boolean

If True and an ST_EDD_BATCH table row exists where EDD_FILE = ebatchEddFile (and ROLLBACK_DATE is null), then this row is used; else, a new ST_EDD_BATCH table row is added.

If False then a new ST_EDD_BATCH table row is added (each time the file changes, regardless of if an ST_EDD_BATCH table row exists where EDD_FILE = ebatchEddFile).

Default Value: True

Example 1 – Path and File Name

The ebatchEddFile's default value is {folderWatcherPath}/{fileNameWithExtension}, and:

OBJECT_TYPE	OBJECT_VALUE	STRING_VALUE
HOBO	folderWatcherPath	C:\EarthSoft\Live\HOBO

If a file is named rainfall-2020-05-26.csv, ST_EDD_BATCH.EDD_FILE is C:\EarthSoft\Live\HOBO\rainfall-2020-05-26.csv.

The same EBATCH number is used for a file (path and name), regardless of how often the file gets data appended to it.

Example 2 – Path and Part of File Name

A date and time, or other info (e.g., everything after the first dash), can be removed from the file name. An example is provided in the table below.

OBJECT_TYPE	OBJECT_VALUE	STRING_VALUE
HOBO	ebatchEddFile	{folderWatcherPath}/{fileNameWithoutExtension => ebatchEddFileRegexReplacements}.{fileExtension}
HOBO	ebatchEddFileRegexReplacements	^(?'a'[^-.]). => ${a}-*
HOBO	folderWatcherPath	C:\EarthSoft\Live\HOBO

If a file is named rainfall-2020-05-26.csv, ST_EDD_BATCH.EDD_FILE is C:\EarthSoft\Live\HOBO\rainfall-*.csv.

The same EBATCH number is used for all files named like rainfall-*.csv.

Example 3 – Path and File Name and Date

A date time variable (e.g., the file's last write date) can be added to the file name. An example is provided in the table below.

OBJECT_TYPE	OBJECT_VALUE	STRING_VALUE
HOBO	ebatchEddFile	{folderWatcherPath}/{fileNameWithoutExtension}-{fileLastWriteTimeUtc:yyyy-MM-dd}.{fileExtension}
HOBO	folderWatcherPath	C:\EarthSoft\Live\HOBO

If a file is named rainfall.csv, ST_EDD_BATCH.EDD_FILE is C:\EarthSoft\Live\HOBO\rainfall-2020-05-26.csv.

A new EBATCH number is used for file rainfall.csv for each day.

Example 4 – Not Used

If ebatchEddFile is defined and NULL or empty, ebatch is not used.

OBJECT_TYPE	OBJECT_VALUE	STRING_VALUE
Default	ebatchEddFile

EDD Rollback

The EDD Rollback option in EDP can be used to remove the data, but it fails if all the data for a selected EBATCH number cannot be removed. A temporary workaround may be to use an SQL statement (which can be executed using SQL Server Management Studio) to delete the DT_LOGGER_DATUM (e.g., delete from DT_LOGGER_DATUM where EBATCH = 1234), and then delete the DT_LOGGER, DT_LOGGER_SERIES, etc., rows manually.