Live File Processor Agent Example

Add the RST-2 settings below to the ST_CONFIG table using the Live File Processor widget:

Settings

OBJECT_TYPE	OBJECT_VALUE	STRING_VALUE	Description
RST-2	dateTimeColumnHeaderRegex	TIMESTAMP	DT_LOGGER_DATUM.DATUM_UTC_DT; column header text contains TIMESTAMP.
RST-2	folderWatcherPath	C:\EarthSoft\Live\RST-2
RST-2	headerLinesRegex	TIMESTAMP\nUNITS	The header has 2 lines that look like this: 1.Contains TIMESTAMP 2.Contains UNITS Note: Lines starting with ,,, are ignored, so the TIMESTAMP and UNITS lines are 'read' one after the other.
RST-2	headerLinesRegexReplacements	(?'a'[^,]+),,,,,(?'b'[^,]+) => ${a},,,,${b}	Replace ,,,,, with ,,,, in the file's header. Note: The example file has an extra comma in the TIMESTAMP line, which is not present in the UNITS and data lines.
RST-2	ignoreColumnHeadersRegex	RECORD	Do not add series or datum for columns whose header text contains RECORD.
RST-2	ignoreLinesRegex	^,,,	Ignore lines starting with ,,,.
RST-2	logDuplicateDatumRowErrors	false	See Example File 2, below.
RST-2	loggerCodeHeaderLinesRegex Groups	^Name,(?'a'[^,]*)	DT_LOGGER.LOGGER_CODE
RST-2	seriesNameColumnHeaderRegex Groups	^(?'a'[^\n]*)	DT_LOGGER_SERIES.SERIES_NAME
RST-2	seriesUnitColumnHeaderRegex Groups	\n(?'a'.*)$	DT_LOGGER_SERIES.SERIES_UNIT
RST-2	utcOffsetDefaultValue	2	DT_LOGGER_DATUM.DATUM_UTC_DT = file dates and times – 2 hours (Solwezi, Zambia time; UTC + 2 hours; not adjusted for DST).

Example File 1

1.Start the agent and copy the following file to folder C:\EarthSoft\Live\RST-2.

Model,DT2055 VW Multi Channel Logger

Name,CGT086

Serial Number,02593

Firmware Version,v1.20,Software Version,v1.31.0

Sampling Status,LOGGING,APPEND

Sampling Rate,FIXED

Current Interval,12 Hours 0 Minutes 0 Seconds

Start Time,Mon Oct 22 18 12:00:00 PM

Current Time,Tue Nov 27 18 10:12:42 AM

Number Of Sensors,8

Number Of Records,72

Sensor Upload Num / Error Code,7 / 80000

"TIMESTAMP","RECORD","BATTERY","VW 1","Therm 1","VW 2","Therm 2","VW 3","Therm 3","VW 4","Therm 4",,,,,"VW 1","Therm 1","VW 2","Therm 2","VW 3","Therm 3","VW 4","Therm 4","LOGGER TEMP","RESET"

,,,,,,,,,,,,,Unit Type,"Pressure","Disabled","Pressure","Disabled","Pressure","Disabled","Pressure","Disabled"

,,,,,,,,,,,,,Input Units,"kPa","Disabled","kPa","Disabled","kPa","Disabled","kPa","Disabled"

,,,,,,,,,,,,,Output Units,"m H2O","Disabled","m H2O","Disabled","m H2O","Disabled","m H2O","Disabled"

,,,,,,,,,,,,,Offset,"8.84600e+001","Disabled","5.69600e+001","Disabled","3.94600e+001","Disabled","2.24600e+001","Disabled"

,,,,,,,,,,,,,Cal A =,"5.64600e-004","Disabled","2.73700e-004","Disabled","2.89500e-004","Disabled","2.98200e-004","Disabled"

,,,,,,,,,,,,,Zero B =,"7.28600e+003","Disabled","7.01000e+003","Disabled","7.70800e+003","Disabled","8.34800e+003","Disabled"

,,,,,,,,,,,,,Cal C =," ","Disabled"," ","Disabled"," ","Disabled"," ","Disabled"

,,,,,,,,,,,,,Temp CF,"4.35600e-001","Disabled","9.30800e-002","Disabled","7.44500e-002","Disabled","1.09500e-001","Disabled"

,,,,,,,,,,,,,Temp Initial,"2.50000e+001","Disabled","2.30000e+001","Disabled","2.50000e+001","Disabled","2.50000e+001","Disabled"

,,,,,,,,,,,,,Linked Channel,"1","Disabled","3","Disabled","5","Disabled","7","Disabled"

"UNITS",,"Volts","m H2O","deg C","m H2O","deg C","m H2O","deg C","m H2O","deg C",,,,"B Units","Ohms","B Units","Ohms","B Units","Ohms","B Units","Ohms"

2018/10/22 12:00:00,1,2.9,88.32,22.86,56.92,22.39,39.42,22.14,22.41,22.11,,,,8110.05,3297.91,8102.74,3368.05,8493.80,3406.23,8994.10,3410.74,29.50,0

2018/10/23 00:00:00,2,2.9,88.32,22.86,56.92,22.39,39.42,22.14,22.41,22.11,,,,8109.56,3297.91,8101.93,3368.05,8493.53,3406.23,8993.73,3410.74,16.60,0

2018/10/23 12:00:00,3,2.9,88.32,22.86,56.92,22.39,39.42,22.14,22.41,22.11,,,,8109.31,3297.91,8102.43,3368.05,8493.80,3406.23,8993.81,3410.74,32.30,0

2.The file's Name line contains the DT_LOGGER.LOGGER_CODE column. Find/add the logger row. For example:

LOGGER_CODE	LIVE_DATA_SOURCE	UTC_OFFSET_HRS
CGT086	COMPUTER_NAME/C:/EarthSoft/Live/RST-2	2

3.The line that contains TIMESTAMP, followed by the line that contains UNITS (lines starting with ,,, are ignored), are the header lines; they contain the DT_LOGGER_DATUM.DATUM_UTC_DT column. The RECORD column is ignored. The other columns are the series; they contain DT_LOGGER_SERIES.SERIES_NAMEs and DT_LOGGER_SERIES.SERIES_UNITs.

4.For each subsequent line:

a.For each series (each column, excluding TIMESTAMP and RECORD columns):

i.Find/add the series row for the SERIES_NAME and SERIES_UNIT.

ii.Add the datum row (e.g., for the first three series in the first data line):

SERIES_NAME	DATUM_UTC_DT	DATUM_VALUE	SERIES_UNIT
BATTERY	2018-10-22 10:00:00 (= 2018-10-22 12:00:00 - utcOffsetDefaultValue)	2.9	Volts
VW 1	2018-10-22 10:00:00	88.32	m H2O
Therm 1	2018-10-22 10:00:00	22.86	deg C

Note: The offsets, between the TIMESTAMP and UNITS, lines:

,,,,,,,,,,,,,Offset,"8.84600e+001","Disabled","5.69600e+001","Disabled","3.94600e+001","Disabled","2.24600e+001","Disabled"

are not added to the datum values.

The Data Computation Agent or Data Deriver Agent may be used to add the offsets.

Example File 2

The following file contains two VW 1, 2, and 3 columns, with the same units (B Units), and values (and dates and times). RST-2.logDuplicateDatumRowErrors = false, does not log errors for these.

Model,DT2055 VW Multi Channel Logger

Name,CPD-002

Serial Number,02574

Firmware Version,1.20,Software Version,1.15.4

Sampling Status,LOGGING

Sampling Rate,FIXED

Current Interval,12:0:0

Start Time,2018/09/24,08:50:33

Current Time,2018/11/27,11:43:40,Logger Time,2018/11/27,11:41:55

Number Of Sensors,6

Number Of Records,129,USB

Sensor Upload Num / Status Code,7,80000,FFFFFFFF,1048704,Bat Type Not set,,WD 4,PO 0,EXT 0,BO 0,SW 255,PDI 255

"TIMESTAMP","RECORD","BATTERY","VW 1","Therm 1","VW 2","Therm 2","VW 3","Therm 3",,,,"VW 1","Therm 1","VW 2","Therm 2","VW 3","Therm 3","LOGGER TEMP","RESET","USB","RSSI","STATUS 0x"

,,,,,,,,,,,Unit Type,"Disabled"," ","Disabled"," ","Disabled"," ",,,,,"[code][P#]"

,,,,,,,,,,,Input Units,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Output Units,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Offset,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Cal A =,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Zero B =,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Cal C =,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Temp CF,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Temp Initial,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Linked Channel,"Disabled"," ","Disabled"," ","Disabled"," "

,,,,,,,,,,,Sensor,"U User","3K NTC","U User","3K NTC","U User","3K NTC"

"UNITS->",,"Volts","B Units","deg C","B Units","deg C","B Units","deg C",,,,"B Units","Ohms","B Units","Ohms","B Units","Ohms","deg C"

2018/09/24 08:50:41,1,3.65,8420.08,23.28,8668.99,22.71,9063.68,22.70,,,,8420.080078,3236.779053,8668.990234,3320.150879,9063.679688,3321.922363,23.30,1,0,0,000

2018/09/24 20:50:33,2,3.00,8421.59,23.28,8669.47,22.70,9064.20,22.69,,,,8421.589844,3237.213867,8669.469727,3321.922363,9064.200195,3323.694336,23.00,0,0,0,000

2018/09/25 08:50:33,3,3.00,8422.64,23.27,8669.40,22.69,9064.05,22.68,,,,8422.639648,3238.083740,8669.400391,3323.251221,9064.049805,3325.023926,23.40,0,0,0,000

Live File Processor Agent Example – RST-2

Live File Processor Agent Example – RST-2

Settings

Example File 1

Example File 2