File Processor Agent

<< Click to Display Table of Contents >>

EQuIS 7  >>  Live > Agents >

File Processor Agent

The File Processor Agent loads file data from the following file types into EQuIS Live:




Installation and Setup


Follow these steps to install, configure, and start this agent.


App Settings


Before using this agent, one or more app settings may be changed in the *.config file as follows:

1.Open the EarthSoft.Live.Agents.File Processor.exe.config file in a text editor (e.g., Notepad++).

2.In the <appSettings> section, find the desired settings key and set its value (more information in table below).

3.Repeat steps 1-2 for additional settings that you want to change.

4.Save and close the *.config file.












The agent adds a NULL to DT_LOGGER_DATUM.DATUM_VALUE if a value is one of these (case-insensitive) strings. Otherwise if a value is not numeric, it does not add it to the database. Set this value to * to add NULL for all non-numeric values, or empty it to not add any NULLs. For example, |NaN|NULL adds NULL if a value is empty, NaN, or NULL, or 999 adds NULL if a value is 999. Separate multiple strings with pipes (|).





The DT_LOGGER_SERIES.SERIES_NAMEs for CNL 004 files, in this order. If this setting is empty, the agent uses the column names in the file, on line 3, e.g. V1, V2, and V3. Separate multiple strings with pipes (|).





The DT_LOGGER_SERIES.SERIES_NAMEs for CNL 014 files, in this order. If this setting is empty, the agent uses V1, V2, V3, etc. Separate multiple strings with pipes (|).


Date and Time



The date and time column name; this column must be present in all files that the agent processes.


MM/dd/yyyy HH:mm:ss



If this setting is not empty, the agent uses this custom date and time format when parsing the dates and times in the files. A list of format specifiers can be found here. Otherwise, if this setting is empty, it lets the DateTime.Parse(String) method determine the date and time format. The latter works well enough in most cases, but can run into problems if the date and time can have a double meaning (e.g., if the month-day-year can be interpreted as either dd/MM/yyyy or MM/dd/yyyy).


Delimited: Used to parse the date in time values in the date and time column.

CNL 004: Used to parse the date and time in the file's first line.





The columns in the files are separated by these delimiters. The most common delimiters are the comma, tab, and colon. If this setting is empty, the agent assumes that the columns are separated by commas. Separate multiple delimiters with pipes (|), e.g. ,|;.





The agent uses this encoding to read the files. If this setting is empty, it uses the computer's default encoding: Encoding.Default. Possible values for this setting are listed in the Name column of the table in the Encoding class topic's Remarks section.





The folder watcher's FileSystemWatcher.InternalBufferSize, in bytes. The default is 8192 bytes (8 KB), the minimum is 4096 bytes (4 KB), and the maximum is 65536 (64 KB). If there are many file changes in a short time, the buffer can overflow, and some files may not be processed. Increasing the size of the buffer can prevent missing file system change events. However, it is expensive, because it comes from non-paged memory that cannot be swapped out to disk. Therefore, keep the buffer as small as possible.


If the buffer overflows, this agent logs an error. For example, InternalBufferOverflowException (Too many changes at once in directory:C:\Live Files\CSV.) - to file %env{temp}/equislogs/{process}.log.





This setting is used to delay loading file data to the database by a defined interval, in milliseconds.


The FileSystemWatcher, used by the agent to watch the files and folders, may detect MULTIPLE near-same-time OnChanged events, for ONE file system operation (e.g., copying a file). To prevent a file from being processed multiple times for one file system operation, the agent starts / resets a timer when it detects an OnChanged event that fires after this number of milliseconds (e.g.,
500 ms) and processes the file.


The timer also frees up the FileSystemWatcher thread, by processing the files on a ThreadPool thread. Thus, the folder watcher's FileSystemWatcher.InternalBufferSize does not need to be too big.


(This setting should not need to change/increase, even for larger files that take longer to copy. The process that's copying the file should have a lock on it, so the agent cannot process it while it is copying, and the FileSystemWatcher should get an OnChanged event when the copying completes.)





If this setting is not empty, the agent only processes new files matching this filter. Otherwise, if this setting is empty, it process all files (*.*).


C:\a folder\delimited|C:\a second folder



The agent watches these path for new files, or new file data, and automatically loads the data into the EQuIS Live database.


CNL: The path(s) to watch must be specified in the DT_LOGGER row(s), in LIVE_DATA_SOURCE. You can use the addlogger command line option to add them. (Please let us know if you would like to add the path(s) in this App Setting instead of using addlogger. We will need to know what to set the DT_LOGGER.LOGGER_CODE to, when creating a DT_LOGGER row (e.g., the file type, something in the files, and/or part of the folder path name.))


True or False



Set to true if the column names end with units, in between parentheses, e.g. Power(kWh), and you want the agent to add the units to RT_UNIT and DT_LOGGER_SERIES.SERIES_UNIT. Otherwise, set to false to add the column names (and units, if present) to DT_LOGGER_SERIES.SERIES_NAME.





DT_LOGGER.LOGGER_CODE column name; this column must be present in all files that the agent processes. The DT_LOGGER.LOGGER_CODE values (and DT_LOGGER.LIVE_DATA_SOURCE, i.e. the computer name and folderWatcherPath) should uniquely identify a DT_LOGGER.




Use this DT_LOGGER.LOGGER_CODE if loggerCodeColumnName is empty, or does not exist in a file.


Some Prefix



If this setting is not empty and the agent adds a new DT_LOGGER row, it sets LOGGER_DESC = loggerDescPrefix + LOGGER_CODE. Otherwise, if this setting is empty, it sets LOGGER_DESC = NULL.





Delimited or CNL. If this setting is empty, the agent uses delimited.





The agent adds DT_LOGGERs with this value's UTC_OFFSET_HRS, and subtracts a DT_LOGGER's UTC_OFFSET_HRS from the dates and times in the files, to get the DT_LOGGER_DATUM.DATUM_UTC_DTs.


Common app settings, for all EQuIS Live Agents that can be installed as Windows Services, are listed here.


Continuous Data Loading


When the agent starts, it connects to the EQuIS Live database, and monitors the folderWatcherPath (delimited) or DT_LOGGER.LIVE_DATA_SOURCE paths (CNL) for file changes. When it detects a new file, or a change to an existing file, it reads the file and adds its data to the EQuIS Live database in tables: DT_LOGGER, DT_LOGGER_SERIES, and DT_LOGGER_DATUM.


Note: The agent inserts new DT_LOGGER_DATUM rows, but it does not update existing DT_LOGGER_DATUM rows (i.e., if a DT_LOGGER_DATUM row already exists with the same LOGGER_SERIES_ID and DATUM_UTC_DT, it will not update its DATUM_VALUE). (Please let us know if you want to update existing DT_LOGGER_DATUM rows. We can add an App Setting for this.)


Note: You can deactivate one or more series (by setting their STATUS_FLAGs = 'R'), and restart the agent. New data rows will not be added.


Command Line Options


EarthSoft.Live.Agents.File Processor.exe supports the following command line options:





Add a DT_LOGGER row for the given DT_LOGGER.LOGGER_DESC and folder, using the *.config file's loggerDescPrefix, processorType, and utcOffsetHrs.


e.g., addlogger "some logger code" "C:\some folder"


The delimited processorType automatically adds DT_LOGGER rows when it loads the file data; addlogger is not needed.


You must restart the agent for it to detect the new DT_LOGGER row(s).


The agent must be running to process files. You can use this option to load data for all files that were added to the watched folder(s), or changed, while the agent was not running.


Watched folder(s) = delimited: all files in the *.config file's folderWatcherPaths; CNL: all files in all CNL DT_LOGGER row folders.

The files must match the file name pattern in the *.config file's folderWatcherFilter.


Common command line options for all EQuIS Live Agents that can be installed as Windows Services are listed here.