EDP EDD Status Widget

<< Click to Display Table of Contents >>

Navigation:  Enterprise > Widgets > Enterprise EDP Widgets >

EDP EDD Status Widget

Introduction

EDP EDD Status Fields

Status

File Name

Channel

Date

Facility

Submitter

Rows

Navigation and Page Count

EDP Workflow Operations

Widget Properties

EDD Processing

 

Introduction

Users can view the status of EDDs that have been submitted and processed by Enterprise EDP with the EDP EDD Status widget. The widget can be viewed on the standard EDP dashboard or can be added to a user dashboard. Users with both EDD Viewer and Facility Viewer permissions will see all EDDs processed by Enterprise EDP.

Ent-EDP_EDD_Status_Widget

The list of EDDs can be sorted or filtered based on status, file name, channel, date, facility, and submitter. The columns are sortable by clicking on the column header (click again to reverse the sort). Use the Status drop-down menu to filter the returned records based on the status of the EDD (All, Successful, Errors, In Progress, or Stuck). To filter the EDDs by file name, channel, date, facility, or submitter, enter a search term (one or more characters) in the appropriate search box beneath the column header.

 

EDP EDD Status Fields

 

Status

 

Status of the EDD is determined by one of the File Validate, File Process, or API Upload tasks. The Status is defined as:

Successful: The status column will have a green icon if the EDD data was successfully committed to the database.

Errors: If any errors prevented the EDD data from being committed to the database, the status column displays a red icon.

In Progress (or Obstructed*):  The status column displays a blue icon if the EDD processing is not complete.

Stuck*: The status column will have a yellow icon if the EDD was rolled back after commit or if the status of the data load cannot be determined.

Rolled Back: The status column will have a yellow icon if the EDD was rolled back after commit or if the status of the data load cannot be determined.

 

*Obstructed EDDs

An EDD becomes obstructed when it causes the FileProcess Agent to fail and all EDDs submitted thereafter form a queue behind the problem EDD, leaving it unable to be processed. The status icon for these EDDs remains blue (in progress) and ST_TASK.COMPLETED_DATE and ST_TASK.STUCK_DATE remain unpopulated. Once an EDD becomes obstructed, the entire workflow comes to a halt.

To convert an obstructed EDD to a stuck EDD and allow queued EDDs to process,

1.Update ST_TASK.STUCK_DATE and ST_FILE_REGISTRATION.REJECTED_DATE to equal the current date (these tables can be filtered on the JOB_ID to find the offending record).

2.Add a comment to ST_FILE_REGISTRATION.REJECTED_REASON.

3.Once the status of the obstructed EDD updates from “in progress” to “stuck”, restart the site to reset the FileProcess Agent and allow it to begin processing the backlog of EDDs that formed during the obstruction.  

 

*Stuck EDDs

If an EDD becomes stuck due to an error, the following occurs:

1.The status icon of the EDD turns yellow in the EDP EDD Status widget.

2.The ST_TASK.STUCK_DATE field is populated in Professional.

When this happens, the EDD stops processing and the FileProcess Agent is free to move on to process other EDDs. When EDDs become stuck, it is often due to issues within the formats themselves. See the following troubleshooting tips.

 

Troubleshooting Stuck EDDs

EDDs can become stuck for several reasons. Following are some errors and troubleshooting suggestions:

 

Error

Issue

Resolution

No error is thrown.

A long wait time or disruption in the workflow can cause the connection to break and an EDD to become stuck.

Delete the stuck date or re-upload the EDD.

(Enterprise): EDD Process Agent alert - Configuration Error

Error occurred while loading data: [EDD NAME].zip

System.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\local\Temp\....'

 

(Professional): “An error has occurred in EarthSoft.EDP, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7b3b91006de2d53d:

 

Could not find a part of the path '[FILE PATH]\[EDD NAME].xls'.

  at EarthSoft.EDP.IO.ZipFile.LoadFile(String filename, EddFormatDefinition efd)

The EDD is nested in a subfolder within the zip folder, or the files/worksheets may be named incorrectly.

Ensure the following for the zip folder being loaded into EDP:  

it only contains data files and does not contain any subfolders  

all *.txt files or excel worksheets (depending on the file type of the EDD) are named accurately to match the corresponding sections of the EDD (e.g., if "LabSample_v1" is expected, an error will be thrown if your tab or filename is "LabSample")

(Enterprise): EDD Process Agent alert - Configuration Error

 

Unable to initialize EddFormatDefinition: [EDD NAME].zip System.Exception: Unable to read format file ([FILE PATH]\[FORMAT NAME].dll): Type 'http://tempuri.org/EQEDD:yes_no_values' is not declared.

(Enterprise): See the "Best Practices for the Temp Folder" section of EarthSoft.Common.Utilities.File.zipTempFolder..

 

(Professional): See the "Cause/Details" section of Error Opening Format (Unable to Read Format File, tempuri.org).

(Enterprise): See the "Best Practices for the Temp Folder" section of EarthSoft.Common.Utilities.File.zipTempFolder.

 

(Professional): See "Resolution/Workaround" steps in Error Opening Format (Unable to Read Format File, tempuri.org).

(Enterprise): EDD Process Agent alert - Configuration Error

Unable to initialize EddFormatDefinition:  [EDD NAME].zip System.Exception: Unable to read format file ([FILE PATH]\[FORMAT NAME].xse): Object reference not set to an instance of an object.

EDP is unable to read the format files, or some of the required format files are missing.

In some cases, this issue can be resolved by loading the format onto the server rather than storing the format in DT_FILE. In others, the site's workflow needs to be restarted to ensure that all necessary format files are loaded to the temporary format folder on the server and can be accessed by EDP.

(Enterprise): EDD Process Agent alert - Configuration Error

 

Error occurred while loading data: [EDD NAME].zip System.Xml.XmlException: Unexpected xml tag. at EarthSoft.EDP.IO.ZipFile.LoadFile(Stream fileStream, EddFormatDefinition efd)

If using a third-party application to create or edit EDDs (such as Visual Basic or Excel), the method in which the EDD is saved may cause this error.

Re-save the EDD as one of the alternate valid file types and attempt to re-load the data.

(Enterprise via EDGE EDD): Error occurred while creating package: [FILE NAME] System.Exception: Transaction connection was lost, unable to continue committing rows. at EarthSoft.EDP.EddPackage.Commit(UpdateModes mode) at EarthSoft.Workflow.Clients.Default.DefaultFileProcess.PackageCommit(EddPackage pkg, EddConfig eddConfig) at EarthSoft.Workflow.Clients.Default.DefaultFileProcess.PerformAction(ReservedTaskInfo taskInfo, TaskReservationMonitor taskReservationMonitor)

The user who pushed an EDGE EDD  (which makes use of the EDGE format's automatic file organization feature) was not assigned an Admin role in Enterprise.

To solve this issue, either:

assign the user an Admin role

have a user who is already assigned an Admin role download the EDD (e.g., via the EDP EDD Upload widget) and re-commit the data

(Enterprise): EDD Process Agent alert - Configuration Error

Unable to initialize EddFormatDefinition: {EDDName}.zip System.ArgumentException: Illegal characters in path. at System.IO.LongPathHelper.Normalize(String path, UInt32 maxPathLength, Boolean checkInvalidCharacters, Boolean expandShortPaths) at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths) at System.IO.Path.GetFullPathInternal(String path) at System.IO.FileInfo.Init(String fileName, Boolean checkHost) at EarthSoft.Workflow.Management.File.FileManagementConfigSettings.GetFormatConfigurations(Connection connection, List`1 configItems) at EarthSoft.Workflow.Management.File.FileManagementConfigSettings.getFileManagementConfig(Connection connection) at EarthSoft.Workflow.Clients.Default.DefaultFileProcess.PerformAction(ReservedTaskInfo taskInfo, TaskReservationMonitor taskReservationMonitor)

An illegal character may exist in the ST_CONFIG.STRING_VALUE for the record where OBJECT_VALUE=’xsdFileName’.

Rename the file path for the format to confirm there is not an illegal character. Check for spaces or hard-returns.

(Enterprise): EDD Process Agent alert - Configuration Error

Error occurred while creating package: {EDDName}.zip System.Exception: Unknown commit type 'merge' at EarthSoft.Workflow.Clients.Default.DefaultFileProcess.PackageCommit(EddPackage pkg, EddConfig eddConfig) at EarthSoft.Workflow.Clients.Default.DefaultFileProcess.PerformAction(ReservedTaskInfo taskInfo, TaskReservationMonitor taskReservationMonitor)

The capitalization may not be correct in ST_CONFIG.STRING_VALUE for the commit type.

ST_CONFIG.STRING_VALUE for the commit type record is case sensitive. For example, 'Merge' must be used for Insert and Merge, not 'merge' or 'MERGE'.

 

 

File Name

 

File Name is the name of the EDD that has been submitted and processed by Enterprise EDP with the EDP EDD Status widget.

 

Channel

 

Channel denotes how the EDD was submitted:

EML – One of the email retrieval agents (See Email EDD Submittals for configuration details.)

FTP – File Transfer Protocol site

Web – Enterprise EDP or REST API

Pro – Professional EDP (EDDs that have been uploaded directly to the database via Professional EDP)

 

Date

 

Date is the date the EDD was submitted and processed by Enterprise EDP with the EDP EDD Status widget.

 

Facility

 

Facility is the name of the facility the EDD is being submitted to.

 

Note: The Facility name must be written in the File Name or must be selected in Enterprise (when using the EDP EDD Upload Widget). If present in both, the Facility names must match.

 

Facility Selected in Enterprise

Facility is part of the File Name

Result

NO

YES

EDP EDD Status Widget will populate the Facility field correctly

YES

NO

EDP EDD Status Widget will populate the Facility field correctly

NO

NO

An error will be thrown

YES

YES (but different Facility)

An error will be thrown

 

Submitter

 

Submitter is the person submitting the EDD.

 

Rows

 

Rows is the number of rows in the uploaded EDD.

 

Navigation and Page Count

 

The bottom of the widget contains paging controls for displaying a limited number of EDDs at a time, as described below.  

Arrows – Use the left/right arrows to navigate through pages.

Go to Page List – Select a specific page to navigate directly to that page.

Rows per Page – Change the number of EDDs to be displayed per page by using the drop-down menu.

Count of Pages and Total Items – This control indicates the total number of EDDs available in the widget, how many pages to display the EDDs, and which page is currently selected.

 

EDP Workflow Operations

 

Operations used by Enterprise EDP to process an EDD file are defined in the ST_SERVICE_RESERVATION table and are executed as items in the ST_TASK table. Possible operations are:

File Monitor – This process monitors the channel folder and loads EDDs to the ST_FILE_REGISTRATION table in the database. Creates a file validate task for the next operation.

File Validate – Validates that the EDD file follows the correct naming convention. Checks the user certificate to validate the user and verify that they have permissions to the specified facility.

File Process – Processes the data in the EDD by checking for errors and committing the data to the database. This is the execution of EDP without a user interface.

Event Monitor – Reviews EIAs and conditionally creates tasks for report generator.

Report Generate – Executes any EIA reports queued in the ST_TASK table.

API Upload – Indicates an EDD uploaded via a REST API. File validation was performed immediately. Thus, an EDD with this status will not have a File Monitor or File Validate task.

 

By clicking the triangle to the left of the color display icon in the status column, details about all Tasks and Notices associated with the job that loaded the EDD will be displayed. Hovering the mouse over the status icon produces a tooltip with the JobID. Hovering the mouse over the EDD file name produces a tooltip displaying the EBATCH and checksum values for the file.The EDD can be downloaded by clicking the hyperlinked EDD file name. Any reports or commit logs generated by the EDD can be downloaded by clicking the hyperlinked file name.

Ent-EDP_EDD_Status_Widget-Expanded

 

Note: Users with at least Editor permission to the EDD's facility can download and review the commit logs. To view error/commit logs of other users, a user must be assigned the Admin role or a Data Manager role. In Builds 7.21.3 through 7.22.2, a Data Manager role is not an option.

 

Widget Properties

 

To change properties of the EDP EDD Status widget, open the Widget Editor by selecting the More Options Ent-More_Options-Icon icon in the upper right corner of the widget to display the drop-down menu and then select the Edit menu option.

 

Ent-EDP_EDD_Status_Widget_Editor

Widget Title – By default, the widget title is EDP EDD Status. The title can be modified.

Widget Color – The color of the widget can be changed using the Widget ThemeEnt-Theme_Color_Iconicon to the left of the widget title.

Facility Code Filter – This is a legacy setting used to filter the list to a specific facility when the facility code is entered. Users still using facility code to modify their widgets are advised to use the Facility Chooser instead.

Facility – Use the Facility Chooser to limit the results to those EDDs submitted to specific facility or for any of the facilities in a facility group. This is the preferred method to filter the list for EDDs submitted.

Status – The Status drop-down menu can be set to filter the returned records based on the status of the EDD (All, Successful, Errors, In Progress, or Stuck). When the status filter has been applied within the widget properties, the status will be displayed as a number on the surface of the widget. The number values are equivalent to the following:

Status: 2 = Successful Edd_Status_Checkmark_icon

Status: 3 = Errors Edd_Status_X_icon

Status: 4 = In Progress Edd_Status_Ellipsis_icon

Status: 5 = Stuck Edd_Status_Exclamation_icon

Status: 6 = Queued Edd_Status_Hourglass_icon

Date – The Date drop-down menu can be set to filter the records based on the date submitted. The options include Today, Last 7 Days, Last Month, Last 3 Months, All, and Custom. When Custom is selected, users must specify the "Last Number of Days" to set the date filter.

EDD Format Definition – Enter the EDD format to restrict EDD status display to only the specified EDD format.

When checked, EDDs loaded via EQuIS Professional are also included in the output – Select this option to also display EDDs that have been uploaded directly to the database via Professional EDP but not using the RegisterFile option.

These EDDs will be denoted with "Pro" in the Channel column.

If the RegisterFile option has been correctly added to ST_CONFIG, any EDDs loaded via EQuIS Professional EDP will display in the EDP EDD Status Widget regardless of the EQuIS Professional Check Box being checked or not. EDDs 'Created' in EDP but not 'Committed' will also be displayed in the widget when the ST_CONFIG setting exists and the EQuIS Professional Check Box checked. These EDDs will appear with Red Error Status and Submitter is the ALS role assigned to the user that created the EDD:

ENT-EDP_EDD_Status_Error

If the RegisterFile record does not exist in ST_CONFIG, but EQuIS Professional Check Box is checked the File Name shown in the widget will be the entire file path and file name.

Only display EDDs submitted by the current user – When checked, only EDDs submitted by the currently authenticated user will be displayed in the widget.

The widget will display EDDs loaded via Enterprise EDP and successful submittals via EQuIS Professional EDP associated to the ALS user when the following are checked: "EDDs loaded via EQuIS Professional are also included in the output" and "Only display EDDs submitted by the current user."

 

Widget Object Editor – To set various appearance properties of the widget, select the Object Editor Ent-Widget-Editor_Properties icon in the widget header to open the Widget Object Editor. See the Widget Editor article for more details.

Help – The Help Ent-Help-Icon icon connects to the online documentation related to the specific widget.

Copy Settings – Select the Copy Settings Ent-Copy_Settings-Icon icon to to copy the settings from another widget of the same type. See the Copy Widget Settings article for more information.

Cancel – Select the Close without Saving Ent-Close_icon icon to exit the Widget Editor without saving changes.

Save – Changes will be applied to the widget by clicking on the Save Ent-Widget_Save-Icon icon. The Widget Editor screen will close after the save operation is complete.

 

 

EDD Processing

 

An EDD is considered fully processed after it has passed all steps—File Monitor, File Validate, and File Process. The 'Accepted' notice is generated after the File Validate step, but before the File Process step that actually loads the EDD and commits it to the database.  

 

Each of these steps work independently and will sleep for a configured time between tasks to allow other agents CPU time. The default sleep time is 5 minutes. The time it takes an EDD to pass all three steps may vary between the sum of the time it takes all three agents to process (e.g., 15 minutes) plus the sum of the time it takes each individual agent to process. To reduce the time it takes to run each agent, go to the workflow settings and reduce the sleep seconds for optimal performance.

 

Sleep Seconds sets the minimum number of seconds the service will sleep. For example, if in the File Monitor tab a Sleep Seconds value of 300 used in conjunction with a Max Tasks value of 1, the File Monitor service will process at most 1 (Max Tasks = 1) EDD every 5 minutes (Sleep Seconds = 300). This setting can be modified by administrator users as described in the Workflow widget article.

 

Note: After an EDD has passed through the File Process step and is marked as "Successfully Completed" in the EDP EDD Status widget, EarthSoft recommends opening the attached Commit log to verify that all records included in the EDD were processed as expected.