<< Click to Display Table of Contents >> Navigation:  Enterprise > Widgets > Enterprise EDP Widgets > EDP EDD Status Widget

•Introduction

•EDP EDD Status Fields

•Status

•File Name

•Channel

•Date

•Facility

•Submitter

•Rows

•Navigation and Page Count

•EDP Workflow Operations

•Widget Properties

•EDD Processing

•Commit Logs

 

Introduction

Users can view the status of EDDs that have been submitted and processed by Enterprise EDP with the EDP EDD Status widget. The widget can be viewed on the standard EDP dashboard or can be added to a user dashboard. The Viewer permission on the Facility and EDD object types is required to download and/or view EDDs for users who are not administrators and who did not submit the EDD(s). Please see the Permissions chapter for more information.

The list of EDDs can be sorted or filtered based on status, file name, channel, date, facility, and submitter. The columns are sortable by clicking on the column header (click again to reverse the sort). Use the Status drop-down menu to filter the returned records based on the status of the EDD (All, Successful, Errors, In Progress, or Stuck). To filter the EDDs by file name, channel, date, facility, or submitter, enter a search term (one or more characters) in the appropriate search box beneath the column header.

 

EDP EDD Status Fields

 

Status

 

Status of the EDD is determined by one of the File Validate, File Process, or API Upload tasks. The Status is defined as:

•Successful: The status column will have a green icon if the EDD data was successfully committed to the database.

•Errors: If any errors prevented the EDD data from being committed to the database, the status column displays a red icon.

•In Progress (or Obstructed*):  The status column displays a blue icon if the EDD processing is not complete.

•Stuck*: The status column will have a yellow icon if the EDD was rolled back after commit or if the status of the data load cannot be determined.

•Rolled Back: The status column will have a yellow icon if the EDD was rolled back after commit or if the status of the data load cannot be determined.

 

*Obstructed EDDs

▪An EDD becomes obstructed when it causes the FileProcess Agent to fail and all EDDs submitted thereafter form a queue behind the problem EDD, leaving it unable to be processed. The status icon for these EDDs remains blue (in progress) and ST_TASK.COMPLETED_DATE and ST_TASK.STUCK_DATE remain unpopulated. Once an EDD becomes obstructed, the entire workflow comes to a halt.

▪To convert an obstructed EDD to a stuck EDD and allow queued EDDs to process,

1.Update ST_TASK.STUCK_DATE and ST_FILE_REGISTRATION.REJECTED_DATE to equal the current date (these tables can be filtered on the JOB_ID to find the offending record).

2.Add a comment to ST_FILE_REGISTRATION.REJECTED_REASON.

3.Once the status of the obstructed EDD updates from “in progress” to “stuck”, restart the site to reset the FileProcess Agent and allow it to begin processing the backlog of EDDs that formed during the obstruction.  

 

*Stuck EDDs

▪If an EDD becomes stuck due to an error, the following occurs:

1.The status icon of the EDD turns yellow in the EDP EDD Status widget.

2.The ST_TASK.STUCK_DATE field is populated in Professional.

▪When this happens, the EDD stops processing and the FileProcess Agent is free to move on to process other EDDs. When EDDs become stuck, it is often due to issues within the formats themselves. See the following troubleshooting tips.

 

Troubleshooting Stuck EDDs

EDDs can become stuck for several reasons. Following are some errors and troubleshooting suggestions:

 

 

 

File Name

 

File Name is the name of the EDD that has been submitted and processed by Enterprise EDP with the EDP EDD Status widget.

 

Channel

 

Channel denotes how the EDD was submitted:

•EML – One of the email retrieval agents (See Email EDD Submittals for configuration details.)

•FTP – File Transfer Protocol site

•Web – Enterprise EDP or REST API

•Pro – Professional EDP (EDDs that have been uploaded directly to the database via Professional EDP)

 

Date

 

Date is the date the EDD was submitted and processed by Enterprise EDP with the EDP EDD Status widget.

 

Facility

 

Facility is the name of the facility the EDD is being submitted to.

 

 

 

Submitter

 

Submitter is the person submitting the EDD.

 

Rows

 

Rows is the number of rows in the uploaded EDD.

 

Navigation and Page Count

 

The bottom of the widget contains paging controls for displaying a limited number of EDDs at a time, as described below.  

•Arrows – Use the left/right arrows to navigate through pages.

•Go to Page List – Select a specific page to navigate directly to that page.

•Rows per Page – Change the number of EDDs to be displayed per page by using the drop-down menu.

•Count of Pages and Total Items – This control indicates the total number of EDDs available in the widget, how many pages to display the EDDs, and which page is currently selected.

 

EDP Workflow Operations

 

Operations used by Enterprise EDP to process an EDD file are defined in the ST_SERVICE_RESERVATION table and are executed as items in the ST_TASK table. Possible operations are:

•File Monitor – This process monitors the channel folder and loads EDDs to the ST_FILE_REGISTRATION table in the database. Creates a file validate task for the next operation.

•File Validate – Validates that the EDD file follows the correct naming convention. Checks the user certificate to validate the user and verify that they have permissions to the specified facility.

•File Process – Processes the data in the EDD by checking for errors and committing the data to the database. This is the execution of EDP without a user interface.

•Event Monitor – Reviews EIAs and conditionally creates tasks for report generator.

•Report Generate – Executes any EIA reports queued in the ST_TASK table.

•API Upload – Indicates an EDD uploaded via a REST API. File validation was performed immediately. Thus, an EDD with this status will not have a File Monitor or File Validate task.

 

By clicking the triangle to the left of the color display icon in the status column, details about all Tasks and Notices associated with the job that loaded the EDD will be displayed. Hovering the mouse over the status icon produces a tooltip with the JobID. Hovering the mouse over the EDD file name produces a tooltip displaying the EBATCH and checksum values for the file.The EDD can be downloaded by clicking the hyperlinked EDD file name. Any reports or commit logs generated by the EDD can be downloaded by clicking the hyperlinked file name.

 

 

Widget Properties

 

To change properties of the EDP EDD Status widget, open the Widget Editor by selecting the More Options icon in the upper right corner of the widget to display the drop-down menu and then select the Edit menu option.

 

Widget Title – By default, the widget title is EDP EDD Status. The title can be modified.

Widget Color – The color of the widget can be changed using the Widget Themeicon to the left of the widget title.

Facility Code Filter – This is a legacy setting used to filter the list to a specific facility when the facility code is entered. Users still using facility code to modify their widgets are advised to use the Facility Chooser instead.

Facility – Use the Facility Chooser to limit the results to those EDDs submitted to specific facility or for any of the facilities in a facility group. This is the preferred method to filter the list for EDDs submitted.

Status – The Status drop-down menu can be set to filter the returned records based on the status of the EDD (All, Successful, Errors, In Progress, or Stuck). When the status filter has been applied within the widget properties, the status will be displayed as a number on the surface of the widget. The number values are equivalent to the following:

Status: 2 = Successful

Status: 3 = Errors

Status: 4 = In Progress

Status: 5 = Stuck

Status: 6 = Queued

Date – The Date drop-down menu can be set to filter the records based on the date submitted. The options include Today, Last 7 Days, Last Month, Last 3 Months, All, and Custom. When Custom is selected, users must specify the "Last Number of Days" to set the date filter.

EDD Format Definition – Enter the EDD format to restrict EDD status display to only the specified EDD format.

When checked, EDDs loaded via EQuIS Professional are also included in the output – Select this option to also display EDDs that have been uploaded directly to the database via Professional EDP but not using the RegisterFile option.

•These EDDs will be denoted with "Pro" in the Channel column.

•If the RegisterFile option has been correctly added to ST_CONFIG, any EDDs loaded via EQuIS Professional EDP will display in the EDP EDD Status Widget regardless of the EQuIS Professional Check Box being checked or not. EDDs 'Created' in EDP but not 'Committed' will also be displayed in the widget when the ST_CONFIG setting exists and the EQuIS Professional Check Box checked. These EDDs will appear with Red Error Status and Submitter is the ALS role assigned to the user that created the EDD:

•If the RegisterFile record does not exist in ST_CONFIG, but EQuIS Professional Check Box is checked the File Name shown in the widget will be the entire file path and file name.

Only display EDDs submitted by the current user – When checked, only EDDs submitted by the currently authenticated user will be displayed in the widget.

The widget will display EDDs loaded via Enterprise EDP and successful submittals via EQuIS Professional EDP associated to the ALS user when the following are checked: "EDDs loaded via EQuIS Professional are also included in the output" and "Only display EDDs submitted by the current user."

 

Widget Object Editor – To set various appearance properties of the widget, select the Object Editor icon in the widget header to open the Widget Object Editor. See the Widget Editor article for more details.

Help – The Help icon connects to the online documentation related to the specific widget.

Copy Settings – Select the Copy Settings icon to to copy the settings from another widget of the same type. See the Copy Widget Settings article for more information.

Cancel – Select the Close without Saving icon to exit the Widget Editor without saving changes.

Save – Changes will be applied to the widget by clicking on the Save icon. The Widget Editor screen will close after the save operation is complete.

 

 

EDD Processing

 

When EDDs are uploaded, they are placed in a queue. Depending on the number specified in the 'Maximum Tasks per wake cycle' field for the FileProcess agent, a number of tasks (in this case, EDDs) will be picked up for processing. When picking these tasks up, the FileProcess agent will pull randomly from the queue without respect for the order in which they were uploaded.  

 

Once a batch of EDDs is picked up, EDP will process each of these EDDs one at a time but in no particular order. In this way, it is possible for smaller EDDs to process before larger EDDs that were submitted first. This only occurs if they are picked up by the FileProcess agent before the older EDD or if they are placed ahead of the older EDD when chosen for the same batch. Once an EDD is picked up and it begins to process, no other EDDs can jump in front of it to be processed.  

 

Reservation Seconds is the maximum amount of time that the service needs to process the task(s). If the Reservation Seconds expires and the task(s) are not finished (I.e., the EDD is not fully processed in EDP), the FileProcess agent will restart the Reservation Seconds for the EDDs that are still processing. No other EDDs will be picked up to process until the entire batch has completed processing and the agent can begin again with a new batch.

 

An EDD is considered fully processed after it has passed all steps—File Monitor, File Validate, and File Process. The 'Accepted' notice is generated after the File Validate step, but before the File Process step that actually loads the EDD and commits it to the database.  

 

Each of these steps work independently and will sleep for a configured time between tasks to allow other agents CPU time. The default sleep time is 5 minutes. The maximum time it takes an EDD to pass all three steps is the sum of the three agents' sleep times (e.g., 15 minutes) plus the time it takes each individual agent to process its task. To reduce the time it takes to run each agent, go to the workflow settings and reduce the sleep seconds for optimal performance.

 

Sleep Seconds sets the minimum number of seconds the service will sleep. For example, if in the File Monitor tab a Sleep Seconds value of 300 is used in conjunction with a Max Tasks value of 1, the File Monitor service will process at most 1 (Max Tasks = 1) EDD every 5 minutes (Sleep Seconds = 300). This setting can be modified by administrator users as described in the Workflow widget article.

 

 

Commit Logs

 

After an EDD has passed through the File Process step and is marked as "Successfully Completed" in the EDP EDD Status widget, EarthSoft recommends opening the attached Commit log to verify that all records included in the EDD were processed as expected.

 

Administrator users can create a "Data Manager" role using the Role Manager widget. Users assigned to this role who have the proper permissions will have the ability to download EDD commit and error logs in the EDP EDD Status widget without the need to be an Administrator or the Data Submitter.