<< Click to Display Table of Contents >> Navigation:  EDP > Home Ribbon > Data > Managing EDDs

The most time-consuming aspects of developing and using an effective EDD are certainly determining the technical requirements, and providing the appropriate tools and enhancements to the EDD so that it is actually used by those collecting and submitting data. However, what happens once those EDDs start rolling in? Where do they go? How are they handled? By considering a few issues regarding the management of incoming EDDs, the problem of having data coming in and not knowing what to do with it can be avoided.

 

Frequency and Format

 

The best procedures for handling incoming data are largely dependent on the volume of data being received. Consider the questions below.

1.Are you a consultant receiving one or two files per week?

2.Are you a state environmental agency or EPA office that may receive hundreds of files in a week?

3.Similarly, is the volume of received data relatively consistent (i.e., the same number of files per * week) or is it submitted in batches, where you might get several files at the beginning of the month * and none for the next few weeks?

4.How are EDDs submitted? In other words, what is the physical media and delivery mechanism?

5.Is data submitted on a CD or USB? If so, is it sent through postal mail or via courier? How can the delivery of data be tracked?

6.Are data submitted electronically, via email or through a web page?

7.What ensures electronic delivery and notification? What if an email submission is lost or delayed?

 

EDD Transmission Methods

 

When using Enterprise EDP, encryption of EDDs depends on the transmission method.

 

* If EQuIS Enterprise is configured to use HTTPS, which is highly recommended. (All EOL sites use HTTPS with TLS 1.2)

 

 

EDD Storage

 

The best means of handling data once it is received depends on how and when it is received. A procedure should be developed for the management of EDD files both prior to and following processing.

 

Is data submitted to a single person? Having a single contact may be preferable to having anyone in the office receive data, but what if that person is unavailable?

 

File Naming

 

What rules have been established and provided to data collectors/providers for the naming of EDD files? While this may seem like a fairly trivial issue, a significant amount of time can be lost if someone runs across an EDD file named Lithology.txt and has no idea where it came from, who submitted it, or what project it applies to.

 

File names must be unique, and may include such information as the EDD file format, date, site or project code. The file Lithology.txt would be easier to keep track of if it were, instead, named Lithology.20010815.Landfill26.txt or something similar indicating its source and/or destination. In addition to uniqueness, files may be named for batch loading.

 

Importing data into EQuIS may be done on a manual, file-by-file basis or as a batched operation. If the volume of data received is low, manually importing data may be sufficient. However, if hundreds of EDD files are received per week, batch importing data is much more practical than loading data file-by-file. The batch import process allows you to automate loading by having the import start at, for example, midnight. Every file in a designated location is processed; if the file is loaded successfully, it is moved to one location. If it fails the import process, it is moved to another location. Regardless, each file is logged so that when you come into the office in the morning, you may see 490 of 500 files have loaded successfully. For the remaining 10, a detailed log identifies the specific problems, which you can address and then reload the files.

 

In order to successfully batch load data files, all of the data files must exist in the same folder, and must fit the following naming convention:

~\<FOLDER>\<UNIQUE_IDENTIFIER>.<PROJECT_CODE>.<IMPORT_FORMAT>.<EXTENSION>

where

•<FOLDER> – the folder that contains all of the data files to be imported

•<UNIQUE_IDENTIFIER> – some combination of letters and numbers that uniquely identifies this file.

•<PROJECT_CODE> – the project code of the EQuIS project into which this particular file is to be imported.

•<IMPORT_FORMAT> – the name of the existing import format that should be used to import this data file.

•<EXTENSION> – either ".csv" for comma-delimited files or ".txt" for tab-delimited files.

 

Checking Submitted EDDs

 

Enhancing an EDD by enforcing data types, field lengths, and reference values will do much to prevent errors during data input and thereby simplify the data loading process. However, the most fail-safe measure to make sure data will load successfully is by using tools such as the EQuIS Data Processor (EDP). Generally, the EDP is used by the lab prior to submitting the EDD. The EDP produces a log indicating when the EDD file was checked, by whom, and that it passed all required checks. To promote accuracy and submission of good data as much as possible, some agencies such as the New Jersey Department of Environmental Protection (NJDEP), require the EDP log to be submitted along with the data file. Furthermore, some organizations implement their own data checking center.

 

As data are received, before ever entering the process for loading into EQuIS, the data are checked according to established requirements.

 

 

When it comes to managing data files prior to and after importing into EQuIS, there is not necessarily a "best way" or right answer for the questions listed below. What is important is to consider these issues regarding storage of incoming EDDs, naming of files, checking of data, and protocol for dealing with good or rejected files.

1.Regardless of whether EDDs are checked internally or externally requiring the submission of logs, are these logs going to be archived? If so, where will logs be stored?

2.If the data passes the required checks, where does it go next?

3.Who needs to be notified that the data is available for loading, and how will they be notified?

4.If the data fails the checking process, what happens to it?

5.Is a notice sent to the submitting party?

6.Is the bad file stored, or rejected with the notice?

7.When corrected and resubmitted, is the data considered a new submission or a resubmittal?

8.If differentiated, how is the resubmitted EDD tracked?