Importing Data

Deleting Data with the DIT

This article describes how the Data Import Tool (DIT) uses DateDeleted metadata to determine if data should be deleted.

Table of Contents

Background on DateDeleted Metadata

DIT-deletable Data Types

Access Role Requirements

Deletion Requirements

Processing Logic

Import Results Listing and Client Record Logging

Appendix

 

The HMIS XML and CSV data formats allow the inclusion of a DateDeleted property for certain fields to indicate if a record was deleted in the source system. The Data Import Tool (DIT) will use DateDeleted metadata to determine if the data should be deleted in Clarity Human Services. This article describes the DIT's process to determine if data should be deleted.

Background on DateDeleted Metadata

DateDeleted=”{dateTime}” metadata found in both HMIS CSV and Clarity XML functions as a directive (or command), behaving similarly to an API call to delete a resource. The DateDeleted directive operates independently of other data transmitted along with the enclosing HMIS CSV record or Clarity XML data element. See the definition in the FY22 HMIS XML Schema. 

This DateDeleted metadata attribute allows a community to synchronize and maintain its HMIS data set using automation by giving the uploading agency a way to indicate that a previously uploaded resource should be deleted as of a given time. Without a DateDeleted metadata attribute, a community would not be able to purge incorrect or out-of-date information that was previously uploaded into an HMIS system without manual human intervention within the target HMIS user interface.

DIT-deletable Data Types

Most HMIS CSV and XML records that include an optional DateDeleted element can be deleted via import through the DIT, as long as the requirements outlined below are met. These HMIS record types include Assessment,* AssessmentQuestions,* Client, CurrentLivingSituation, Disabilities, Employment, Education, Enrollment, EnrollmentCoC, Exit, HealthAndDV, IncomeBenefits, Services,* and YouthEducationStatus.

*Services, Assessment, and AssessmentQuestions use a different DateDeleted directive logic described in the Appendix.

Exceptions that may not be deleted via the DIT are the following HMIS record types: Affiliation, AssessmentResults, Funder, Inventory, Project, ProjectCoC, and Organization. These record types are not handled by the DIT, so they may not be deleted using the DIT DateDeleted metadata directive described in this article.  

Access Role Requirements

Certain permissions are required to delete data through the DIT. Without these permissions, the DateDeleted value will be ignored by the DIT, and the rest of the record will be processed as usual. For a complete list of permissions and their descriptions, visit the Rights Glossary for Access Roles.

  • Clients:
    • To delete client records, the user must have Delete Any Client enabled.
  • Enrollments:
    • To delete enrollments at the same agency as the user uploading the file, the user must have Delete Agency Programs enabled.
    • To delete enrollments at a different agency, the user must have Delete Any Agency Programs enabled.
  • Services
    • To delete services at the same agency as the user uploading the file, the user must have Delete Agency Services enabled.
    • To delete services at a different agency, the user must have Delete Any Agency Services enabled.

Deletion Requirements

To delete records in Clarity Human Services:

  1. The record that is being imported must have a DateDeleted value.
  2. There must be an existing record in Clarity Human Services that the imported data matches with. When importing a record that does not match with an existing record in Clarity Human Services (i.e., a new record), the DIT will skip processing that record and not create a new record. This match failure will be logged in the results, which can be obtained by clicking the “Results” button after the import is performed.

Processing Logic

The CSV or XML submitted in the DIT should not include irrelevant or redundant information, since matching and import attempts will be made for all data, including additional data elements within records also containing DateDeleted metadata.

No data elements should be sent solely for the purpose of determining a record match, except for record IDs. Required data elements also must be sent, and may contain new information. Only outdated or invalid data will be disregarded and logged as such by the DIT. The DIT attempts to import all submitted records, except records submitted solely for the purpose of deleting records already existing within the customer’s Clarity Human Services instance (by means of DateDeleted directives).

Ideally, a DateDeleted metadata containing record, whether within HMIS CSV or Clarity XML, should only additionally contain:

  1. The source system ID of the record to be deleted in the target system.
  2. Any other required elements.

To ensure processing of the record in one simple step, optional updated data elements should be left out and sent in a separate record. 

However, due to the flexible structure of HMIS CSV and Clarity XML (based on HMIS XML), a DateDeleted metadata containing record may also additionally contain optional data elements. These optional data elements could contain information that is either back-dated or post-dated from the DateDeleted metadata timestamp. In this case, more processing is required, and this additional information must be sequentially handled, independently of the DateDeleted directive (see "Step 2" in the flowchart below). 

The logic for this process is outlined in the diagram below.

Flowchart for either scenario

Flowchart of DateDeleted processing


An example of how this two-step processing logic might execute is included in the following illustration.

Scenario 1 - Ideal: No new info in the record, except the DateDeleted directive

Scenario 2 - Ideal: New info is submitted along with the DateDeleted directive

Two-step DateDeleted processing example


Also, any data structures that depend on, or are substructures of, a deleted record will also be marked deleted with the same DateDeleted deletion effective date.

Import Results Listing and Client Record Logging

The results of DIT processing are available to the authorized DIT users, to verify intended processing. The DateDeleted import results are available by clicking a “Results” button located at the bottom of the DIT screen, after the import process has completed.

  • All successfully imported record identifiers are logged in the Client Audit Log as positive confirmation to users. The record is also listed in the import results.

Client logs showing successful DIT record deletion and later manual restoration

  • All errors and disregarded inputs are listed in the "Results" with an explanation, such as:
    • A DateDeleted directive that is ignored (for various reasons, such as the matching ID is not found in Clarity, or the deleted record is older than the DateUpdated in the Clarity system)
    • New information in the same CSV/XML data element after DateDeleted that is ignored (because the record contains an outdated DateUpdated value, when compared to the target Clarity system’s DateUpdated value).

success_xml_test1

Example of listed decision results: a successful deletion

 

results_explaining_why-1

Example of listed decision results: rejected records

Appendix

Services

  1. If a DIT-uploaded Services record
    • doesn’t have a match in the Clarity instance (ie new record), and
    • the record also contains DateDeleted metadata,

      then the entire uploaded Services record is rejected.

  2. If a DIT-uploaded Services record
    • has a match in the Clarity instance, and
    • the record also contains DateDeleted metadata, and
    • has Services record deletion allowed in the DIT configuration settings, and
    • is a daily or multiple attendance Services (for DIT update and deletion, long-term is always processed as a new record),

      then delete the Services record.

    Assessments

    1. If a DIT-uploaded Assessment record:
      • doesn’t have a match in the Clarity instance (i.e. new record), and
      • the record also contains DateDeleted metadata,

        then the entire uploaded Assessment record is rejected.

    2. If a DIT-uploaded Assessment record:
      • has a match in the Clarity instance, and
      • the record also contains DateDeleted metadata, and
      • the record has a DateUpdated more recent than the matched Clarity Assessment record,

        then ignore the DateDeleted directive, and update the Assessment record.

      AssessmentQuestions

      1. For CSV file imports of AssessmentQuestions records with the DIT:
        • DateDeleted data elements are ignored, as are all other date metadata for AssessmentQuestions records,
        • only the imported record’s parent Assessment.AssessmentDate values are taken into account for the update time determination.
      2. For Clarity XML file imports of AssessmentQuestions data elements with the DIT:
        • dateDeleted element attributes on AssessmentQuestions records are ignored. [Also, the imported AssessmentQuestions element will be skipped if its dateUpdated attribute value is older than the matching parent Assessment in the Clarity instance.]


      Updated: 8/3/2023