Importing Data Using the XML API Import Tool – Bitfocus

Bitfocus's RESTfuls XML API Import tool was developed to allow customers to import household information, client demographics, program participation, assessments, and service data.

The HUD XML FY2020 Schema Documentation can be found at:
https://github.com/hmis-interop/xml/blob/FY2020-latest/src/HUD_HMIS.xsd
Additional Resources can be found at:
- HMIS XML Schema Overview (pdf)
- http://www.hudhdx.info/VendorResources.aspx

Required XML FY2020 Header For Data XML Files

<?xml version="1.0" encoding="UTF-8"?>
<hmis:Sources xmlns:airs="http://www.clarityhumanservices.com/schema/2020/1/AIRS_3_0_mod.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:hmis="http://www.clarityhumanservices.com/schema/2020/1/HUD_HMIS.xsd">
 <hmis:Source>

Required XML FY2020 Header For MAP XML Files

<?xml version="1.0" encoding="UTF-8"?>

<hmis:Sources xmlns:airs="http://www.clarityhumanservices.com/schema/2020/1/AIRS_3_0_mod.xsd" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:hmis="http://www.clarityhumanservices.com/schema/2020/1/HUD_HMIS.xsd">

<hmis:Source>

Sending Import Requests

Your API username, password, and crypt key for any specific agency can be found in the Manage>Sharing section within Clarity Human Services under API Credentials. For more information, see API Credentials.

Non-working Example of PHP using CURL

$fullPathToXmlFile = 'Your-XML-File.xml';
$autUserName = 'YOUR-API-USER-KEY';
$autUserPassword = 'YOUR-API-USER-PASSWORD';
$apiUrl = 'https://api-YOUR-INSTANCE-NAME.clarityhs.com/import/xml/data';
$client = new XmlImportClient($autUserName, $autUserPassword, $apiUrl);
$result = $client->importFile($fullPathToXmlFile);



class XmlImportClient {
    /**
     * @var string full path to file being imported. File should not be encrypted!
     */
    protected $file;
    /**
     * @var string HTTP auth user name
     */
    protected $username;
    /**
     * @var string HTTP auth password.
     */
    protected $password;
    /**
     * @var string url to target instance
     */
    protected $requestUrl;

    public function __construct($username, $password, $requestUrl) {
        $this->username = $username;
        $this->password = $password;
        $this->requestUrl = $requestUrl;
    }

    /**
     * Provides import of XML file to Clarity instance.
     *
     * @var string $file full path to file that should be imported.
     *
     * @return mixed import result.
     */
    public function importFile($file) {
        set_time_limit(0);
        $this->file = $file;

        $request = $this->createRequest();

        $result = $this->sendRequest($request);

        return $result;
    }

    protected function createRequest() {
        $request = curl_init();
        $this->initRequestTarget($request);

        $this->attachFileToRequest($request);

        $this->attachImportPropertiesToRequest($request);

        $this->attachAutInfoToRequest($request);

        return $request;
    }

    protected function initRequestTarget($request) {
        $requestUrl = $this->requestUrl;
        curl_setopt($request, CURLOPT_URL, $requestUrl);
        curl_setopt($request, CURLOPT_TIMEOUT, 0);
        curl_setopt($request, CURLOPT_RETURNTRANSFER, 1);
    }

    protected function attachFileToRequest($request) {
        curl_setopt($request, CURLOPT_POST, true);
        $dataFile = $this->file;
        $fileName = basename($dataFile);
        $fileType = 'text/xml';
        $postData = array(
            'data_file' => curl_file_create($dataFile, $fileType, $fileName), // Available since PHP 5.5
        );
        curl_setopt($request, CURLOPT_POSTFIELDS, $postData);
    }

    protected function attachAutInfoToRequest($request) {
        curl_setopt($request, CURLOPT_USERPWD, $this->username . ':' . $this->password);
        curl_setopt($request, CURLOPT_SSL_VERIFYPEER, 0);
        curl_setopt($request, CURLOPT_SSL_VERIFYHOST, 0);
    }


    protected function attachImportPropertiesToRequest($request) {
        curl_setopt($request, CURLOPT_HTTPHEADER, array(
            'Standard-Version: 2020', // Indicates what standard to use
            'Content-Encrypted: 0', // Indicates that data is not encrypted and MCrypt should not be used
        ));
    }

    protected function sendRequest($request) {
        $result = curl_exec($request);
        if (!$result) {
            curl_close($request);
            throw new Exception('Unable to receive response from the Clarity instance.');
        }
        curl_close($request);
        return $result;
    }
}

Incoming XML Format

The incoming XML payload must conform to the conditions described in the HUD HMIS FY2020 Schema.

 <hmis:Sources xmlns:hmis="http://www.clarityhumanservices.com/schema/2020/1/HUD_HMIS.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

XML 1.0 vs XML 1.1

While the official XML FY2020 specification calls for the use of XML 1.1, there are several issues we have run into with the adoption of XML 1.1 that have required us to create a local copy of the XSD, and make adjustments to make the XSD XML 1.0 compliant rather than XML 1.1 compliant. The key difference is that XML 1.1 allows the use of Assert expressions. These expressions allow dependency validation on specific fields. By removing these Assert dependency validations, the items become non-required, and we rely on proper XML FY2020 development to ensure that the dependencies are met.

XML 1.0 is currently seen as the XML Standard, and XML 1.1 is only used in special circumstances. Because of this, many programming frameworks do not include libraries that deal with XML 1.1. This has caused significant issues with adoption of XML 1.1-based imports. Customers looking to import data were not able to use their native XML libraries and PHP-based libraries did not include support for XML 1.1. In order to make the XML FY2020 compliant with the standard XML 1.0, we made the XML FY2020 Schema XML 1.0 compliant by pointing the XSD to a local location, and adjusted the XSD to remove assets, such as the following:

  <!--Testing if DisabilityType is 8, before allowing use of TCellCountAvailable-->
  <xs:assert test="if (hmis:TCellCountAvailable) then boolean(hmis:DisabilityType/text()='8') else true()"/>

  <!--Testing if TCellCountAvailable is 1, before allowing use of TCellCount-->
  <xs:assert test="if (hmis:TCellCountAvailable/text()='1') then boolean(hmis:TCellCount) else (if (not(hmis:TCellCountAvailable/text()='1')) then (not(boolean(hmis:TCellCount))) else false())"/>

  <!--Testing if TCellCount is not null, before allowing use of TCellSource-->
  <xs:assert test="if (hmis:TCellCount) then boolean(hmis:TCellSource) else (if (not(hmis:TCellCount)) then (not(boolean(hmis:TCellSource))) else false())"/>

  <!--Testing if DisabilityType is 8, before allowing use of ViralLoadAvailable-->
  <xs:assert test="if (hmis:ViralLoadAvailable) then boolean(hmis:DisabilityType/text()='8') else true()"/>
		
  <!--Testing if ViralLoadAvailable is 1, before allowing use of ViralLoad-->
  <xs:assert test="if (hmis:ViralLoadAvailable/text()='1') then boolean(hmis:ViralLoad) else (if (not(hmis:ViralLoadAvailable/text()='1')) then (not(boolean(hmis:ViralLoad))) else false())"/>
		
  <!--Testing if  ViralLoad is not null, before allowing use of  ViralLoadSource-->
  <xs:assert test="if (hmis:ViralLoad) then boolean(hmis:ViralLoadSource) else (if (not(hmis:ViralLoad)) then (not(boolean(hmis:ViralLoadSource))) else false())"/>

Clarity Human Services Mapping Document

To ensure mapping between source programs and services and the target Clarity Human Services Programs and Service Items, Clarity Human Services utilizes a mapping schema. Below is a sample XML file used to provide mapping. The YOUR-Organization-ID is a numeric ID only.

<?xml version="1.0"  encoding="UTF-8"?>
<map:Map xmlns:map="http://www.clarityhumanservices.com/schema/2020/1/mapping.xsd">
    <Source>
        <SourceID>YOUR-Organization-ID</SourceID>
   </Source>
    <Programs>
        <Program>
            <ClarityID>677</ClarityID>  // Target Clarity Program ID
            <ExternalID>1</ExternalID>  // Source System Program ID
        </Program>
        <Program>
            <ClarityID>678</ClarityID>
            <ExternalID>2</ExternalID>
       </Program>
        <Program>
            <ClarityID>679</ClarityID>
            <ExternalID>3</ExternalID>
        </Program>
    </Programs>
    <ServiceItems>
        <ServiceItem>
            <ClarityID>370</ClarityID>  // Target Clarity Service Item ID
            <ExternalID>583</ExternalID> //Source System Service iD
        </ServiceItem>        
    </ServiceItems>
</map:Map>

RESTful XML API Web Interface

There may be times you want to manually import your XML files. We provide a web interface that allows you to import your XML files once you have uploaded your mapping XML file.

RESTful Web URL
RESTful XML API URL:	https://rest-client.clarityhs.com
File Status
File Status URL:	https://api-Your-Instance-Name.clarityhs.com/import/xml/status/000
Error Status
Error Status URL	https://api-Your-Instance-Name.clarityhs.com/import/xml/errors/000
Import Results:
Import Status URL:	https://api-Your-Instance-Name.clarityhs.com/import/xml/result/000

The URLs above take you to one of three pages: Import Progress, Import Errors and Import Results.

IMPORT URL

STATUS URL

This XML response is generated every time a Mapping or Data XML file is imported into Clarity Human Services via the rest API.

This information will be provided to you by your system administrator. Please ensure you keep this information in a secure location as defined by your organization's security policies.

Organization ID Usage

It is important that you use the same Organization ID in your Data & Mapping XML documents. This ID is the unique key that ties your imported data and mapped programs and services to your agency.

Below are examples where the identifier needs to be consistently the same. In these examples, the Organization ID is "1234."

(1): XML Mapping File – SourceID

 <Source>
  <SourceID>1234</SourceID>
 </Source>

(2): XML Data File – SourceID

 <hmis:Source>
        <hmis:SourceID>1234></hmis:SourceID>

Error Messages

You may receive the following error messages:

Error Message

<Category>program status</Category>
<Data>
	<EntryID>N/A</EntryID>
	<ExternalID>-8186407364975394814</ExternalID>
	<Message>Client program status wasn't processed because client program was not found(Client program was probably not imported).</Message>
</Data>

<Category>enrollment</Category>
<Data>
	<EntryID>N/A</EntryID>
	<ExternalID>2406058958600011778</ExternalID>
	<Message>Client program exit wasn't processed because enrollment not found(Enrollment was probably not imported).</Message>
</Data>

Possible Causes:

The enrollment wasn’t imported because program mappings weren’t configured for the enrollment program.
The enrollment hasn’t imported yet because of tag order in the input XML file (e.g., the “Exit” tag is located higher than the “Enrollment” tag). These messages must be located before the log message about enrollment, in this case.