Testing your import

Owned by Kristina Spurgin
Last updated: Oct 14, 2020
7 min read

Before running a large import job, we strongly encourage all users of the CSV import tool to prepare and load a test file containing a record (or a few records) representative of the data you will be loading.

How many records to test?

Best practice would be to include a value in each field you plan to populate with the CSV import tool. Often you can do this in one row.

However, if a non-repeatable field can be populated from multiple authorities, you would need to include two rows.

Example: Core Media “Creator” is non-repeatable and may be populated from Person/Local or Organization/Local. If you expect to add both person and organizational creators to the Media records you are loading, you would want to test two rows, as shown below.

| REPEATABLE FIELD? | n | n | n | | IN REPEATING FIELD GROUP? | n/a | n/a | n/a | | VALUE SOURCE | | personauthorities/person | orgauthorities/organization | | CSVHEADER | identificationNumber | creatorPerson | creatorOrganization | |---------------------------+----------------------+--------------------------+-----------------------------| | | MH2020.1.1 | Ann Acme | | | | MH2020.1.2 | | The Museum |

Testing recommendations

We recommend you pay particularly close attention to the following data patterns and field types in your testing.

Fields in repeating field groups, especially if the field group repeats

Formatting the CSV data in fields like those shown below can be tricky, so you want to make sure you are getting the expected results.

Fields populated by authorities

The Value Source in the CSV template will mention “authorities”.

Sample CSV data (core collectionobject):

| VALUE SOURCE | | conceptauthorities/concept | | CSVHEADER | objectNumber | contentConceptAssociated | |--------------+--------------+----------------------------| | | 2020.1.1 | Teapot |

What you want to see

The term is included as expected in the “Terms Used” list on the right sidebar:

 

Problem signs

Not imported as authority

Expected term does not appear in the “Terms Used” list on the right sidebar at all.

Imported as wrong authority

Expected term appears in the “Terms Used” list on the right sidebar, but as part of the wrong authority.

Fields populated by term lists

The Value Source in the CSV template will start with “vocabulary:”.

Sample import data:

| VALUE SOURCE | | vocabulary: inventorystatus | | CSVHEADER | objectNumber | inventoryStatus | |--------------+--------------+-----------------------------| | | 2020.10.14.1 | accession status unclear |

What you want to see

Click in the populated field in the imported record. The term list will drop down. You should see a check mark next to the term in the term list.

Problem signs

Apparently blank field

The data was sent through and is in the under-the-hood XML as shown below. However, it is not formatted properly as a reference to a term list term, so the user interface does not display it.

Term appears in record, but no check mark next to the value in the dropdown.

The data was sent through as a reference to a term list term as shown below, but something in the way the reference was created doesn’t match the actual term list configuration expectations in the application.

Fields populated from static dropdown lists (option lists)

The Value Source in the CSV template will be a comma separated list of allowable values.

Sample CSV data:

The collection field value for 2020.10.14.2 will import as expected. The other two rows will not.

What you want to see

Click in the populated field in the imported record. The term list will drop down. You should see a check mark next to the term in the term list.

This is from the record imported for 2020.10.14.2:

XML for 2020.10.14.2:

Problem signs

Apparently blank field

The data was sent through and is in the under-the-hood XML as shown below. However, it does not exactly match the key value for the option list, so the user interface does not display it.

XML for 2020.10.14.3:

XML for 2020.10.14.4:

Date fields

This is particularly important for fields where the CSV template data type is “structured date group”.

Ensure that the date format(s) you are using are being parsed as expected.

If you have sent “03/04/96” through as Object production date, this initially looks fine:

However, you should click on the calendar icon in the field to expand and examine the detailed date parsing:

The importer was unable to parse the date at all, so this object would not come up in a search for objects with production date 1996.

If this date had parsed, you’d want to check whether it was interpreted as March 4 or April 3!