User Manual: CSV Importer: Error and Warning Reference

 

This page lists the errors and warnings you might encounter while using the CSV Importer. It includes those you might see in the application web interface, and those you might find in the CSV reports generated by the Processing and Transferring steps.

The errors and warnings are listed in the order you would likely encounter them in a typical workflow.

There is some explanation of what each error means, and what you should try in order to fix it.

If you encounter an error or warning you do not find explained here, please let us know and we will add it!

Creating a connection

Request error user account lookup failed

The CSV Importer could not log in to your CollectionSpace instance using the Url, Username, and Password you entered.

Check that the Url, Username, and Password are correct. The Url should be the base Url for the Services API associated with your CollectionSpace instance.

Creating a batch

Uploaded CSV is invalid

The CSV you uploaded is not readable by the CSV Importer. As the Open Data Institute says, “CSV looks easy, but it can be hard to make a CSV file that other people can work with.” Issues could include things such as:

  • invalid character encoding

  • different operating systems' end-of-line characters

  • rows with different numbers of columns

  • stray/unclosed quotes

  • …and many more

For a CSV smaller than 100MB, upload the file at https://csvlint.io/ and click “Validate”. This service will give you information on why your file is unreadable. See their “About” page for more info on how this works, and the issues they will report.

Try clicking the “Download Standardised CSV File” button on the results page, saving that file, and creating your batch with it.

If that didn’t work, try opening your CSV file with LibreOffice Calc and doing “Save as…” a new Text CSV file. As a dedicated tabular data creation and editing application, LibreOffice Calc is able to do fairly sophisticated automatic detection and remediation of different CSV dialects and other common structural issues with the format.

If your CSV is larger than about 100MB, we recommend you split it into smaller files. Then you can use csvlint.io to diagnose any validity issues. We also assume that there is some number of records in a single batch at which CSV Importer performance will degrade. We have not identified a specific number, but 100MB of textual CSV data would be a lot of records, and might cause you further issues in import.

Batch processing steps

About errors vs. warnings
You will see information about issues in your data in two places: in the Messages box on the right of the step screen, and in the downloadable CSV reports in the Files section on the left of the step screen. Note that the Preprocessing step never produces downloadable CSV reports.

The meaning of error is different depending on where you see that term.

On the batch processing step screens, an error means something is so wrong that the overall batch process failed or cannot be carried out. You will not be able to proceed to the next step of the import. On this screen, a warning means there is something you probably want to check before you proceed because it might indicate something unexpected and potentially destructive will happen if you continue to the next step.

In the CSV reports, you may see errors on an individual row, without seeing any error on the batch processing step screen. This means that record could not be processed or transferred. You will be able to move forward to the next batch processing step, but individual records with errors on one step will also have an error on the next step. For instance, if a row could not be processed into a valid CollectionSpace record, there is no record to transferred to CollectionSpace, so there will also be an error for that row in the Transferring step.

When there are individual-row errors in the CSV report, you will always be shown a warning about those problems on the batch processing step screen.

The reasoning behind this slightly confusing handling is: if you are importing 100 rows, and 1 rows can’t be processed, it’s likely you’d prefer to go ahead and batch transfer the 99 rows that were successfully processed. Then you can just enter the problem record manually.

Likewise, if transfer of 2 out of 5000 records fails due to an internet blip, you certainly do not want the entire transfer job to fail and stop with an error. You’ll want to know which record(s) did not transfer so you can enter them manually, or with a new batch import of just 2 records.

 

 

About CSV reports

Any issues found with your CSV file during the batch processing steps will appear in the Messages box.

The Processing and Transferring steps also produce downloadable CSV reports in the Files box.

If the Messages box indicates any issues, download your file from the CSV Importer to view more detailed error and warning messages.

The downloadable files are named via the following pattern: {your batch name}-{date-time the step was initiated}_{report type}.csv

Specific report types are described under the relevant steps below.

Preprocessing

The Preprocessing step does not produce downloadable CSV reports. The warnings and errors listed here will be displayed in the Messages box.

WARNING: Fields that will not import

If any field names are listed here, data from those columns will not go into CollectionSpace.

It is fine to go forward if you know your data has extra, un-importable columns. If this is unexpected, check your column names against the relevant CSV template.

Note: column headers/field names are case insensitive. The header is downcased for processing, so column names are returned in all lower case.

ERROR: One or more headers in spreadsheet are empty

There is at least one column that contains data, but nothing in the header row. The Processing step will not know what to do with the data in this column.

Column C is has an empty header

Either add a field name in C1, or delete column C

ERROR: Required field missing

At least one required column is missing from your CSV. The message will tell you specifically what column(s) you need to add.

Note that use of the CSV Importer requires every row/record have a unique record id. This is not a requirement of the CollectionSpace application itself, so you may get errors here about required fields that you don’t have to populate if you are entering one record at a time manually.

ERROR: In one or more rows, required field empty

Row 3 lacks the required objectNumber value

The required column is present in your CSV, but one or more rows has no value in that column. The message tells you what column(s) to examine in your data.

Open your CSV and sort or filter on those columns to find the blank ones. Enter appropriate values or remove affected rows from the CSV.

Processing

Report type: processing_report

  • Contains all data from your originally uploaded file

  • Error and Warning information is added to columns at the end (right) of your file. The presentation of the error information is designed to allow you to sort/filter to rows with the same error.

  • Warnings may or may not be true problems for you and your data. They are intended to flag possible data quality issues. Some of these may be expected and acceptable to you, so you can proceed with warnings, but it is important to always review warnings.
    For example, you may receive an “Unknown value in option list” warning. If you discover you have put “Permanent collection” in your data instead of the required “permanent-collection”, then you need to fix that before load. If you are adding “new-collection” to the collection dropdown but your admin hasn’t added it to the list of usable terms, you can go ahead and load this data with the warning. Once your admin has added the term, everything should work as expected. You just need to ensure you are using the system-expected under-the-hood value, not the display value. 

Report type: uniq_missing_terms

This report is only available if your data included authority or vocabulary terms that are not already present in your CollectionSpace instance

  • The report contains one row per unique not-found term. That is, if you have used a new term for a given vocabulary/authority in 100 different records or fields, it will appear here once. 

  • The report has four columns: term type (i.e. personauthorities), term subtype (i.e. person or ulan_pa), identifier (the “shortID” the mapping process created for the new term), and value (the term value as it was found in your original data)

  • The intent of this report is two-fold: 

    • If new terms are unexpected, you can use this information and searching your CollectionSpace instance to investigate whether case differences, punctuation differences, or typos caused your terms not to match. Then you can update your original file with the expected term form. OR…

    • If you want to create fuller term records, before linking this data to the terms, you can use the values in this form to create new CSVs for separate batch import of term records. To do that:

      • Filter this report to a given type/subtype combination (i.e. local person names)

      • Copy the values out into the termDisplayName column of the Person CSV template

      • Add in any more details you’d like to import

      • Batch import each of those CSV forms (one batch per term type).

      • After importing your new Terms, you need to delete the batch you started with start over with it. The processing step on the new batch will match to the terms you just imported.

        • If you proceed with this batch, it will create new stub records with the identifiers generated by the batch processor, which don’t match the identifiers generated by CollectionSpace when you imported the fuller term records. You will end up with duplicate term records!

    • If you do not want to create fuller term records, you can proceed with this batch without importing the terms separately, and the CSV Importer will create “stub” records for the new Terms that only include a Display name.

Transferring

WARNING: Some records did not transfer. See CSV report for details

This one is pretty self-explanatory. Look in the CSV report in the Files box to learn which records did not transfer. Filter the spreadsheet to rows where “XFER_status” column = “failure”. Details on the failure(s) will be in the “XFER_message” column.

ERROR: no appropriate transfer action detected for record

There was a mismatch between the status of this record in CollectionSpace and the type of transfer action(s) you selected. Examples could include:

  • Record status = new, but you checked only “Update existing records?” or “Delete existing records?”

  • Record status = existing, but you checked only “Create new records?”

If any of these errors was unexpected and the Transferring step is finished, you will need to start over:

  • Use the CSV report to create a new import file

    • Delete rows for records that transferred successfully

    • Delete any INFO, WARNING, or ERROR data columns added by the CSV Importer

  • Create a new batch and make sure to select appropriate transfer actions on the Transferring step

ERROR: Could not delete because other records in the system are still referencing this authority term

As in the web application, you cannot delete an authority term if it is still being used in other records.

Visit the un-deleted term authority record in CollectionSpace and look in the right sidebar to see what records it is “Used by”. Visit those records and remove the term from them. Then you should be able to delete the term manually or via a new CSV Importer batch.

If your term does not appear to be “Used by” any other records, but you are getting this message, it is likely that records using this term were “soft deleted” at some point. These records no longer show up in the application, but still exist in the underlying data used by the services API. Contact your CollectionSpace admin support and ask them to check for and delete from the database any soft deleted records using the term you wish to delete.