CSV Import Tool: User Story Candidates

Admin

Sysadmin can install and maintain a single deployment of the CSV Import Tool that can be used by many CSpace users/instances

JL: Hmmm… considering UCB, would that be one instance for all 5 museums? Not necessary for UCB.

MC: Could be 1 instance for UCB, or 5.

Sysadmin can create, update, and inactivate all User accounts

Sysadmin can create a session as another user in the system, allowing an an admin to reset a user’s password, etc.

User can create or edit a single User account

Sysadmin can create and edit all Group accounts

MC: Can assign users to groups, but there should also be a (configurable) system to assign a user to a group on sign up.

User can create and edit a single Group account

MC: Only allow admins to create groups (at least initially).

Admin can create a Connection between the Tool and any CSpace instance

User can create a Connection between the Tool and one or more CSpace instance to which they can authenticate

MC: To use the system a user will need to create at least one “connection” to cspace, but can create more (i.e. single user creates staging / prod connections).

Sysadmin can grant a User access to a Connection

MC: use of connections should be user specific.

User can grant a single Group access to a single Connection

MC: we probably need to define groups better, and decide if we want / need this. I currently view a group as a way to connect multiple users so they can see each others records (batches etc.). An admin can see / do everything, but if a user is not an admin they can either be a “Manager” or “Member” of a single group. A member can see content from their group, but only directly create / update their own stuff. A manager is more like an admin but group scoped only. This would allow us (LYRASIS) to have groups like “OHC” and “Breman” with users who can see what their colleagues have done / are doing, and group managers who can enable / disable group users and so on.

We could also consider abandoning the group system and have a binary authorization implementation differentiating only between admin (see / do all) and regular users (see / do own stuff). This is simpler and seamless in terms of “sign up and start using”, but wouldn’t allow users from the same “group” (organization) to see what each other is doing in the app ... if that’s important … as each user would be sandboxed from every other user (except for admins). They’d have to share logins or show each what is up in a browser.

Admin or Manager can upload the mapping configuration for a Connection

Import/Update

User can create import/update job for a given record type in a given profile by uploading a properly formatted CSV file

User can import new records to a CollectionSpace instance

User can update existing records in a CollectionSpace instance

User can delete existing records in a CollectionSpace instance

UCB: not needed by us, and might be a dangerous feature.

User can ingest a CSV file containing rows representing both records to be newly created and existing records to update

Data will be imported, validated, transformed, checked for quality issues the same way for all, and flagged in the system as whether it will be a new record in CSpace or an update to an existing record

User can ingest a CSV file containing rows representing records to delete. (A batch record deletion workflow is treated as a different process in the system than an import/update workflow)

Deletion workflow is a separate workflow, requiring only a record ID match. No data quality checks or mapping will be done.

Question: does there need to be some form of validation/messaging that transparently says “No record with ID 123.456 found - nothing deleted”, or should the found records be deleted and others just skipped?

As part of an import/update workflow, User can select one or multiple records and remove them from the batch

This has no effect on existing records in CSpace

As part of an import/update workflow, User can select one or multiple records flagged as already existing in CSpace and initiate deletion of those records from CSpace

This process also removes the records from the processing batch

After ingesting a CSV file for import/update, User can clearly see how many records are new and how many will update existing records.

After ingesting a CSV file for import/update, if there are any records in the batch with duplicate record IDs (or Short IDs, for authorities/vocabulary terms), User is warned and can view a duplicate report from which they can select records to remove from the batch

The current design for the Mapper has it only knowing about one row of data at a time. So this checking for ID collisions across a batch needs to be handled in the import tool.

After the initial ingest, data checks, and mapping, User can separately initiate just the import of new records into CSpace, just the update of existing records in CSpace, or both in one step.

Here I am imagining something like three buttons: Add New, Update Existing, Do Both

User can specify (and save) default import batch preferences that will be applied to all their import jobs unless other options are selected. If they do not specify default preferences, the system defaults will be applied.

These include: repeating field value delimiter, repeating field-in-subgroup delimiter, date order (month-day-year or day-month-year). These preferences are not specific to a given record type, profile, or job.

User can specify (and save/reuse) profile/record type/job level preferences that may override their default preferences.

These include default field values and field-specific data transformations, as well as the default batch preference

User can validate a CSV file against an existing import Profile

This validation checks that required fields are included in CSV and populated in all rows. Presence of invalid data means the data cannot be mapped. Workflow stops until data is remediated.

User can receive clear error messaging for invalid data

Error specifies what field is missing, and, if field is not missing but is unpopulated, says what row is missing the value

User can receive clear data quality warnings prior to the mapping process. “Clear” means the user is told what field and what row contains the possible issue. User can opt to proceed with mapping data that receives a warning.

These warnings include: use of non-established values in option-list source fields; use of non-established vocabulary or authority terms; data type mismatches; uneven numbers of values in repeating field groups

After receiving validation errors and/or data quality warnings, user can export CSV of the rows having errors or warnings.

This CSV contains the original rows/columns, plus two additional columns: errors and warnings

User can export CSV of the rows having NO errors or warnings.

If user receives validation errors and/or data quality warnings, they can choose to proceed with processing only the rows without errors, or only rows without errors or warnings.

User can generate an CSV import template for each record type supported by the import tool

User can view a progress indicator for imports or updates in progress

User can delete a file that has been successfully imported

User can “redo” or “undo” any step in the process

JL: except to the extent that changes are necessarily permanent??

User can mark a data import batch as “complete and immutable” (i.e. “archived”)

JL: What should happen in that case? Should the batch be hidden somewhere to reduce clutter?

User can obtain stats/analytics about their batch(es): count, type/token counts, report on valid/invalid values, etc.

User can (directly) add new authority and vocabulary terms in support of adds and updates of records in a batch that use them.

Users can download missing/invalid terms from a batch

JL: in order to analyze/correct/re-upload them.

Users can download valid terms and other values from a batch

On every screen where some set of records is listed/displayed, User can click “Select All”, “Select None” or individual checkboxes to select individual records for operations such as Export to CSV, Remove from this batch, etc.