{"type":"doc","content":[{"type":"layoutSection","content":[{"type":"layoutColumn","attrs":{"width":33.33},"content":[{"type":"paragraph","content":[{"text":"Table of Contents","type":"text","marks":[{"type":"strong"}]}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"2"},"maxLevel":{"value":"2"},"outline":{"value":"false"},"style":{"value":"disc"},"type":{"value":"list"},"printable":{"value":"true"}},"macroMetadata":{"macroId":{"value":"beba9063-27af-481a-b1b6-4f53f95d2c58"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"9a67f727-9c6f-45db-a8bd-48ce54138ff5"}}]},{"type":"layoutColumn","attrs":{"width":66.66},"content":[{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"All CollectionSpace records are defined as XML documents. To ingest data via the API (using the CSV Importer or any other tool), you must provide an XML record.","type":"text"}]},{"type":"paragraph","content":[{"text":"To understand data in CollectionSpace today, one must have some understanding of XML concepts.","type":"text"}]},{"type":"paragraph","content":[{"text":"One of the reasons Nuxeo was chosen as the backend for CollectionSpace was its ability to be given arbitrary XML record type definitions, and handle most of the work of (1) generating a database structure for persisting data from XML documents; (2) running an API with different record types as endpoints; and (3) providing abstractions and interface for building a user-facing application.","type":"text"}]}]}]}]},{"type":"rule"},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"This page is a rough draft. It also has not yet been through technical review.","type":"text"}]},{"type":"paragraph","content":[{"text":"Initial form is a Kristina-brain-dump of information learned over years of building/maintaining tools to support the process of converting user data into CollectionSpace XML, and to generate information about the CollectionSpace data model in a way that humans can review it.","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Namespaces","type":"text"}]},{"type":"paragraph","content":[{"text":"XML is very modular, in that you can mix and match elements from different schema in the same document. CollectionSpace was initially intended to also be very modular, with a base “core” implementation to which extensions could be added in a mix-and-match fashion to meet the needs of different kinds of museum.","type":"text"}]},{"type":"paragraph","content":[{"text":"In any standard or system intended to modular in this way, it is important to ensure that elements with the same name, defined in different schema or extensions, do not conflict or overwrite each other. A common approach to this is the use of namespaces. This is what allows the community-supported anthro collectionobject record to support two fields defined with the element name “reference”. In the underlying XML document, a record with both values populated looks like:","type":"text"}]},{"type":"codeBlock","content":[{"text":" \n refTest1\n \n \n urn:cspace:anthro.collectionspace.org:citationauthorities:name(citation):item:name(Book11741991095945)'Book\n 1'\n p. 15\n \n \n \n \n \n \n 3\n Article 2\n urn:cspace:anthro.collectionspace.org:taxonomyauthority:name(taxon):item:name(Caracalcaracal1741991165486)'Caracal\n caracal'\n false\n \n \n ","type":"text"}]},{"type":"paragraph","content":[{"text":"The two ","type":"text"},{"text":"reference","type":"text","marks":[{"type":"code"}]},{"text":" elements do not collide because one is defined in the ","type":"text"},{"text":"collectionobjects_common","type":"text","marks":[{"type":"code"}]},{"text":" namespace and one is defined in ","type":"text"},{"text":"collectionobjects_naturalhistory_extension","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Xpaths / hierarchical organization of record data","type":"text"}]},{"type":"paragraph","content":[{"text":"The other reason that the two ","type":"text"},{"text":"reference","type":"text","marks":[{"type":"code"}]},{"text":" elements above do not collide is that, in XML, it is assumed that the way to unambiguously refer to an element is by the path of hierarchical elements one traverses to find that element.","type":"text"}]},{"type":"paragraph","content":[{"text":"/referenceGroupList/referenceGroup/reference","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"/taxonomicIdentGroupList/taxonomicIdentGroup/reference","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":" The two elements above could be distinguished by having different paths, even without considering namespaces. HOWEVER, in these paths, we don’t know if this is a ","type":"text"},{"text":"reference","type":"text","marks":[{"type":"code"}]},{"text":" element defined once for use in different contexts, or two separately defined elements with the same name.","type":"text"}]},{"type":"paragraph","content":[{"text":"Only by combining the use of namespaces with the use of full path to a field element can fields be identified unambiguously.","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"The actual XML xpaths to these fields are thus:","type":"text"}]},{"type":"paragraph","content":[{"text":"/document/collectionobjects_common/referenceGroupList/referenceGroup/reference","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"/document/collectionobjects_naturalhistory_extension/taxonomicIdentGroupList/taxonomicIdentGroup/reference","type":"text","marks":[{"type":"code"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Namespaces + paths in action","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"In JIRA","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://collectionspace.atlassian.net/browse/DRYD-113"}},{"text":" suggests using just namespace + individual field name to refer to fields.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"In the initial ToolTips output format: namespace/path ","type":"text"},{"text":"as used","type":"text","marks":[{"type":"em"}]},{"text":" only","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":714,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":714,"alt":"image-20250314-233817.png","id":"8ae40587-49d5-41ee-bc96-a973ae09aeb5","collection":"contentId-3579117569","type":"file","height":308}},{"type":"caption","content":[{"text":"The ID and Name columns of an initial prototype of data model info output from CollectionSpace","type":"text"}]}]},{"type":"paragraph","content":[{"text":"The full path to each field is used as the ID in the initial ToolTip data entry prototype Google Sheets Ray produced.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"In the “all fields” CSVs produced by cspace-config-untangler: namespace/path ","type":"text"},{"text":"as defined","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":" versus ","type":"text"},{"text":"as used","type":"text","marks":[{"type":"strong"},{"type":"em"}]}]},{"type":"paragraph","content":[{"text":"Why do the non-”friendly” versions of the Untangler’s “all-fields” spreadsheets lead with so much namespace, id, and path junk? Because it is really important for analyzing the data model!","type":"text"}]},{"type":"paragraph","content":[{"text":"As an example, let’s look at the ","type":"text"},{"text":"addressCountry","type":"text","marks":[{"type":"code"}]},{"text":" field in the core, anthro, and publicart profiles:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"wide","width":800,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":1410,"alt":"image-20250315-000537.png","id":"7e70c03f-3897-4fbd-bcdc-177b80e9ba43","collection":"contentId-3579117569","type":"file","height":330}},{"type":"caption","content":[{"text":"Untangler “all fields” CSV, filtered to the addressCountry field in anthro, core, and public art profiles","type":"text"}]}]},{"type":"paragraph","content":[{"text":"This filtered view of the “all fields” CSV shows 13 usages of the ","type":"text"},{"text":"addressCountry","type":"text","marks":[{"type":"code"}]},{"text":" field across 3 profiles.","type":"text"}]},{"type":"paragraph","content":[{"text":"The non-namespace part of the path-to-field is captured in the ","type":"text"},{"text":"xml_path","type":"text","marks":[{"type":"code"}]},{"text":" column.","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"As-defined field analysis","type":"text"}]},{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"namespace_for_id","type":"text","marks":[{"type":"code"}]},{"text":" column attempts to capture where the field represented by the row is actually defined. This allows us to see there are actually two different ","type":"text"},{"text":"addressCountry","type":"text","marks":[{"type":"code"}]},{"text":" fields used in 13 different places. One is an extension in core (defined ","type":"text"},{"text":"in application","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/collectionspace/application/blob/main/tomcat-main/src/main/resources/defaults/base-other-address.xml"}}]},{"text":"; ","type":"text"},{"text":"cspace-ui","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/collectionspace/cspace-ui.js/tree/main/src/plugins/extensions/address"}}]},{"text":"; not found in services). The other is a sub-record type (contact) defined in ","type":"text"},{"text":"application","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/collectionspace/application/blob/2ee4ccf299194be471699b7dbb44ff5c183598f0/tomcat-main/src/main/resources/defaults/base-authority-contact.xml#L11"}}]},{"text":"; ","type":"text"},{"text":"cspace-ui","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/collectionspace/cspace-ui.js/tree/main/src/plugins/recordTypes/contact"}}]},{"text":"; ","type":"text"},{"text":"services","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/collectionspace/services/tree/d44481eb01cfd8d4db5dfb8d5da3d0b2167e312f/services/contact"}}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"The 2 fields are indeed defined differently: the extension-defined field is populated from place authorities, while the contact subrecord-defined field is populated from a static term list hard-coded with country codes. Further, we can see that the publicart profile overrides which authority vocabularies/subtypes control the extension-defined field.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"wide","width":800,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":1866,"alt":"image-20250315-003009.png","id":"34faca2d-6c8a-4724-82b8-b6631ff684c7","collection":"contentId-3579117569","type":"file","height":660}},{"type":"caption","content":[{"text":"Data source info for the same fields, from the same CSV","type":"text"}]}]},{"type":"paragraph","content":[{"text":" There are many uses for an as-defined view of fields. Consider also the power of this in writing field documentation/tool tip content. Across all profiles there are 37 uses of these 2 fields. I know I would rather document 2 fields rather than 37!","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"As-used field analysis","type":"text"}]},{"type":"paragraph","content":[{"text":"Of course there are also many reasons we need to get the details of each use of a defined field.","type":"text"}]},{"type":"paragraph","content":[{"text":"A user wants to know what fields are in Person records in their domain profile. They don’t care where that field is defined.","type":"text"}]},{"type":"paragraph","content":[{"text":"The CSV Importer CSV templates and JSON “record mappers” (instructions on how to transform CSV data to CS XML) for each record type need to have all fields for each record type, with the “as-used” namespace and path of the field.","type":"text"}]},{"type":"paragraph","content":[{"text":"Whereas the full “as-defined” namespace + path id for the ","type":"text"},{"text":"addressCountry","type":"text","marks":[{"type":"code"}]},{"text":" field in core place and work records would be:","type":"text"}]},{"type":"paragraph","content":[{"text":"ext.address/addrGroupList/addrGroup/addressCountry","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"This field would have two separate “as-used” namespace + path ids in core:","type":"text"}]},{"type":"paragraph","content":[{"text":"ns2:places_common/addrGroupList/addrGroup/addressCountry","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"ns2:works_common/addrGroupList/addrGroup/addressCountry","type":"text","marks":[{"type":"code"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"What are the types of data structures in CollectionSpace?","type":"text"}]},{"type":"paragraph","content":[{"text":"The main field structures are:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"NONREPEATABLE STANDALONE FIELD","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Only one value can be recorded in field AND field is NOT semantically/logically connected to other fields in the data structure.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Object record examples: objectNumber, collection, descriptionLevel, recordStatus, distinguishing features.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"REPEATABLE STANDALONE FIELD:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Multiple values can be recorded in the field AND field is NOT semantically/logically connected to other fields in the data structure.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Object record examples: namedCollection, briefDescription, comment, responsibleDepartment.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"REPEATABLE FIELD GROUP:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Two or more single-value fields semantically/logically connected to other fields in data structure (e.g. nested hierarchically under the same element) AND multiple instances of the field grouping can be added/populated","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Object record examples: Other number value and other number type; Annotation type, date, author, note.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"REPEATABLE FIELD GROUP WITHIN REPEATABLE FIELD GROUP","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"A repeatable field group, but instead of there just being single value fields in each iteration of the group, there is also a nested repeatable field group","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Object record examples: The Translation repeatable subgroup within the repeatable Title group; Dimension subgroup repeatable within MeasuredPart group","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"REPEATABLE FIELD (NOT FIELD GROUP) WITHIN REPEATABLE FIELD GROUP:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"A repeatable field group, but instead of there just being single value fields in each iteration of the group, there is also a nested repeatable standalone field","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Example: Currently this structure only exists in the Associated Authorities extension. See the repeatable associatedPersonCitation field within the associatedPerson repeatable group where all other fields are single value.","type":"text"}]}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"How is data structure related to where you will find/access field values in the database?","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"NONREPEATABLE STANDALONE FIELD:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Field value stored in main table for record type namespace in which field is used (collectionobjects_common, collectionobjects_fineart, etc.)","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"REPEATABLE STANDALONE FIELD:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Field value stored in separate table joinable directly to main record type table (recordtypes_common) on id without hierarchy table","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Table name often follows pattern: ","type":"text"},{"text":"#{namespace where used}_#{pluralized fieldname}","type":"text","marks":[{"type":"code"}]},{"text":", eg ","type":"text"},{"text":"collectionobjects_common_responsibledepartments","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"REPEATABLE FIELD GROUP:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Each field group is in a separate table in db, joinable to main recordtype table via hierarchy table","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"REPEATABLE FIELD GROUP WITHIN REPEATABLE FIELD GROUP:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Database tables: titlegroup (related to collectionobjects_common through hierarchy); titletranslationsubgroup (not sure but betting this is related to titlegroup as hierarchy.parentid.)","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"REPEATABLE FIELD (NOT FIELD GROUP) WITHIN REPEATABLE FIELD GROUP:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Tables: assocpersongroup, assocpersoncitations. I have not actually needed to write a query to get at these","type":"text"}]}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"How do these structures appear in the UI and why should I not trust my eyes?","type":"text"}]},{"type":"paragraph","content":[{"text":"todo","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"How to determine the structure of a field from “all fields” CSV?","type":"text"}]},{"type":"paragraph","content":[{"text":"todo","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"What is the underlying XML structure of each pattern?","type":"text"}]},{"type":"paragraph","content":[{"text":"todo","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Structured dates","type":"text"}]},{"type":"paragraph","content":[{"text":"todo","type":"text"}]}],"version":1}

Browser not supported