Detail overload for experiences with "java.lang.NullPointerException" error

Note: The following are notes made by a novice deployer as learning how not to break things in CollectionSpace. The notes below ramble and include a number of irrelevant details, tangential thoughts, dead ends, and loose strings. These should be edited for relevancy now that the problem has apparently been identified.

"Status:400:Create request failed: javax.ejb.EJBException: java.lang.NullPointerException"

Status: Workaround found. Progress is blocked until this problem is resolved.
Scope: This appears to affect only collectionobject. Other types of records and authorities save without experiencing this problem.
Background: Deployment was working nearly perfectly, but one repeating custom field would not save data to database. The field was named "nagpraInventory", and I had named the repeating list "nagpraInventorys" to follow a pattern I found (in collectionobjects_common.xsd) of appending "s" to the field:

    <xs:element name="briefDescriptions" type="xs:string">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="briefDescription" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
    <xs:element name="comments">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="comment" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>

Some time after creating nagpraInventory and nagpraInventorys, I changed the name of the field to better reflect its contents and to prepare for an upcoming fuller schema for information about NAGPRA inventories. I renamed "nagpraInventory" as "nagpraInventoryName", but neglected to change "nagpraInventorys" to "nagpraInventoryNames". Still, the only effect this mistake appeared to have was possibly being the reason that data entered in this field was not saved to the database. The rest of the fields saved to the database and no error message was displayed when saving the record.

When trying to find the reason why nagpraInventory data was not being saved to the database, I saw that I had not renamed "nagpraInventorys" as "nagpraInventoryNames". I changed the name in collectionobjects_anthropology.xsd and also in our-tenant-tenant.xml. The relevant code now read as follows for our-tenant-tenant.xml:

          <section id="objectNagpraComplianceInformation">
            	<repeat id="nagpraInventoryNames" section="anthropology">
	            	<field id="nagpraInventoryName" section="anthropology" seperate_ui_container="true">
	            		<selector>collection-object-nagpraInventoryName</selector>
	            		<options>

and the relevant code in collectionobjects_anthropology.xsd read as follows (non-relevant lines omitted):

	<xs:element name="collectionobjects_anthropology">
		<xs:complexType>
			<xs:sequence>
				<xs:element name="nagpraInventoryName" type="xs:string"/>
			</xs:sequence>
		</xs:complexType>
	</xs:element>

	<xs:element name="nagpraInventoryNames">
		<xs:complexType>
			<xs:sequence>
				<xs:element name="nagpraInventoryName" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:complexType>
	</xs:element>

It turns out that while I thought I made the schema change to the anthropology domain extension schema (services/v1.8.2/services/collectionobject/3rdparty/nuxeo-platform-collectionobject-pahma/src/main/resources/schemas/collectionobjects_anthropology.xsd), I actually made the change instead to the JAXB version of the schema (services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_anthropology.xsd). The (primary) anthropology domain extension schema was left unchanged:

	<xs:element name="nagpraInventorys">
		<xs:complexType>
			<xs:sequence>
				<xs:element name="nagpraInventory" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:complexType>
	</xs:element>

I edited the primary version of collectionobjects_anthropology.xsd to align with the JAXB version and ran "mvn clean install -DskipTests" for just the collectionobject service, following the suggestion given here:

If you're customizing only a single record type, it's often feasible to run

mvn clean install -DskipTests

just within the services module whose record type you're customizing; e.g. within services/collectionobject, services/intake, or services/media, which is faster than running it from the top of the services source code tree. You can then change the tenant bindings, as needed, to reflect that change to the schema, and run

ant undeploy deploy

from the top of the services tree.

After running just the mvn clean install -DskipTests, however, I got a BUILD ERROR message in the terminal:

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Unable to parse input schema(s). Error messages should have been provided.
[INFO] ------------------------------------------------------------------------

Looking at the messages in the terminal window, I saw these errors (excerpted):

[ERROR] Error while parsing schema(s).Location [ file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_anthropology.xsd{23,55}].
org.xml.sax.SAXParseException: s4s-elt-must-match.1: The content of 'sequence' must match (annotation?, (element | group | choice | sequence | any)*). A problem was found starting at: complexType.

[ERROR] Error while parsing schema(s).Location [ file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_anthropology.xsd{28,51}].
org.xml.sax.SAXParseException: s4s-elt-must-match.1: The content of 'sequence' must match (annotation?, (element | group | choice | sequence | any)*). A problem was found starting at: complexType.

[ERROR] Error while parsing schema(s).Location [ file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_anthropology.xsd{36,56}].
org.xml.sax.SAXParseException: s4s-elt-must-match.1: The content of 'sequence' must match (annotation?, (element | group | choice | sequence | any)*). A problem was found starting at: complexType.

[ERROR] Error while parsing schema(s).Location [ file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_anthropology.xsd{41,52}].
org.xml.sax.SAXParseException: s4s-elt-must-match.1: The content of 'sequence' must match (annotation?, (element | group | choice | sequence | any)*). A problem was found starting at: complexType.

[ERROR] Error while parsing schema(s).Location [ file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_anthropology.xsd{20,85}].
org.xml.sax.SAXParseException: src-resolve: Cannot resolve the name 'nagpraClaimsMadeGroupList' to a(n) 'type definition' component.

[ERROR] Error while parsing schema(s).Location [ file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_anthropology.xsd{21,87}].
org.xml.sax.SAXParseException: src-resolve: Cannot resolve the name 'accessRestrictionGroupList' to a(n) 'type definition' component.

Looking at the the JAXB version of the schema, I saw that while it was well-formed, some complex types (those specified in the above error messages) were included within the <xs:sequence> structure (inside the <xs:element name="collectionobjects_anthropology"> structure), while others were enumerated later in the JAXB schema. I moved the former <xs:complexType> definitions to be with the latter, and ran "mvn clean install -DskipTests" again, this time resulting in a BUILD SUCCESSFUL message.

I restarted both servers and created a new cataloging record with only the Museum number (Identification number) filled in. Upon save, however, I was still faced with the same "Status:400:Create request failed: javax.ejb.EJBException: java.lang.NullPointerException" that initiated this debugging exercise.

I ran "ant undeploy deploy" from the top of the services/v1.8.2 tree, saw no error messages, and got nothing but BUILD SUCCESSFUL messages. In fact, the only thing that wasn't perfect about this build was the handful of these messages:

  [exec] [WARNING] DEPRECATED [systemProperties]: Use systemPropertyVariables instead.

In the interest of being thorough, I first tested the deployment by just reloading the page and re-logging in (same error), then I tried restarting just the CSpace server (same error), and then I restarted both Nuxeo and CSpace servers (same error).

Looking at the log messages in the CSpace terminal window, I found the following errors (excerpted):

12:00:32,635 ERROR [RepositoryJavaClientImpl] Could not close the repository session
javax.ejb.NoSuchEJBException: Could not find stateful bean: 1y103q52-5uld4n-gr2c37uj-1-gr2dnxf8-n
        at org.jboss.ejb3.cache.simple.SimpleStatefulCache.get(SimpleStatefulCache.java:390)
( . . . )
12:00:32,784 ERROR [CollectionObjectResource] org.collectionspace.services.collectionobject.CollectionObjectResource detail: org.collectionspace.services.common.document.DocumentException: javax.ejb.EJBException: java.lang.NullPointerException
        at org.collectionspace.services.nuxeo.client.java.RepositoryJavaClientImpl.create(RepositoryJavaClientImpl.java:172)
( . . . )
Caused by: javax.ejb.EJBException: java.lang.NullPointerException
        at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:63)
( . . . )
Caused by: java.lang.NullPointerException
        at org.nuxeo.ecm.core.api.impl.DataModelImpl.getSchema(DataModelImpl.java:90)
( . . . )
12:00:32,791 ERROR [CollectionObjectResource] org.collectionspace.services.collectionobject.CollectionObjectResource detail: javax.ws.rs.WebApplicationException
        at org.collectionspace.services.common.AbstractCollectionSpaceResourceImpl.bigReThrow(AbstractCollectionSpaceResourceImpl.java:419)
( . . . )
12:00:32,793 ERROR [SynchronousDispatcher] failed to execute
javax.ws.rs.WebApplicationException
        at org.collectionspace.services.common.AbstractCollectionSpaceResourceImpl.bigReThrow(AbstractCollectionSpaceResourceImpl.java:419)

Next, I used Firebug to examine the POST that was being generated and sent when I was attempting to save a record. This is the entire Post source, unabridged except for the addition of whitespace for readability:

{"termsUsed":[],
	"relations":{},
	"fields":{
		"inscriptionContentMethod":"",
		"nagpraApplicabilitys":[{
			"nagpraApplicability":"nonNagpra","_primary":true}],
		"inscriptionDescriptionPosition":"",
		"culturalCareNotes":[{"_primary":true}],
		"fieldCollectionMethods":[{
			"fieldCollectionMethod":"","_primary":true}],
		"assocEventPeoples":[{"_primary":true}],
		"nagpraCulturalDeterminations":[{"_primary":true}],
		"responsibleDepartments":[{
			"responsibleDepartment":"","_primary":true}],
		"nagpraInventoryNames":[{
			"nagpraInventoryName":"notOnAnInventory","_primary":true}],
		"ownershipAccess":"",
		"contentPositions":[{
			"contentPosition":"","_primary":true}],
		"inscriptionContentPosition":"",
		"styles":[{"_primary":true}],
		"ownershipExchangeMethod":"",
		"objectNameGroup":[{
			"objectNameCurrency":"current",
			"objectNameLanguage":"eng",
			"objectNameSystem":"",
			"objectNameType":"simple",
			"objectNameLevel":"wholeObject","_primary":true}],
		"ownershipExchangePriceCurrency":"",
		"contentObjectGroup":[{
			"contentObjectType":"","_primary":true}],
		"owners":[{"_primary":true}],
		"inscriptionDescriptionDate":{
			"dateEarliestSingleQualifier":"none",
			"dateEarliestSingleEra":"ce",
			"dateEarliestSingleCertainty":"none",
			"dateLatestEra":"ce",
			"dateLatestCertainty":"none",
			"dateLatestQualifier":"none",
			"dateEarliestSingleQualifierUnit":"years",
			"dateLatestQualifierUnit":"years"},
		"objectProductionPlaceGroup":[{
			"objectProductionPlaceRole":"","_primary":true}],
		"referenceGroup":[{"_primary":true}],
		"objectProductionReasons":[{"_primary":true}],
		"contentLanguages":[{
			"contentLanguage":"eng","_primary":true}],
		"pahmaAltNumGroup":[{
			"pahmaAltNumType":"notStated","_primary":true}],
		"assocPlaceGroup":[{
			"assocPlaceType":"","_primary":true}],
		"collection":"",
		"contentDate":{
			"dateEarliestSingleQualifier":"none",
			"dateEarliestSingleEra":"ce",
			"dateEarliestSingleCertainty":"none",
			"dateLatestEra":"ce",
			"dateLatestCertainty":"none",
			"dateLatestQualifier":"none",
			"dateEarliestSingleQualifierUnit":"years",
			"dateLatestQualifierUnit":"years"},
		"inscriptionDescriptionType":"",
		"assocConceptGroup":[{
			"assocConceptType":"","_primary":true}],
		"assocPersonGroup":[{
			"assocPersonType":"","_primary":true}],
		"inscriptionContentScript":"",
		"contentEventNameGroup":[{
			"contentEventNameType":"","_primary":true}],
		"ageQualifier":"",
		"objectStatusList":[{
			"objectStatus":"","_primary":true}],
		"assocEventNameType":"",
		"techniqueGroup":[{
			"techniqueType":"","_primary":true}],
		"assocEventPersons":[{"_primary":true}],
		"comments":[{"_primary":true}],
		"briefDescriptions":[{"_primary":true}],
		"objectProductionDate":{
			"dateEarliestSingleQualifier":"none",
			"dateEarliestSingleEra":"ce",
			"dateEarliestSingleCertainty":"none",
			"dateLatestEra":"ce",
			"dateLatestCertainty":"none",
			"dateLatestQualifier":"none",
			"dateEarliestSingleQualifierUnit":"years",
			"dateLatestQualifierUnit":"years"},
		"dimensionGroup":[{
			"dimension":"",
			"measurementUnit":"",
			"valueQualifier":"","_primary":true}],
		"materialGroup":[{"_primary":true}],
		"taxonomicIdentGroup":[{
			"identKind":"",
			"qualifier":"","_primary":true}],
		"usageGroup":[{"_primary":true}],
		"assocActivityGroup":[{
			"assocActivityType":"","_primary":true}],
		"assocCulturalContextGroup":[{
			"assocCulturalContextType":"","_primary":true}],
		"ageUnit":"",
		"pahmaEthnographicFileCodes":[{
			"pahmaEthnographicFileCode":"","_primary":true}],
		"contentOrganizations":[{"_primary":true}],
		"ownershipCategory":"",
		"ownershipDates":{
			"dateEarliestSingleQualifier":"none",
			"dateEarliestSingleEra":"ce",
			"dateEarliestSingleCertainty":"none",
			"dateLatestEra":"ce",
			"dateLatestCertainty":"none",
			"dateLatestQualifier":"none",
			"dateEarliestSingleQualifierUnit":"years",
			"dateLatestQualifierUnit":"years"},
		"contentActivities":[{"_primary":true}],
		"otherNumber":[{"numberType":"notStated"}],
		"contentOtherGroup":[{"_primary":true}],
		"contentPersons":[{"_primary":true}],
		"objectComponentGroup":[{"_primary":true}],
		"contentScripts":[{
			"contentScript":"","_primary":true}],
		"colors":[{"_primary":true}],
		"contentConcepts":[{"_primary":true}],
		"fieldColEventNames":[{"_primary":true}],
		"inscriptionContentType":"",
		"assocEventPlaces":[{"_primary":true}],
		"titleGroup":[{
			"titleTranslationLanguage":"eng",
			"titleType":"",
			"titleLanguage":"eng","_primary":true}],
		"contentPlaces":[{"_primary":true}],
		"phase":"",
		"inscriptionContentLanguage":"eng",
		"assocPeopleGroup":[{
			"assocPeopleType":"","_primary":true}],
		"contentPeoples":[{"_primary":true}],
		"sex":"",
		"inscriptionDescriptionMethod":"",
		"assocDateGroup":[{
			"assocDateType":"","_primary":true}],
		"recordStatus":"minimalInProgress",
		"technicalAttributeGroup":[{
			"technicalAttributeMeasurement":"",
			"technicalAttribute":"",
			"technicalAttributeMeasurementUnit":"","_primary":true}],
		"pahmaNagpraCodeLegacys":[{
			"pahmaNagpraCodeLegacy":"noCode","_primary":true}],
		"objectProductionOrganizationGroup":[{
			"objectProductionOrganizationRole":"","_primary":true}],
		"fieldCollectionSources":[{"_primary":true}],
		"forms":[{"form":"","_primary":true}],
		"objectProductionPersonGroup":[{
			"objectProductionPersonRole":"","_primary":true}],
		"fieldCollectors":[{"_primary":true}],
		"assocOrganizationGroup":[{
			"assocOrganizationType":"","_primary":true}],
		"objectProductionPeopleGroup":[{
			"objectProductionPeopleRole":"","_primary":true}],
		"assocEventOrganizations":[{"_primary":true}],
		"accessRestrictionGroup":[{
			"accessRestrictionType":"","_primary":true}],
		"nagpraClaimsMadeGroup":[{
			"nagpraClaimStatus":""}],
		"inscriptionContentDate":

I suspect that something is wrong with one or more of the above fields. To test this hypothesis, I made an abridged version of the above list, containing only those fields I had worked on in any way prior to the onset of the error I'm trying to diagnose:

lifierUnit":"years"},
		"assocObjectGroup":[{
		"assocObjectType":"","_primary":true}],
		"objectNumber":"12"},
	"csid":""}

I suspect that something is wrong with one or more of the above fields. To test this hypothesis, I made an abridged version of the above list, containing only those fields I had worked on in any way prior to the onset of the error I'm trying to diagnose:

"nagpraApplicabilitys":[{
	"nagpraApplicability":"nonNagpra","_primary":true}],

"culturalCareNotes":[{"_primary":true}],

"nagpraCulturalDeterminations":[{"_primary":true}],

"nagpraInventoryNames":[{
	"nagpraInventoryName":"notOnAnInventory","_primary":true}],

"pahmaAltNumGroup":[{
	"pahmaAltNumType":"notStated","_primary":true}],

"pahmaEthnographicFileCodes":[{
	"pahmaEthnographicFileCode":"","_primary":true}],

"otherNumber":[{"numberType":"notStated"}],

"pahmaNagpraCodeLegacys":[{
	"pahmaNagpraCodeLegacy":"noCode","_primary":true}],

"accessRestrictionGroup":[{
	"accessRestrictionType":"","_primary":true}],

"nagpraClaimsMadeGroup":[{
	"nagpraClaimStatus":""}],

In checking the JAXB version of the collectionobjects_pahma.xsd, I noticed that in cloning "pahmaAltNumGroup" from "otherNumbers", I inadvertently left the type as:

type="ns:pahmaAltNumGroup"

Instead of being "ns", it should have been "xs". This had already been corrected in the primary version of the collectionobjects_pahma.xsd schema.

I attempted a maven clean install of just collectionobject, but got this error:

[ERROR] Error while parsing schema(s).Location [ file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_pahma.xsd{30,77}].
org.xml.sax.SAXParseException: src-resolve.4.2: Error resolving component 'xs:pahmaAltNumGroupList'. It was detected that 'xs:pahmaAltNumGroupList' is in namespace 'http://www.w3.org/2001/XMLSchema', but components from this namespace are not referenceable from schema document 'file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_pahma.xsd'. If this is the incorrect namespace, perhaps the prefix of 'xs:pahmaAltNumGroupList' needs to be changed. If this is the correct namespace, then an appropriate 'import' tag should be added to 'file:/Users/michaelblack/Documents/CSpaceSourceCode/svn4/trunk/services/v1.8.2/services/collectionobject/jaxb/src/main/resources/collectionobjects_pahma.xsd'.

Really? That error message needs an error message. On a hunch supplied by the word "import" in the above error message, I thought that perhaps a maven clean install was insufficient for the task at hand, and that perhaps "ant undeploy deploy create_db import" was the way to go. I ran this ant at the root of services/v1.8.2, and restarted both servers. And guess what? The error is still there.

So maybe pahmaAltNumGroupList really should be of type "ns". I made that change to both primary and JAXB versions of the PAHMA collectionobjects schema and ran "mvn clean install -DskipTests" on just the collectionobject service. Build completed without errors this time. Both servers restarted, and still I get the same error when I try to save a cataloging record.

Time to nuke the pahmaAltNumGroup fields and structure. Commented out all references to field and structure in both versions of the PAHMA schema, in CatalogingTemplate.html, and in our-tenant-tenant.xml. Used "ant undeploy deploy create_db import" on services/v1.8.2 root, and restarted both servers.