I am trying to apply a patch to our Test environment running Dspace 5.1 (with some additional modules) to fix a OAI harvesting issue due to improper date granularity. The issue is DS-2542 XOAI does not support non granular YYYY-MM-DD harvesting properly
First I need to grab the patch for Dspace (dspace-oai), from https://github.com/DSpace/DSpace/pull/912. That gives me the following patch (as a diff), which I put in /tmp/
diff --git a/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java b/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
index 1995fc0..cdb17d9 100644
--- a/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
+++ b/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
@@ -50,6 +50,11 @@ public boolean isShown(DSpaceItem item)
public SolrFilterResult buildSolrQuery()
{
String format = dateProvider.format(date).replace("Z", ".999Z"); // Tweak to set the millisecon
+ if (format.substring(11, 19).equals("00:00:00"))
+ {
+ format = format.substring(0, 11) + "23:59:59" + format.substring(19);
+ }
+
return new SolrFilterResult("item.lastmodified:[* TO "
+ ClientUtils.escapeQueryChars(format) + "]");
}
diff --git a/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java b/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
index e968414..b73955d 100644
--- a/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
+++ b/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
@@ -31,8 +31,7 @@ public static String format(Date date)
}
public static String format(Date date, boolean init)
{
- SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.'000Z'");
- if (!init) sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.'999Z'");
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");
// We indicate that the returned date is in Zulu time (UTC) so we have
// to set the time zone of sdf correct.
sdf.setTimeZone(TimeZone.getTimeZone("ZULU"));
I have applied the patch using the following sequence of commands:
sudo su - builder
cd /usr/local/src/
git clone 'https://github.com/lyncode/xoai.git' xoai
cd xoai/
git branch -a
git checkout 3.x
less pom.xml <-- verify that com.lyncode version is 3.2.10-SNAPSHOT
mvn package install
cd ..
cd dspace-src/
patch -p1 < /tmp/473f2faaba99671b55372bcca1604aea2acf9601.diff # (needed tweaking)
vim dspace-oai/pom.xml # change <xoai.version> from 3.2.9 to 3.2.10-SNAPSHOT
mvn package -U -P \!dspace-jspui,\!dspace-lni,\!dspace-rest,\!dspace-swordv2,\!dspace-rdf,\!dspace-xmlui-mirage2
exit
cd /usr/local/src/dspace-src/dspace/target/dspace-installer/
ant update
chown -Rc tomcat:tomcat /usr/local/dspace/
service tomcat6 restart
By post-patching results are the same as my pre-patching. Using 2015-04-07 still fails, but should pass after patching. 2015-04-07T00:00:00Z works the same and passes.
$ curl 'https://test-dspace.example.com/oai/request?verb=ListRecords&metadataPrefix=oai_dc&until=2015-04-07'
...<error code="badArgument">Invalid date given in until parameter</error></OAI-PMH>
$ curl 'https://test-dspace.example.com/oai/request?verb=ListRecords&metadataPrefix=oai_dc&until=2015-04-07T00:00:00Z'
... completeListSize="3731" cursor="0">oai_dc//2015-04-07T00:00:00.000Z//100</resumptionToken></ListRecords></OAI-PMH>
I have seen, and read multiple times, a DSpace - Tech email entitled 'Building DSpace after bug fix (DS-2542)', and although I have done everything as I should have according to that email trail (including both the "proper" -- edit your POM -- and "quick and dirty" -- replace the jar -- methods listed therein), the problem remains.
xoai-3.2.10-SNAPSHOT.jar in /usr/local/dspace/lib/ and /usr/local/dspace/webapps/oai/WEB-INF/lib/~/.m2/repository and build again from scratch (including all modules).I'm confident that the jar is being created as expected, and is well installed, and that there is no old jar in play.
I'm not confident that the code I think should be run is actually getting run, or if something else is overshadowing the code... and I don't know how to diagnose that in Tomcat.
Is there a command (presumably dspace dsrun ...) I could use to replicate this test outside of Tomcat?
I should point out that I'm a Systems Engineer, not a Java Developer.
Added at the end of the appender section for dspace.log
log4j.logger.com.lyncode.xoai=DEBUG, A1
log4j.logger.org.dspace.xoai=DEBUG, A1
This let me see logs such as:
2015-07-28 02:34:07,222 DEBUG org.dspace.xoai.services.impl.solr.DSpaceSolrServerResolver @ Solr Server Initialized
2015-07-28 02:34:07,233 DEBUG com.lyncode.xoai.dataprovider.OAIRequestParameters @ RootParameterMap 'until' = '2015-04-07'
and if I edit /usr/local/src/xoai/src/main/java/com/lyncode/xoai/dataprovider/OAIDataProvider.java, I can turn that into
2015-07-28 02:50:15,374 DEBUG com.lyncode.xoai.dataprovider.OAIRequestParameters @ CAMERON-RootParameterMap 'until' = '2015-04-07'
but adding CAMERON to be beginning of the seemingly only place where the error text is generated does not show a change in the test output (this in OAIDataProvider.java from that same directory):
throw new BadArgumentException("CAMERON Invalid date given in until parameter");
# git diff
diff --git a/build.properties b/build.properties
index 844cb93..09cb5c3 100644
--- a/build.properties
+++ b/build.properties
@@ -166,6 +166,7 @@ http.proxy.port = 3128
loglevel.other = WARN
# loglevel.other: Log level for other third-party tools/APIs used by DSpace
# Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
-loglevel.dspace = INFO
+#loglevel.dspace = INFO
+loglevel.dspace = DEBUG
# loglevel.dspace: Log level for all DSpace-specific code (org.dspace.*)
# Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
diff --git a/dspace-oai/pom.xml b/dspace-oai/pom.xml
index 6e56a52..136a10f 100644
--- a/dspace-oai/pom.xml
+++ b/dspace-oai/pom.xml
@@ -16,7 +16,7 @@
<!-- This is the path to the root [dspace-src] directory. -->
<root.basedir>${basedir}/..</root.basedir>
<spring.version>3.2.5.RELEASE</spring.version>
- <xoai.version>3.2.9</xoai.version>
+ <xoai.version>3.2.10-SNAPSHOT</xoai.version>
<jtwig.version>2.0.1</jtwig.version>
</properties>
diff --git a/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java b/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
index 1995fc0..cdb17d9 100644
--- a/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
+++ b/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
@@ -50,6 +50,11 @@ public class DateUntilFilter extends DSpaceFilter
public SolrFilterResult buildSolrQuery()
{
String format = dateProvider.format(date).replace("Z", ".999Z"); // Tweak to set the millisecon
+ if (format.substring(11, 19).equals("00:00:00"))
+ {
+ format = format.substring(0, 11) + "23:59:59" + format.substring(19);
+ }
+
return new SolrFilterResult("item.lastmodified:[* TO "
+ ClientUtils.escapeQueryChars(format) + "]");
}
diff --git a/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java b/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
index e968414..b73955d 100644
--- a/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
+++ b/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
@@ -31,8 +31,7 @@ public class DateUtils
}
public static String format(Date date, boolean init)
{
- SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.'000Z'");
- if (!init) sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.'999Z'");
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");
// We indicate that the returned date is in Zulu time (UTC) so we have
// to set the time zone of sdf correct.
sdf.setTimeZone(TimeZone.getTimeZone("ZULU"));
commit f6721be7ef5bf75d790e220ee81821e8eb70986e
Merge: dd9ef83 71ae14e
Author: João Melo <[email protected]>
Date: Thu Oct 9 18:33:57 2014 +0100
Merge pull request #34 from kosarko/dates_fix
from and until granularity
commit dd9ef830a16209e624bd03c3e91e7c5d7bdbd449
Merge: 8e778c2 c04b0bc
Author: João Melo <[email protected]>
Date: Thu Oct 9 18:33:29 2014 +0100
Merge pull request #33 from kosarko/filter_fix
fixed typo
commit 71ae14ed5ee9e6a5a6e3d6253424c212ac98081a
Author: Ondřej Košarko <[email protected]>
Date: Thu Oct 9 19:05:03 2014 +0200
from and until granularity
Regardles of the repository setting the from and until parameters must
handle day granularity. Error message fixed
commit c04b0bcd177007f618f18008e53a61a81aabdb06
Author: Ondřej Košarko <[email protected]>
Date: Thu Oct 9 19:01:08 2014 +0200
fixed typo
$ sudo updatedb
$ locate xoai- | grep '\.jar$' | grep -v bak | xargs ls -l
-rwxr-xr-x 1 tomcat tomcat 321775 Jul 28 03:04 /usr/local/dspace/lib/xoai-3.2.10-SNAPSHOT.jar
-rwxr-xr-x 1 tomcat tomcat 321775 Jul 28 03:04 /usr/local/dspace/webapps/oai/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder 321832 Jul 28 01:34 /usr/local/src/dspace-src/dspace/modules/oai/target/oai-5.1/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder 321832 Jul 28 01:34 /usr/local/src/dspace-src/dspace-oai/target/dspace-oai-5.1/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder 321832 Jul 28 01:34 /usr/local/src/dspace-src/dspace/target/dspace-installer/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder 321832 Jul 28 01:34 /usr/local/src/dspace-src/dspace/target/dspace-installer/webapps/oai/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder 321775 Jul 28 03:04 /usr/local/src/xoai/target/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder 1571092 Jul 28 03:04 /usr/local/src/xoai/target/xoai-3.2.10-SNAPSHOT-javadoc.jar
-rw-rw-r-- 1 builder builder 222305 Jul 28 03:04 /usr/local/src/xoai/target/xoai-3.2.10-SNAPSHOT-sources.jar
Verifying that tomcat should see the same (in case of broken links, of which there appear to be none)
$ find -L /usr/share/tomcat6 -type f -name 'xoai-*.jar' -print | xargs ls -l
-rwxr-xr-x 1 tomcat tomcat 321775 Jul 28 03:04 /usr/share/tomcat6/webapps/oai/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
All com.lyncode.xoai. classes come from file:/usr/local/dspace/webapps/oai/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
Many many thanks for reading,
Cameron
Have you tried clearing the oai response cache? I don't know whether this type of thing gets cached, but it's the only thing I can think of that you haven't mentioned yet.
[dspace]/bin/dspace oai clean-cache