Upgrade of OSS Nexus Repository Manager from 3.29.2 to 3.48.0 fails on exception during index creation

Question

I am trying to upgrade Nexus from version 3.29.2 to 3.48.0. For deployment of Nexus we use Official Docker container (originally 3.29.2; target version 3.48.0). Environment i test it on is:

OS: Ubuntu 22.04.2 LTS (Jammy Jellyfish)
Docker: 23.0.1 Used DB: Embedded OrientDB

Procedure used for upgrade (based on this):

1. Stop and remove current container using command

docker stop --time=120 nexus | xargs docker rm

2. Run new container with the same persistent data using command

docker run --name=nexus --volume /mnt/nexus-data:/nexus-data -d -p 8081:8081 sonatype/nexus3:3.48.0

Then the startup of a container fails with exit code 255. Complete output of docker logs nexus is here.

Seems like problem with index creation in component database according to this line

2023-03-09 13:24:16,103+0000 ERROR [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.index.OIndexUnique - $ANSI{green {db=component}} Exception during index 'asset_bucket_rid_idx' creation

First i tried to upgrade gradually, i.e. to apply same procedure mentioned above to upgrade e.g. from 3.29.2 to 3.30.0.

Here follows matrix of versions i tried to upgrade to (from 3.29.2) and corresponding errors that have arised.

Nexus version	Exception text
3.30.0	Exception during index 'deleted_blob_index_blobstore_blob_id_idx' creation
3.30.1	Exception during index 'deleted_blob_index_blobstore_blob_id_idx' creation
3.33.1	Exception during index 'deleted_blob_index_blobstore_blob_id_idx' creation
3.37.3	Exception during index 'coordinate_download_count_repository_name_namespace_name_version_date_ip_address_username_idx' creation
3.42.0	Exception during index 'asset_bucket_rid_idx' creation
3.48.0	Exception during index 'asset_bucket_rid_idx' creation

As next step i tried to use OrientDB Console to migrate records from component DB and then to build indexes manually while running both Nexus versions simultaneously. The config and security databases were migrated whole.

Procedure i used is following:

1. Start Nexus 3.29.2 with all the data

docker run --name=nexus-3.29.2 --volume /mnt/nexus-data:/nexus-data -d -p 8081:8081 sonatype/nexus3:3.29.2

2. Perform DB exports from ODB Console

docker exec -it nexus-3.29.2 bash -c 'cd /nexus-data/console && java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar'

connect plocal:/nexus-data/db/component admin admin
export database component -excludeAll -includeCluster="asset bucket component"
disconnect

connect plocal:/nexus-data/db/config admin admin
export database config
disconnect

connect plocal:/nexus-data/db/security admin admin
export database security
disconnect

3. Copy exports to new location

mkdir -p /mnt/nexus-data-new/console
chown -R 200:200 /mnt/nexus-data-new
cp /mnt/nexus-data/console/{component,config,security}.json.gz /mnt/nexus-data-new/console/

4. Start clean Nexus 3.48.0 with no data

docker run --name=nexus-3.48.0 --volume /mnt/nexus-data-new:/nexus-data -d -p 8881:8081 sonatype/nexus3:3.48.0

5. Perform DB imports from ODB Console

docker exec -it nexus-3.48.0 bash -c 'cd /nexus-data/console && java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar'

connect plocal:/nexus-data/db/component admin admin
import database component.json.gz -merge=true
disconnect

During import a following error occured but i decided to ignore it for this time as it was only test run and most of the data were imported.

Error on database import happened just before line 0, column 178093048
com.orientechnologies.orient.core.db.tool.ODatabaseImportException: Invalid format. Found unsupported tag 'brokenRids'
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:190)
    at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:2214)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405)
    at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260)
    at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131)
    at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:145)
    at org.sonatype.nexus.orient.console.Main.main(Main.java:63)

Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'component' from file: component-clusters.json.gz

Error: com.orientechnologies.orient.core.db.tool.ODatabaseImportException: Invalid format. Found unsupported tag 'brokenRids'

Then import of security and config databases.

connect plocal:/nexus-data/db/security admin admin
import database security
disconnect

connect plocal:/nexus-data/db/config admin admin
export database config
disconnect

6. Rebuild indexes in component db

connect plocal:/nexus-data/db/component admin admin
rebuild index *

During this operation following error occured

Error: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #23:10: found duplicated key 'maven-central' in index 'bucket_repository_name_idx' previously assigned to the record #23:3
    DB name="component" INDEX=bucket_repository_name_idx RID=#23:3

To fix this it was neccessary to locate all these duplicates in class bucket and for each perform

truncate record #23:3

Then repeat rebuild index *

7. Stop both Nexuses and move blobstore to a new location

docker stop --time=120 nexus-3.48.0
docker stop --time=120 nexus-3.29.2
rm -rf /mnt/nexus-data-new/blobs && mv /mnt/nexus-data/blobs /mnt/nexus-data-new/

8. Start Nexus 3.48.0

Nexus has started and authentication to GUI worked. Also browsing and downloading content worked. I had assumed inconsistency in data so i ran Task Repair - Reconcile component database from blob store. This Task stuck in state Running and never finished. Also log for the Task stopped to grow. So Tasks were broken and data presumably inconsistent but reachable.

Any suggestions how to perform the upgrade?

Answer 1

Summary

After some research and trials and errors i had came up with solution.
It was no go to try to solve it via direct db operations. Rather i used nexus3-cli in combination with Maven CLI to download packages from old instance and then upload into clean Nexus3 instance.
My case comprises only of Maven and npm repos.
After this i was able to upgrade Nexus3 to latest version using official procedure.

Prepare the environment

• Install python3, pip3 and prep two nexus3-cli versions with virtualenv - link to example
• Install java JDK and maven - link to example

Migration procedure

1. Start latest Nexus 3
2. Manually create repositories (maven, npm)
3. Allow scripting by inserting line nexus.scripts.allowCreation=true to $nexus-data/etc/nexus.properties and restarting Nexus (This is needed for REST API to work, hence nexus3-cli to work)
4. Download contents of hosted repos
   1. npm - use nexus3-cli
      nexus3 download $repo_name/ $repo_name/
   2. maven - use custom bash script based on this
5. Upload contents to new instance
   1. npm - Use nexus3-cli - beware of upload versions in proper order because of subsequent usage of semVer wildcards

      for i in $(find $repo_name -name '*tgz' | sort --version-sort) ; do
          nexus3 upload "$i" "$i" >> npm.err 2>&1
      done
      

   2. maven - use mvn cli in bash script based on this
6. turn off scripting as per 3.