TL;DR: When updating system apps on android, the updated app is only started after reboot.
I am developing a custom ROM based on Android 8.1.2 for a custom device based on Rockchip RK3126c. I took the AOSP source from the board manufacturer and only modified the platform key and added two custom apps as system apps; one is privileged (shared system user), and one isn't, i.e. the first is located in /system/priv-app and signed with the platform key, and the second in /system/app and signed with its own key.
Both apps work, but when I try to update any of the two (via adb shell pm install -r ... or via Android Studio) the update doesn't take effect; the old version still runs, even after killing (adb shell kill -9 ... or throwing a null pointer exception) and restarting it. Only after a full reboot is the new version started. I did increment the versionCode. Before the reboot, getPackageManager().getPackageInfo(...).versionCode yields the new version number, while BuildConfig.VERSION_CODE gives the old one. This is both for eng and user builds.
While updating, adb logcat shows:
10-17 15:45:19.706 1048 1048 D AndroidRuntime: Calling main entry com.android.commands.pm.Pm
10-17 15:45:19.725 1058 1058 E asset : setgid: Operation not permitted
10-17 15:45:22.563 336 362 I ActivityManager: Start proc 1062:com.android.defcontainer/u0a11 for service com.android.defcontainer/.DefaultContainerService
10-17 15:45:22.617 1062 1062 I zygote : The ClassLoaderContext is a special shared library.
10-17 15:45:22.739 336 362 I chatty : uid=1000(system) PackageManager expire 1 line
10-17 15:45:22.820 1077 1077 I dex2oat : /system/bin/dex2oat --input-vdex-fd=-1 --output-vdex-fd=14 --compiler-filter=quicken --classpath-dir=/data/app/com.example.myapp-cdF84NESHVxl5UTZHVbRdg== --class-loader-context=PCL[]
10-17 15:45:22.832 1077 1077 W dex2oat : Could not reserve sentinel fault page
10-17 15:45:26.117 1077 1080 W dex2oat : Verification of boolean ao0.a(java.lang.String, long) took 130.568ms
10-17 15:45:28.960 1077 1077 I dex2oat : dex2oat took 6.142s (10.744s cpu) (threads: 4) arena alloc=17KB (17568B) java alloc=4MB (4274432B) native alloc=6MB (6543768B) free=1801KB (1844840B)
10-17 15:45:28.997 336 350 I ActivityManager: Force stopping com.example.myapp appid=1000 user=-1: installPackageLI
10-17 15:45:29.009 336 362 W PackageManager: Trying to update system app code path from /system/priv-app/MyApp to /data/app/MyApp-cdF84NESHVxl5UTZHVbRdg==
10-17 15:45:29.010 336 362 W PackageManager: Code path for com.example.myapp changing from /system/priv-app/MyApp to /data/app/MyApp-cdF84NESHVxl5UTZHVbRdg==
10-17 15:45:29.010 336 362 W PackageManager: Resource path com.example.myapp changing from /system/priv-app/MyApp to /data/app/MyApp-cdF84NESHVxl5UTZHVbRdg==
10-17 15:45:29.205 242 242 E : Couldn't opendir /data/app/vmdl1968223466.tmp: No such file or directory
10-17 15:45:29.205 242 242 E installd: Failed to delete /data/app/vmdl1968223466.tmp: No such file or directory
10-17 15:45:29.213 336 362 I ActivityManager: Force stopping com.example.myapp appid=1000 user=0: pkg removed
10-17 15:45:29.214 1048 1048 I Pm : Package com.example.myapp installed in 9500 ms
10-17 15:45:29.239 1048 1048 I app_process: System.exit called, status: 0
10-17 15:45:29.239 1048 1048 I AndroidRuntime: VM exiting with result code 0.
After the reboot, further updates to the app do work - apparently, the problem only exists when changing the code path from /system/[priv-]app to /data/app, but not when the path is already /data/app.
The same thing worked fine on Android 7.1.2. Deep within the android framework there is probably some cache for the code paths of the apps which doesn't get updated after the app is overwritten. Anyone knows where that is located and how to fix this problem?
After a lot of fiddling, I found it myself: The app was marked as persistent in the manifest, which causes the mentioned behaviour. It only worked on Android 7.1.2 because there was a vendor-specific modification that ignored the persistent flag.