In Silverlight it was once possible to register an app for the edit-dialog in the photo hub (https://msdn.microsoft.com/en-us/library/windows/apps/hh202966(v=vs.105).aspx).
Is there an UWP-equivalent now? Or is there any other way, my app can be started from within the photos hub?
This functionality is not currently available in the Universal Platform. You can use the Windows Feedback tool ("Developer" section) or UserVoice to request that it be made available.