hdiutil can attach and detach a DMG file containing HFS+ partition, into selected mount-point. However, it uses the private API of DiskImages.framework. My aim is to make this task fully programmatic, so I look for alternative in the kernel drivers realm.
while delving into the KPI represented by <sys/mount.h> I found useful method for unmount filesystem according to its matching fsid:
int vfs_unmountbyfsid(fsid_t *fsid, int flags, vfs_context_t ctx)
However, do we have the opposite operation for mount ?
There isn't a public KPI for initiating a mount, and I'm not even aware of a function in com.apple.kpi.private. In user space however, there's a lot more than DiskImages.framework at your disposal: there's the DiskArbitration.framework, and of course the POSIX syscall of mount(2).
I think you might be conflating 2 concepts here, which are actually entirely separate:
Disk image support is not an intrinsic part of the xnu kernel. They're implemented in the IOHDIXController object (code for this is in a kext), which you'll find attached to IOResources in the IORegistry. When the user double-clicks a .dmg file or similar, the diskimages-helper daemon opens & parses it and instructs the IOHDIXController to create a new IODiskImageBlockStorageDeviceOutKernel instance (an IOBlockStorageDevice subclass). This appears to the OS like a physical block device, and the usual stack of IOBlockStorageDriver -> IOMedia -> IOPartitionScheme -> IOMedia -> IOMediaBSDClient objects configures itself on top of it. This then causes device discovery events to fire in diskarbitrationd, which moves on to the second part of the process: calling mount(2) on the newly discovered IOMedia objects' /dev/diskXsY node(s) as appropriate.
The HDIX subsystem is not open as far as I'm aware. So if you want to implement your own disk image format, you'll need to recreate something similar to Apple's diskimage-helper and IOHDIXController.kext. You can probably get away with implementing it entirely in-kernel if you prefer, although it's probably not a great idea.
The second part, mounting, is done by diskarbitrationd automatically if you are using the IOStorage stack, but you can influence it via a disk arbitration dissenter. See DiskArbitration.framework for details. But this also lets you influence mounting of disk images handled by diskimages-helper, so if you are using an image format supported by it, you won't need to write your own and can simply intercept the mounting and do whatever it is you want to do yourself.
You can probably also bypass the IOStorage stack entirely and only create the BSD dev node in your kext. In that case, diskarbitrationd shouldn't notice it, and you will need to call mount() explicitly from your daemon.
I hope that clarifies things.