This question has already been asked on Stack Overflow in 2013, but it is in need of an update. Functionality of Core Bluetooth for macOS was changed to bring it more in line with iOS since macOS 10.13.
The answers in that question are
CoreBluetooth is an iOS framework for Bluetooth LE communication. IOBluetooth is an OS X framework
This is not true anymore, Core Bluetooth is not exclusively an iOS framework, though IOBluetooth is exclusive to macOS
CoreBluetooth is for accessing Bluetooth Low Energy APIs. and IOBluetooth is for Bluetooth Classic Interface
This is also not true as Core Bluetooth is not exclusively for BLE.
CoreBluetooth documentation states:
Communicate with Bluetooth low energy and BR/EDR (“Classic”) Devices.
which seems straight forward; Core Bluetooth is used for Bluetooth comms, both BLE and Classic.
IOBluetooth documentation states:
Gain user-space access to Bluetooth devices.
but this is not particularly enlightening.
In the age of CoreBluetooth on macOS 10.13+, what is the primary modern use case of IOBluetooth vs CoreBluetooth on macOS?
CoreBluetooth supports BLE, GATT over BR/EDR, and L2CAP. Despite it's summary, it does not support most classic profiles. In particular, it only supports GATT over BR/EDR. It does not support other protocols. (GATT over BR/EDR is pretty rare in my experience. I've never encountered a device that actually supports it.) It does not support SDP and cannot be used to discover most classic devices.
IOBluetooth supports various classic protocols including SDP, RFCOMM, and L2CAP. It supports things like audio and HCI devices. It doesn't directly support BLE to my knowledge; it embeds CoreBluetooth for that. (I believe it can do some very low-level discovery of BLE, but it's not what you'd want to use.)
I don't believe this has radically changed in 10.13. What mostly changed (mostly I think to support Catalyst) is that Core Bluetooth was made more uniform between the platforms. I don't think this radically changed what the frameworks are used for.