Office dialog API with Office.context.ui.displayDialogAsync() opens a dialog (verified in Office 2021 version). Do Office 365 supports same API or any other to show a dialog?
The same API is used for opening a dialog in desktop applications as well as O365 (browsers). Read more about dialog APIs in the Use the Office dialog API in Office Add-ins article.
You may also find the Best practices and rules for the Office dialog API page helpful.