In an Excel formula, is it possible to get a custom document property without an external add-in or help from VBA code?
No, you need to call a macro for that. However, you can link a custom property to the cell content, see Using “Link to content” to Define custom file properties in Excel for more information. In that case you could refer to the cell which delivers a custom property value for your formula calculations.