This screenshot was taken while creating a cluster on the Azure portal. As given in the picture below,
In the documentation here and here it is explained in a very clear way,
Custom endpoints: This field allows you to enter a comma-separated list of ports that you want to expose through the Azure Load Balancer to the public Internet for your applications. For example, if you plan to deploy a web application to your cluster, enter "80" here to allow traffic on port 80 into your cluster.
Application Ports (Start|End): are the ports that are used by the Service Fabric applications. The application port range should be large enough to cover the endpoint requirement of your applications. This range should be exclusive from the dynamic port range on the machine, that is, the EphemeralPorts range as set in the configuration. Service Fabric uses these ports whenever new ports are required and takes care of opening the firewall for these ports.
In Summary:
Custom endpoints are ports opened in the Load Balancer to enable external access.
Application Ports is a range of reserved ports to be opened in the nodes and assigned to services when using dynamic allocated ports, but not externally accessible.