I'm learning the difference between these to concepts. I want to build a public Sharepoint site, and is one better than the other for this use, or is it too general of a question?
At a very general level; application pages are stored on the web server's file system (also known as the "12 hive") and run in a more priviledged security context. You tend to use these to create administrative interfaces.
Site pages (such as default.aspx) are user-targeted, and available for customization via SharePoint designer.