In order to stay backward compatible with existing customers, I have to be able to handle attachments and files in Salesforce. Basically, some customers need to be able to insert files and other attachments in our community portal. I would like to do this without the user knowing which one they are inserting.
The easiest way I can think to do this is to leave the current apex pages used for uploading alone. Then in the Attachment trigger, I can determine (using a flag somewhere) if the attachment should stay or be changed to a File (ContentVersion). If it should be a file; then I would create all the necessary records, ContentVersion, ContentDocumentLink and then skip the attachment. However, I can't figure out a way to stop the attachments from inserting without generating an error to the user. I really don't want to allow the attachment to insert and then delete it in the after insert trigger because it seems like a lot of overhead.
Using the trigger means I can handle multiple places where community users can add files. Otherwise, I will have to make changes in all locations to determine which type of record to insert.
So after all of that, the main question is whether or not there is a way to stop inserting records without the addError method?
No, there isn't. You cannot stop the insertion of the Attachment record in a trigger without forcing a rollback of the entire transaction or savepoint region, which would include any DML your Attachment trigger performed to add Content objects instead.
The only options that I see are to perform the deletion as you describe, or to change the front-end code to directly insert Content records instead of Attachments.