Out-of-the-box, an Alfresco user can read a document based on:
Now, I want to add a new factor: Whether the document is currently part of a workflow.
Alfresco's permissionDefinitions.xml allows me to define permissions based on authorities such as ROLE_LOCK_OWNER etc, but it does not seem to be the right place to add permission conditions.
I guess I will have to write some Java source code, but I am not sure what classes are responsible for this, and whether there is an Alfresco way to customize them?
So, I assume you want to somehow have nodes that are attached to a workflow have different access rights? You need to think about the behavior you want in all of the UIs and protocols you are exposing (e.g. share, WebDAV, CIFS, FTP, etc.).
If you want to set a permission on a node, you can do that via JavaScript as well as Java (See http://docs.alfresco.com/5.2/references/API-JS-setPermission.html and http://docs.alfresco.com/5.2/references/dev-services-permission.html). As was mentioned in one of the comments, you can also get the number of active workflows on a node by referencing the activeWorkflows property in JavaScript (http://docs.alfresco.com/5.2/references/API-JS-ScriptNode.html) or in Java
Depending on the specifics, I might implement this in different ways, but if all you want to do is have the permission change, you could just update it at the beginning and end of your workflow with a simple javascript call. The only thing bad about that is that it doesn't take into consideration the workflow getting canceled. You could also create a policy/behavior on an aspect you attach or even have a rule or job run that updates content based on the activeWorkflows values.