Unsubscribe from delegate passed through ref keyword to the subscription method?

I've got the following class:

public class Terminal : IDisposable
{
    readonly List<IListener> _listeners;

    public Terminal(IEnumerable<IListener> listeners)
    {
        _listeners = new List<IListener>(listeners);
    }

    public void Subscribe(ref Action<string> source)
    {
        source += Broadcast;
        //Store the reference somehow?
    }

    void Broadcast(string message)
    {
        foreach (var listener in _listeners) listener.Listen(message);
    }

    public void Dispose()
    {
        //Unsubscribe from all the stored sources?
    }
}

I've searched for a while and it appears that an argument passed with the ref keyword can't be stored. Trying to add the source argument to a list or to assign it to a field variable doesn't allow it to keep a reference to the actual delegate's original reference; so my questions are:

Is there a way to unsubscribe from all the sources without passing their references again?
If not, how can the class be changed in order to support it, but still maintain the subscription by passing a delegate through a method?
Is it possible to achieve it without using Reflection?
Is it possible to achieve it without wrapping the delegate/event in a class and then passing the class as a parameter for the subscription?

Thank you.

EDIT: It appears that without using a Wrapper or Reflection, there's no solution to the given problem. My intention was to make the class as much portable as possible, without having to wrap delegates in helper classes. Thanks everyone for the contributions.

Solution

Edit: Ok, that was a bad idea, so back to the basics:

I recommend creating a wrapper class over an Action:

class ActionWrapper
{
    public Action<string> Action;
}

And restructuring your initial class to work with wrappers:

private ActionWrapper localSource;

public void Subscribe(ActionWrapper source)
{
    source.Action += Broadcast;
    localSource = source;        
}

public void Dispose()
{
    localSource.Action -= Broadcast;
}

Now you should get the desired results.