The above statement is mentioned in the SwingWorker javadoc.
In an application I have seen a lengthy background task runs in a distinct thread and updates the UI as well without a problem (a reference to a Swing component was accessible).
Could it be that something bad can happen?
It's because the Java memory model does not guarantee that memory writes by one thread will be visible to other threads unless you use some form of synchronization. For performance and simplicity, Swing is not synchronized. Therefore, writes from other threads may never be visible to the EDT.
The application you've seen may work most of the time, and it may even work all of the time in some environments. But when it doesn't work, it'll fail in really strange ways that are difficult to reproduce.