how to print jupyter background job output in cell it was launched from?

Question

When launching a background job from an IPython Jupyter Notebook, how can I make the printed output appear in the cell it was launched from, rather than in the cell I am currently working in?

The print() command seems to print into the current working cell, not into the cell that the background job was launched from. Is there a way to make it print nicely in the cell it was launched from? This is particularly relevant when running multiple background job sets, as to determine which jobset was responsible for that line of output.

Edit: it occurs with any code but here is a small snippet to reproduce it:

from IPython.lib import backgroundjobs as bg
import time
def launchJob():
    for i in range(100):
       print('x')
       time.sleep(1)
jobs = bg.BackgroundJobManager()
for r in range(3):
    jobs.new("launchJob()")

that does exactly what you'd expect it to do, print 3 x's every second in the output under the cell. Now go to the next cell and type 1+1 and execute it. The output 2 appears, but also any remaining x's get printed in this new cell rather in the original cell.

I am looking for a way to specifically tell my job to always print to the original cell it was executed from, as to obtain sort of a log in the background, or to generate a bunch of plots in the background, or any kind of data that I want in one place rather than appearing all over my notebook.

Answer 1

The IPython.lib.backgoundjobs.BackgroundJobManager.new documentation states:

All threads running share the same standard output. Thus, if your background jobs generate output, it will come out on top of whatever you are currently writing. For this reason, background jobs are best used with silent functions which simply return their output.

In GitHub pull request #856 of IPython (October 2011), the developers discussed this issue and concluded that keeping the output in the original cell would be difficult to implement. Hence, they decided to table the idea and in the latest version (7.9.0 at the time of writing) it still has not been solved.

In the mean time, a workaround could be to store the output in a variable, or to write/pickle the output to a file and print/display it once the background jobs are finished.