Files not synching in Azure databricks

Question

So I am writing a few scripts, one of those scripts is a helper script. When I make changes to the helper script (make sure the changes are saved) I cannot see those changes through the other script. For example I changed the value of a variable in the helper script and I want to use that updated value in the other script, when I run the other script I can not see the change. I have to restart the cluster to see the change.

Is there a reason why I cannot see instant changes in databricks like we can see in local machines?

Answer 1

When using Python files (modules) -- and not to be confused with Python notebooks -- the module is interpreted and loaded upon import. Once you have imported a module, changes to the module are not reflected unless the Python interpreter is restarted, or you can do it dynamically at runtime using the importlib built-in module, as detailed in the Answer here: How do I unload (reload) a Python module?

from importlib import reload  # Python 3.4+
import my_module

# Do some things
print(my_module.x) 

# Change things in `my_module`

# Reload the module at runtime
my_module = reload(my_module)

# Do things again; changes are reflected
print(my_module.x)