pythondatabricksazure-databricksfb-hydra
I'm trying to use Config Management Tool Hydra(As a decorator) for a Multirun experimentation in Azure Databricks Environment. It does not work
I'm trying to use Config Management Tool Hydra in a Azure Databricks environment(Client Specific requirement). The idea is to do multiruns using config files. I have tried a few codes on my local, it works like a charm. But the same does not work in my Databricks set-up. I have installed the latest Hydra Wheel file downloaded from the Hydra official website onto my Databricks cluster.
Here is how my code looks like. Initially Im trying to simply print my config file and subsequently I want to have multirun from this possible and see if the parameters are being sweeped or not. I face an error while using a basic Hydra decorator style function itself. Using hydra's compose API's I can read one single YAML config, but according to Hydra's official documentation, compose does not allow multirun and my task is to have multirun. Has anybody faced a similar error or Is there any quick fix I can try?
config_test.yaml
network:
inputs: 2
outputs: 3
layer_size: 60
nr_layers: 8
optimiser:
lr: 1e-3
scheduler: exponential_lr
iter: 5000
hydra:
mode: MULTIRUN
sweeper:
params:
+n: 5,10
Code
import hydra
from omegaconf import DictConfig, OmegaConf
@hydra.main(version_base= None, config_path="conf", config_name="config_test.yaml")
def main(cfg : DictConfig) -> None:
print(OmegaConf.to_yaml(cfg))
if __name__ == '__main__':
main()
The high level error says
- **LexerNoViableAltException: 39187**
and
- **object of type 'NoneType' has no len()</sub>**
**The detailed error looks something like this**
<sub>
LexerNoViableAltException: 39187
^
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
ERROR:root:Internal Python error in the inspect module.
Below is the traceback from this internal error.
Traceback (most recent call last):
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
cfg = self.compose_config(
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 594, in compose_config
cfg = self.config_loader.load_configuration(
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
return self._load_configuration_impl(
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
parsed_overrides = parser.parse_overrides(overrides=overrides)
File "/databricks/python/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py", line 96, in parse_overrides
raise OverrideParseException(
hydra.errors.OverrideParseException: LexerNoViableAltException: 39187
^
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/databricks/python/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<command-229383469694947>", line 28, in <module>
main()
File "/databricks/python/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py", line 303, in run_and_report
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py", line 1101, in get_records
return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
File "/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py", line 248, in wrapped
return f(*args, **kwargs)
File "/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py", line 281, in _fixed_getinnerframes
records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
File "/usr/lib/python3.8/inspect.py", line 1515, in getinnerframes
frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
AttributeError: 'tuple' object has no attribute 'tb_frame'
The Main Error looks something like this
TypeError: object of type 'NoneType' has no len()
---------------------------------------------------------------------------
OverrideParseException Traceback (most recent call last)
/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py in run_and_report(func)
219 try:
--> 220 return func()
221 except Exception as ex:
/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py in <lambda>()
457 run_and_report(
--> 458 lambda: hydra.run(
459 config_name=config_name,
/databricks/python/lib/python3.8/site-packages/hydra/_internal/hydra.py in run(self, config_name, task_function, overrides, with_log_configuration)
104 ) -> JobReturn:
--> 105 cfg = self.compose_config(
106 config_name=config_name,
/databricks/python/lib/python3.8/site-packages/hydra/_internal/hydra.py in compose_config(self, config_name, overrides, run_mode, with_log_configuration, from_shell, validate_sweep_overrides)
593
--> 594 cfg = self.config_loader.load_configuration(
595 config_name=config_name,
/databricks/python/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py in load_configuration(self, config_name, overrides, run_mode, from_shell, validate_sweep_overrides)
141 try:
--> 142 return self._load_configuration_impl(
143 config_name=config_name,
/databricks/python/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py in _load_configuration_impl(self, config_name, overrides, run_mode, from_shell, validate_sweep_overrides)
243 self.ensure_main_config_source_available()
--> 244 parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
245 config_name, overrides
/databricks/python/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py in _parse_overrides_and_create_caching_repo(self, config_name, overrides)
227 parser = OverridesParser.create()
--> 228 parsed_overrides = parser.parse_overrides(overrides=overrides)
229 caching_repo = CachingConfigRepository(self.repository)
/databricks/python/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py in parse_overrides(self, overrides)
95 msg = f"Error parsing override '{override}'" f"\n{e}"
---> 96 raise OverrideParseException(
97 override=override,
OverrideParseException: LexerNoViableAltException: 39187
^
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details
During handling of the above exception, another exception occurred:
SystemExit Traceback (most recent call last)
[... skipping hidden 1 frame]
<command-229383469694947> in <module>
27 if __name__ == '__main__':
---> 28 main()
29
/databricks/python/lib/python3.8/site-packages/hydra/main.py in decorated_main(cfg_passthrough)
93 # multiple times (--multirun)
---> 94 _run_hydra(
95 args=args,
/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py in _run_hydra(args, args_parser, task_function, config_path, config_name, caller_stack_depth)
393 run_mode = hydra.get_mode(config_name=config_name, overrides=overrides)
--> 394 _run_app(
395 run=args.run,
/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py in _run_app(run, multirun, mode, hydra, config_name, task_function, overrides)
456 if mode == RunMode.RUN:
--> 457 run_and_report(
458 lambda: hydra.run(
/databricks/python/lib/python3.8/site-packages/hydra/_internal/utils.py in run_and_report(func)
302 raise ex
--> 303 sys.exit(1)
304
SystemExit: 1
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call last)
[... skipping hidden 1 frame]
/databricks/python/lib/python3.8/site-packages/IPython/core/interactiveshell.py in showtraceback(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)
2052 stb = ['An exception has occurred, use %tb to see '
2053 'the full traceback.\n']
-> 2054 stb.extend(self.InteractiveTB.get_exception_only(etype,
2055 value))
2056 else:
/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py in get_exception_only(self, etype, value)
752 value : exception value
753
--> 754 return ListTB.structured_traceback(self, etype, value)
755
756 def show_exception_only(self, etype, evalue):
/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py in structured_traceback(self, etype, evalue, etb, tb_offset, context)
627 chained_exceptions_tb_offset = 0
628 out_list = (
--> 629 self.structured_traceback(
630 etype, evalue, (etb, chained_exc_ids),
631 chained_exceptions_tb_offset, context)
/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py in structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
1365 else:
1366 self.tb = tb
-> 1367 return FormattedTB.structured_traceback(
1368 self, etype, value, tb, tb_offset, number_of_lines_of_context)
1369
/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py in structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
1265 if mode in self.verbose_modes:
1266 # Verbose modes need a full traceback
-> 1267 return VerboseTB.structured_traceback(
1268 self, etype, value, tb, tb_offset, number_of_lines_of_context
1269 )
/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py in structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
1122 Return a nice text document describing the traceback.
1123
-> 1124 formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
1125 tb_offset)
1126
/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py in format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)
1080
1081
-> 1082 last_unique, recursion_repeat = find_recursion(orig_etype, evalue, records)
1083
1084 frames = self.format_records(records, last_unique, recursion_repeat)
/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py in find_recursion(etype, value, records)
380 # first frame (from in to out) that looks different.
381 if not is_recursion_error(etype, value, records):
--> 382 return len(records), 0
383
384 # Select filename, lineno, func_name to track frames with
TypeError: object of type 'NoneType' has no len()
Now if I use COMPOSE API, it does work, but my sweep parameters do not get printed. For ex, in the above YAML my sweep parameter is n. N should be changing once to 5 and once to 10 in two different runs(Multiruns). But I dont see that happening
Code:
from hydra import compose, initialize_config_dir
from omegaconf import OmegaConf
def main() -> None:
with initialize_config_dir(version_base="1.3", config_dir="/Workspace/Repos/some_path/conf"):
cfg = compose(config_name="config_test.yaml", overrides=[])
print(OmegaConf.to_yaml(cfg))
if __name__ == "__main__":
main()Output:
experiment_name: 10-08-2023_t2
network:
inputs: 2
outputs: 3
layer_size: 60
nr_layers: 8
optimiser:
lr: 0.001
scheduler: exponential_lr
iter: 5000As you can see above the config gets printed only once without the sweep multirun parameter (n).
When I try it with decorator on my local this is the output I get
[2023-08-15 02:36:02,432][HYDRA] Launching 2 jobs locally
[2023-08-15 02:36:02,433][HYDRA] #0 : +n=5
experiment_name: 10-08-2023_t2
network:
inputs: 2
outputs: 3
layer_size: 60
nr_layers: 8
optimiser:
lr: 0.001
scheduler: exponential_lr
iter: 5000
'n': 5
[2023-08-15 02:36:02,525][HYDRA] #1 : +n=10
experiment_name: 10-08-2023_t2
inputs: 2
outputs: 3
layer_size: 60
nr_layers: 8
optimiser:
lr: 0.001
scheduler: exponential_lr
iter: 5000As you can see above the config has run with respect to two sweep parameters, n=5 and n=10
Solution
I tried with .py file and called in notebook using %sh command.
Below is your data i used.
Next, created new .py file.
%sh
echo 'import hydra
from omegaconf import DictConfig, OmegaConf
@hydra.main(version_base=None, config_path="/Workspace/Users/Your_folder_path/", config_name="config")
def my_app(cfg : DictConfig) -> None:
print(OmegaConf.to_yaml(cfg))
if __name__ == "__main__":
my_app()
' > /Workspace/Users/Your_folder_path/hydra_py.py
Next, call it.
%sh
python3 /Workspace/Users/Your_folder_path/hydra_py.py
Output:
