I am very new to Python and am working through the Dagster hello tutorial
I have set up the following from the tutorial
import csv
from dagster import execute_pipeline, execute_solid, pipeline, solid
@solid
def hello_cereal(context):
# Assuming the dataset is in the same directory as this file
dataset_path = 'cereal.csv'
with open(dataset_path, 'r') as fd:
# Read the rows in using the standard csv library
cereals = [row for row in csv.DictReader(fd)]
context.log.info(
'Found {n_cereals} cereals'.format(n_cereals=len(cereals))
)
return cereals
@pipeline
def hello_cereal_pipeline():
hello_cereal()
However pylint shows
a no value for parameter
message.
What have I missed?
When I try to execute the pipeline I get the following
D:\python\dag>dagster pipeline execute -f hello_cereal.py -n hello_cereal_pipeline 2019-11-25 14:47:09 - dagster - DEBUG - hello_cereal_pipeline - 96c575ae-0b7d-49cb-abf4-ce998865ebb3 - PIPELINE_START - Started execution of pipeline "hello_cereal_pipeline". 2019-11-25 14:47:09 - dagster - DEBUG - hello_cereal_pipeline - 96c575ae-0b7d-49cb-abf4-ce998865ebb3 - ENGINE_EVENT - Executing steps in process (pid: 11684) event_specific_data = {"metadata_entries": [["pid", null, ["11684"]], ["step_keys", null, ["{'hello_cereal.compute'}"]]]} 2019-11-25 14:47:09 - dagster - DEBUG - hello_cereal_pipeline - 96c575ae-0b7d-49cb-abf4-ce998865ebb3 - STEP_START - Started execution of step "hello_cereal.compute". solid = "hello_cereal" solid_definition = "hello_cereal" step_key = "hello_cereal.compute" 2019-11-25 14:47:10 - dagster - ERROR - hello_cereal_pipeline - 96c575ae-0b7d-49cb-abf4-ce998865ebb3 - STEP_FAILURE - Execution of step "hello_cereal.compute" failed. cls_name = "FileNotFoundError" solid = "hello_cereal" solid_definition = "hello_cereal" step_key = "hello_cereal.compute"
File "c:\users\kirst\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\errors.py", line 114, in user_code_error_boundary yield File "c:\users\kirst\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\engine\engine_inprocess.py", line 621, in _user_event_sequence_for_step_compute_fn for event in gen: File "c:\users\kirst\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\execution\plan\compute.py", line 75, in _execute_core_compute for step_output in _yield_compute_results(compute_context, inputs, compute_fn): File "c:\users\kirst\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\execution\plan\compute.py", line 52, in _yield_compute_results for event in user_event_sequence: File "c:\users\kirst\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\definitions\decorators.py", line 418, in compute result = fn(context, **kwargs) File "hello_cereal.py", line 10, in hello_cereal with open(dataset_path, 'r') as fd:
2019-11-25 14:47:10 - dagster - DEBUG - hello_cereal_pipeline - 96c575ae-0b7d-49cb-abf4-ce998865ebb3 - ENGINE_EVENT - Finished steps in process (pid: 11684) in 183ms event_specific_data = {"metadata_entries": [["pid", null, ["11684"]], ["step_keys", null, ["{'hello_cereal.compute'}"]]]} 2019-11-25 14:47:10 - dagster - ERROR - hello_cereal_pipeline - 96c575ae-0b7d-49cb-abf4-ce998865ebb3 - PIPELINE_FAILURE - Execution of pipeline "hello_cereal_pipeline" failed.
[Update] From Rahul's comment I realised I had not copied the whole example. When I corrected that I got a FileNotFoundError
To answer the original question about why you are receiving a "no value for parameter" pylint message -
This is because the pipeline function calls don't include any parameters in the constructors and the @solid functions have parameters defined. This is intentional from dagster and can be ignored by adding the following line either at the beginning of the module, or to the right of the line with the pylint message. Note that putting the python comment below at the beginning of the module tells pylint to ignore any instance of the warning in the module, whereas putting the comment in-line tells pylint to ignore only that instance of the warning.
# pylint: disable=no-value-for-parameter
Lastly, you could also put a similar ignore statement in a .pylintrc file too, but I'd advise against that as that would be project-global and you could miss true issues.
hope this helps a bit!