Search code examples
pythonautocompletepycharmaws-glue

PyCharm has no auto-completion for AWS Glue Python

I'm new to Python and AWS Glue and I have trouble to enable auto-completion for certain case.

First case:

dynamic_frame has no API list

enter image description here

If I force the creation of spark DataFrame, no API list is showed, it seems DataSource0 is unknown.

enter image description here

But glueContext has the API list shown:

enter image description here

Second case:

Spark DataFrame can be created and API list is showed. Since I'm new to AWS Glue, I'm not sure this is the best practice to use the object of DynamicFrame for the DF conversion, instead of using .toDF() directly and what would be the impact.

enter image description here

enter image description here

Third case:

Define the type to have API list showed. and again, I'm new to Python and I come from a background of C/JAVA/SCALA, so I've no idea if this would be weird or "non-python" code style.

enter image description here

Environnment: Python 3.6 installed by Anaconda pyspark and AWS Glue installed via pip

enter image description here

Solution

  • To solve the first case try putting a type hint in front of the variable DataSource0 : DynamicFrame.

    The Linter of the IDE tries running your code in a separate background process to resolve the types and populate the auto-completion list. If you use a type hint you are helping the Linter determine the type of the attribute and so the IDE doesn't have to determine the type dynamically.

    The reason auto-completion works with the library functions in the other examples is because in those cases no dynamic code needs to resolved by the Linter. The library functions by themselves are clearly determined.

    Sometimes it also takes PyCharm a few seconds to resolve auto-completions, so don't force the menu, give it a few seconds for the background Linter process to complete between writing the variable and the dot (Ctrl+Space afterwards to show the auto-complete).

    In your case those library functions also have several levels of depth, that makes resolving the auto-suggestions heavier for the IDE. Flat really is better than nested in this case. But this isn't a problem just an inconvenience, provided the code is correct it will execute as expected at run-time (it does make writing the code more difficult without the occasional help of auto-suggestions).