Running a glueetl, GlueVersion 2.0, python 3, AWS Glue job I am trying to do the recommended python3 lazy loading log messages with
logger.info("Attempting to run python module 1 {entrypoint}".format(entrypoint=entrypoint))
logger.info("Attempting to run python module 2 %s", entrypoint)
This produces an error on the second line but the first line succeeds and prints the string.
2021-06-25 04:12:58,782 INFO [Thread-7] log.GlueLogger (GlueLogger.scala:info(8)): Attempting to run python module 1 main.test_program
2021-06-25 04:12:58,818 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Error from Python:Traceback (most recent call last):
File "/tmp/main_etl_script.py", line 103, in <module>
spark=spark)
File "/tmp/main_etl_script.py", line 32, in main_handler
logger.info("Attempting to run python module 2 %s", entrypoint)
File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 332, in get_return_value
format(target_id, ".", name, value))
py4j.protocol.Py4JError: An error occurred while calling o85.info. Trace:
py4j.Py4JException: Method info([class java.lang.String, class java.lang.String]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:274)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
My logger setup is simple:
import logging
...
glue_context = GlueContext(SparkContext())
logger = glue_context.get_logger()
Why does this occur?
It doesn't like the comma after the 2 %s":
logger.info("Attempting to run python module 2 %s", entrypoint)
try this with a f string:
logger.info(f"Attempting to run python module 2 {entrypoint}")