Блог

Я хотел бы распечатать трассировку стека исключения, возникшего во время выполнения кода pyspark. Я использую собственный регистратор pyspark (py4j), но при его выполнении logger.exception() происходит сбой. Я также пытался logger.error()

 log4jLogger = sc._jvm.org.apache.log4j  logger = log4jLogger.LogManager.getLogger(__name__)  logger.info('first statement.')  try:  raise Exception('Dummy exception')   except Exception as e:  logger.error('Something awful happened') # Does not print stacktrace  logger.exception('Something awful happened') # print stacktrace but crashes    logger.info('Importing module.')  

Вывод выглядит так:

 21/12/08 11:50:56 INFO __main__: First statement 21/12/08 11:50:56 ERROR __main__: Something awful happened Traceback (most recent call last):  File "/home/gaurav.gupta/projects/PoCs/brandMention/pyspark-scripts/dist/main.py", line 94, in lt;modulegt;  raise Exception('Dummy exception')  Exception: Dummy exception  During handling of the above exception, another exception occurred:  Traceback (most recent call last):  File "/home/gaurav.gupta/projects/PoCs/brandMention/pyspark-scripts/dist/main.py", line 97, in lt;modulegt;  logger.exception('Something awful happened')  File "/home/gaurav.gupta/miniconda3/envs/venv_pyspark/lib/python3.7/site-packages/pyspark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__  File "/home/gaurav.gupta/miniconda3/envs/venv_pyspark/lib/python3.7/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco  File "/home/gaurav.gupta/miniconda3/envs/venv_pyspark/lib/python3.7/site-packages/pyspark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 332, in get_return_value py4j.protocol.Py4JError: An error occurred while calling o37.exception. Trace: py4j.Py4JException: Method exception([class java.lang.String]) does not exist  at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)  at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)  at py4j.Gateway.invoke(Gateway.java:274)  at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  at py4j.commands.CallCommand.execute(CallCommand.java:79)  at py4j.GatewayConnection.run(GatewayConnection.java:238)  at java.lang.Thread.run(Thread.java:748)   21/12/08 11:50:56 INFO SparkUI: Stopped Spark web UI at http://192.168.1.13:4040 21/12/08 11:50:56 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!   

Я смог найти обходной путь для решения этой проблемы. Я получаю трассировку стека в строковом формате, а затем печатаю ее как обычно.

 import traceback strace = ''.join(traceback.format_exception(etype=type(ex), value=ex, tb=ex.__traceback__)) logger.error(strace)