Блог

я отправил задание spark в emr. он хорошо работает в режиме клиента. но при запуске в режиме кластера он показывает следующее error.in в коде я использую sqs-приемник из GitHub — imapi / spark-sqs-receiver. и пытается получить доступ с Amazon s3 с помощью s3a.

 16/10/21 07:10:00 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 71, ip-172-31-54-127.ec2.internal, partition 0,NODE_LOCAL, 2115 bytes)
16/10/21 07:10:00 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-172-31-54-127.ec2.internal:43789 (size: 2.4 KB, free: 511.5 MB)
16/10/21 07:10:01 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 72, ip-172-31-54-127.ec2.internal, partition 1,NODE_LOCAL, 2115 bytes)
16/10/21 07:10:01 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 71, ip-172-31-54-127.ec2.internal): java.lang.ExceptionInInitializerError
        at sqs.sqstest.Streaming$1.call(Streaming.java:146)
        at sqs.sqstest.Streaming$1.call(Streaming.java:97)
        at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$fn$1$1.apply(JavaDStreamLike.scala:170)
        at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$fn$1$1.apply(JavaDStreamLike.scala:170)
        at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
        at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:189)
        at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: A master URL must be set in your configuration
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
        at sqs.sqstest.Streaming.<clinit>(Streaming.java:41)
        ... 15 more

16/10/21 07:10:01 INFO YarnClusterSchedulerBackend: Disabling executor 1.
16/10/21 07:10:01 INFO DAGScheduler: Executor lost: 1 (epoch 1)
16/10/21 07:10:01 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
16/10/21 07:10:01 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, ip-172-31-54-127.ec2.internal, 43789)
16/10/21 07:10:01 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor
16/10/21 07:10:02 INFO YarnAllocator: Completed container container_1477032067457_0003_02_000002 on host: ip-172-31-54-127.ec2.internal (state: COMPLETE, exit status: 50)
16/10/21 07:10:02 WARN YarnAllocator: Container marked as failed: container_1477032067457_0003_02_000002 on host: ip-172-31-54-127.ec2.internal. Exit status: 50. Diagnostics: Exception from container-launch.
Container id: container_1477032067457_0003_02_000002
Exit code: 50
Stack trace: ExitCodeException exitCode=50:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
        at org.apache.hadoop.util.Shell.run(Shell.java:456)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 50

16/10/21 07:10:02 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1477032067457_0003_02_000002 on host: ip-172-31-54-127.ec2.internal. Exit status: 50. Diagnostics: Exception from container-launch.
Container id: container_1477032067457_0003_02_000002
Exit code: 50
Stack trace: ExitCodeException exitCode=50:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
        at org.apache.hadoop.util.Shell.run(Shell.java:456)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)