Блог

версия hadoop :2.10.1

Я установил несколько версий Hadoop, но проблема остается прежней.

журналы приведены ниже.

 org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$NodeEventDispatcher  2021-12-09 22:52:49,500 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties  2021-12-09 22:52:49,656 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).  2021-12-09 22:52:49,656 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics system started  2021-12-09 22:52:49,672 INFO org.apache.hadoop.yarn.security.YarnAuthorizationProvider: org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.  2021-12-09 22:52:49,674 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.RMAppManagerEventType for class org.apache.hadoop.yarn.server.resourcemanager.RMAppManager  2021-12-09 22:52:49,680 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncherEventType for class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher  2021-12-09 22:52:49,684 INFO org.apache.hadoop.yarn.server.resourcemanager.RMNMInfo: Registered RMNMInfo MBean  2021-12-09 22:52:49,684 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.monitor.RMAppLifetimeMonitor: Application lifelime monitor interval set to 3000 ms.  2021-12-09 22:52:49,689 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list  2021-12-09 22:52:49,694 INFO org.apache.hadoop.conf.Configuration: found resource capacity-scheduler.xml at file:/Users/salishkumar/Downloads/hadoop-2.10.1/etc/hadoop/capacity-scheduler.xml  2021-12-09 22:52:49,711 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: Minimum allocation = lt;memory:1024, vCores:1gt;  2021-12-09 22:52:49,712 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: Maximum allocation = lt;memory:8192, vCores:4gt;  2021-12-09 22:52:49,743 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE  2021-12-09 22:52:49,743 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE  2021-12-09 22:52:49,743 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration: max alloc mb per queue for root is undefined  2021-12-09 22:52:49,743 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration: max alloc vcore per queue for root is undefined  2021-12-09 22:52:49,760 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root, capacity=1.0, absoluteCapacity=1.0, maxCapacity=1.0, absoluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APP:*ADMINISTER_QUEUE:*, labels=*,  , reservationsContinueLooking=true, orderingPolicy=utilization, priority=0  2021-12-09 22:52:49,761 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Initialized parent-queue root name=root, fullname=root  2021-12-09 22:52:49,770 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE  2021-12-09 22:52:49,770 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE  2021-12-09 22:52:49,770 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration: max alloc mb per queue for root.default is undefined  2021-12-09 22:52:49,770 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration: max alloc vcore per queue for root.default is undefined  2021-12-09 22:52:49,776 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default  capacity = 1.0 [= (float) configuredCapacity / 100 ]  absoluteCapacity = 1.0 [= parentAbsoluteCapacity * capacity ]  maxCapacity = 1.0 [= configuredMaxCapacity ]  absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]  userLimit = 100 [= configuredUserLimit ]  userLimitFactor = 1.0 [= configuredUserLimitFactor ]  maxApplications = 10000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]  maxApplicationsPerUser = 10000 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]  usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)]  absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]  maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]  minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ]  maximumAllocation = lt;memory:8192, vCores:4gt; [= configuredMaxAllocation ]  numContainers = 0 [= currentNumContainers ]  state = RUNNING [= configuredState ]  acls = SUBMIT_APP:*ADMINISTER_QUEUE:* [= configuredAcls ]  nodeLocalityDelay = 40  rackLocalityAdditionalDelay = -1  labels=*,  reservationsContinueLooking = true  preemptionDisabled = true  defaultAppPriorityPerQueue = 0  priority = 0  maxLifetime = -1 seconds  defaultLifetime = -1 seconds  2021-12-09 22:52:49,776 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager: Initialized queue: default: capacity=1.0, absoluteCapacity=1.0, usedResources=lt;memory:0, vCores:0gt;, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0  2021-12-09 22:52:49,777 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager: Initialized queue: root: numChildQueue= 1, capacity=1.0, absoluteCapacity=1.0, usedResources=lt;memory:0, vCores:0gt;usedCapacity=0.0, numApps=0, numContainers=0  2021-12-09 22:52:49,778 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager: Initialized root queue root: numChildQueue= 1, capacity=1.0, absoluteCapacity=1.0, usedResources=lt;memory:0, vCores:0gt;usedCapacity=0.0, numApps=0, numContainers=0  2021-12-09 22:52:49,778 INFO org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule: Initialized queue mappings, override: false  2021-12-09 22:52:49,778 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.WorkflowPriorityMappingsManager: Initialized workflow priority mappings, override: false  2021-12-09 22:52:49,779 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized CapacityScheduler with calculator=class org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator, minimumAllocation=lt;lt;memory:1024, vCores:1gt;gt;, maximumAllocation=lt;lt;memory:8192, vCores:4gt;gt;, asynchronousScheduling=false, asyncScheduleInterval=5ms  2021-12-09 22:52:49,781 INFO org.apache.hadoop.conf.Configuration: dynamic-resources.xml not found  2021-12-09 22:52:49,782 INFO org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain: Initializing AMS Processing chain. Root Processor=[org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor].  2021-12-09 22:52:49,789 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: TimelineServicePublisher is not configured  2021-12-09 22:52:49,866 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog  2021-12-09 22:52:49,871 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.  2021-12-09 22:52:49,875 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.resourcemanager is not defined  2021-12-09 22:52:49,882 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)  2021-12-09 22:52:49,886 INFO org.apache.hadoop.http.HttpServer2: Added filter RMAuthenticationFilter (class=org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter) to context cluster  2021-12-09 22:52:49,886 INFO org.apache.hadoop.http.HttpServer2: Added filter RMAuthenticationFilter (class=org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter) to context static  2021-12-09 22:52:49,886 INFO org.apache.hadoop.http.HttpServer2: Added filter RMAuthenticationFilter (class=org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter) to context logs  2021-12-09 22:52:49,886 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context cluster  2021-12-09 22:52:49,886 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static  2021-12-09 22:52:49,887 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs  2021-12-09 22:52:49,889 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /cluster/*  2021-12-09 22:52:49,889 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*  2021-12-09 22:52:50,094 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state STARTED; cause: com.google.inject.internal.util.$ComputationException: java.lang.ExceptionInInitializerError  com.google.inject.internal.util.$ComputationException: java.lang.ExceptionInInitializerError  at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:553)  at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:419)  at com.google.inject.internal.util.$CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)  at com.google.inject.internal.FailableCache.get(FailableCache.java:50)  at com.google.inject.internal.ConstructorInjectorStore.get(ConstructorInjectorStore.java:49)  at com.google.inject.internal.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:125)  at com.google.inject.internal.InjectorImpl.initializeBinding(InjectorImpl.java:507)  at com.google.inject.internal.AbstractBindingProcessor$Processor$1.run(AbstractBindingProcessor.java:159)  at com.google.inject.internal.ProcessedBindingData.initializeBindings(ProcessedBindingData.java:44)  at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:122)  at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)  at com.google.inject.Guice.createInjector(Guice.java:95)  at com.google.inject.Guice.createInjector(Guice.java:72)  at com.google.inject.Guice.createInjector(Guice.java:62)  at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:356)  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:401)  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1133)  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1242)  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1443)  Caused by: java.lang.ExceptionInInitializerError  at com.google.inject.internal.cglib.reflect.$FastClassEmitter.lt;initgt;(FastClassEmitter.java:67)  at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72)  at com.google.inject.internal.cglib.core.$DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25)  at com.google.inject.internal.cglib.core.$AbstractClassGenerator.create(AbstractClassGenerator.java:216)  at com.google.inject.internal.cglib.reflect.$FastClass$Generator.create(FastClass.java:64)  at com.google.inject.internal.BytecodeGen.newFastClass(BytecodeGen.java:207)  at com.google.inject.internal.DefaultConstructionProxyFactory.create(DefaultConstructionProxyFactory.java:53)  at com.google.inject.internal.ProxyFactory.create(ProxyFactory.java:153)  at com.google.inject.internal.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:89)  at com.google.inject.internal.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:28)  at com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:36)  at com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:32)  at com.google.inject.internal.FailableCache$1.apply(FailableCache.java:39)  at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:549)  ... 19 more  Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @1a27aae3  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)  at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)  at java.base/java.lang.reflect.Method.setAccessible(Method.java:193)  at com.google.inject.internal.cglib.core.$ReflectUtils$2.run(ReflectUtils.java:56)  at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)  at com.google.inject.internal.cglib.core.$ReflectUtils.lt;clinitgt;(ReflectUtils.java:46)  ... 33 more  2021-12-09 22:52:50,099 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state  2021-12-09 22:52:50,100 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state  2021-12-09 22:52:50,100 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager  com.google.inject.internal.util.$ComputationException: java.lang.ExceptionInInitializerError  at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:553)  at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:419)  at com.google.inject.internal.util.$CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)  at com.google.inject.internal.FailableCache.get(FailableCache.java:50)  at com.google.inject.internal.ConstructorInjectorStore.get(ConstructorInjectorStore.java:49)  at com.google.inject.internal.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:125)  at com.google.inject.internal.InjectorImpl.initializeBinding(InjectorImpl.java:507)  at com.google.inject.internal.AbstractBindingProcessor$Processor$1.run(AbstractBindingProcessor.java:159)  at com.google.inject.internal.ProcessedBindingData.initializeBindings(ProcessedBindingData.java:44)  at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:122)  at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)  at com.google.inject.Guice.createInjector(Guice.java:95)  at com.google.inject.Guice.createInjector(Guice.java:72)  at com.google.inject.Guice.createInjector(Guice.java:62)  at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:356)  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:401)  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1133)  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1242)  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1443)  Caused by: java.lang.ExceptionInInitializerError  at com.google.inject.internal.cglib.reflect.$FastClassEmitter.lt;initgt;(FastClassEmitter.java:67)  at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72)  at com.google.inject.internal.cglib.core.$DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25)  at com.google.inject.internal.cglib.core.$AbstractClassGenerator.create(AbstractClassGenerator.java:216)  at com.google.inject.internal.cglib.reflect.$FastClass$Generator.create(FastClass.java:64)  at com.google.inject.internal.BytecodeGen.newFastClass(BytecodeGen.java:207)  at com.google.inject.internal.DefaultConstructionProxyFactory.create(DefaultConstructionProxyFactory.java:53)  at com.google.inject.internal.ProxyFactory.create(ProxyFactory.java:153)  at com.google.inject.internal.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:89)  at com.google.inject.internal.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:28)  at com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:36)  at com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:32)  at com.google.inject.internal.FailableCache$1.apply(FailableCache.java:39)  at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:549)  ... 19 more  Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @1a27aae3  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)  at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)  at java.base/java.lang.reflect.Method.setAccessible(Method.java:193)  at com.google.inject.internal.cglib.core.$ReflectUtils$2.run(ReflectUtils.java:56)  at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)  at com.google.inject.internal.cglib.core.$ReflectUtils.lt;clinitgt;(ReflectUtils.java:46)  ... 33 more  2021-12-09 22:52:50,106 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:   /************************************************************  SHUTDOWN_MSG: Shutting down ResourceManager at Salishs-MacBook-Pro.local/127.0.0.1  ************************************************************/