#java #apache-spark #data-binding #version
Вопрос:
Для проекта я использую структурированную потоковую передачу spark с кафкой.
У меня есть это признание :
<spark.version>3.1.1</spark.version>
<slf4j.version>1.7.7</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<scala.version>2.12</scala.version>
Но когда я бегу, у меня возникает эта ошибка :
Вызвано: com.fasterxml.jackson.databind.Исключение JsonMappingException: для модуля Scala 2.12.3 требуется версия Jackson Databind >= 2.12.0 и >
Я в замешательстве, потому что я уже импортировал привязку данных в 2.12.3. Только здесь, выписка из моего pom.xml :
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_${scala.version}</artifactId>
<version>2.12.3</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.12.3</version>
</dependency>
заранее благодарю вас за помощь
Ответ №1:
Аллен Д. Болл, мой ответ слишком длинный для комментариев. Итак, pom.xml :
4.0.0
<parent>
<groupId>fr.cnp.kfk</groupId>
<artifactId>jee</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>kfk-spark-structured-streams</artifactId>
<packaging>jar</packaging>
<name>Spark Structured Streams</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<spark.version>3.1.1</spark.version>
<slf4j.version>1.7.7</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<scala.version>2.12</scala.version>
<jackson-bom.version>2.12.3</jackson-bom.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.flink:force-shading</exclude>
<exclude>com.google.code.findbugs:jsr305</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>fr.cnp.kfk.streams.Application</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerId>jdt</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-jdt</artifactId>
<version>0.21.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.eclipse.m2e</groupId>
<artifactId>lifecycle-mapping</artifactId>
<version>1.0.0</version>
<configuration>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<versionRange>[2.4,)</versionRange>
<goals>
<goal>single</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore />
</action>
</pluginExecution>
<pluginExecution>
<pluginExecutionFilter>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<versionRange>[3.1,)</versionRange>
<goals>
<goal>testCompile</goal>
<goal>compile</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore />
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<dependencies>
<dependency>
<groupId>fr.cnp.kfk</groupId>
<artifactId>kfk-commons</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.codehaus.janino</groupId>
<artifactId>janino</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.janino</groupId>
<artifactId>commons-compiler</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.12</artifactId>
<version>3.1.1</version>
<exclusions>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>com.opencsv</groupId>
<artifactId>opencsv</artifactId>
<version>4.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_${scala.version}</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<!-- Dépendences pour RecksDB -->
<!-- <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-statebackend-rocksdb_2.12</artifactId>
<version>${flink.version}</version> <scope>provided</scope> </dependency> -->
<dependency>
<artifactId>janino</artifactId>
<groupId>org.codehaus.janino</groupId>
<version>3.0.8</version>
</dependency>
<dependency>
<artifactId>commons-compiler</artifactId>
<groupId>org.codehaus.janino</groupId>
<version>3.0.8</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.9.0</version>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
<version>${jackson-bom.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
зависимость.дерево :
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ kfk-spark-structured-streams ---
[INFO] fr.cnp.kfk:kfk-spark-structured-streams:jar:0.0.1-SNAPSHOT
[INFO] - fr.cnp.kfk:kfk-commons:jar:0.0.1-SNAPSHOT:compile
[INFO] - org.apache.spark:spark-sql_2.12:jar:3.1.1:compile
[INFO] | - com.univocity:univocity-parsers:jar:2.9.1:compile
[INFO] | - org.apache.spark:spark-sketch_2.12:jar:3.1.1:compile
[INFO] | - org.apache.spark:spark-core_2.12:jar:3.1.1:compile
[INFO] | | - org.apache.avro:avro:jar:1.8.2:compile
[INFO] | | | - org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] | | | - org.tukaani:xz:jar:1.5:compile
[INFO] | | - org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:compile
[INFO] | | | - org.apache.avro:avro-ipc:jar:1.8.2:compile
[INFO] | | - com.twitter:chill_2.12:jar:0.9.5:compile
[INFO] | | | - com.esotericsoftware:kryo-shaded:jar:4.0.2:compile
[INFO] | | | - com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] | | | - org.objenesis:objenesis:jar:2.6:compile
[INFO] | | - com.twitter:chill-java:jar:0.9.5:compile
[INFO] | | - org.apache.hadoop:hadoop-client:jar:3.2.0:compile
[INFO] | | | - org.apache.hadoop:hadoop-common:jar:3.2.0:compile
[INFO] | | | | - com.google.guava:guava:jar:23.0:compile
[INFO] | | | | | - com.google.errorprone:error_prone_annotations:jar:2.0.18:compile
[INFO] | | | | | - com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] | | | | | - org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile [INFO] | | | | - commons-cli:commons-cli:jar:1.3.1:compile
[INFO] | | | | - org.apache.httpcomponents:httpclient:jar:4.5.5:compile
[INFO] | | | | | - org.apache.httpcomponents:httpcore:jar:4.4.9:compile
[INFO] | | | | - commons-io:commons-io:jar:2.5:compile
[INFO] | | | | - org.eclipse.jetty:jetty-servlet:jar:9.4.8.v20171121:compile
[INFO] | | | | - javax.servlet.jsp:jsp-api:jar:2.2:provided
[INFO] | | | | - org.apache.commons:commons-configuration2:jar:2.1.1:compile
[INFO] | | | | - com.google.re2j:re2j:jar:1.1:compile
[INFO] | | | | - com.google.code.gson:gson:jar:2.8.2:compile
[INFO] | | | | - org.apache.hadoop:hadoop-auth:jar:3.2.0:compile
[INFO] | | | | | - com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
[INFO] | | | | | | - com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile [INFO] | | | | | - net.minidev:json-smart:jar:2.3:compile
[INFO] | | | | | - net.minidev:accessors-smart:jar:1.2:compile
[INFO] | | | | - org.apache.curator:curator-client:jar:2.12.0:compile
[INFO] | | | | - org.apache.htrace:htrace-core4:jar:4.1.0-incubating:compile
[INFO] | | | | - org.apache.kerby:kerb-simplekdc:jar:1.0.1:compile
[INFO] | | | | | - org.apache.kerby:kerb-client:jar:1.0.1:compile
[INFO] | | | | | | - org.apache.kerby:kerby-config:jar:1.0.1:compile
[INFO] | | | | | | - org.apache.kerby:kerb-core:jar:1.0.1:compile
[INFO] | | | | | | | - org.apache.kerby:kerby-pkix:jar:1.0.1:compile
[INFO] | | | | | | | - org.apache.kerby:kerby-asn1:jar:1.0.1:compile
[INFO] | | | | | | | - org.apache.kerby:kerby-util:jar:1.0.1:compile
[INFO] | | | | | | - org.apache.kerby:kerb-common:jar:1.0.1:compile
[INFO] | | | | | | | - org.apache.kerby:kerb-crypto:jar:1.0.1:compile
[INFO] | | | | | | - org.apache.kerby:kerb-util:jar:1.0.1:compile
[INFO] | | | | | | - org.apache.kerby:token-provider:jar:1.0.1:compile
[INFO] | | | | | - org.apache.kerby:kerb-admin:jar:1.0.1:compile
[INFO] | | | | | - org.apache.kerby:kerb-server:jar:1.0.1:compile
[INFO] | | | | | | - org.apache.kerby:kerb-identity:jar:1.0.1:compile
[INFO] | | | | | - org.apache.kerby:kerby-xdr:jar:1.0.1:compile
[INFO] | | | | - org.codehaus.woodstox:stax2-api:jar:3.1.4:compile
[INFO] | | | | - com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile
[INFO] | | | | - dnsjava:dnsjava:jar:2.1.7:compile
[INFO] | | | - org.apache.hadoop:hadoop-hdfs-client:jar:3.2.0:compile
[INFO] | | | | - com.squareup.okhttp:okhttp:jar:2.7.5:compile
[INFO] | | | | - com.squareup.okio:okio:jar:1.6.0:compile
[INFO] | | | - org.apache.hadoop:hadoop-yarn-api:jar:3.2.0:compile
[INFO] | | | | - javax.xml.bind:jaxb-api:jar:2.3.0:provided
[INFO] | | | - org.apache.hadoop:hadoop-yarn-client:jar:3.2.0:compile
[INFO] | | | - org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.2.0:compile
[INFO] | | | | - org.apache.hadoop:hadoop-yarn-common:jar:3.2.0:compile
[INFO] | | | | - com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.3:compile
[INFO] | | | | | - jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
[INFO] | | | | | - jakarta.activation:jakarta.activation-api:jar:1.2.1:compile [INFO] | | | | - com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.3:compile
[INFO] | | | | - com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.12.3:compile
[INFO] | | | - org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:3.2.0:compile [INFO] | | | | - org.apache.hadoop:hadoop-mapreduce-client-common:jar:3.2.0:compile [INFO] | | | - org.apache.hadoop:hadoop-annotations:jar:3.2.0:compile
[INFO] | | - org.apache.spark:spark-launcher_2.12:jar:3.1.1:compile
[INFO] | | - org.apache.spark:spark-kvstore_2.12:jar:3.1.1:compile
[INFO] | | | - org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] | | - org.apache.spark:spark-network-common_2.12:jar:3.1.1:compile
[INFO] | | - org.apache.spark:spark-network-shuffle_2.12:jar:3.1.1:compile
[INFO] | | - org.apache.spark:spark-unsafe_2.12:jar:3.1.1:compile
[INFO] | | - javax.activation:activation:jar:1.1.1:provided
[INFO] | | - org.apache.curator:curator-recipes:jar:2.13.0:compile
[INFO] | | | - org.apache.curator:curator-framework:jar:2.13.0:compile
[INFO] | | - org.apache.zookeeper:zookeeper:jar:3.4.14:compile
[INFO] | | | - org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] | | - jakarta.servlet:jakarta.servlet-api:jar:4.0.3:compile
[INFO] | | - org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] | | - com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO] | | - org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] | | - org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] | | - org.slf4j:jcl-over-slf4j:jar:1.7.25:compile
[INFO] | | - log4j:log4j:jar:1.2.17:compile
[INFO] | | - org.slf4j:slf4j-log4j12:jar:1.7.25:compile
[INFO] | | - com.ning:compress-lzf:jar:1.0.3:compile
[INFO] | | - org.xerial.snappy:snappy-java:jar:1.1.8.2:compile
[INFO] | | - org.lz4:lz4-java:jar:1.7.1:compile
[INFO] | | - com.github.luben:zstd-jni:jar:1.4.8-1:compile
[INFO] | | - org.roaringbitmap:RoaringBitmap:jar:0.9.0:compile
[INFO] | | | - org.roaringbitmap:shims:jar:0.9.0:runtime
[INFO] | | - commons-net:commons-net:jar:3.1:compile
[INFO] | | - org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:compile
[INFO] | | - org.scala-lang:scala-reflect:jar:2.12.10:compile
[INFO] | | - org.json4s:json4s-jackson_2.12:jar:3.7.0-M5:compile
[INFO] | | | - org.json4s:json4s-core_2.12:jar:3.7.0-M5:compile
[INFO] | | | - org.json4s:json4s-ast_2.12:jar:3.7.0-M5:compile
[INFO] | | | - org.json4s:json4s-scalap_2.12:jar:3.7.0-M5:compile
[INFO] | | - org.glassfish.jersey.core:jersey-client:jar:2.25.1:compile
[INFO] | | | - javax.ws.rs:javax.ws.rs-api:jar:2.1:provided
[INFO] | | | - org.glassfish.hk2:hk2-api:jar:2.5.0-b32:compile
[INFO] | | | | - org.glassfish.hk2:hk2-utils:jar:2.5.0-b32:compile
[INFO] | | | | - org.glassfish.hk2.external:aopalliance-repackaged:jar:2.5.0-b32:compile
[INFO] | | | - org.glassfish.hk2.external:javax.inject:jar:2.5.0-b32:compile
[INFO] | | | - org.glassfish.hk2:hk2-locator:jar:2.5.0-b32:compile
[INFO] | | - org.glassfish.jersey.core:jersey-common:jar:2.25.1:compile
[INFO] | | | - javax.annotation:javax.annotation-api:jar:1.2:compile
[INFO] | | | - org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.25.1:compile
[INFO] | | | - org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
[INFO] | | - org.glassfish.jersey.core:jersey-server:jar:2.25.1:compile
[INFO] | | | - org.glassfish.jersey.media:jersey-media-jaxb:jar:2.25.1:compile
[INFO] | | | - javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] | | - org.glassfish.jersey.containers:jersey-container-servlet:jar:2.25.1:compile
[INFO] | | - org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:compile
[INFO] | | - org.glassfish.jersey.inject:jersey-hk2:jar:2.30:compile
[INFO] | | | - org.javassist:javassist:jar:3.21.0-GA:compile
[INFO] | | - io.netty:netty-all:jar:4.1.51.Final:compile
[INFO] | | - com.clearspring.analytics:stream:jar:2.9.6:compile
[INFO] | | - io.dropwizard.metrics:metrics-core:jar:3.1.5:compile
[INFO] | | - io.dropwizard.metrics:metrics-jvm:jar:3.1.5:compile
[INFO] | | - io.dropwizard.metrics:metrics-json:jar:3.1.5:compile
[INFO] | | - io.dropwizard.metrics:metrics-graphite:jar:3.1.5:compile
[INFO] | | - io.dropwizard.metrics:metrics-jmx:jar:4.1.1:compile
[INFO] | | - org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] | | - oro:oro:jar:2.0.8:compile
[INFO] | | - net.razorvine:pyrolite:jar:4.30:compile
[INFO] | | - net.sf.py4j:py4j:jar:0.10.9:compile
[INFO] | | - org.apache.commons:commons-crypto:jar:1.1.0:compile
[INFO] | - org.apache.spark:spark-catalyst_2.12:jar:3.1.1:compile
[INFO] | | - org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.1.2:compile [INFO] | | - org.antlr:antlr4-runtime:jar:4.8-1:compile
[INFO] | | - commons-codec:commons-codec:jar:1.10:compile
[INFO] | | - org.apache.arrow:arrow-vector:jar:2.0.0:compile
[INFO] | | | - org.apache.arrow:arrow-format:jar:2.0.0:compile
[INFO] | | | - org.apache.arrow:arrow-memory-core:jar:2.0.0:compile
[INFO] | | | - com.google.flatbuffers:flatbuffers-java:jar:1.9.0:compile
[INFO] | | - org.apache.arrow:arrow-memory-netty:jar:2.0.0:compile
[INFO] | - org.apache.spark:spark-tags_2.12:jar:3.1.1:compile
[INFO] | - org.apache.orc:orc-core:jar:1.5.12:compile
[INFO] | | - org.apache.orc:orc-shims:jar:1.5.12:compile
[INFO] | | - com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | | - commons-lang:commons-lang:jar:2.6:compile
[INFO] | | - io.airlift:aircompressor:jar:0.10:compile
[INFO] | | - org.threeten:threeten-extra:jar:1.5.0:compile
[INFO] | - org.apache.orc:orc-mapreduce:jar:1.5.12:compile
[INFO] | - org.apache.hive:hive-storage-api:jar:2.7.2:compile
[INFO] | - org.apache.parquet:parquet-column:jar:1.10.1:compile
[INFO] | | - org.apache.parquet:parquet-common:jar:1.10.1:compile
[INFO] | | - org.apache.parquet:parquet-encoding:jar:1.10.1:compile
[INFO] | - org.apache.parquet:parquet-hadoop:jar:1.10.1:compile
[INFO] | | - org.apache.parquet:parquet-format:jar:2.4.0:compile
[INFO] | | - org.apache.parquet:parquet-jackson:jar:1.10.1:compile
[INFO] | | - org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | | - org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | - org.apache.xbean:xbean-asm7-shaded:jar:4.15:compile
[INFO] | - org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] - org.apache.spark:spark-sql-kafka-0-10_2.12:jar:3.1.1:compile
[INFO] | - org.apache.spark:spark-token-provider-kafka-0-10_2.12:jar:3.1.1:compile
[INFO] | - org.apache.kafka:kafka-clients:jar:2.6.0:compile
[INFO] - org.apache.spark:spark-streaming_2.12:jar:3.1.1:compile
[INFO] | - org.scala-lang:scala-library:jar:2.12.10:compile
[INFO] - com.opencsv:opencsv:jar:4.1:compile
[INFO] | - org.apache.commons:commons-lang3:jar:3.5:compile
[INFO] | - org.apache.commons:commons-text:jar:1.1:compile
[INFO] | - commons-beanutils:commons-beanutils:jar:1.9.3:compile
[INFO] | - commons-collections:commons-collections:jar:3.2.2:compile
[INFO] - com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.12.3:compile [INFO] | - org.yaml:snakeyaml:jar:1.17:compile
[INFO] | - com.fasterxml.jackson.core:jackson-core:jar:2.12.3:compile
[INFO] - com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.12.3:runtime
[INFO] | - com.fasterxml.jackson.core:jackson-annotations:jar:2.12.3:compile
[INFO] | - com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] - com.fasterxml.jackson.core:jackson-databind:jar:2.12.3:compile
[INFO] - org.codehaus.janino:janino:jar:3.0.8:compile
[INFO] - org.codehaus.janino:commons-compiler:jar:3.0.8:compile
[INFO] - org.apache.commons:commons-pool2:jar:2.9.0:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.334 s [INFO] Finished at: 2021-06-02T09:19:26 02:00
[INFO] ------------------------------------------------------------------------
Ошибка :
Error: Bad target:null
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at fr.cnp.kfk.streams.Application.main(Application.java:14)
Caused by: org.apache.spark.sql.streaming.StreamingQueryException: null
[...]
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$runStream(StreamExecution.scala:354)
at org.apache.spark.sql.execution.streaming.StreamExecution$anon$1.run(StreamExecution.scala:244)
Caused by: java.lang.ExceptionInInitializerError
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:338)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2$(WriteToDataSourceV2Exec.scala:336)
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.writeWithV2(WriteToDataSourceV2Exec.scala:297)
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.run(WriteToDataSourceV2Exec.scala:304)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:46)
at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3696)
at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2965)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
at org.apache.spark.sql.Dataset.collect(Dataset.scala:2965)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:589)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$15(MicroBatchExecution.scala:584)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:584)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$runStream(StreamExecution.scala:333)
... 1 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.5 requires Jackson Databind version >= 2.10.0 and < 2.11.0
at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:747)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
... 40 more
Комментарии:
1. В вашем pom.xml, измените
<jackson-bom.version>2.12.3</jackson-bom.version><jackson-bom.version>2.10.5.20201202</jackson-bom.version>на мой ответ выше, и вы должны быть настроены,
Ответ №2:
Вы уверены в ошибке? Для Scala 2.12 требуется Джексон 2.10.x.
Для отладки запустите mvn dependency:tree , чтобы посмотреть, какие зависимости/версии на самом деле перетаскиваются. Вы, вероятно, получаете другие банки Джексона, которые не являются 2.10.x.
Если вы создаете проект Spring (с родительскими зависимостями POM spring-boot), установите:
<jackson-bom.version>2.10.5.20201202</jackson-bom.version>
для последней версии Jackson, которая удовлетворяет требованиям Scala. В противном случае вам нужно будет добавить следующее в свой <dependencyManagement/> :
<dependencyManagement>
<dependencies>
...
<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
<version>${jackson-bom.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
...
</dependencies>
</dependencyManagement>
и удалите <version> из вас зависимости Джексона.
Комментарии:
1. Спасибо за ваш ответ. Когда я пробую ваше точное предложение, у меня возникает следующая ошибка : Вызвано: com.fasterxml.jackson.databind. Исключение JsonMappingException: Для модуля Scala 2.10.5 требуется версия Jackson Databind >= 2.10.0 и >< 2.11.0 Я также пытался сделать : <jackson-bom.версия>2.12.3<jackson-bom.версия></jackson-bom.версия> В этом случае у меня та же ошибка, что и при запуске. Итак : Вызвано: com.fasterxml.jackson.databind. Исключение JsonMappingException: для модуля Scala 2.12.3 требуется версия Jackson Databind >= 2.12.0 и >
2. Пожалуйста, добавьте всю свою pom.xml к вопросу и выводу
mvn dependency:tree. В идеале, также, пожалуйста, добавьте вырезанное и вставленное исключение, которое вы получаете.3. Привет, я отвечаю вам в комментариях, потому что это слишком долго. Спасибо