Environmnet

• Кластеры (устройства VMware)
  • Apache Hadoop v2.6.5 (официальная версия Apache)
  • Apache Hive v1.2.1 (официальная версия Apache)
  • MySQL Community Server v5.1.73
• Хост (физическое устройство)
  • JetBrains IntelliJ IDEA 2020.2.3 (Ultimate Edition)
  • Принять OpenJDK 1.8.0_265

Проблема

После запуска HDFS, YARN, Hadoop History Server, Hive Metastore и HiveServer2 я подключился к Hive удаленно через плагин IntelliJ IDEA «Database». Я УСПЕШНО подключился и выполнил свои *.hql скрипты без каких-либо исключений, но в плагине «Базы данных» ничего не было показано, кроме самой конфигурации. Базы данных, таблицы или столбцы отсутствовали.

При подключении к MySQL не было НИЧЕГО ПЛОХОГО. Параметры сортировки, пользователи, схемы, таблицы, столбцы и даже ключи (первичные или внешние) были показаны в плагине.

Содержимое, отображаемое в плагине «Database»

Драйверы для Apache Hive в IntelliJ IDEA

Я использовал эти файлы JAR в качестве драйверов Hive. Все файлы JAR были скопированы из моего кластера через Xftp6.

• commons-httpclient-3.0.1
• commons-logging-1.1.3
• куратор-клиент-2.6.0
• куратор-фреймворк-2.6.0
• hadoop-auth-2.6.5
• hadoop-common-2.6.5
• hive-common-1.2.1
• hive-exec-1.2.1
• hive-jdbc-1.2.1
• hive-metastore-1.2.1
• hive-service-1.2.1
• httpclient-4.4
• httpcore-4.4
• slf4j-api-1.7.5
• zookeeper-3.4.6

Прилагаемые сведения

24 октября 2020 г.

Я переключил драйвер Apache Hive на ver.3.1.2 [stable] предоставленный JetBrains, и я получил эти исключения при «тестовом подключении»:

 [ 08S01] Could not open client transport with JDBC Uri: jdbc:hive2://master:10000: Could not establish connection to jdbc:hive2://master:10000: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000, use:database=default})
org.apache.hive.org.apache.thrift.TApplicationException: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000, use:database=default}).
  

Я также переключился на ver.3.1.1 [stable] и получил те же исключения, что и выше.

В 3-й раз я переключился на «Пользовательские банки …» и добавил hadoop-common-2.6.5.jar и hive-jdbc-1.2.1-standalone.jar загрузил из центрального репозитория Maven. При «Тестовом подключении» не возникает исключений:

 DBMS: Apache Hive (ver. 1.2.1)
Case sensitivity: plain=mixed, delimited=mixed
Driver: Hive JDBC (ver. 1.2.1, JDBC3.0)
Ping: 265 ms
  

«OK» and refresh «Database» panel again. I got a popup and these messages shown in «Event log»:

Popups by «Event log»

 2020/10/24
17:23   Connected

17:23   Hive (master)
            enabling autocommit is not supported.
  

Went to «Help -> Tail Log in Console», these were exceptions when I refresh Hive connection:

 2020-10-24 17:37:18,068 [2272449]   INFO - re.component.SaveActionManager - [ ] Start SaveActionManager#beforeAllDocumentsSaving 
2020-10-24 17:37:18,068 [2272449]   INFO - re.component.SaveActionManager - Locating psi files for 0 documents: [] 
2020-10-24 17:37:18,068 [2272449]   INFO - re.component.SaveActionManager - End SaveActionManager#beforeAllDocumentsSaving 
2020-10-24 17:37:18,100 [2272481]   INFO - atabaseCredentialsAuthProvider - Connecting as: root 
2020-10-24 17:37:18,100 [2272481]   INFO - .DatabaseConnectionEstablisher - Connecting to: jdbc:hive2://master:10000 
2020-10-24 17:37:18,104 [2272485]   WARN - ution.rmi.RemoteProcessSupport - 10月 24, 2020 9:37:18 上午 org.apache.hive.jdbc.Utils parseURL 
2020-10-24 17:37:18,105 [2272486]   WARN - ution.rmi.RemoteProcessSupport - 信息: Supplied authorities: master:10000 
2020-10-24 17:37:18,105 [2272486]   WARN - ution.rmi.RemoteProcessSupport - 10月 24, 2020 9:37:18 上午 org.apache.hive.jdbc.Utils parseURL 
2020-10-24 17:37:18,105 [2272486]   WARN - ution.rmi.RemoteProcessSupport - 信息: Resolved authority: master:10000 
2020-10-24 17:37:18,109 [2272490]   WARN - ution.rmi.RemoteProcessSupport - 10月 24, 2020 9:37:18 上午 org.apache.hive.jdbc.HiveConnection openTransport 
2020-10-24 17:37:18,109 [2272490]   WARN - ution.rmi.RemoteProcessSupport - 信息: Will try to open client transport with JDBC Uri: jdbc:hive2://master:10000 
2020-10-24 17:37:18,184 [2272565]   INFO - ution.rmi.RemoteProcessSupport - Driver: Hive JDBC 1.2.1 
2020-10-24 17:37:18,188 [2272569]   INFO - ution.rmi.RemoteProcessSupport - getDatabaseProductName: Apache Hive 
2020-10-24 17:37:18,188 [2272569]   INFO - ution.rmi.RemoteProcessSupport - release_version: 1.2.1 
2020-10-24 17:37:18,188 [2272569]   INFO - ution.rmi.RemoteProcessSupport - Detected: HIVE 1.2.1 
2020-10-24 17:37:18,391 [2272772]   WARN - lij.database.util.ErrorHandler - enabling autocommit is not supported 
enabling autocommit is not supported
The SQL statement:
connect

    at com.intellij.database.remote.jdba.jdbc.BaseExceptionRecognizer.recognizeException(BaseExceptionRecognizer.java:48)
    at com.intellij.database.remote.jdba.jdbc.JdbcIntermediateFacade.connect(JdbcIntermediateFacade.java:87)
    at com.intellij.database.remote.jdba.jdbc.UnknownDatabaseIntermediateFacade.connect(UnknownDatabaseIntermediateFacade.java:41)
    at com.intellij.database.remote.jdba.impl.RemoteFacadeImpl.connect(RemoteFacadeImpl.java:23)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359)
    at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
    at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
    at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
    at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
    at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
    at java.rmi/java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:217)
    at java.rmi/java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:162)
    at com.sun.proxy.$Proxy189.connect(Unknown Source)
    at com.intellij.database.dialects.base.introspector.AuditFacade.connect(AuditFacade.kt:59)
    at com.intellij.database.layoutedQueries.impl.AdaptIntermediateFacade.connect(AdaptIntermediateFacade.java:26)
    at com.intellij.database.layoutedQueries.BaseFacade.connect(BaseFacade.java:32)
    at com.intellij.database.dialects.base.introspector.DBFacadeProvider.createFacadeWithSingleLeasedConnection(DBFacadeProvider.kt:34)
    at com.intellij.database.dialects.base.introspector.jdbc.JdbcIntrospector.attachToDB(JdbcIntrospector.java:138)
    at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.initIntrospector(DatabaseModelLoader.java:560)
    at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.prepareIntrospection(DatabaseModelLoader.java:547)
    at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$introspect$1(DatabaseModelLoader.java:314)
    at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.withFacade(DatabaseModelLoader.java:570)
    at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$introspect$2(DatabaseModelLoader.java:312)
    at com.intellij.database.dataSource.DataSourceSyncManager.lambda$static$0(DataSourceSyncManager.java:51)
    at com.intellij.database.console.session.SessionsUtil$function$1.invoke(SessionsUtil.kt:20)
    at com.intellij.database.console.session.DatabaseSessionManager$Facade.compute(DatabaseSessionManager.kt:338)
    at com.intellij.database.console.session.DatabaseSessionManager$Facade.runSync(DatabaseSessionManager.kt:273)
    at com.intellij.database.dataSource.DataSourceSyncManager.lambda$static$1(DataSourceSyncManager.java:49)
    at com.intellij.database.dataSource.DataSourceSyncManager$SyncProcessor$1.perform(DataSourceSyncManager.java:306)
    at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.introspect(DatabaseModelLoader.java:312)
    at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$run$0(DatabaseModelLoader.java:200)
    at com.intellij.database.dataSource.LocalDataSource.performBatch(LocalDataSource.java:1333)
    at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.run(DatabaseModelLoader.java:198)
    at com.intellij.database.dataSource.DataSourceSyncManager$SyncProcessor.performSync(DataSourceSyncManager.java:308)
    at com.intellij.database.util.AsyncTask$Frame$compute$$inlined$supply$1$1.run(AsyncTask.kt:486)
    at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:588)
    at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:60)
    at com.intellij.database.util.AsyncTask$Frame$compute$$inlined$supply$1.get(AsyncTask.kt:515)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:629)
    at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:581)
    at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:60)
    at com.intellij.database.dataSource.AsyncUtil.underProgress(AsyncUtil.java:180)
    at com.intellij.database.dataSource.AsyncUtil.lambda$asyncFriendlyExecutor$1(AsyncUtil.java:56)
    at com.intellij.util.RunnableCallable.call(RunnableCallable.java:20)
    at com.intellij.util.RunnableCallable.call(RunnableCallable.java:11)
    at com.intellij.openapi.application.impl.ApplicationImpl$1.call(ApplicationImpl.java:268)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: enabling autocommit is not supported
    at org.apache.hive.jdbc.HiveConnection.setAutoCommit(HiveConnection.java:1199)
    at com.intellij.database.remote.jdba.jdbc.pooling.ConnectionPool.prepareConnectionAfterConnected(ConnectionPool.java:103)
    at com.intellij.database.remote.jdba.jdbc.pooling.ConnectionPool.obtainOneConnectionIntoPool(ConnectionPool.java:86)
    at com.intellij.database.remote.jdba.jdbc.pooling.ConnectionPool.connect(ConnectionPool.java:66)
    at com.intellij.database.remote.jdba.jdbc.JdbcIntermediateFacade.connect(JdbcIntermediateFacade.java:84)
    at com.intellij.database.remote.jdba.jdbc.UnknownDatabaseIntermediateFacade.connect(UnknownDatabaseIntermediateFacade.java:41)
    at com.intellij.database.remote.jdba.impl.RemoteFacadeImpl.connect(RemoteFacadeImpl.java:23)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359)
    at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
    at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more
2020-10-24 17:37:49,447 [2303828]   INFO - re.component.SaveActionManager - [ ] Start SaveActionManager#beforeAllDocumentsSaving 
2020-10-24 17:37:49,447 [2303828]   INFO - re.component.SaveActionManager - Locating psi files for 0 documents: [] 
2020-10-24 17:37:49,448 [2303829]   INFO - re.component.SaveActionManager - End SaveActionManager#beforeAllDocumentsSaving 
2020-10-24 17:37:49,875 [2304256]   INFO - rationStore.ComponentStoreImpl - Saving appBigDataIdeGlobalConnectionSettings took 38 ms 
2020-10-24 17:37:52,641 [2307022]   INFO - re.component.SaveActionManager - [ ] Start SaveActionManager#beforeAllDocumentsSaving 
2020-10-24 17:37:52,641 [2307022]   INFO - re.component.SaveActionManager - Locating psi files for 0 documents: [] 
2020-10-24 17:37:52,641 [2307022]   INFO - re.component.SaveActionManager - End SaveActionManager#beforeAllDocumentsSaving 
2020-10-24 17:37:55,642 [2310023]   INFO - re.component.SaveActionManager - [ ] Start SaveActionManager#beforeAllDocumentsSaving 
2020-10-24 17:37:55,642 [2310023]   INFO - re.component.SaveActionManager - Locating psi files for 0 documents: [] 
2020-10-24 17:37:55,642 [2310023]   INFO - re.component.SaveActionManager - End SaveActionManager#beforeAllDocumentsSaving 
<Nearest above 3 lines repeated many times ...>
  

Вот журналы SQL, созданные сегодня:

 -- 2020-10-24 17:17:57 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943027] connected
-- [ --------------------------------------------------
-- 2020-10-24 17:17:57 [C06_Spark-TV-Massive] [Hive (master)] [unknown] [session id: 1539943027] [statement id: 1846946385] 
SELECT 'keep alive'
-- ] --------------------------------------------------
-- 2020-10-24 17:18:00 [C06_Spark-TV-Massive] [Hive (master)] [unknown] [session id: 1539943027] [statement id: 1846946385] finished - execution time: 3 s 211 ms
-- 2020-10-24 17:18:00 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943027] disconnected
-- 2020-10-24 17:21:05 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943028] connected
-- [ --------------------------------------------------
-- 2020-10-24 17:21:05 [C06_Spark-TV-Massive] [Hive (master)] [unknown] [session id: 1539943028] [statement id: 1846946386] 
SELECT 'keep alive'
-- ] --------------------------------------------------
-- 2020-10-24 17:21:05 [C06_Spark-TV-Massive] [Hive (master)] [unknown] [session id: 1539943028] [statement id: 1846946386] finished - execution time: 211 ms
-- 2020-10-24 17:21:05 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943028] disconnected
-- 2020-10-24 17:22:24 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943029] connected
-- 2020-10-24 17:22:25 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943029] disconnected
-- 2020-10-24 17:23:50 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943030] connected
-- 2020-10-24 17:23:51 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943030] disconnected
-- 2020-10-24 17:33:43 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943031] connected
-- 2020-10-24 17:33:43 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943031] disconnected
-- 2020-10-24 17:37:18 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943032] connected
-- 2020-10-24 17:37:18 [C06_Spark-TV-Massive] [Hive (master)] [session id: 1539943032] disconnected
  

На панели «База данных» по-прежнему не отображаются схемы, таблицы или столбцы…

Есть ли что-нибудь, что я не настроил или не включил, чтобы Hive не мог отображать схемы и таблицы для меня? Я мог только переключить службу IntelliLang с уровня «Проверки» на уровень «Синтаксис», чтобы избежать неожиданных «ошибок» в *.hql сценариях на данный момент.

Спасибо!