首页 > 开发 > Java > 正文

Hbase、elasticsearch整合中jar包冲突的问题解决

2024-07-13 10:13:22
字体:
来源:转载
供稿:网友

问题背景

再数据平台中,项目搭建需要使用es和HBASE搭建数据查询接口,整合的过程中出现jar包冲突的bug :com.google.common.base.Stopwatch.()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:239)    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:150)    ...    at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntil    ...    at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)    ... 45 more

解决办法

经排查,确认是因为 com.google.guava 这个包引起的冲突。es依赖于18及以上版本,而HBASE只支持16及以上版本。而guava这个包从17开始内部发生变化,方法发生改变,所以18并不会向下兼容16版本。在项目运行的过程中,如果同时引入16、18版本,es及hbase的调用过程会发生混乱。那么接下来就好办了,我们可以重新打包,将guava18打入es内部,再在pom文件中显示引用guava16版本的包。这样,es调用包内部打入的guava18,而hbase调用外部的guava16。

重新打包

新建maven工程,在pom文件中进行如下配置:

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0"     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">  <modelVersion>4.0.0</modelVersion>  <!--    this maven project is aim to fix the jar comflic in hbase & elasticsearch.    in es, required guava 18+ or up. but in hbase you should use guava 16- or blow.    cd the project directory and run 'sh ./cleanbuild.sh',you will get a new-self jar.    then, adjust your project pom.xml whth:    <dependency>      <groupId>douguo.shaded.elasticsearch</groupId>      <artifactId>douguo_shaded_elasticsearch</artifactId>      <version>1.0-SNAPSHOT</version>      <exclusions>        <exclusion>          <groupId>org.elasticsearch</groupId>          <artifactId>elasticsearch</artifactId>        </exclusion>      </exclusions>    </dependency>    finally, your jar comflic will be fixed.    @date:2017-11-30    @author:zhangjianfei    @since:douguo.shaded.elasticsearch-1.0.0  -->  <groupId>douguo.shaded.elasticsearch</groupId>  <artifactId>douguo_shaded_elasticsearch</artifactId>  <version>1.0-SNAPSHOT</version>  <properties>    <elasticsearch.version>2.4.1</elasticsearch.version>  </properties>  <dependencies>    <dependency>      <groupId>org.elasticsearch</groupId>      <artifactId>elasticsearch</artifactId>      <version>${elasticsearch.version}</version>    </dependency>    <dependency>      <groupId>org.elasticsearch.plugin</groupId>      <artifactId>shield</artifactId>      <version>${elasticsearch.version}</version>    </dependency>  </dependencies>  <build>    <plugins>      <plugin>        <groupId>org.apache.maven.plugins</groupId>        <artifactId>maven-shade-plugin</artifactId>        <version>2.4.1</version>        <configuration>          <createDependencyReducedPom>false</createDependencyReducedPom>        </configuration>        <executions>          <execution>            <phase>package</phase>            <goals>              <goal>shade</goal>            </goals>            <configuration>              <relocations>                <relocation>                  <pattern>com.google.guava</pattern>                  <shadedPattern>douguo.shaded.elasticsearch.guava</shadedPattern>                </relocation>                <relocation>                  <pattern>org.joda</pattern>                  <shadedPattern>douguo.shaded.elasticsearch.joda</shadedPattern>                </relocation>                <relocation>                  <pattern>com.google.common</pattern>                  <shadedPattern>douguo.shaded.elasticsearch.common</shadedPattern>                </relocation>                <relocation>                  <pattern>com.google.thirdparty</pattern>                  <shadedPattern>douguo.shaded.elasticsearch.thirdparty</shadedPattern>                </relocation>              </relocations>              <transformers>                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />              </transformers>            </configuration>          </execution>        </executions>      </plugin>    </plugins>  </build>  <repositories>    <repository>      <id>elasticsearch-releases</id>      <url>http://maven.elasticsearch.org/releases</url>      <releases>        <enabled>true</enabled>        <updatePolicy>daily</updatePolicy>      </releases>      <snapshots>        <enabled>false</enabled>      </snapshots>    </repository>  </repositories></project>

将org.joda等4个可能有冲突的jar包通过maven-shade-plugin插件迁移后重新打个jar包从而使得在引入这个jar包时能够使用该jar包自己的依赖而不是使用外部依赖。这里需要注意的是,需要将com.google.common等4个包全部重新迁移,否则会出现java.lang.IllegalAccessError: tried to access method com.google.common.base的错误

项目打包

mvn clean install

新的依赖包会在.m2 maven仓库中,如果公司搭建了仓库的话,需要上传jar包。如果直接运行jar包的话,记得重新编译项目,并替换lib目录

项目载入新包

只需要在pom文件中配置:

<!-- douguo.shaded.elasticsearch --><dependency>  <groupId>douguo.shaded.elasticsearch</groupId>  <artifactId>douguo_shaded_elasticsearch</artifactId>  <version>1.0-SNAPSHOT</version>  <exclusions>    <exclusion>      <groupId>org.elasticsearch</groupId>      <artifactId>elasticsearch</artifactId>    </exclusion>  </exclusions></dependency><!-- https://mvnrepository.com/artifact/com.google.guava/guava --><!-- this guava is only used in habse   in es, 18.0+ is required, but hbase only supported 16.0 or blow.   clean as install douguo.shaded.elasticsearch --><dependency>  <groupId>com.google.guava</groupId>  <artifactId>guava</artifactId>  <version>16.0</version></dependency>

这样,guava18包就在douguo.shaded.elasticsearch下,es会优先调用。而外部配置的guava16就会被HBASE调用。2个版本的jar包互相独立存在!

到此,问题就解决了!

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持VeVb武林网。


注:相关教程知识阅读请移步到JAVA教程频道。
发表评论 共有条评论
用户名: 密码:
验证码: 匿名发表