学习 Druid (二):搭建 Druid 集群

更新至 Druid 0.15.1 版本

依赖

  • Metadata Storage:MySQL 5.6.17
  • ZooKeeper:Apache Zookeeper 3.4.5
  • Deep Storage:Apache Hadoop HDFS 2.6.0

下载

下载 Druid Binary 分发包,解压缩至本地。

配置

配置 Metadata Storage

1. 安装 MySQL Driver 软件包

下载 mysql-connector-java.jar(百度网盘,提取码:mk7u)到 extensions/mysql-metadata-storage/ 目录。

2. 创建 druid 数据库

登录 MySQL 数据库,执行 SQL 创建 druid 数据库:

CREATE DATABASE druid DEFAULT CHARACTER SET utf8mb4;  
3. 编辑配置文件

编辑 conf/druid/cluster/_common/common.runtime.properties 文件。

加载 MySQL Metadata Storage 扩展:

druid.extensions.loadList=["mysql-metadata-storage"]  

配置 Metadata storage:

# For MySQL:
druid.metadata.storage.type=mysql  
druid.metadata.storage.connector.connectURI=jdbc:mysql://<METADATA_STORAGE_HOST>:3306/druid  
druid.metadata.storage.connector.user=<USER>  
druid.metadata.storage.connector.password=<SECRET>  

配置 ZooKeeper

编辑 conf/druid/cluster/_common/common.runtime.properties 文件。

配置 ZooKeeper 集群,多个 IP 使用逗号分隔:

druid.zk.service.host=<ZK_HOST>:2181,<ZK_HOST>:2181,<ZK_HOST>:2181  
druid.zk.paths.base=/druid  

配置 Deep Storage

1. 编辑配置文件

编辑 conf/druid/cluster/_common/common.runtime.properties 文件。

加载 HDFS 存储扩展:

druid.extensions.loadList=["druid-hdfs-storage"]  

配置 Deep Storage 目录:

druid.storage.type=hdfs  
druid.storage.storageDirectory=/druid/segments  

编辑索引服务日志目录:

druid.indexer.logs.type=hdfs  
druid.indexer.logs.directory=/druid/indexing-logs  
2. 拷贝 HDFS 配置文件

拷贝 Hadoop 配置文件 core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml 到 conf/druid/_common/ 目录下。

启动服务

启动 Master Server:

bin/start-cluster-master-no-zk-server  

启动 Data Server:

bin/start-cluster-data-server  

启动 Query Server:

bin/start-cluster-query-server  

浏览器访问:http://DRUID_MASTER:8888

Trouble Shooting

时区不一致

Druid 时间比系统时间慢了八个小时。

编辑以下文件:

  • conf/druid/cluster/master/coordinator-overlord/jvm.config
  • conf/druid/cluster/query/broker/jvm.config
  • conf/druid/cluster/query/router/jvm.config
  • conf/druid/cluster/data/middleManager/jvm.config
  • conf/druid/cluster/data/historical/jvm.config

修改 -Duser.timezone=UTC-Duser.timezone=Asia/Shanghai

编辑 conf/druid/cluster/data/middleManager/runtime.properties 文件,修改 druid.indexer.runner.javaOpts 配置项中 -Duser.timezone=UTC-Duser.timezone=Asia/Shanghai

启动 Master Server 报 Index column size too large. 错误

异常信息:

Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Index column size too large. The maximum column size is 767 bytes. [statement:"null", located:"null", rewritten:"null", arguments:null]  
    at org.skife.jdbi.v2.Batch.execute(Batch.java:131) ~[jdbi-2.63.1.jar:2.63.1]
    at org.apache.druid.metadata.SQLMetadataConnector$2.withHandle(SQLMetadataConnector.java:189) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
    at org.apache.druid.metadata.SQLMetadataConnector$2.withHandle(SQLMetadataConnector.java:179) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
    at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
    ... 15 more
Caused by: java.sql.BatchUpdateException: Index column size too large. The maximum column size is 767 bytes.  
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_212]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_212]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[?:?]
    at com.mysql.jdbc.Util.getInstance(Util.java:387) ~[?:?]
    at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1154) ~[?:?]
    at com.mysql.jdbc.StatementImpl.executeBatchInternal(StatementImpl.java:1049) ~[?:?]
    at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:959) ~[?:?]
    at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
    at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
    at org.skife.jdbi.v2.Batch.execute(Batch.java:121) ~[jdbi-2.63.1.jar:2.63.1]
    at org.apache.druid.metadata.SQLMetadataConnector$2.withHandle(SQLMetadataConnector.java:189) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
    at org.apache.druid.metadata.SQLMetadataConnector$2.withHandle(SQLMetadataConnector.java:179) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
    at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
    ... 15 more
Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes.  
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957) ~[?:?]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878) ~[?:?]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814) ~[?:?]
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2478) ~[?:?]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2625) ~[?:?]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2547) ~[?:?]
    at com.mysql.jdbc.StatementImpl.executeUpdateInternal(StatementImpl.java:1541) ~[?:?]
    at com.mysql.jdbc.StatementImpl.executeBatchInternal(StatementImpl.java:1023) ~[?:?]
    at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:959) ~[?:?]
    at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
    at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
    at org.skife.jdbi.v2.Batch.execute(Batch.java:121) ~[jdbi-2.63.1.jar:2.63.1]
    at org.apache.druid.metadata.SQLMetadataConnector$2.withHandle(SQLMetadataConnector.java:189) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
    at org.apache.druid.metadata.SQLMetadataConnector$2.withHandle(SQLMetadataConnector.java:179) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
    at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
    ... 15 more

解决方案:

参考 Specified key was too long; max key length is 767 bytes · issue #6781 的解决方案,使用 utf8 代替 utf8mb4 重建数据库:

DROP DATABASE druid;  
CREATE DATABASE druid DEFAULT CHARACTER SET utf8;  

参考