学习 Airflow(二):部署

更新至 Airflow 1.10.6 版本

环境:

  • CentOS 7
  • Python 3.6.8
  • pip 9.0.3
  • MariaDB 10.4

数据库

编辑 /etc/my.cnf.d/server.cnf 文件:

[mysqld]
explicit_defaults_for_timestamp=1  

重启数据库:

systemctl restart mariadb  

创建 airflow 用户:

create user 'airflow'@'%' identified by '<password>';  

创建 airflow 数据库:

create database airflow character set = 'utf8';  

授权:

grant all privileges on airflow.* to 'airflow'@'%';  

安装

安装 gcc 和 python-devel 库:

yum install gcc python3-devel  

使用 pip 安装 Airflow:

pip install apache-airflow  

安装子包 mysql:

pip install 'apache-airflow[mysql]'  

(可选)编辑 ~/.bash_profile 文件,添加环境变量 $AIRFLOW_HOME,默认为 ~/airflow。

执行 airflow webserver 在 $AIRFLOW_HOME 下生成配置文件。编辑 airflow.cfg 文件,修改数据库连接配置:

sql_alchemy_conn = mysql://airflow:<password>@localhost:3306/airflow  

初始化数据库:

airflow initdb  

在 8080 端口启动服务:

airflow webserver -p 8080 -D  

Trouble shooting

安装 mysqlclient 报错 oserror mysql_config not found

报错信息:

/bin/sh: mysql_config: command not found
Traceback (most recent call last):  
  File "<string>", line 1, in <module>
  File "/tmp/pip-build-arjr4837/mysqlclient/setup.py", line 16, in <module> metadata, options = get_config()
  File "/tmp/pip-build-arjr4837/mysqlclient/setup_posix.py", line 53, in get_config libs = mysql_config("libs_r")
  File "/tmp/pip-build-arjr4837/mysqlclient/setup_posix.py", line 28, in mysql_config raise EnvironmentError("%s not found" % (mysql_config.path,))
OSError: mysql_config not found  

解决方案:

yum install mariadb-devel  

初始化数据库报错 Unknown system variable 'explicit_defaults_for_timestamp'

报错信息:

Traceback (most recent call last):  
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 581, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 255, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib64/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
  File "/usr/local/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 252, in execute
    res = self._query(query)
  File "/usr/local/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 378, in _query
    db.query(q)
  File "/usr/local/lib64/python3.6/site-packages/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1193, "Unknown system variable 'explicit_defaults_for_timestamp'")  

解决方案:

数据库版本过低,MariaDB 需要 10.1.8 以上,MySQL 需要 5.6.4 以上。

参考