ITEEDU

编程学习网


  • 首页

  • 标签

  • 分类

  • 归档

  • 关于

  • 旧站归档

  • 搜索

azkaban集群部署方案

发表于 2018-07-31 更新于 2019-06-29 分类于 架构 , 任务调度

集群部署就是部署一个server再部署多个executor。

executor要先一个个启动,启动后会向mysql的executors表插入数据。

最后启动server,server会从executors表找executor加载。

数据包正常在linux下按网上的步骤编译就行,这里mark一下相关配制。

server的conf配制

azkaban.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# Azkaban Personalization Settings
azkaban.name=azkaban-dev
azkaban.label=azkaban-dev
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects

# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties. #jetty服务器属性
jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.keystore=keystore
jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=keystore
jetty.trustpassword=azkaban
# Azkaban Executor settings
executor.port=12321
# mail settings #邮件配置(暂没有配置)
mail.sender=
mail.host=
mail.user=
mail.password=
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes

database.type=mysql
mysql.port=3306
mysql.host=xxx
mysql.database=xxx
mysql.user=xxxx
mysql.password=xxxxx
mysql.numconnections=100

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

log4j.properties

1
2
3
4
5
6
7
8
9
10
11
log4j.rootLogger=INFO, Console
log4j.logger.azkaban=INFO, server
log4j.appender.server=org.apache.log4j.RollingFileAppender
log4j.appender.server.layout=org.apache.log4j.PatternLayout
log4j.appender.server.File=logs/azkaban-web.log
log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
log4j.appender.server.MaxFileSize=102400MB
log4j.appender.server.MaxBackupIndex=2
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n

executor的conf配制

azkaban.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# cat azkaban.properties 
# Azkaban Personalization Settings
default.timezone.id=Asia/Shanghai
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes

database.type=mysql
mysql.port=3306
mysql.host=192.168.1.205
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30

log4j.properties

1
2
3
4
5
6
7
8
9
10
11
log4j.rootLogger=INFO, Console
log4j.logger.azkaban=INFO, server
log4j.appender.server=org.apache.log4j.RollingFileAppender
log4j.appender.server.layout=org.apache.log4j.PatternLayout
log4j.appender.server.File=logs/azkaban-exe.log
log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
log4j.appender.server.MaxFileSize=102400MB
log4j.appender.server.MaxBackupIndex=2
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n

jobtypes错误解决

在executor的根目录下创建plugins\jobtypes\commonprivate.properties

1
2
3
# set execute-as-user
execute.as.user=false
azkaban.native.lib=false
证心 微信支付

微信支付

# 大数据 # 任务调度 # azkaban
azkaban集群任务执行状态status一直为:Preparing
Springboot配制文件外置方法
  • 文章目录
  • 站点概览

证心

367 日志
45 分类
78 标签
GitHub 微博
  1. 1. server的conf配制
    1. 1.1. azkaban.properties
    2. 1.2. log4j.properties
  2. 2. executor的conf配制
    1. 2.1. azkaban.properties
    2. 2.2. log4j.properties
    3. 2.3. jobtypes错误解决
京ICP备16069454号 © 2020 证心