网站推广优化外包便宜,win 2012网站建设,招网站开发人员,wordpress主机记录请先完成HDFS的前置部署#xff0c;部署方式可查看:本地部署HDFS集群https://blog.csdn.net/m0_73641796/article/details/145998092?spm1001.2014.3001.5502 部署说明
组件配置文件启动进程备注Hadoop HDFS需修改 需启动: NameNode作为主节点 DataNode作为从节点 Secondary…请先完成HDFS的前置部署部署方式可查看:本地部署HDFS集群https://blog.csdn.net/m0_73641796/article/details/145998092?spm1001.2014.3001.5502 部署说明
组件配置文件启动进程备注Hadoop HDFS需修改 需启动: NameNode作为主节点 DataNode作为从节点 SecondaryNameNode主节点辅助 分布式文件系统Hadoop YARN需修改 需启动 ResourceManager作为集群资源管理者 NodeManager作为单机资源管理者 ProxyServer代理服务器提供安全性 JobHistoryServer记录历史信息和日志 分布式资源调度Hadoop MapReduce需修改 无需启动任何进程 MapReduce程序运行在YARN容器内 分布式数据计算
集群规划
主机角色node1 ResourceManager NodeManager ProxyServer JobHistoryServer node2NodeManagernode3NodeManager
修改MapReduce配置文件
在$HADOOP_HOME/etc/hadoop文件夹内修改mapred-env.sh文件添加如下环境变量
export JAVA_HOME/export/server/jdk
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE1000
export HADOOP_MAPRED_ROOT_LOGGERINFO,RFA
修改mapred-site.xml文件添加如下配置信息
configurationpropertynamemapreduce.framework.name/namevalueyarn/valuedescription/description/propertypropertynamemapreduce.jobhistory.address/namevaluenode1:10020/valuedescription/description/propertypropertynamemapreduce.jobhistory.webapp.address/namevaluenode1:19888/valuedescription/description/propertypropertynamemapreduce.jobhistory.intermediate-done-dir/namevalue/data/mr-history/tmp/valuedescription/description/propertypropertynamemapreduce.jobhistory.done-dir/namevalue/data/mr-history/done/valuedescription/description/propertypropertynameyarn.app.mapreduce.am.env/namevalueHADOOP_MAPRED_HOME$HADOOP_HOME/value/propertypropertynamemapreduce.map.env/namevalueHADOOP_MAPRED_HOME$HADOOP_HOME/value/propertypropertynamemapreduce.reduce.env/namevalueHADOOP_MAPRED_HOME$HADOOP_HOME/value/property
/configuration
修改YARN配置文件
在$HADOOP_HOME/etc/hadoop文件夹内修改yarn-env.sh文件添加如下环境变量
export JAVA_HOME/export/server/jdk
export HADOOP_HOME/export/server/hadoop
export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR$HADOOP_HOME/logs修改yarn-site.xml文件添加如下内容
configuration!-- Site specific YARN configuration properties --
propertynameyarn.log.server.url/namevaluehttp://node1:19888/jobhistory/logs/valuedescription/description
/propertypropertynameyarn.web-proxy.address/namevaluenode1:8089/valuedescriptionproxy server hostname and port/description/propertypropertynameyarn.log-aggregation-enable/namevaluetrue/valuedescriptionConfiguration to enable or disable log aggregation/description/propertypropertynameyarn.nodemanager.remote-app-log-dir/namevalue/tmp/logs/valuedescriptionConfiguration to enable or disable log aggregation/description/property!-- Site specific YARN configuration properties --propertynameyarn.resourcemanager.hostname/namevaluenode1/valuedescription/description/propertypropertynameyarn.resourcemanager.scheduler.class/name
valueorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler/valuedescription/description/propertypropertynameyarn.nodemanager.local-dirs/namevalue/data/nm-local/valuedescriptionComma-separated list of paths on the local filesystem where intermediate data is written./description/propertypropertynameyarn.nodemanager.log-dirs/namevalue/data/nm-log/valuedescriptionComma-separated list of paths on the local filesystem where logs are written./description/propertypropertynameyarn.nodemanager.log.retain-seconds/namevalue10800/valuedescriptionDefault time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled./description/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/valuedescriptionShuffle service that needs to be set for Map Reduce applications./description/property
/configuration 分发配置文件
scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node2:pwd/
scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node3:pwd/ 启动HDFS
start-dfs.sh 启动YARN
# 一键启动YARN集群
start-yarn.sh# 一键停止YARN集群
stop-yarn.sh# 在当前机器单独启动或停止进程
yarn --daemon start|stop resourcemanager|nodemanager|proxyserver# 历史服务器启动和停止
mapred --daemon start|stop historyserver 可通过jps命令查看当前运行的进程 查看YARN的WEB UI页面
打开 http://node1:8088 即可看到YARN集群的监控页面ResourceManager的WEB UI
