网站建设要学哪些,网站开发需要哪些知识和工具,怎么给网站做aap,郑州自助建站软件hive运行失败会导致临时目录无法自动清理#xff0c;因此需要自己写脚本去进行清理
实际发现hive临时目录有两个#xff1a; /tmp/hive/{user}/* /warehouse/tablespace//hive/**/.hive-staging_hive
分别由配置hive.exec.scratchdir和hive.exec.stagingdir决定: 要注意的…hive运行失败会导致临时目录无法自动清理因此需要自己写脚本去进行清理
实际发现hive临时目录有两个 /tmp/hive/{user}/* /warehouse/tablespace//hive/**/.hive-staging_hive
分别由配置hive.exec.scratchdir和hive.exec.stagingdir决定: 要注意的是stagingdir可能存在多个层级的目录中比如xxx.db/.hive-staging_xxxxxx.db/${table}/.hive-staging_xxx。这里为了偷懒只清理两层即可如果发现有更多层再多加一层的调用即可
创建清理脚本clean_hive_tmpfile.sh
#!/bin/bash
usageUsage: cleanup.sh [days]if [ ! $1 ]thenecho $usageexit 1
finow$(date %s)
days$1cleanTmpFile() {echo clean path:$1su hdfs -c hdfs dfs -ls -d $1 | grep ^d | while read f; dodir_dateecho $f | awk {print $6}difference$(( ( $now - $(date -d $dir_date %s) ) / (24 * 60 * 60 ) ))if [ $difference -gt $days ]; thenecho $fnameecho $f| awk { print $8 }echo delete:$namesu hdfs -c hadoop fs -rm -r -skipTrash $namefidone}cleanTmpFile /tmp/hive/*/*
cleanTmpFile /warehouse/tablespace/*/hive/*/.hive-staging_*
cleanTmpFile /warehouse/tablespace/*/hive/*/*/.hive-staging_*配置crontab每天凌晨2点执行清理30天以前的目录
0 2 * * * sh /data/script/clean_hive_tmpfile.sh 30 /tmp/clean_hive_tmpfile.log
参考 https://cwiki.apache.org/confluence/display/Hive/ConfigurationProperties#ConfigurationProperties-hive.start.cleanup.scratchdir https://www.cnblogs.com/ucarinc/p/11831280.html https://blog.csdn.net/zhoudetiankong/article/details/51800887 http://t.zoukankan.com/telegram-p-10748530.html
