集团网站推广,用vs做购物网站下载,ip查询地址精准地图,大美南京网站目录#xff1a; 一、准备环境#xff08;1#xff09;安装JDK#xff08;2#xff09;检查Python版本#xff08;3#xff09;DataX 解压及测试 二、同步测试1、配置清单2、操作示例#xff08;1#xff09;同步测试环境#xff08;2#xff09;准备测试表#xf… 目录 一、准备环境1安装JDK2检查Python版本3DataX 解压及测试 二、同步测试1、配置清单2、操作示例1同步测试环境2准备测试表postgresql3准备测试表Oracle4创建并编辑json文件5执行json配置文件 一、准备环境
1、创建用户组和目录
groupadd -g 1500 datax
useradd -g datax -u 1500 datax
mkdir /datax
chown datax:datax /datax安装datax两端服务器都可以但是要保证网络畅通 2、系统环境 LinuxJDK(1.8以上推荐1.8)Python(2或3都可以)Apache Maven 3.x (Compile DataX)本文使用datax工具包直接解压未采用编译方式 1安装JDK
安装JDK1.8.0_361
# rpm -ivh jdk-8u361-linux-x64.rpm
warning: jdk-8u361-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Preparing... ################################# [100%]
Updating / installing...1:jdk1.8-2000:1.8.0_361-fcs ################################# [100%]
Unpacking JAR files...tools.jar...plugin.jar...javaws.jar...deploy.jar...rt.jar...jsse.jar...charsets.jar...localedata.jar...检查JDK版本
# java -version
java version 1.8.0_361
Java(TM) SE Runtime Environment (build 1.8.0_361-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.361-b09, mixed mode)2检查Python版本
# python -V
Python 2.7.5我这边使用的是linux7自带的python版本 3DataX 解压及测试
Download DataX下载地址上传到/datax目录
解压缩
# tar -zxvf datax.tar.gz自检脚本
python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json示例
# python /datax/datax/bin/datax.py /datax/datax/job/job.jsonDataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.2023-10-10 11:31:06.527 [main] INFO MessageSource - JVM TimeZone: GMT08:00, Locale: zh_CN
2023-10-10 11:31:06.531 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[idGMT08:00,offset28800000,dstSavings0,useDaylightfalse,transitions0,lastRulenull]
2023-10-10 11:31:06.571 [main] INFO VMInfo - VMInfo# operatingSystem class sun.management.OperatingSystemImpl
2023-10-10 11:31:06.577 [main] INFO Engine - the machine info osInfo: Oracle Corporation 1.8 25.361-b09jvmInfo: Linux amd64 5.4.17-2011.6.2.el7uek.x86_64cpu num: 4totalPhysicalMemory: -0.00GfreePhysicalMemory: -0.00GmaxFileDescriptorCount: -1currentOpenFileDescriptorCount: -1GC Names [PS MarkSweep, PS Scavenge]MEMORY_NAME | allocation_size | init_size PS Eden Space | 256.00MB | 256.00MB Code Cache | 240.00MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB PS Survivor Space | 42.50MB | 42.50MB PS Old Gen | 683.00MB | 683.00MB Metaspace | -0.00MB | 0.00MB 2023-10-10 11:31:06.595 [main] INFO Engine -
{content:[{reader:{name:streamreader,parameter:{column:[{type:string,value:DataX},{type:long,value:19890604},{type:date,value:1989-06-04 00:00:00},{type:bool,value:true},{type:bytes,value:test}],sliceRecordCount:100000}},writer:{name:streamwriter,parameter:{encoding:UTF-8,print:false}}}],setting:{errorLimit:{percentage:0.02,record:0},speed:{channel:1}}
}2023-10-10 11:31:06.615 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-10-10 11:31:06.617 [main] INFO PerfTrace - PerfTrace traceIdjob_-1, isEnablefalse, priority0
2023-10-10 11:31:06.617 [main] INFO JobContainer - DataX jobContainer starts job.
2023-10-10 11:31:06.620 [main] INFO JobContainer - Set jobId 0
2023-10-10 11:31:06.641 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-10-10 11:31:06.643 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do prepare work .
2023-10-10 11:31:06.643 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2023-10-10 11:31:06.643 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-10-10 11:31:06.643 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels.
2023-10-10 11:31:06.644 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2023-10-10 11:31:06.645 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2023-10-10 11:31:06.667 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-10-10 11:31:06.674 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-10-10 11:31:06.677 [job-0] INFO JobContainer - Running by standalone Mode.
2023-10-10 11:31:06.685 [taskGroup-0] INFO TaskGroupContainer - taskGroupId[0] start [1] channels for [1] tasks.
2023-10-10 11:31:06.688 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-10-10 11:31:06.688 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-10-10 11:31:06.701 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-10-10 11:31:07.003 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[304]ms
2023-10-10 11:31:07.004 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed its tasks.
2023-10-10 11:31:16.696 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.050s | All Task WaitReaderTime 0.060s | Percentage 100.00%
2023-10-10 11:31:16.696 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-10-10 11:31:16.696 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] do post work.
2023-10-10 11:31:16.697 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do post work.
2023-10-10 11:31:16.697 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-10-10 11:31:16.698 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /datax/datax/hook
2023-10-10 11:31:16.700 [job-0] INFO JobContainer - [total cpu info] averageCpu | maxDeltaCpu | minDeltaCpu -1.00% | -1.00% | -1.00%[total gc info] NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s PS Scavenge | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s 2023-10-10 11:31:16.700 [job-0] INFO JobContainer - PerfTrace not enable!
2023-10-10 11:31:16.701 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.050s | All Task WaitReaderTime 0.060s | Percentage 100.00%
2023-10-10 11:31:16.705 [job-0] INFO JobContainer -
任务启动时刻 : 2023-10-10 11:31:06
任务结束时刻 : 2023-10-10 11:31:16
任务总计耗时 : 10s
任务平均流量 : 253.91KB/s
记录写入速度 : 10000rec/s
读出记录总数 : 100000
读写失败总数 : 0自检脚本无法通过报错处理https://developer.aliyun.com/ask/376622?spma2c6h.13148508.0.0.1d734f0eKYBCG7 二、同步测试
1、配置清单
在两个不同数据库之间使用datax数据迁移可以通过命令查看配置模板 python datax.py -r {Sourcedb_READER} -w {Targetdb_WRITER}
示例查看PostgreSQL到Oracle数据传输json文件模板
# python /datax/datax/bin/datax.py -r postgresqlreader -w oraclewriterDataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.Please refer to the postgresqlreader document:https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md Please refer to the oraclewriter document:https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md Please save the following configuration as a json file and usepython {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json
to run the job.{job: {content: [{reader: {name: postgresqlreader, parameter: {connection: [{jdbcUrl: [], table: []}], password: , username: }}, writer: {name: oraclewriter, parameter: {column: [], connection: [{jdbcUrl: , table: []}], password: , preSql: [], username: }}}], setting: {speed: {channel: }}}
}详情查看 Please refer to the postgresqlreader document: https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md Please refer to the oraclewriter document: https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md 2、操作示例
1同步测试环境
OS版本CentOS Linux release 7.6.1810 (Core)Ol7.9DB版本PostgreSQL12.3Oracle19c (经典模式)同步方向ReaderWriterIP192.168.245.9192.168.245.10端口54321521数据库名htdborcl用户/密码htuser/htuser123shuaige/shuaige123表名httabhttab 两个虚拟机均使用OracleLinux7.9操作系统最小化安装关闭防火墙和selinux 2准备测试表postgresql
-- 1、登录postgresql
------------------------------------------------------------------------------------
$ psql -h192.168.245.9 -p5432 -Uhtuser htdb
Password for user htuser:
psql (12.3)
Type help for help.htdb# \c
You are now connected to database htdb as user htuser.
htdb#-- 2、创建测试表
------------------------------------------------------------------------------------
htdb# create table httab(id serial primary key,name text,age int);
CREATE TABLE-- 3、检查新建表
------------------------------------------------------------------------------------
htdb# \d httabTable public.httabColumn | Type | Collation | Nullable | Default
-------------------------------------------------------------------------id | integer | | not null | nextval(httab_id_seq::regclass)name | text | | | age | integer | | |
Indexes:httab_pkey PRIMARY KEY, btree (id)htdb#-- 4、插入测试数据
------------------------------------------------------------------------------------
htdb# insert into httab(name,age) select md5(random()::text),rn from generate_series(1,10000) rn;
INSERT 0 10000-- 5、查询测试数据
------------------------------------------------------------------------------------
htdb# select * from httab;id | name | age
------------------------------------------------10001 | 4c943d9e2be750b836365166b8bbfdf1 | 110002 | ddbad0963e08c9642d1692e9961677f7 | 210003 | 40551c9e3f00b4e7ea7c81062a94b20b | 3
...3准备测试表Oracle
-- 1、登录oracle
------------------------------------------------------------------------------------
$ sqlplus / as sysdbaSQL*Plus: Release 19.0.0.0.0 - Production on Tue Oct 10 16:06:40 2023
Version 19.3.0.0.0Copyright (c) 1982, 2019, Oracle. All rights reserved.Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0SYSorcl-- 1、创建测试表
------------------------------------------------------------------------------------
SYSorcl create table shuaige.httab (id number(30) primary key,name VARCHAR2(1000) not null,age int);Table created.注意更改用户名和表空间,不指定使用用户默认表空间 4创建并编辑json文件
在/datax/datax/job目录下用postgresqlreader to oraclewriter模板配置json文件
vi /datax/datax/job/postgresql2oracle_httab.json编辑配置模板按照实际环境修改
{job: {content: [{reader: {name: postgresqlreader, parameter: {connection: [{jdbcUrl: [jdbc:postgresql://192.168.245.9:5432/htdb], table: [httab]}], column: [*],splitPk: id,password: htuser123, username: htuser}}, writer: {name: oraclewriter, parameter: {column: [*], connection: [{jdbcUrl: jdbc:oracle:thin:192.168.245.10:1521:orcl, table: [httab]}], password: shuaige123, preSql: [truncate table httab], username: shuaige}}}], setting: {speed: {channel: 4}}}
}5执行json配置文件
# python /datax/datax/bin/datax.py /datax/datax/job/postgresql2oracle_httab.jsonDataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.2023-10-10 17:04:26.033 [main] INFO MessageSource - JVM TimeZone: GMT08:00, Locale: zh_CN
2023-10-10 17:04:26.035 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[idGMT08:00,offset28800000,dstSavings0,useDaylightfalse,transitions0,lastRulenull]
2023-10-10 17:04:26.060 [main] INFO VMInfo - VMInfo# operatingSystem class sun.management.OperatingSystemImpl
2023-10-10 17:04:26.066 [main] INFO Engine - the machine info osInfo: Oracle Corporation 1.8 25.361-b09jvmInfo: Linux amd64 5.4.17-2011.6.2.el7uek.x86_64cpu num: 4totalPhysicalMemory: -0.00GfreePhysicalMemory: -0.00GmaxFileDescriptorCount: -1currentOpenFileDescriptorCount: -1GC Names [PS MarkSweep, PS Scavenge]MEMORY_NAME | allocation_size | init_size PS Eden Space | 256.00MB | 256.00MB Code Cache | 240.00MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB PS Survivor Space | 42.50MB | 42.50MB PS Old Gen | 683.00MB | 683.00MB Metaspace | -0.00MB | 0.00MB 2023-10-10 17:04:26.080 [main] INFO Engine -
{content:[{reader:{name:postgresqlreader,parameter:{column:[*],connection:[{jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb],table:[httab]}],password:*********,splitPk:id,username:htuser}},writer:{name:oraclewriter,parameter:{column:[*],connection:[{jdbcUrl:jdbc:oracle:thin:192.168.245.10:1521:orcl,table:[httab]}],password:**********,preSql:[truncate table httab],username:shuaige}}}],setting:{speed:{channel:4}}
}2023-10-10 17:04:26.094 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-10-10 17:04:26.095 [main] INFO PerfTrace - PerfTrace traceIdjob_-1, isEnablefalse, priority0
2023-10-10 17:04:26.096 [main] INFO JobContainer - DataX jobContainer starts job.
2023-10-10 17:04:26.097 [main] INFO JobContainer - Set jobId 0
2023-10-10 17:04:26.372 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:postgresql://192.168.245.9:5432/htdb.
2023-10-10 17:04:26.373 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列当您的表字段个数、类型有变动时可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-10-10 17:04:26.576 [job-0] INFO OriginalConfPretreatmentUtil - table:[httab] all columns:[
ID,NAME,AGE
].
2023-10-10 17:04:26.576 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*当您的表字段个数、类型有变动时可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-10-10 17:04:26.578 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (ID,NAME,AGE) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:oracle:thin:192.168.245.10:1521:orcl]
2023-10-10 17:04:26.578 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-10-10 17:04:26.578 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] do prepare work .
2023-10-10 17:04:26.578 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] do prepare work .
2023-10-10 17:04:26.627 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[truncate table httab]. context info:jdbc:oracle:thin:192.168.245.10:1521:orcl.
2023-10-10 17:04:26.648 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-10-10 17:04:26.648 [job-0] INFO JobContainer - Job set Channel-Number to 4 channels.
2023-10-10 17:04:26.667 [job-0] INFO SingleTableSplitUtil - split pk [sqlSELECT MIN(id),MAX(id) FROM httab] is running...
2023-10-10 17:04:26.690 [job-0] INFO SingleTableSplitUtil - After split(), allQuerySql[
select * from httab where (10001 id AND id 10501)
select * from httab where (10501 id AND id 11001)
select * from httab where (11001 id AND id 11501)
select * from httab where (11501 id AND id 12001)
select * from httab where (12001 id AND id 12501)
select * from httab where (12501 id AND id 13001)
select * from httab where (13001 id AND id 13501)
select * from httab where (13501 id AND id 14001)
select * from httab where (14001 id AND id 14501)
select * from httab where (14501 id AND id 15001)
select * from httab where (15001 id AND id 15501)
select * from httab where (15501 id AND id 16001)
select * from httab where (16001 id AND id 16501)
select * from httab where (16501 id AND id 17001)
select * from httab where (17001 id AND id 17501)
select * from httab where (17501 id AND id 18001)
select * from httab where (18001 id AND id 18501)
select * from httab where (18501 id AND id 19001)
select * from httab where (19001 id AND id 19501)
select * from httab where (19501 id AND id 20000)
select * from httab where id IS NULL
].
2023-10-10 17:04:26.691 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] splits to [21] tasks.
2023-10-10 17:04:26.692 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] splits to [21] tasks.
2023-10-10 17:04:26.712 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-10-10 17:04:26.756 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-10-10 17:04:26.758 [job-0] INFO JobContainer - Running by standalone Mode.
2023-10-10 17:04:26.771 [taskGroup-0] INFO TaskGroupContainer - taskGroupId[0] start [4] channels for [21] tasks.
2023-10-10 17:04:26.775 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-10-10 17:04:26.776 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-10-10 17:04:26.782 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[14] attemptCount[1] is started
2023-10-10 17:04:26.785 [0-0-14-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (17001 id AND id 17501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.786 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started
2023-10-10 17:04:26.787 [0-0-3-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (11501 id AND id 12001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.801 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[13] attemptCount[1] is started
2023-10-10 17:04:26.802 [0-0-13-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (16501 id AND id 17001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.807 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started
2023-10-10 17:04:26.807 [0-0-2-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (11001 id AND id 11501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.831 [0-0-14-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (17001 id AND id 17501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.836 [0-0-2-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (11001 id AND id 11501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.851 [0-0-13-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (16501 id AND id 17001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.865 [0-0-3-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (11501 id AND id 12001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.110 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[13] is successed, used[323]ms
2023-10-10 17:04:27.157 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started
2023-10-10 17:04:27.157 [0-0-4-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (12001 id AND id 12501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.180 [0-0-4-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (12001 id AND id 12501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.258 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] is successed, used[456]ms
2023-10-10 17:04:27.258 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[14] is successed, used[477]ms
2023-10-10 17:04:27.259 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[18] attemptCount[1] is started
2023-10-10 17:04:27.261 [0-0-18-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (19001 id AND id 19501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.262 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-10-10 17:04:27.263 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (10001 id AND id 10501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.282 [0-0-18-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (19001 id AND id 19501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.300 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (10001 id AND id 10501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.363 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] is successed, used[578]ms
2023-10-10 17:04:27.363 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] is successed, used[252]ms
2023-10-10 17:04:27.365 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[15] attemptCount[1] is started
2023-10-10 17:04:27.366 [0-0-15-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (17501 id AND id 18001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.368 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] attemptCount[1] is started
2023-10-10 17:04:27.368 [0-0-6-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (13001 id AND id 13501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.387 [0-0-15-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (17501 id AND id 18001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.402 [0-0-6-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (13001 id AND id 13501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.569 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[309]ms
2023-10-10 17:04:27.571 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] attemptCount[1] is started
2023-10-10 17:04:27.571 [0-0-5-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (12501 id AND id 13001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.590 [0-0-5-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (12501 id AND id 13001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.672 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] is successed, used[306]ms
2023-10-10 17:04:27.672 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[15] is successed, used[307]ms
2023-10-10 17:04:27.673 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[18] is successed, used[414]ms
2023-10-10 17:04:27.674 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started
2023-10-10 17:04:27.675 [0-0-1-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (10501 id AND id 11001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.677 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[10] attemptCount[1] is started
2023-10-10 17:04:27.677 [0-0-10-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (15001 id AND id 15501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.679 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[17] attemptCount[1] is started
2023-10-10 17:04:27.679 [0-0-17-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (18501 id AND id 19001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.692 [0-0-1-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (10501 id AND id 11001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.709 [0-0-17-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (18501 id AND id 19001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.725 [0-0-10-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (15001 id AND id 15501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.981 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[10] is successed, used[306]ms
2023-10-10 17:04:27.981 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[17] is successed, used[303]ms
2023-10-10 17:04:27.983 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[11] attemptCount[1] is started
2023-10-10 17:04:27.983 [0-0-11-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (15501 id AND id 16001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.985 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[16] attemptCount[1] is started
2023-10-10 17:04:27.985 [0-0-16-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (18001 id AND id 18501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.008 [0-0-11-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (15501 id AND id 16001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.021 [0-0-16-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (18001 id AND id 18501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.086 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] is successed, used[516]ms
2023-10-10 17:04:28.090 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[19] attemptCount[1] is started
2023-10-10 17:04:28.090 [0-0-19-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (19501 id AND id 20000)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.217 [0-0-19-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (19501 id AND id 20000)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.307 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] is successed, used[633]ms
2023-10-10 17:04:28.309 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] attemptCount[1] is started
2023-10-10 17:04:28.309 [0-0-7-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (13501 id AND id 14001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.376 [0-0-7-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (13501 id AND id 14001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.511 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[19] is successed, used[423]ms
2023-10-10 17:04:28.513 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[8] attemptCount[1] is started
2023-10-10 17:04:28.514 [0-0-8-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (14001 id AND id 14501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.617 [0-0-8-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (14001 id AND id 14501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.717 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[11] is successed, used[735]ms
2023-10-10 17:04:28.717 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[16] is successed, used[734]ms
2023-10-10 17:04:28.718 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[9] attemptCount[1] is started
2023-10-10 17:04:28.719 [0-0-9-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (14501 id AND id 15001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.720 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[12] attemptCount[1] is started
2023-10-10 17:04:28.720 [0-0-12-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (16001 id AND id 16501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.814 [0-0-9-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (14501 id AND id 15001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.892 [0-0-12-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (16001 id AND id 16501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.248 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[8] is successed, used[736]ms
2023-10-10 17:04:29.250 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[20] attemptCount[1] is started
2023-10-10 17:04:29.250 [0-0-20-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where id IS NULL
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.269 [0-0-20-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where id IS NULL
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.502 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[12] is successed, used[783]ms
2023-10-10 17:04:29.706 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] is successed, used[1398]ms
2023-10-10 17:04:29.806 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[9] is successed, used[1088]ms
2023-10-10 17:04:29.806 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[20] is successed, used[557]ms
2023-10-10 17:04:29.807 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed its tasks.
2023-10-10 17:04:36.773 [job-0] INFO StandAloneJobContainerCommunicator - Total 10000 records, 408894 bytes | Speed 39.93KB/s, 1000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.010s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-10-10 17:04:36.773 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-10-10 17:04:36.774 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] do post work.
2023-10-10 17:04:36.774 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] do post work.
2023-10-10 17:04:36.774 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-10-10 17:04:36.775 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /datax/datax/hook
2023-10-10 17:04:36.775 [job-0] INFO JobContainer - [total cpu info] averageCpu | maxDeltaCpu | minDeltaCpu -1.00% | -1.00% | -1.00%[total gc info] NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime PS MarkSweep | 1 | 1 | 1 | 0.018s | 0.018s | 0.018s PS Scavenge | 1 | 1 | 1 | 0.009s | 0.009s | 0.009s 2023-10-10 17:04:36.776 [job-0] INFO JobContainer - PerfTrace not enable!
2023-10-10 17:04:36.776 [job-0] INFO StandAloneJobContainerCommunicator - Total 10000 records, 408894 bytes | Speed 39.93KB/s, 1000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.010s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-10-10 17:04:36.777 [job-0] INFO JobContainer -
任务启动时刻 : 2023-10-10 17:04:26
任务结束时刻 : 2023-10-10 17:04:36
任务总计耗时 : 10s
任务平均流量 : 39.93KB/s
记录写入速度 : 1000rec/s
读出记录总数 : 10000
读写失败总数 : 0执行成功检查Oracle数据库查看数据是否同步成功
SYSorcl col name for a50
SYSorcl select * from shuaige.httab where rownum 10;ID NAME AGE
---------- -------------------------------------------- ----------16809 5f19fd4c4f65352c79318089634ee459 680916810 039773606c114ae52af27fd6bb2603e7 681016811 9dd5a56b63e6192e6f091c3dfd600f50 681116812 564147edcc95fe83af7b5bf2cc42268a 681216813 1dc07f8e48bd6baf258c0a341ad8753a 681316814 7569553c59834aff4e0c78c8a5292a9f 681416815 b93d01952acc42d345e06a062166693f 681516816 d60332e3fef6292da8c98d963435ec9e 681616817 9f3271cd20b8e473ae9e98c4b258f45d 68179 rows selected.数据传输成功如果在执行json文件报错一定要先看好json文件格式用户密码大小写[ ] “等“channel参数至少为“1”“preSql”:[””] SQL条件可以不加但是有主键冲突的行会报错传输失败。
