php做数据网站,物联网的发展前景,英文网站建设怎么样,百度ocpc怎么优化在处理插入milvus数据时#xff0c;设置了字段长度为512. 明明考虑了预留#xff0c;插入的数据中没有这么长的#xff0c;但还是会有报错 类似#xff1a;MilvusException: (code0, messagethe length (564) of 78th string exceeds max length (512) 查找max(len(x) for …在处理插入milvus数据时设置了字段长度为512. 明明考虑了预留插入的数据中没有这么长的但还是会有报错 类似MilvusException: (code0, messagethe length (564) of 78th string exceeds max length (512) 查找max(len(x) for x in temp_list)之类 都没有超过512过也没超过256过不知道哪里的数据有问题.. 反复截段文本等测试后发现例如用len(x)看到的字符串长度是10但保存进milus的长度并不是..
举例把数据库长度设为一个小值16 FieldSchema(namequestion, dtypeDataType.VARCHAR, auto_idFalse, max_length16)
再把数据缩到只有一行 测试结果插入成功
line contents is : 你好呀你好 and length is 5 Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:0100:00, 1.02s/it] index handle result: Status(code0, message) insert result: (insert count: 1, delete count: 0, upsert count: 0, timestamp: 449735609509740549, success count: 1, err count: 0)
再增加一点文字长度 就报错了
line contents is : 你好呀你好呀 and length is 6 Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:0000:00, 1.03it/s] index handle result: Status(code0, message) [2024-05-13 20:59:27,915 decorators.py:134 ERROR] RPC error: [batch_insert], MilvusException: (code0, messagethe length (18) of 0th string exceeds max length (16)), Time:{RPC start: 2024-05-13 20:59:27.912751, RPC error: 2024-05-13 20:59:27.915058} Traceback (most recent call last): File /root/temp_dir/run_task.py, line 55, in module XXX().create_insert_vector_db() File /root/temp_dir/app/service/vector_db/xx_pre_handle.py, line 63, in create_insert_vector_db ).get_or_create_db(fields, description, possible_question_embeddings, entities) File /root/temp_dir/app/service/vector_db/milvus_db.py, line 23, in get_or_create_db return self.create_and_insert(fields, description, index_field_name, entities) File /root/temp_dir/app/service/vector_db/milvus_db.py, line 28, in create_and_insert self.insert_db(entities) File /root/temp_dir/app/service/vector_db/milvus_db.py, line 40, in insert_db insert_result self.collection.insert(entities) File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/orm/collection.py, line 497, in insert res conn.batch_insert( File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py, line 135, in handler raise e from e File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py, line 131, in handler return func(*args, **kwargs) File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py, line 170, in handler return func(self, *args, **kwargs) File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py, line 110, in handler raise e from e File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py, line 74, in handler return func(*args, **kwargs) File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py, line 566, in batch_insert raise err from err File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py, line 560, in batch_insert check_status(response.status) File /root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/utils.py, line 54, in check_status raise MilvusException(status.code, status.reason, status.error_code) pymilvus.exceptions.MilvusException: MilvusException: (code0, messagethe length (18) of 0th string exceeds max length (16)) 所以可能是因为UTF-8或其他编码的原因一些非ASCII字符可能被编码成多个字节 以保存进milvus。 所以解决方案是 建表时FieldSchema中把max_length 设置为4倍或其他倍数于预期的最大长度。
