MySQL导入大量数据时,可能会遇到性能瓶颈,如导入速度慢、占用大量内存和磁盘空间等问题。这通常是因为MySQL在处理大量数据时,需要逐行读取文件并进行解析、插入操作,导致效率低下。
innodb_buffer_pool_size
、innodb_log_file_size
等,可以提高数据库的性能。LOAD DATA INFILE
语句导入。SOURCE
命令导入。LOAD DATA INFILE 'data.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
START TRANSACTION;
LOAD DATA INFILE 'data.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
COMMIT;
编辑MySQL配置文件(如my.cnf
或my.ini
),调整以下参数:
[mysqld]
innodb_buffer_pool_size = 1G
innodb_log_file_size = 256M
innodb_flush_log_at_trx_commit = 2
import mysql.connector
db = mysql.connector.connect(
host="localhost",
user="yourusername",
password="yourpassword",
database="yourdatabase"
)
cursor = db.cursor()
batch_size = 1000
for i in range(0, total_rows, batch_size):
sql = f"INSERT INTO table_name (column1, column2, column3) VALUES "
values = []
for j in range(i, min(i + batch_size, total_rows)):
values.append((data[j][0], data[j][1], data[j][2]))
sql += ', '.join(map(lambda x: f"({x[0]}, {x[1]}, {x[2]})", values))
cursor.execute(sql)
db.commit()
领取专属 10元无门槛券
手把手带您无忧上云