7.0 to 8.0 migration stops abruptly with psql log showing "Broken Pipe"

[raviprasadmr]

Dear OCA,

I am migrating my database from OpenERP 7 to Odoo 8. During stock migration, it gets to a point logger.info("Prefetching records") in the post-migration.py.

After this, the migration abruptly ends with no error logs in the migration.log file.

The postgresql logs show:
2022-10-08 15:56:23.722 CEST [170317] odoo@mydb_migrated LOG: could not send data to client: Broken pipe 2022-10-08 15:56:23.722 CEST [170317] odoo@mydb_migrated STATEMENT: SELECT "stock_move_operation_link".id FROM "stock_move_operation_link" WHERE ("stock_move_operation_link"."move_id" in (1493,1492,1491,.........,1760330,1760332,1760340,1760341)) ORDER BY "stock_move_operation_link"."id"

The SELECT statement takes a long time to fetch all the records from ID 1493 till 1760341.

Tail from the migration.log file:
`...
2022-10-08 16:59:08,817 184169 INFO mydb_migrated OpenUpgrade.stock: ~~~~~~~~~~~~~
2022-10-08 16:59:08,817 184169 INFO mydb_migrated OpenUpgrade.stock: stock::post::migrate_stock_qty
2022-10-08 16:59:08,817 184169 INFO mydb_migrated OpenUpgrade.stock: ~~~~~~~~~~~~~
2022-10-08 17:07:05,876 184169 DEBUG mydb_migrated OpenUpgrade: 1831897 rows affected after 0:07:57.058871 running
UPDATE stock_move SET restrict_lot_id = openupgrade_legacy_8_0_prodlot_id

2022-10-08 17:07:06,320 184169 DEBUG mydb_migrated OpenUpgrade: -1 rows affected after 0:00:00.443253 running
CREATE INDEX IF NOT EXISTS stock_pack_operation_location_id_index
ON stock_pack_operation USING BTREE(location_id)

2022-10-08 17:07:06,979 184169 DEBUG mydb_migrated OpenUpgrade: -1 rows affected after 0:00:00.659316 running
CREATE INDEX IF NOT EXISTS stock_pack_operation_product_id_index
ON stock_pack_operation USING BTREE(product_id)

2022-10-08 17:07:07,432 184169 DEBUG mydb_migrated OpenUpgrade: -1 rows affected after 0:00:00.452107 running
CREATE INDEX IF NOT EXISTS stock_move_operation_link_move_id_index
ON stock_move_operation_link USING BTREE(move_id)

2022-10-08 17:07:07,962 184169 DEBUG mydb_migrated OpenUpgrade: -1 rows affected after 0:00:00.530414 running
CREATE INDEX IF NOT EXISTS stock_move_operation_link_operation_id_index
ON stock_move_operation_link USING BTREE(operation_id)

2022-10-08 17:07:08,316 184169 DEBUG mydb_migrated OpenUpgrade: -1 rows affected after 0:00:00.353677 running
CREATE INDEX IF NOT EXISTS stock_move_operation_link_reserved_quant_id_index
ON stock_move_operation_link USING BTREE(reserved_quant_id)

2022-10-08 17:07:08,317 184169 INFO mydb_migrated OpenUpgrade.stock: Prefetching records
...`

Could you please help me solve this issue?

[legalsylvain]

Could you provides sizes of the "stock" tables ? Table size, number of rows.

[raviprasadmr]

Here is the detail about stock tables.

table_name	pg_relation_size	number_of_rows
stock_journal	8192	1
stock_location_path	8192	1
stock_incoterms	8192	16
stock_production_barcode_rule	8192	8
stock_tracking	8192	38
stock_warehouse	8192	23
stock_production_lot_revision	16384	123
stock_location	24576	89
stock_production_lot_numberspace	65536	732
stock_production_lot_numberspace_product_rel	180224	4836
stock_case_product_rel	180224	4887
stock_inventory_move_rel	385024	10535
stock_warehouse_orderpoint	663552	4771
stock_inventory	1384448	10567
stock_inventory_line	2039808	11015
stock_production_lot	13967360	120570
stock_picking	21323776	84551
stock_move_history_ids	110305280	3043068
stock_move	388833280	1831897

[raviprasadmr]

Found out that the SIGKILL was sent due to memory running full.

I increased the Swap space on my linux and restarted the migration. It surpassed this point and is still running. Unfortunately takes several days just to finish 7.0 to 8.0 migration since my stock_move table is gigantic. Very unrealistic when you want to upgrade to 13.0 over the weekend so that the company would not go through a downtime. But it is what it is.

This particular problem is solved now.

[StefanRijnhart]

I'm not sure the prefetching strategy from #2080 will actually work to improve performance that much (unless it is the case that those records are not written at all which is clearly not the case here), and the price indeed is the increased memory usage. I have had more success wrapping the record set in the main loop in chunked (https://oca.github.io/OpenUpgrade/API.html#openupgradelib.openupgrade.chunked), which will optimize both the number of queries as well as keeping memory usage low.