Search code examples
pythonnested-loopslarge-files

Fastest way to iterate over multiple lists

I'm trying to get 2 giant text files to combine their data into one text file.

One text file containing 200 000 - 500 000 lines of [node_id, x, y, z, temperature] and the other a 203 000 line file which contains vectors.

The idea is to identify the nodes from the one file, to the vectors in the other file, and combine them into a new text file. I've managed to get both files into blocks of 3-6k lines to manage times, but I can only think of:

  1. A for loop on the vectors to get starting and ending coordinates.

  2. A for loop every node to check the coordinate to the box of the vector.

  3. Calculate the distance of the nodes to the vector and select the vectors therein.

  4. get the next vector.

This still creates nestled for loops of 5000 iterations in one for loop, in which another 5000 iterations for loop is nestled. Is there a faster way to do this?

Example of the node file: [node_id, x, y, z, temp]

[21, -10.0, -12.0, 4.0, 160,0]

Example of the vector file: [vector_time, x, y, z, wattage]

[8.83, -9.82, -3.16, 0.05. 150.00]

I've tried:

with open(rf'Node_Temp_result.txt', 'r') as nodeFile:
    nodesInfo = nodeFile.readlines()

with open(rf'laser.txt', 'r') as laserFile:
    laserInfo = laserFile.readlines()
    
for laser in tqdm(range(currentLaserMin, currentLaserNext)):
        local_laser = Split(laserInfo, laser)
        laser_search = Laser(local_laser.col0(), local_laser.col1(), local_laser.col2(), local_laser.col3(), local_laser.col4())
        splits = 0

        x_search_laser = laser_search.laser_x()
        y_search_laser = laser_search.laser_y()
        z_search_laser = laser_search.laser_layer()

        if layer - 1 < z_search_laser <= layer and laser_search.laser_power() > 0:
            laser_next = Split(laserInfo, laser + 1)
            laser_search_next = Laser(laser_next.col0(), laser_next.col1(), laser_next.col2(), laser_next.col3(), laser_next.col4())

            x_start = x_search_laser
            x_end = laser_search_next.laser_x()

            y_start = y_search_laser
            y_end = laser_search_next.laser_y()

            num = 0

            for node in range(currentNodeMin, currentNodeNext):
                local_node = Split(nodesInfo, node)
                node_search = Node(local_node.col0(), local_node.col1(), local_node.col2(), local_node.col3(), local_node.col4())

                x_search_node = node_search.node_x()
                y_search_node = node_search.node_y()
                z_search_node = node_search.node_layer()

                if node_search.node_temp() > 250:
                    if layer - 1 < z_search_node <= layer:
                        if x_start <= x_search_node <= x_end and y_start <= y_search_node <= y_end:

                            crossLocations = cross_finder(x_start, x_end, y_start, y_end)

                            if crossLocations[3] < 0.07:
                                splits += 1
                                'append the node to the vector'
Solution

  • Some options that you could consider to do this task are: