mysql - remove empty line printed from hive query output using python

Question

i am performing a hive query and storing the output in a tsv file in the local FS. I am running a for loop for the hive query and passing different parameters. If the hive query returns no output once in the for loop it prints an empty line in the tsv file. This causes NULL values to be pushed to my DB in the backend. Hence, after the for loop runs and the file is created - i have the below code to remove all the empty lines printed, but it doesn't work.

How do i remove the empty line from this file?

` 395.9   429.61  PT  
                       `

code:

with open('file.tsv','r+w') as file:
        for line in file:
          if line.strip():
            file.write(line)

thanks

Answer 1

Usually you would open the input file and write the non-empty lines to a second file:

with open('file.tsv') as infile, open('filtered_file.tsv', 'w') as outfile:
    for line in infile:
        if line.strip():
            outfile.write(line)

If you want to filter the file inplace you can use FileInput with the inplace option:

import fileinput
for line in fileinput.FileInput("infile", inplace=1):
    if line.strip():
        print line

however, this uses an intermediate file and may not work in low disk space situations.

To filter the file inplace without allocating any additional disk space you could try something like this:

with open('file.tsv', 'r+') as infile:
    read_pos = write_pos = 0
    line = infile.readline()
    while line:
        read_pos += len(line)
        if line.strip():
            infile.seek(write_pos)
            infile.write(line)
            write_pos += len(line)
        infile.seek(read_pos)
        line = infile.readline()
    # update file size to the new, possibly reduced, size
    infile.truncate(write_pos)