Python Hash not being updated in csv file output

Question

I have working code that takes a directory of csv files and hashes one column of each line, then aggregates all files together. The issue is the output only displays the first hash value, not re-running the hash for each line. Here is the code:

 import glob
 import hashlib

 files = glob.glob( '*.csv' )
 output="combined.csv"

 with open(output, 'w' ) as result:
     for thefile in files:
        f = open(thefile)
        m = f.readlines()
        for line in m[1:]:
            fields = line.split()       
            hash_object = hashlib.md5(b'(fields[2])')
            newline = fields[0],fields[1],hash_object.hexdigest(),fields[3]
            joined_line = ','.join(newline)
            result.write(joined_line+ '
')
  f.close()

Answer 1

You are creating a hash of a fixed bytestring b'(fields[2])'. That value has no relationship to your CSV data, even though it uses the same characters as are used in your row variable name.

You need to pass in bytes from your actual row:

hash_object = hashlib.md5(fields[2].encode('utf8'))

I am assuming your fields[2] column is a string, so you'd need to encoding it first to get bytes. The UTF-8 encoding can handle all codepoints that could possibly be contained in a string.

You also appear to be re-inventing the CSV reading and writing wheel; you probably should use the csv module instead:

 import csv

 # ...

 with open(output, 'w', newline='') as result:
     writer = csv.writer(result)

     for thefile in files:
        with open(thefile, newline='') as f:
            reader = csv.reader(f)
            next(reader, None)  # skip first row
            for fields in reader:
                hash_object = hashlib.md5(fields[2].encode('utf8'))
                newrow = fields[:2] + [hash_object.hexdigest()] + fields[3:]
                writer.writerow(newrow)