numpy - Extracting data from multiple files with python

Question

I'm trying to extract data from a directory with 12 .txt files. Each file contains 3 columns of data (X,Y,Z) that i want to extract. I want to collect all the data in one df(InforDF), but so far i only succeeded in creating a df with all of the X,Y and Z data in the same column. This is my code:

import pandas as pd
import numpy as np
import os
import fnmatch

path = os.getcwd()

file_list = os.listdir(path)

InfoDF = pd.DataFrame()

for file in file_list:
    try:
        if fnmatch.fnmatch(file, '*.txt'):
            filedata = open(file, 'r')
            df = pd.read_table(filedata, delim_whitespace=True, names={'X','Y','Z'})

    except Exception as e:
        print(e)

What am i doing wrong?

Answer 1

df = pd.read_table(filedata, delim_whitespace=True, names={'X','Y','Z'})

this line replace df at each iteration of the loop, that's why you only have the last one at the end of your program.

what you can do is to save all your dataframe in a list and concatenate them at the end

df_list = []
for file in file_list:
    try:
        if fnmatch.fnmatch(file, '*.txt'): 
            filedata = open(file, 'r')
            df_list.append(pd.read_table(filedata, delim_whitespace=True, names={'X','Y','Z'}))
df = pd.concat(df_list)

alternatively, you can write it:

df_list = pd.concat([pd.read_table(open(file, 'r'), delim_whitespace=True, names={'X','Y','Z'})  for file in file_list if fnmatch.fnmatch(file, '*.txt')])