python - Unable to allocate that much memory using modified LLE, hessian LLE and Local Tangent Space Alignment

Question

Hello,

I'm trying to use LLE and other methods of LLE like modified-, hessian- and ltsa on a external Dataset (you can find it here: https://www.kaggle.com/mlg-ulb/creditcardfraud) I got it working with LLE, but the modified Version definitely need way too much RAM. For example:

def clean_dataset(df):
    assert isinstance(df, pd.DataFrame), "df needs to be a pd.DataFrame"
    df.dropna(inplace=True)
    indices_to_keep = ~df.isin([np.nan, np.inf, -np.inf]).any(1)
    return df[indices_to_keep].astype(np.float64)

data = pd.read_csv('C:/Users/yazar/Downloads/creditcardfraud/creditcard.csv')
clean_dataset(data)
X_features = data.drop('Class', axis=1)
y_targets = data['Class']

clf = manifold.LocallyLinearEmbedding(n_neighbors=n_neighbors, n_components=2, method='modified')
clf.fit(X=X_features, y=y_targets)

t0 = time()
print("Done. Reconstruction error: %g" %clf.reconstruction_error_)
X_mllecf=clf.transform(X_features)

gives the following Error:

MemoryError: Unable to allocate 604. GiB for an array with shape (284807, 284807) and data type float64

How can I minimize the needed memory or if necessary minimize the dataset to get some results?

Answer 1

My solution was to reduce the dataframe by just choosing a random sample of my data with the function:

sample = data.sample(n=3000, random_state=1)

But using it you have to reset the index numbers because else your plotting function wont work:

sample = sample.reset_index(drop=True)