c# - How can I free-up memory used by a Parallel.Task?

Question

I have a program that does a memory intensive simulation. Below I've written a small console application that replicates the problem I'm having.

class Program {
    static void Main(string[] args) {
        var t = new Task(() => DoMemoryHog(20000000));
        t.Start();
        t.Wait();
        t.Dispose();
        t = null;
        GC.Collect();
        Console.WriteLine("Done");
        Console.ReadLine();
    }

    static void DoMemoryHog(int n) {
        ConcurrentBag<double> results = new ConcurrentBag<double>();

        Parallel.For(0, n, (i) => {
            results.Add(Math.Sqrt(i.GetHashCode()));
        });
    }
}

When I run the program, I can see the amount of used memory increasing in the windows task manager, but when the task is finished (and "Done" is displayed) the memory doesn't go back to it's original level, that only happens when I close the application.

Does anyone know how to free up memory used by a parallel task, while the main application keeps running? As you can see, I've already tried disposing it, setting it's reference to null and running the garbage collector manually (which you shouldn't do, I know).

Answer 1

This is due to the nature of a concurrentBag (see my earlier question regarding ConcurrentBag ( Possible memoryleak in ConcurrentBag? )).

Basically, a concurrent bag stores items on the local thread. If you dont consume the items, the items will stay on the local thread. Please check the following example:

    class TrackedItem
    {
        public TrackedItem()
        {
            Console.WriteLine("Constructor!");
        }
        ~TrackedItem()
        {
            Console.WriteLine("Destructor!");
        }
    }

    static void Main(string[] args)
    {
        Action allCollect = () =>
            {
                GC.Collect();
                GC.WaitForPendingFinalizers();
                GC.Collect();
            };

        // create a single item and loose it directly thereafter
        TrackedItem item = new TrackedItem();
        item = null;
        allCollect();
        // Constructor!, Destructor!

        ConcurrentBag<TrackedItem> bag = new ConcurrentBag<TrackedItem>();
        bag.Add(new TrackedItem());
        bag = null;
        allCollect();
        // Constructor!
        // Note that the destructor was not called since there is still a collection on the local thread

        Console.ReadLine();
    }

The concurrentBag makes use of the ThreadLocal class which makes it convenient to hold 1 instance per thread. The intended way for disposing data in a ThreadLocal is by calling the Dispose method on ThreadLocal (ThreadLocal implements IDisposable). This disposes only the data for the current thread. The concurrentBag doesn't dispose it's ThreadLocals though. Instead it relies on all items being consumed- or a thread hosting the ThreadLocal being disposed. this can be very nasty however when you share threads like within a ThreadPool.