c# - What is the fastest way to read data from a DbDataReader?

Question

In the following code, command is a DbCommand that has already been set up:

using( var dataReader = command.ExecuteReader() /*The actual execution of the query takes relatively little time.*/ ) {
                while( dataReader.Read() ) {
                    // These are what take all of the time. Replacing them all with reader.GetValues( myArray ) has no impact.
                    val0 = dataReader.GetValue( 0 );
                    val1 = dataReader.GetValue( 1 );
                    val2 = dataReader.GetValue( 2 );
                }
            }

The bulk of the time for the query I am currently working with is spent doing the GetValue calls. Is it making a round trip to the database for each GetValue call? It seems like it is, and this seems very inefficient. As the code notes, attempting to do it in one shot using GetValues() does not make a difference. Is there a way to get the entire row in one shot? Better yet, is there a way to get the entire result set in one shot?

Thanks.

Answer 1

I did some benchmarking myself with various approaches:

public DataTable Read_using_DataTable_Load(string query)
{
    using (var cmd = conn.CreateCommand())
    {
        cmd.CommandText = query;
        cmd.Connection.Open();
        var table = new DataTable();
        using (var r = cmd.ExecuteReader())
            table.Load(r);
        return table;
    }
}

public DataTable Read_using_DataSet_Fill<S>(string query) where S : IDbDataAdapter, IDisposable, new()
{
    using (var da = new S())
    {
        using (da.SelectCommand = conn.CreateCommand())
        {
            da.SelectCommand.CommandText = query;
            DataSet ds = new DataSet();
            da.Fill(ds);
            return ds.Tables[0];
        }
    }
}

public IEnumerable<S> Read_using_yield_selector<S>(string query, Func<IDataRecord, S> selector)
{
    using (var cmd = conn.CreateCommand())
    {
        cmd.CommandText = query;
        cmd.Connection.Open();
        using (var r = cmd.ExecuteReader())
            while (r.Read())
                yield return selector(r);
    }
}

public S[] Read_using_selector_ToArray<S>(string query, Func<IDataRecord, S> selector)
{
    using (var cmd = conn.CreateCommand())
    {
        cmd.CommandText = query;
        cmd.Connection.Open();
        using (var r = cmd.ExecuteReader())
            return ((DbDataReader)r).Cast<IDataRecord>().Select(selector).ToArray();
    }
}

public List<S> Read_using_selector_into_list<S>(string query, Func<IDataRecord, S> selector)
{
    using (var cmd = conn.CreateCommand())
    {
        cmd.CommandText = query;
        cmd.Connection.Open(); 
        using (var r = cmd.ExecuteReader())
        {
            var items = new List<S>();
            while (r.Read())
                items.Add(selector(r));
            return items;
        }
    }
}

1 and 2 returns DataTable while the rest strongly typed result set, so its exactly not apples to apples, but I while time them accordingly.

Just the essentials:

Stopwatch sw = Stopwatch.StartNew();
for (int i = 0; i < 100; i++)
{
    Read_using_DataTable_Load(query); // ~8900 - 9200ms

    Read_using_DataTable_Load(query).Rows.Cast<DataRow>().Select(selector).ToArray(); // ~9000 - 9400ms

    Read_using_DataSet_Fill<MySqlDataAdapter>(query); // ~1750 - 2000ms

    Read_using_DataSet_Fill<MySqlDataAdapter>(query).Rows.Cast<DataRow>().Select(selector).ToArray(); // ~1850 - 2000ms

    Read_using_yield_selector(query, selector).ToArray(); // ~1550 - 1750ms

    Read_using_selector_ToArray(query, selector); // ~1550 - 1700ms

    Read_using_selector_into_list(query, selector); // ~1550 - 1650ms
}

sw.Stop();
MessageBox.Show(sw.Elapsed.TotalMilliseconds.ToString());

The query returned about 1200 rows and 5 fields (run for 100 times). Apart from Read_using_Table_Load all performed well.

Of all I prefer Read_using_yield_selector which returns data lazily, as enumerated. This is great for memory if you only need to enumerate it. To have a copy of the collection in memory, you're better off with Read_using_selector_ToArray or Read_using_selector_into_list as you please.