c# - HttpWebRequest.GetResponse() keeps getting timed out

Question

i wrote a simple C# function to retrieve trade history from MtGox with following API call:

https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>

documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades

here's the function:

string GetTradesOnline(Int64 tid)
{
    Thread.Sleep(30000);

    // communicate
    string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    StreamReader reader = new StreamReader(response.GetResponseStream());

    string json = reader.ReadToEnd();
    reader.Close();
    reader.Dispose();
    response.Close();

    return json;
}

i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

here are the facts:

• catching the exception and retying keeps causing the same exception
• the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
• changing HttpWebRequest.KeepAlive to false didn't solve anything either
• it seems to always work in the browser even while the function is failing
• it has no problems retrieveing the response from https://www.google.com
• the amount of successful responses before the exceptions varies from day to day (but browser always works)
• starting at the trade id that failed last time causes the exception immediately
• calling this function from the main thread instead still caused the exception
• running it on a different machine didn't work
• running it from a different IP didn't work
• increasing Thread.Sleep inbetween requests does not help

any ideas of what could be wrong?

Answer 1

I had the very same issue. For me the fix was as simple as wrapping the HttpWebResponse code in using block.

using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
    // Do your processings here....
}

Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.