c# - How should i properly invoke a WebBrowser using multiplethreads?

Question

Problem Scope:

I'm writing an aplication to save the HTML's retrieved from the Bing and Google searches. I know there are classes to execute the Web Requests using stream such as this example, but since Google and Bing both use Javascript and Ajax to render the results into the HTML, there's no way i can simply read the stream and use get to the result i need.

The solution to this, is to use the WebBrowser class and navigate to the url i want, so that the Browser itself will handle all the Javascript and Ajax scripting executions.

MultiThreading:

In order to make it more efficient, i have the same Form aplication firing a thread for each service (one for Bing, and one for Google).

Problem:

Since i need the WebBrowser, i have instantiated one for each thread (which are 2, at this moment). According to Microsoft, there is a known bug that prevents the DocumentCompleted event from firing if the WebBrowser is not visible and is not added to a visible form aswell (for more information, follow this link).

Real Problem:

The main issue is that, the DocumentCompleted event of the browser, never fires. Never.

I have wrote a proper handler for the DocumentCompleted event that never gets the callback. For handling the wait needed for the Browser event to fire, i have implemented a AutoResetEvent with a high timeout (5 minutes), that will dispose the webbrowser thread if it does not fire the event i need after 5 minutes.

At the moment, i have the Browser created and added into a WindowsForm, both are visible, and the event is still not firing.

Some Code:

        // Creating Browser Instance
        browser = new WebBrowser ();

        // Setting up Custom Handler to "Document Completed" Event
        browser.DocumentCompleted += DocumentCompletedEvent;

        // Setting Up Random Form
        genericForm = new Form();
        genericForm.Width = 200;
        genericForm.Height = 200;
        genericForm.Controls.Add (browser);
        browser.Visible = true;  

As for the Navigation i have the Following (method for the browser) :

    public void NavigateTo (string url)
    {
        CompletedNavigation = false;

        if (browser.ReadyState == WebBrowserReadyState.Loading) return;

        genericForm.Show (); // Shows the form so that it is visible at the time the browser navigates
        browser.Navigate (url);
    }

And, for the call of the Navigation i have this :

            // Loading URL
            browser.NavigateTo(URL);

            // Waiting for Our Event To Fire
            if (_event.WaitOne (_timeout))
            {
               // Success
            }
            { // Error / Timeout From the AutoResetEvent } 

TL:DR:

My WebBrowser is instantiated into a another STAThread, added to a form, both are visible and shown when the Browser Navigation fires, but the DocumentCompleted event from the Browser is never fired, so the AutoResetEvent always times out and i have no response from the browser.

Thanks in Advance and sorry for the long post

Answer 1

Although this seems a strange way, here is my attempt.

var tasks = new Task<string>[]
{
    new MyDownloader().Download("http://www.stackoverflow.com"),
    new MyDownloader().Download("http://www.google.com")
};

Task.WaitAll(tasks);
Console.WriteLine(tasks[0].Result);
Console.WriteLine(tasks[1].Result);

---

public class MyDownloader
{
    WebBrowser _wb;
    TaskCompletionSource<string> _tcs;
    ApplicationContext _ctx;
    public Task<string> Download(string url)
    {
        _tcs = new TaskCompletionSource<string>();

        var t = new Thread(()=>
        {
            _wb = new WebBrowser();
            _wb.ScriptErrorsSuppressed = true;
            _wb.DocumentCompleted += _wb_DocumentCompleted;
            _wb.Navigate(url);
            _ctx = new ApplicationContext();
            Application.Run(_ctx);
        });

        t.SetApartmentState(ApartmentState.STA);
        t.Start();

        return _tcs.Task;
    }

    void _wb_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
    {
        //_tcs.TrySetResult(_wb.DocumentText);
        _tcs.TrySetResult(_wb.DocumentTitle);
        _ctx.ExitThread();
    }
}