php - Make a JavaScript-aware Crawler

Question

I want to make a script that's crawling a website and it should return the locations of all the banners showed on that page.

The locations of banners are most of the time from known domains. But banners are not in the HTML as an easy image or swf-file. Most of the times a Javascript is used to show the banner.

So if a .swf-file or image-file is loaded from a banner-domain, it should return that url.

Is that possible to do? And how could I do that roughly?

Best would be if it can also returns the landing page of that ad. How to solve that?

Answer 1

You could use selenium to open the pages in a real browser and then access the DOM.

PhantomJS might also be worth a look - it's a headless version of WebKit (the engine behind Chrome, Safari, etc.).

However, none of those solutions are pure php - if that's a requirement, you'll probably have to write your own JavaScript engine in PHP (which is nothing I'd ask my worst enemy to do ;))