javascript - Scraping dynamically generated html inside Android app

Question

I am currently writing an Android app that, among other things, uses text information from websites which I do not own. In addition, some of the pages require authentification.

For some pages I have been able to log in and retrieve the html code using BasicNameValuePairs and an HTTPClient with its associated objects.

Unfortunately, these methods retrieve the webpage source without running any javascript functions that a browser (Android Webview even) would normally run. I need the text that some of these scripts are retrieving.

I've done my research, but everything I've found is guesswork & extremely confusing. I'm okay with ignoring pages that require login for now. Also, I am willing to post any code that may be useful for constructing a solution; It is an independent project.

Any concrete solutions for scraping the html result from javascript calls? An example would be absolutely top-notch.

Answer 1

Final Success:

• Rhino. Used this jar file.

Other Things I Tried:

• HttpClient provided by Android
  • Cannot run javascript
• HtmlUnit
  • 4 hours, no success. Also huge, added 12 mb to my apk.
• SL4A
  • Finally compiled. Used THIS guide to set-up. Abandoned as overkill for a simple rhino jar.

Things That Might Work:

• Selenium

Further results will be posted. Others results will be added if posted.

Note: many of the options listed above reference each other. I think rhino is included in both sl4a and htmlunit. Also, I think htmlunit contains selenium.