javascript - How to scrape a hidden hyperlink from a button with Python?

Question

I am trying to scrape the following website for a project: https://www.tunefind.com/show/chicago-fire/season-1/12210 The last step is to scrape the links to the spotify songs mentioned on a page. Normally I look into the source code and it is clear from there. However, in this case not. Looking into the source around the spotify button I cannot find a hyperlink directing me to the song. Probably done on purpose, to prevent scraping? (Oops)

Is there a way to get the hyperlink from the button? I am aware of an 'internet' interface in Python which clicks on the buttons, but I would rather not use this, as this will affect the load time tremendiously.

Thanks!

question from:https://stackoverflow.com/questions/65946145/how-to-scrape-a-hidden-hyperlink-from-a-button-with-python

Answer 1

If using Chorme, Look at DevTools, go to Network tab an reload the site. You'll find that the data yoou need is in this url:

https://www.tunefind.com/api/frontend/show/chicago-fire/season/1?fields=episodes,theme-song,music-supervisors,hot-songs,top-users,related-questions-season,composers,albums&metatags=1