i want to create a script that returns all the urls found in a page a google for example , so i create this script : (using BeautifulSoup)
import urllib2
from BeautifulSoup import BeautifulSoup
page = urllib2.urlopen("https://www.google.dz/search?q=see")
soup = BeautifulSoup(page.read())
links = soup.findAll("a")
for link in links:
print link["href"]
and it return this 403 forbidden result :
Traceback (most recent call last):
File "C:Python27sqlsql.py", line 3, in <module>
page = urllib2.urlopen("https://www.google.dz/search?q=see")
File "C:Python27liburllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:Python27liburllib2.py", line 400, in open
response = meth(req, response)
File "C:Python27liburllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:Python27liburllib2.py", line 438, in error
return self._call_chain(*args)
File "C:Python27liburllib2.py", line 372, in _call_chain
result = func(*args)
File "C:Python27liburllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
any idea to avoid this error or another methode to get the urls from of the search result ?
See Question&Answers more detail:os
