i scraping web page multiple pages. appreciate following problem:
i have built loop around url of web page. however, when looking tags in html code information page 1 appears. seems loop not flowing through. unfortunately cannot find mistake in following code:
for pagenumber in range(1,50): url = "http://suchen.mobile.de/fahrzeuge/auto/search.html?zipcoderadius=100&scopeid=c&ambitcountry=de&makemodelvariant1.makeid=3500&makemodelvariant1.modelid=115%2c98%2c80%2c99%2c102%2c81%2c100%2c83%2c105%2c82%2c101%2c120%2c121&makemodelvariant1.modelgroupid=53&issearchrequest=true&pagenumber + str(pagenumber)" r = requests.get(url) soup = beautifulsoup(r.content,"lxml") # parsing data webpage cartypetemp=[] cartypeweb = soup.find_all("span", {"class":"h3"}) # writing car type/description in list in range(0,len(cartypeweb),2): cartypetemp.extend((cartypeweb[i]))
in forloop doing:
url = "* + str(pagenumber)" this literally url be, , isn't concatenating think is.
>>> "a url + str(pagenumber)" "a url + str(pagenumber)" you want:
url = "*" + str(pagenumber) or use string formatters, whatever prefer.
edit: didn't catch difference between names / capitalization noted in comment.
you want pagenumber not pagenumber. pagenumber doesn't exist.
