为何curl取不到内容？

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

For Existing Member Sign In

This topic created in 5033 days ago, the information mentioned may be changed or developed.

各位试试这个 curl "http://brand.tmall.com/azIndexInside.htm?firstLetter=A&prt=1342414752421&prc=5" 能否取到内容。

初步研究貌似跟referer、useragent等无关。

curl

useragent

referer

7 replies • 1970-01-01 08:00:00 +08:00

yujnln

Jul 16, 2012

可以。
>>> print len(content)
87031

bigdude

Jul 16, 2012

@yujnln 你用的python？我用urllib2老是告诉我
urllib2.HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Moved Temporarily

yujnln

Jul 16, 2012

@bigdude 是的。
你可以参考下这里 http://stackoverflow.com/questions/9113652/how-do-i-set-cookies-using-python-urlopen

bigdude

Jul 16, 2012

抓狂·······
>>> a=urllib.urlopen('http://brand.tmall.com/azIndexInside.htm?firstLetter=A&prt=1342414752421&prc=5')
>>> len(a.read())
0

bigdude

Jul 16, 2012

@yujnln ok了，必须要带cookie，不带不让抓。

est

Jul 16, 2012

@bigdude 不需要cookie也可以的

curl -vL "http://brand.tmall.com/azIndexInside.htm?firstLetter=A&prt=1342414752421&prc=5"

bigdude

Jul 16, 2012

@est 了解了，强制让curl follow这个链接，用-L就行了，搞不懂淘宝为何搞这么多跳转