Welcome to ShenZhenJia Knowledge Sharing Community for programmer and developer-Open, Learning and Share
menu search
person
Welcome To Ask or Share your Answers For Others

Categories

今天尝试了一下用php拉取头条文章内容,但并不是那么顺利,整体分成三个步骤。

首先,如果我们直接请求文章内容,以postman举例,它会返回如下一段Javascript代码:

image.png

<html><head><meta charset="UTF-8" /></head><body></body><script src='https://sf1-ttcdn-tos.pstatp.com/obj/rc-web-sdk/acrawler.js'></script><script>window.byted_acrawler.init({aid:99999999,dfp:!0});var b;a:{for(var c=document.cookie.split(/[;&]/),d,e=0;e<c.length;e++){for(d=c[e];" "===d.charAt(0);)d=d.substring(1,d.length);if(0===d.indexOf("__ac_nonce=")){b=d.substring(11,d.length);break a}}b=""}var f=b;var g=window.byted_acrawler.sign("",f);document.cookie="__ac_signature=; expires=Mon, 20 Sep 1970 00:00:00 UTC; path=/;";
document.cookie="__ac_signature="+g+"; expires="+(new Date((new Date).getTime()+18E5)).toGMTString()+"; path=/;";window.location.reload();</script></html>

分析下上面这段代码就知道,它从cookie中获取__ac_nonce这个值,然后用acrawler.js文件中的方法加密成一串__ac_signature,这些都可以理解,没有问题。

有个疑问:__ac_nonce是怎么来的??

继续看刚才用post请求的响应信息,我发现这个值是通过头条响应头带过来的。

image.png

尝试不断请求,是否每次头条都会响应__ac_nonce值,于是我尝试写了如下一个demo脚本:

$ch = curl_init();
$headers = [
    'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
];

curl_setopt($ch, CURLOPT_URL, 'https://www.toutiao.com/a6846918379043815950');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 3);

$rs = curl_exec($ch);
// 查看具体返回值
var_dump($rs, curl_error($ch));
// 响应头中匹配cookie部分,为了获取__ac_nonce值
preg_match_all('/^Set-Cookie:s*([^;]*)/mi', $rs, $matches);
$cookies = [];
$nonce = "";
foreach($matches[1] as $item) {
    parse_str($item, $cookie);
    $cookies = array_merge($cookies, $cookie);
}
if (! empty($cookies['__ac_nonce'])) {
    $nonce = $cookies['__ac_nonce'];
}

echo 'ac_nonce: ' . $nonce . PHP_EOL;

很遗憾,结果极度不稳定:

bool(false)
string(67) "Operation timed out after 2540 milliseconds with 682 bytes received"

但还是有部分请求给我重新分配了__ac_nonce值,只是没有的情况比较多,这是什么问题?

附言

拿到__ac_nonce配合nodejs生成__ac_signature我最后在拉取详情的时候把请求头带上cookie成功的拉到了头条内容, 如下:

// ...
$headers = [
    'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
    'cookie: __ac_nonce=xxx;__ac_signature=xxx;',
];
// ...

所以,问题是:什么是__ac_nonce?怎么100%稳定得到它的值?


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
thumb_up_alt 0 like thumb_down_alt 0 dislike
516 views
Welcome To Ask or Share your Answers For Others

1 Answer

问题自己已经找到了答案,我发现用postman每次请求,响应头中的__ac_nonce都附有新的值,但body的响应速度不稳定,有时候非常慢。

解决方案是:不要响应body内容,仅要求得到响应头信息,这样就不会导致body超时而无法正确设置__ac_nonce值,php代码:

curl_setopt($ch, CURLOPT_NOBODY, true); // 最核心的一行代码解决问题,拒绝body返回

每次都会得到响应的__ac_nonce值:

HTTP/2 200
server: Tengine
content-type: text/html
content-length: 682
date: Thu, 09 Jul 2020 12:58:31 GMT
vary: Accept-Encoding
set-cookie: __ac_nonce=05f07147700e7b19045f9; Path=/; Max-Age=1800
server-timing: inner; dur=4
x-tt-trace-host: 0183a88c5cb521b4b793d71cf69322c4d084c3d1b6113acf546d6ca2abcd1ffb2c74c68419495975cdb2f8d8caa8132033892b1d35699d42ecd925fe0c7a278105f02a151ccfda62976bffffade8bbdce3
x-tt-trace-tag: id=3;cdn-cache=miss
server-timing: cdn-cache;desc=MISS,edge;dur=0,origin;dur=48
via: vcache49.cn2658[48,0]
timing-allow-origin: *
eagleid: b73d7cc515942995115473548e

"

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
thumb_up_alt 0 like thumb_down_alt 0 dislike
Welcome to ShenZhenJia Knowledge Sharing Community for programmer and developer-Open, Learning and Share
...