Using PHP cURL to downloading URL with special characters

Question

I'm trying to download the following URL https://www.astegiudiziarie.it/vendita-asta-appartamento-genova-via-san-giovanni-d’acri-14-1360824 with PHP cURL:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.astegiudiziarie.it/vendita-asta-appartamento-genova-via-san-giovanni-d’acri-14-1360824');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$f = curl_exec($ch);
curl_close($ch);
echo $f;

but the server always returns an error page. Navigating the same URL in a web browser works fine. Manually comparing the HTML source returned by curl_exec with the HTML source in a web browser, the difference is immediately noticeable.

I tried to utf8_decode() the URL without success.

I cannot simply wrap the url in urlencode() because it will encode even normal characters like : and /.

That URLs are retrieved programmatically (scraping) and won't always have the same structure, so it would be difficult to split them and urlencode just some parts.

Btw, it seems that modern web browsers handle this case very well. There is a solution for that in PHP?

Answer 1

Your URL is already encoded. Do not call urlencode() on it as that the reason you get 404, as server decodes only once. Just remove the call.