Downloading remote file using WordPress forbidden

Question

I am trying to use WordPress' media_sideload_image with a remotely hosted image in S3 in order to save it into WordPress' media gallery.

But for whatever reason, I always get a forbidden response no matter what I try and do regarding request options for the WordPress request. Visiting the URL directly in the browser works, wget works, postman works.

Does anyone have any ideas on how to make WordPress be able to successfully download this file?

Here's the code I'm using:

$attachment_ID = media_sideload_image('https://s3.amazonaws.com/mlsgrid/images/0788b2c2-d865-496b-bad3-69ebe9c1db79.png');

And here's the WordPres error response I get:

object(WP_Error)[2090]
  public 'errors' => 
    array (size=1)
      'http_404' => 
        array (size=1)
          0 => string 'Forbidden' (length=9)
  public 'error_data' => 
    array (size=1)
      'http_404' => 
        array (size=2)
          'code' => int 403
          'body' => string '<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>39B59073BBC1205F</RequestId><HostId>6TwMl4cMbLXzr7jbx6ykQKaQuk0Rn5Oyc2Q3+02zmgtNoDqUvcg8VY32qGuS1ZMzgpZuLAefK3g=</HostId></Error>' (length=243)
  protected 'additional_data' => 
    array (size=0)
      empty

Thanks!

question from:https://stackoverflow.com/questions/65942851/downloading-remote-file-using-wordpress-forbidden

Answer 1

After digging around WordPress' request functionality, it looks like it is setting a referer header on each request to be the same as the URL being fetched and I guess Amazon S3 rejects requests with a referer header set? (not sure if that is specific to the bucket I'm fetching images from or true across every single bucket).

Here's how I got it working by removing the referer header from the request, basically just filter for all S3 URLs and remove the referer request header.

    // Remove referer from request headers if the URL is an S3 bucket.
    add_action( 'http_api_curl', function ($ch, $parsed_args, $url) {
        $s3_url = 'https://s3.amazonaws.com';

        if (substr($url, 0, strlen($s3_url)) === $s3_url) {
            curl_setopt($ch, CURLOPT_HTTPHEADER, ['Referer:']);
        }
    }, 10, 3);