google cloud ml - Deploying a model to the regional endpoint on AI Platform Prediction succeeds, but deploying the same model to the global endpoint fails

Question

I have a scikit-learn model saved in Cloud Storage which I am attempting to deploy with AI Platform Prediction. When I deploy this model to a regional endpoint, the deployment completes successfully:

? gcloud ai-platform versions describe regional_endpoint_version --model=regional --region us-central1
Using endpoint [https://us-central1-ml.googleapis.com/]
autoScaling:
  minNodes: 1
createTime: '2020-12-30T15:21:55Z'
deploymentUri: <REMOVED>
description: testing deployment to a regional endpoint
etag: <REMOVED>
framework: SCIKIT_LEARN
isDefault: true
machineType: n1-standard-4
name: <REMOVED>
pythonVersion: '3.7'
runtimeVersion: '2.2'
state: READY

However, when I try to deploy the exact same model, using the same Python/runtime versions, to the global endpoint, the deployment fails, saying there was an error loading the model:

(aiz) ?  stanford_nlp_a3 gcloud ai-platform versions describe public_object --model=global
Using endpoint [https://ml.googleapis.com/]
autoScaling: {}
createTime: '2020-12-30T15:12:11Z'
deploymentUri: <REMOVED>
description: testing global endpoint deployment
errorMessage: 'Create Version failed. Bad model detected with error:  "Error loading
  the model"'
etag: <REMOVED>
framework: SCIKIT_LEARN
machineType: mls1-c1-m2
name: <REMOVED>
pythonVersion: '3.7'
runtimeVersion: '2.2'
state: FAILED

I tried making the .joblib object public to make sure there wasn't a permissions difference when trying to deploy to the two endpoints causing the issue, but the deployment to the global endpoint still failed. I removed the deploymentUri from the post since I have been experimenting with the permissions on this model object, but the paths are identical in the two different model versions.

The machine types for the two deployments have to be different, and for the regional deployment I use min nodes = 1 while for global I can use min nodes = 0, but other than that and the etags everything else is exactly the same.

I couldn't find any information in the AI Platform Prediction regional endpoints docs page which indicated certain models could only be deployed to a certain type of endpoint. The "Error loading the model" error message doesn't give me a lot to go on since it doesn't appear to be a permissions issue with the model file.

When I add the --log-http option to the create version command, I see that the errorcode is 3, but the message doesn't reveal any additional information:

?  ~ gcloud ai-platform versions create $VERSION_NAME 
  --model=$MODEL_NAME 
  --origin=$MODEL_DIR 
  --runtime-version=2.2 
  --framework=$FRAMEWORK 
  --python-version=3.7 
  --machine-type=mls1-c1-m2 --log-http

Using endpoint [https://ml.googleapis.com/]
=======================
==== request start ====
...
...
the final response from the server looks like this:
---- response start ----
status: 200
-- headers start --
<headers>
-- headers end --
-- body start --
{
  "name": "<name>",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.ml.v1.OperationMetadata",
    "createTime": "2020-12-30T22:53:30Z",
    "startTime": "2020-12-30T22:53:30Z",
    "endTime": "2020-12-30T22:54:37Z",
    "operationType": "CREATE_VERSION",
    "modelName": "<name>",
    "version": {
      <version info>
    }
  },
  "done": true,
  "error": {
    "code": 3,
    "message": "Create Version failed. Bad model detected with error:  "Error loading the model""
  }
}

-- body end --
total round trip time (request+response): 0.096 secs
---- response end ----
----------------------
Creating version (this might take a few minutes)......failed.
ERROR: (gcloud.ai-platform.versions.create) Create Version failed. Bad model detected with error:  "Error loading the model"

Can anyone explain what I am missing here?

Answer 1

等待大神答复