python - In AWS, Execute notebook from sagemaker

Question

Im using AWS SageMaker Notebooks.

What is the best way to execute notebook from sagemaker?

• My idea is to have an S3 bucket.
• When a new file is putted there i want to execute a notebook that reads from S3 and puts the output in other bucket.

The only way i have from now is to start an S3 event, execute a lambda function that starts a sagemaker instance and execute the notebook. But is getting too much time to start and it doesnt work yet for me with a big notebook.

Maybe is better to export the notebook and execute it from another place in aws (in order to be faster), but i dont know where.

Thanks in advance

question from:https://stackoverflow.com/questions/66061721/in-aws-execute-notebook-from-sagemaker

Answer 1

I have limited experience, but..If you are talking about training jobs, the only other way to launch one is to create you own container, push to ECR and launch directly the training job without dealing with notebooks. I am working on a similar project where an S3 upload triggers a lambda function which start a container (it's not sagemaker but the concept is more or less the same). The problem with this approach is that however AWS takes time to launch an instance, minutes I would say. Another approach could be to have a permanent running endpoint and trigger in some way the elaboration.