Amazon ECS agent on ubuntu not starting

Question

I am currently trying to build a custom ubuntu ami for AWS batch and following the document mentioned here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html

However when I try to start the docker agent on that machine it always keeps giving me this error

2018-07-04T23:34:01Z [INFO] Amazon ECS agent Version: 1.18.0, Commit: c0defea9
2018-07-04T23:34:01Z [INFO] Loading state! module="statemanager"
2018-07-04T23:34:01Z [INFO] Event stream ContainerChange start listening...
2018-07-04T23:34:01Z [INFO] Creating root ecs cgroup: /ecs
2018-07-04T23:34:01Z [INFO] Creating cgroup /ecs
2018-07-04T23:34:01Z [WARN] Disabling TaskCPUMemLimit because agent is unabled to setup '/ecs' cgroup: cgroup create: unable to create controller: mkdir /sys/fs/cgroup/systemd/ecs: read-only file system
2018-07-04T23:34:01Z [WARN] Error getting valid credentials (AKID ): NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-07-04T23:34:01Z [INFO] Registering Instance with ECS
2018-07-04T23:34:01Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-07-04T23:34:01Z [ERROR] Error registering: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

I made sure the instance has the ecsInstanceRole associated with that. Can you guys let me know what I am missing?

Answer 1

Not certain how you are starting the ecs-agent. Ran into the error of

Disabling TaskCPUMemLimit because agent is unabled to setup '/ecs cgroup: cgroup create: unable to create controller: /sys/fs/cgroup/systemd/ecs: read-only file system

We resolved this by adding the volume --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro to the systemd unit file that we having launching ecs.

Outside of that, I assume the issue resides with the ecsInstanceRole. Can you verify it has the following permissions? AmazonEC2ContainerRegistryFullAccess, AmazonEC2ContainerServiceFullAccess, AmazonEC2ContainerServiceforEC2Role

Below is the full systemd file for ecs-agent.

[Unit]
Description=Docker Container %I
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStartPre=-/usr/bin/docker rm -f %i
ExecStart=/usr/bin/docker run --name %i 
--restart=on-failure:10 
--volume=/var/run:/var/run 
--volume=/var/log/ecs/:/log:Z 
--volume=/var/lib/ecs/data:/data:Z 
--volume=/etc/ecs:/etc/ecs 
--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro 
--net=host 
--env-file=/etc/ecs/ecs.config 
--env LOGSPOUT=ignore 
amazon/amazon-ecs-agent:latest
ExecStop=/usr/bin/docker stop %i

[Install]
WantedBy=default.target