EC2 Docker Instance No Space Left
$ docker-machine env s2-dynamodb-publisher
SSH cmd error!
command: if grep -xq 127.0.1.1.* /etc/hosts; then sudo sed -i 's/^127.0.1.1.*/127.0.1.1 s2-dynamodb-publisher/g' /etc/hosts; else echo '127.0.1.1 s2-dynamodb-publisher' | sudo tee -a /etc/hosts; fi
err : exit status 4
output : sed: couldn't flush /etc/sedYfFTgP: No space left on device
We have ran out of space. Let's log in the EC2 instance.
Let's see whats in the containers log
ubuntu@s2-dynamodb-publisher:~$ sudo ls -lh /var/lib/docker/containers/5e73d53180ddc3131ea51aadfc791d02a45db34f6ba472aa2e843d639b810fdd
total 13G
-rw------- 1 root root 13G Oct 5 17:36 5e73d53180ddc3131ea51aadfc791d02a45db34f6ba472aa2e843d639b810fdd-json.log
-rw-r--r-- 1 root root 3.0K Oct 5 17:14 config.json
-rw-r--r-- 1 root root 663 Oct 5 17:14 hostconfig.json
-rw-r--r-- 1 root root 13 Oct 5 17:14 hostname
-rw-r--r-- 1 root root 267 Oct 5 17:14 hosts
-rw-r--r-- 1 root root 191 Oct 5 17:14 resolv.conf
-rw------- 1 root root 71 Oct 5 17:14 resolv.conf.hash
Let's get rid of that log file
sudo rm /var/lib/docker/containers/5e73d53180ddc3131ea51aadfc791d02a45db34f6ba472aa2e843d639b810fdd/5e73d53180ddc3131ea51aadfc791d02a45db34f6ba472aa2e843d639b810fdd-json.log
Lets prevent it from happening again.
The docker-compose.tpl.yml
file I was using was the following:
publisher:
build: .
command: node index.js
environment:
- DEBUG=*
- NODE_AMQP_ENDPOINT=${NODE_AMQP_ENDPOINT}
- NODE_HONEYBADGER_KEY=${NODE_HONEYBADGER_KEY}
- NODE_AWS_ACCESS_KEY_ID=${NODE_AWS_ACCESS_KEY_ID}
- NODE_AWS_SECRET_ACCESS_KEY=${NODE_AWS_SECRET_ACCESS_KEY}
restart: always
I'm using docker 1.8.2 which has a json file option which enable you to define a log rotation policy- using compose:
publisher:
build: .
command: node index.js
environment:
- DEBUG=*
- NODE_AMQP_ENDPOINT=${NODE_AMQP_ENDPOINT}
- NODE_HONEYBADGER_KEY=${NODE_HONEYBADGER_KEY}
- NODE_AWS_ACCESS_KEY_ID=${NODE_AWS_ACCESS_KEY_ID}
- NODE_AWS_SECRET_ACCESS_KEY=${NODE_AWS_SECRET_ACCESS_KEY}
restart: always
log_opt:
max-size: "1g"
max-file: "4"