$ docker-machine env s2-dynamodb-publisher
SSH cmd error!
command: if grep -xq 127.0.1.1.* /etc/hosts; then sudo sed -i 's/^127.0.1.1.*/127.0.1.1 s2-dynamodb-publisher/g' /etc/hosts; else echo '127.0.1.1 s2-dynamodb-publisher' | sudo tee -a /etc/hosts; fi
err    : exit status 4
output : sed: couldn't flush /etc/sedYfFTgP: No space left on device

We have ran out of space. Let's log in the EC2 instance.


Let's see whats in the containers log

ubuntu@s2-dynamodb-publisher:~$ sudo ls -lh /var/lib/docker/containers/5e73d53180ddc3131ea51aadfc791d02a45db34f6ba472aa2e843d639b810fdd
total 13G
-rw------- 1 root root  13G Oct  5 17:36 5e73d53180ddc3131ea51aadfc791d02a45db34f6ba472aa2e843d639b810fdd-json.log
-rw-r--r-- 1 root root 3.0K Oct  5 17:14 config.json
-rw-r--r-- 1 root root  663 Oct  5 17:14 hostconfig.json
-rw-r--r-- 1 root root   13 Oct  5 17:14 hostname
-rw-r--r-- 1 root root  267 Oct  5 17:14 hosts
-rw-r--r-- 1 root root  191 Oct  5 17:14 resolv.conf
-rw------- 1 root root   71 Oct  5 17:14 resolv.conf.hash

Let's get rid of that log file

sudo rm /var/lib/docker/containers/5e73d53180ddc3131ea51aadfc791d02a45db34f6ba472aa2e843d639b810fdd/5e73d53180ddc3131ea51aadfc791d02a45db34f6ba472aa2e843d639b810fdd-json.log

Lets prevent it from happening again.

The docker-compose.tpl.yml file I was using was the following:

publisher:
  build: .
  command: node index.js
  environment:
    - DEBUG=*
    - NODE_AMQP_ENDPOINT=${NODE_AMQP_ENDPOINT}
    - NODE_HONEYBADGER_KEY=${NODE_HONEYBADGER_KEY}
    - NODE_AWS_ACCESS_KEY_ID=${NODE_AWS_ACCESS_KEY_ID}
    - NODE_AWS_SECRET_ACCESS_KEY=${NODE_AWS_SECRET_ACCESS_KEY}
  restart: always

I'm using docker 1.8.2 which has a json file option which enable you to define a log rotation policy- using compose:

publisher:
  build: .
  command: node index.js
  environment:
    - DEBUG=*
    - NODE_AMQP_ENDPOINT=${NODE_AMQP_ENDPOINT}
    - NODE_HONEYBADGER_KEY=${NODE_HONEYBADGER_KEY}
    - NODE_AWS_ACCESS_KEY_ID=${NODE_AWS_ACCESS_KEY_ID}
    - NODE_AWS_SECRET_ACCESS_KEY=${NODE_AWS_SECRET_ACCESS_KEY}
  restart: always
  log_opt:
      max-size: "1g"
      max-file: "4"