This is the first blog that someone's actually asked me to write. Most of these I write simply because it's a good way for me to remember and explore the ideas covered in them. In this case though, it came from a conversation about how they couldn't find any good docker tutorials that explained how they set it up. So I'm going to have a go :)

So what is Docker then? It's magic. That's pretty much all you need to know to begin with. There's stuff in there are virtualisation layers and shared operating systems, but it's only stuff you only kinda need to have heard, once, in a loud and crowded place. If you do want to learn more, then their docs tell you a bit more.

What you really need to know about Docker is that you have containers and you have images. A container is a running image. The aim is to have containers do only one thing and then use multiple containers to create your application. These will end up being things like a MySQL server in one, a Redis server in another, etc, etc.

You don't necessarily have to have just one thing in a Docker container, but they do expect a single command to run. For instance, I prefer to run PHP and Apache in a single container as I view it as a single unit and if either one is not working then there's an issue. For this I use another tool like supervisord running inside my container, managing the PHP and Apache processes.

Images, then, are what we build our Dockerfiles into and what we use for creating containers. You can use pre-built images for composing your application. A good source of these is hub.docker.com but at some point it's likely you'll need to build your own.

The images have a very cool, onion-like approach to storage in that they're stored as an ordered list of layers. This means that if you always use the same base image for your own images, such as using debian:8, Docker will only download one copy of it. On that point, when you run a Docker container it will need to first download any images it needs to the local filesystem.

The Dockerfile

As I've already touched upon a couple of times so far, Docker uses things called Dockerfiles. These are how you create an image that does what you want. They are used to define what base image you're going to use, if any, and also to define any additional instructions you want to run after the previous image has completed it's own instructions.

If you read the documentation for Dockerfiles you'll see there are loads of commands you can use. When starting out though, I'd recommend using only a few of them. Below is an example Dockerfile I used for testing my Alexa code.

FROM            debian:8

# Install basic software for server
RUN             apt-get update && \
                apt-get upgrade -y && \
                apt-get install -y \
                    curl \
                    git \
                    vim \
                    wget

# Install node and serverless
RUN             curl -sL https://deb.nodesource.com/setup_6.x -o nodesource_setup.sh && \
                bash nodesource_setup.sh && \
                apt-get install -y \
                    nodejs \
                    build-essential && \
                npm install serverless -g

# adds aws command line
RUN             curl -O https://bootstrap.pypa.io/get-pip.py && \
                python get-pip.py && \
                pip install awscli
COPY            ./etc/aws.config ~/.aws/config
COPY            ./etc/aws.credentials ~/.aws/credentials

# Clean apt-get
RUN             apt-get clean && \
                rm -rf /var/lib/apt/lists/*

# Fix vim controls
RUN             echo "set term=xterm-256color" >> ~/.vimrc

WORKDIR         /var/alexa-test
CMD             ["tail", "-F", "-n0", "/etc/hosts"]

I hope it's relatively easy to follow. The first command says we're going to use the debian:8 base image. This is because Debian is quite a small distribution and if you know Ubuntu then you can find your way around Debian.

The next 3 steps are all RUN commands. These will just run a normal command as if you were logged into a Debian 8 server as root. I use them for setting up the specialised software I need for my testing. There may only be 3 RUN commands, but each of these uses && \ to include multiple commands.

# Install basic software for server
RUN             apt-get update && \
                apt-get upgrade -y && \
                apt-get install -y \
                    curl \
                    git \
                    vim \
                    wget

# Install node and serverless
RUN             curl -sL https://deb.nodesource.com/setup_6.x -o nodesource_setup.sh && \
                bash nodesource_setup.sh && \
                apt-get install -y \
                    nodejs \
                    build-essential && \
                npm install serverless -g

# adds aws command line
RUN             curl -O https://bootstrap.pypa.io/get-pip.py && \
                python get-pip.py && \
                pip install awscli

Each of those RUN commands will become a layer in our final image. The layers have to be built if we want to run our Docker container. We don't want too many as there's an upper limit on the number you can have, and having multiple layers slows down our deployment too much.

On the other hand, if we compressed all of those RUN commands into a single one then whenever we altered a command in it, the layer would have to be rebuilt. This can sometimes be quite a slow process depending on what you're doing. Therefore, you want to find a balance between having too few and too many commands. I personally prefer to group them to make them more readable than any alternative method.

COPY            ./etc/aws.config ~/.aws/config
COPY            ./etc/aws.credentials ~/.aws/credentials

The next commands up are COPY commands. We use these when we want to copy a file from our local file system into the container. In this case I'm using it to copy a couple of local config files into the home directory of the container. These are useful when you want to populate files in various locations in your container. There is a limitation here though, in that you can't copy files from outside of your local directory (so no copying from /etc/hosts, for instance).

WORKDIR         /var/alexa-test

The WORKDIR command is quite simple. It's the directory in the container that everything's going to be relative to. If you're using absolute paths for everything that it could affect (as we are), then the only consequence this command has is to set the directory you'll land in if you decide to log into the container.

Lastly, we use the CMD command. As mentioned above, Docker containers are based around running a single command. When that command triggers an exit code, the Docker container will also exit.

CMD             ["tail", "-F", "-n0", "/etc/hosts"]

Sometimes this isn't desirable though, as you may want to keep a container open for development or other purposes. In these instances, it's useful to set your CMD to be something that will never end. In this case it's going to tail the hosts file forever.

This CMD could just as easily be a call to supervisord using [/usr/bin/supervisord]. Do remember though, that you can't point your command at, say, Apache and expect it to work as a lot of services tend to run as background tasks. You'd need to run these services as foreground tasks.

The last thing to mention regarding this CMD command is the format. It's basically just replacing the spaces with a comma, and turning the entire string into an array.

Compose files

Now we have a Dockerfile, we can create a docker image and container if we choose to. This is great for going live and I'll cover the commands to do that later with, but I mentioned earlier how we can compose multiple Docker containers together to create full apps easily. With Docker, this is done with a docker-compose.yaml file. Below is an example of a docker-compose file:

version: '2.0'
services:

    alexa-test:
        build:
            context: .
            dockerfile: Dockerfile
        container_name: alexa-test
        environment:
            SLS_DEBUG: "*"
        ports:
            - "180:443"
        networks:
            - test_network
        volumes:
            - .:/var/alexa-test

    alexa-dynamo-db:
        image: cnadiminti/dynamodb-local
        container_name: alexa-dynamo-db
        expose:
            - "8000"
        networks:
            - test_network


networks:
    test_network:
        external:
            name: test_network

There are multiple versions of the docker-compose files, so our first declaration is to say what version we're going to use. Aside from that, we're defining our services and networks. Let's cover the networks first as they're the easiest one.

networks:
    mother_network:
        external:
            name: mother_network

You don't have to define a network. It's entirely optional, but it is very useful. By defining a network and then telling your services to use it, you can have multiple, entirely separate projects using the same network and therefore able to access each other's services.

This becomes handy when you have a large project with a number of standalone services that can optionally use other services if they exist.

The services, then, will becomes our containers. The above example has two services defined. One that's to be built from our Dockerfile and one that's pulled from an image stored on hub.docker.com (If you don't define a full URL then Docker will automatically check locally and in the Docker hub).

    alexa-dynamo-db:
        image: cnadiminti/dynamodb-local
        container_name: alexa-dynamo-db
        expose:
            - "8000"
        networks:
            - test_network

The second service is basically just specifying it's container name and what network it belongs to and a couple of other things. We define the name if we don't want Docker giving us a random one. Having a fixed name isn't always useful, as without one you can run multiple identical containers without having to worry about name collisions. With a fixed name, however, you can call it directly, as seen in the commands below.

The last option here not yet covered is the exposeoption. This option allows us to send information to this container on port 8000 (in the example above) from other containers. It doesn't, however, allow any external access. We'll cover that in a sec.

    alexa-test:
        build:
            context: .
            dockerfile: Dockerfile
        container_name: alexa-test
        environment:
            SLS_DEBUG: "*"
        ports:
            - "180:443"
        networks:
            - test_network
        volumes:
            - .:/var/alexa-test

The first service is our main service so obviously has more options. It shows how to define environment variables and what networks it belongs to and a couple of other things.

        ports:
            - "180:443

Docker containers, by default, are relatively secure from outside access. They won't expose any ports unless told to, so we have to define them as they have been above.

This definition allows us to say that any request to the port 180 will be translated into port 443 inside the container. Therefore, to send data too port 443 in the container, we can access http://localhost:180 on our local system.

        volumes:
            - .:/var/alexa-test

The last part we're going to cover here is the volume option. Unfortunately, this doesn't define how loud you want your container to be. Instead, this allows us to share a folder in our local system with the container we're building. In this case I share my local directory as it contains all of my code, which makes this perfect for development as my container will share the code I'm working on using an external IDE.

You probably don't want to do this for production Docker containers as it does increase overhead. Instead, you'd probably want to update your Dockerfile to just COPY over the directory. Using a COPY command in Dockerfile instead of a docker-compose volume also gives you the benefit of being able to store everything as an image, which is great for deployments and rolling back.

Overriding docker-compose files

These compose files are very useful from a development perspective, but they're even more useful when you start overriding parts of them. You can define another docker-compose file, like:

version: '2.0'
services:

    alexa-test:
        environment:
            env_name: testing
            secret: wibble
        ports:
            - "160:443"

This can then be composited with the previous docker-compose file to produce a full file with these above values overridden as described. The command for this is shown below.

The times you'll use this are relatively limited, but possibly for a test server, or a build server, or even if each development environment needs different variables for some reason.

Commands

I've grouped all of the commands in one place here, as I find them useful myself and it's easier to read them all in one place.

Let's start by covering how you'd run a development box, based on having created the files above.

docker-compose up -d --build

This command turns your docker-compose file into images and then into running containers. You'll probably use this one a lot. If your Dockerfile and docker-compose.yaml files are present and correct then that's the one command you need to get your containers up and running and usable.

The --build command is useful for when you're still making changes to your Dockerfile. When you're done with that though, you can drop the --build from the command as it means docker-compose will just use the pre-built images, which is quicker to start up from than of checking their up-to-date / building them again each time.

docker ps -a

To see how your containers are doing, run the above command. It'll show all running and recently exited containers, together with some vaguely useful information about them.

Let's assume you have a nice, happily running container but you need to use it for development or debug some issues on it:

docker exec -it alexa-test bash

This command will allow you to run "bash" on your "alexa-test" container (which is the fixed name of the container from the docker-compose file above). Once run, you'll now be inside your container and can use it as if it was whatever flavour of linux is running inside.

You can also choose to run any command you like instead of "bash" such as "docker exec -it alexa-test whoami" or "docker exec -it alexa-test curl -I http://www.google.co.uk". Basically, any command that isn't infinitely long and gives a good output is a good thing to run like this.

If you want to shut down your composed containers, try either of the following commands:

docker-compose down # or docker rm -f (docker ps -aq)

The first will, when run from the same folder as the "up" command, simply stop all the containers created by the "up" command. The second, however, is a bit more nuclear. The second command is a nice, two-step process that first returns all of Docker's running container ids, and then forcibly shuts them down.

If, as mentioned above, you wish to use composite multiple docker-compose files together in a given order then run:

docker-compose -f docker-compose.yaml -f docker-compose.test.yaml up -d --build

This is like the docker-compose command above, but specifies which docker-compose yaml files to use and in which order.

Also mentioned above was the concept of the Docker images and their layers. If you wish to see which ones have been downloaded locally, you can run:

docker images

This isn't really that necessary day-to-day however, as you rarely care about the images that much, and when you do it's quite often the next command:

docker rmi -f $(docker images -q)

This is a step beyond the nuclear option shown above. If you have no running docker containers, use this command when you either want to completely wipe out any old docker images on your local system, or if Docker is playing up and not deploying containers correctly. From experience, I find this one being called into practise a lot more that should otherwise be expected as Docker seems to sometimes corrupt images and it's a quick way to fix the issue.

The last two commands I'm going to cover will be covered together as they're useful together. These will allow you to create an image and then run a container from that image without having to use docker-compose.

docker build . -t alexa-test:latestdocker run --name alexa-test -itd alexa-test:latest

The first command manually turns a local Dockerfile into an image tagged as alexa-test:latest. We can then choose to run that manually as well using the above notation. There's only a few settings shown above as that's merely a fraction of all of the options available.

If you want to see all of the possible options, try the manual, although there are a lot of them in there.

That's all I'm going to cover so far as Docker's a large subject but I hope this is enough to get someone off the ground with Docker. It really is a very good tool, but it's just a bit complex to dive into immediately.