Docker layers

Useful links

No time for reading the whole post ? Directly go to the sum up section

Layers, Image and Container

When you build a new docker image (using a Dockerfile), you add a series of layers above a preexisting image (using the FROM keyword).

Building an image

To build a new docker image you use a Dockerfile which contains a series of instruction. Consider the Dockerfile below:

FROM alpine:latest
LABEL name="my-image"
RUN apk add --no-cache bash
WORKDIR /app
COPY my-file ./
CMD bash

These instructions create one layer:

FROM creates a layer from the alpine:latest Docker image.
COPY adds files from your Docker client’s current directory.
RUN execute the command in argument.
CMD specifies what command to run within the container.

These instructions DO NOT create a layer:

LABEL: Only modify metadatas of the image.
WORKDIR: Add metadata to the image configuration

Let's build and inspect the image to see what's happening:

docker build -t test:layer .

The output should be:

Sending build context to Docker daemon  3.072kB
Step 1/7 : FROM alpine:latest
 ---> c059bfaa849c
Step 2/7 : FROM alpine:latest
 ---> c059bfaa849c
Step 3/7 : LABEL name="my-image"
 ---> Running in e8c297101cc1
Removing intermediate container e8c297101cc1
 ---> 69bd1b374541
Step 4/7 : RUN apk add --no-cache bash
 ---> Running in e7da897fc16a
# Some apk logs
OK: 8 MiB in 18 packages
Removing intermediate container e7da897fc16a
 ---> 1aed100951f8
Step 5/7 : WORKDIR /app
 ---> Running in 1a16e1154f9a
Removing intermediate container 1a16e1154f9a
 ---> 2e134818c0d1
Step 6/7 : COPY my-file ./
 ---> fc04f457965d
Step 7/7 : CMD bash
 ---> Running in 922bf6938594
Removing intermediate container 922bf6938594
 ---> 0ec60ce9656c
Successfully built 0ec60ce9656c
Successfully tagged test:layer

Here we see the step by step construction of the docker image.

Note : Step does not mean that a layer is created !.

To see layers of this image use the following command :

docker image inspect --format "{{json .RootFS}}" test:layer

The output shows that only 4 layers are created, these layer correspond to the description above.

{
  "Type": "layers",
  "Layers": [
    "sha256:8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759",
    "sha256:1c1a058c984d8a97d6729d6b268dff0a898ec3cbeb04335f9f7892515cd55077",
    "sha256:1ceda311321ce6884cd40c1b8f3183777d43ab102bb6b7ca0ae32f8180d699a4",
    "sha256:8db629f13a587580ba5d897b2865836a2a9788b80e62ff408e9dc5bf8f366d61"
  ]
}

N.B: you might not have a pretty json output. In this case just pipe the command with jq

docker image inspect --format "{{json .RootFS}}" test:layer | jq`

To have more details of the docker image step size, use:

docker image history test:layer

Output:

IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
0ec60ce9656c   24 minutes ago   /bin/sh -c #(nop)  CMD ["/bin/sh" "-c" "bash…   0B        
fc04f457965d   24 minutes ago   /bin/sh -c #(nop) COPY file:7f83a9d95802131a…   5B        
2e134818c0d1   24 minutes ago   /bin/sh -c #(nop) WORKDIR /app                  0B        
1aed100951f8   24 minutes ago   /bin/sh -c apk add --no-cache bash              2.15MB    
69bd1b374541   24 minutes ago   /bin/sh -c #(nop)  LABEL name=my-image          0B        
c059bfaa849c   6 weeks ago      /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B        
<missing>      6 weeks ago      /bin/sh -c #(nop) ADD file:9233f6f2237d79659…   5.59MB

What about the image's size ?

As we have seen before an image is composed of stacked layers. But what if we have multiple image sharing a part of their layers ?

let's figure it out with a alpine image.

Pull the alpine image
```
docker pull alpine:latest
```
Use docker system df -v to get details on file system
```
docker system df -v
```
Output:

Images space usage:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
alpine              latest              c059bfaa849c        6 weeks ago         5.586 MB            0 B                 5.586 MB            0

Containers space usage:

CONTAINER ID        IMAGE               COMMAND             LOCAL VOLUMES       SIZE                CREATED             STATUS              NAMES

Local Volumes space usage:

VOLUME NAME         LINKS               SIZE

About size we can see three column:

UNIQUE SIZE: The real size of the image
SHARED SIZE: The share volume due to shared layer (Thanks to the copy on write strategy)
SIZE: Sum of UNIQUE SIZE and SHARED SIZE

Let' build two new image from the alpine:latest.

First with bash installed
Second with vim installed

First:

echo "FROM alpine:latest
RUN apk add bash" >> Dockerfile.alpine.bash

Second:

echo "FROM alpine:latest
RUN apk add vim" >> Dockerfile.alpine.vim

Build both images:

docker build -t test:alpine-bash -f Dockerfile.alpine bash .
docker build -t test:alpine-vim -f Dockerfile.alpine.vim .

Redo the docker system df -v and see what happened:

docker system df -v

Output:

Images space usage:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
test                alpine-vim          f0250a4255df        2 minutes ago       33.72 MB            5.586 MB            28.14 MB            0
test                alpine-bash         1f73220af2ef        5 minutes ago       7.883 MB            5.586 MB            2.297 MB            0
alpine              latest              c059bfaa849c        6 weeks ago         5.586 MB            5.586 MB            0 B                 0

Containers space usage:

CONTAINER ID        IMAGE               COMMAND             LOCAL VOLUMES       SIZE                CREATED             STATUS              NAMES

Local Volumes space usage:

VOLUME NAME         LINKS               SIZE

We can see that both new images are increasing the host file system size just by the modifications we did by installing package (bash in one hand and vim in another.

This is a good and simple example to see how docker manages layer from a docker image point of view.

What about the container's size ?

By now we saw that a docker image is composed of a stack of layers directly linked to the Dockerfile's reference.

The following paragraph is a extract from the official docker's documentation and perfectly explain how a container is different from an image:

The layers are stacked on top of each other. When you create a new container, you add a new writable layer on top of the underlying layers. This layer is often called the “container layer”. All changes made to the running container, such as writing new files, modifying existing files, and deleting files, are written to this thin writable container layer. The diagram below shows a container based on an ubuntu:15.04 image.

We can sum up this as: An image is composed of a stack of read only layers. A container inherits those layers PLUS a thin read-write layer that allows the user to edit files.

About the container size

Containers are based on the copy-on-write strategy implying multiple containers started from the same exact image shared the same R/O layers.

Get container's size information from the CLI

Let's start with creating a new container from an alpine:latest image

docker run -tid --name my-container alpine:latest sh

To see the container size use the following command:

docker ps --size

You can filter the ouput using this command (-s stand for --size):

docker ps -s --format "table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Size}}"

output:

CONTAINER ID   IMAGE           NAMES          SIZE
fe4d3baa8afc   alpine:latest   my-container   0B (virtual 5.59MB)

What are these size informations ?

SIZE which is 0B stands for the R/W layer (e.g the "container layer")
virtual is the sum of the R/O layers and the R/W layer

Let's increase by updating the apk package manager:

docker exec -it my-container apk update

And now let's see what happened on the container's size

> $ docker ps -s --format "table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Size}}"
CONTAINER ID   IMAGE           NAMES          SIZE
fe4d3baa8afc   alpine:latest   my-container   2.3MB (virtual 7.88MB)

We can see that both size have increased and that we can get the initial image with virtual - SIZE

This extract of the official documentation tells us more about how estimate the size comsumption:

The total disk space used by all of the running containers on disk is some combination of each container’s size and the virtual size values. If multiple containers started from the same exact image, the total size on disk for these containers would be SUM (size of containers) plus one image size (virtual size- size).

About Heavy-write application

An heavy-write application such as database or just a script that produced a lot of file should NOT store its datas in the thin R/W container layer.

It is better to use docker volume to store app datas in.

Sum up

These docker instructions add a layer:
- FROM
- COPY
- RUN
- CMD
An image is composed of a stack of read only layers
A container is compose of a stack of read only layers (inherited from the image) PLUS a thin read-write layer called "container layer"

Useful command

Build an image: docker build -t my-image:my-tag . (in a directory which contains the Dockerfile)
Show the layers: docker image inspect --format "{{json .RootFS}}" my-image:my-tag
Show each layer size: docker image history my-image:my-tag
Show container size: docker ps -s
For wrtie-heavy application (such as databases) you better use docker volumes

Mickael's place