volumeMounts[0].name: Not found in k8s Even Though the volume Should Exist

It turns out there are cases where duplicate volume names can be defined

Posted on Fri, Nov 19, 2021
Tags kubernetes
📝 This article is a translation of the original Japanese post. View original

Falling Into an Unexpected Trap with ConfigMap volumeMounts

  • Version tested: kubernetes 1.20.11

You sometimes use a ConfigMap via volumeMounts in a Deployment and the like, as below.

yaml-1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    spec:
      containers:
      - image: httpd
        name: httpd
        volumeMounts:
        - name: configmap-volume
          mountPath: /var/tmp/configmap
      volumes:
      - name: configmap-volume
        configMap:
          name: configmap-sample-data

Regarding the volume above, normally you cannot apply duplicates with the same name.

yaml-2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

    spec:
      containers:
      - image: httpd
        name: httpd
        volumeMounts:
        - name: configmap-volume
          mountPath: /var/tmp/configmap
      volumes:
      - name: configmap-volume
        configMap:
          name: configmap-sample-data
      - name: configmap-volume
        configMap:
          name: configmap-sample-data

1

The Deployment "test" is invalid: spec.template.spec.volumes[1].name: Duplicate value: "configmap-volume"

However, if you apply yaml-1 and then apply yaml-2, the volume with the duplicate name gets applied (!!!)

After applying, two configmap-volume entries have been generated in last-applied-configuration.

Then applying yaml-1 produces a Not found error.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# this part is normal
$ kubectl apply -f ~/Desktop/yaml-1.yml
deployment.apps/test created

# this one gets accepted
$ kubectl apply -f ~/Desktop/yaml-2.yml
deployment.apps/test configured

# this becomes Not found
$ kubectl apply -f ~/Desktop/yaml-1.yml
The Deployment "test" is invalid:
* spec.template.spec.containers[0].volumeMounts[0].name: Not found: "configmap-volume"

I skimmed the upstream Kubernetes issues, but it’s unclear whether this is by design or a bug (it depends on how volumes, being defined as an array, handles additions after apply).

As an investigation technique, after the first apply, run kubectl get deployment test -oyaml and expand the last-applied-configuration json inside the deployment to check it.

As is often said, Kubernetes is a declarative system, but the master copy of the declarative configuration lives in etcd, and it can be viewed via last-applied-configuration.

This is a case you might run into while experimenting with kustomize or helm, and I got well and truly stuck in it.

Verifying on minikube

You can reproduce it as follows.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198

$ minikube kubectl -- apply -f <(cat <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: test
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-sample-data
  namespace: test
data:
  test: "test"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: configmap-volume
          mountPath: /var/tmp/configmap
      volumes:
      - name: configmap-volume
        configMap:
          name: configmap-sample-data  
EOF
)
namespace/test created
configmap/configmap-sample-data created
deployment.apps/nginx-deployment created


$ minikube kubectl -- apply -f <(cat <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: configmap-volume
          mountPath: /var/tmp/configmap
      volumes:
      - name: configmap-volume
        configMap:
          name: configmap-sample-data 
      - name: configmap-volume # this one actually gets applied
        configMap:
          name: configmap-sample-data
EOF
)
deployment.apps/nginx-deployment configured

$ minikube kubectl -- -ntest get deploy nginx-deployment -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx-deployment","namespace":"test"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"nginx"}},"template":{"metadata":{"labels":{"app":"nginx"}},"spec":{"containers":[{"image":"nginx:latest","name":"nginx","ports":[{"containerPort":80}],"volumeMounts":[{"mountPath":"/var/tmp/configmap","name":"configmap-volume"}]}],"volumes":[{"configMap":{"name":"configmap-sample-data"},"name":"configmap-volume"},{"configMap":{"name":"configmap-sample-data"},"name":"configmap-volume"}]}}}}
  creationTimestamp: "2021-11-20T14:27:43Z"
  generation: 2
  labels:
    app: nginx
...

# let us look at last-applied-configuration
{
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "metadata": {
        "annotations": {},
        "labels": {
            "app": "nginx"
        },
        "name": "nginx-deployment",
        "namespace": "test"
    },
    "spec": {
        "replicas": 1,
        "selector": {
            "matchLabels": {
                "app": "nginx"
            }
        },
        "template": {
            "metadata": {
                "labels": {
                    "app": "nginx"
                }
            },
            "spec": {
                "containers": [
                    {
                        "image": "nginx:latest",
                        "name": "nginx",
                        "ports": [
                            {
                                "containerPort": 80
                            }
                        ],
                        "volumeMounts": [
                            {
                                "mountPath": "/var/tmp/configmap",
                                "name": "configmap-volume"
                            }
                        ]
                    }
                ],
                "volumes": [
                    {
                        "configMap": {
                            "name": "configmap-sample-data"
                        },
                        "name": "configmap-volume"     # <-- two entries with the same name can exist...
                    },
                    {
                        "configMap": {
                            "name": "configmap-sample-data"
                        },
                        "name": "configmap-volume"     # <-- two entries with the same name can exist..
                    }
                ]
            }
        }
    }
}


# this one fails
$ minikube kubectl -- apply -f <(cat <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: configmap-volume
          mountPath: /var/tmp/configmap
      volumes:
      - name: configmap-volume
        configMap:
          name: configmap-sample-data
EOF
)
The Deployment "nginx-deployment" is invalid: spec.template.spec.containers[0].volumeMounts[0].name: Not found: "configmap-volume"

Share

Edit on GitHub