Bad Gateway 502 when upload many files

[ishugaley]

Hi! We have allure server in K8s with following specs:
spec: replicas: 1 selector: matchLabels: app.kubernetes.io/instance: allure app.kubernetes.io/name: allure-server template: metadata: creationTimestamp: null labels: app.kubernetes.io/instance: allure app.kubernetes.io/managed-by: spinnaker app.kubernetes.io/name: allure-server annotations: artifact.spinnaker.io/location: allure artifact.spinnaker.io/name: allure-server-deployment artifact.spinnaker.io/type: kubernetes/deployment artifact.spinnaker.io/version: '' kubectl.kubernetes.io/restartedAt: '2023-04-19T10:19:34Z' spec: volumes: - name: storage persistentVolumeClaim: claimName: pvc-claim-allure containers: - name: allure-server image: kochetkovma/allure-server:2.11.2 ports: - name: http containerPort: 8080 protocol: TCP env: - name: ALLURE_BLOBSTORAGE_S3_BUCKET value: allure - name: API_RESPONSE_LESS_VERBOSE value: "1" - name: ALLURE_BLOBSTORAGE_S3_ENDPOINT value: s3.amazonaws.com - name: ALLURE_BLOBSTORAGE_S3_REGION value: us-west-2 - name: ALLURE_BLOBSTORAGE_TYPE value: S3 - name: SPRING_SERVLET_MULTIPART_MAXFILESIZE value: 500MB - name: SPRING_SERVLET_MULTIPART_MAXREQUESTSIZE value: 500MB - name: SPRING_DATASOURCE_URL value: >- jdbc:postgresql://store-dev1-postgres/allure?&targetServerType=master&useSSL=false - name: SPRING_JPA_DATABASE value: postgresql - name: SPRING_JPA_HIBERNATE_DDL-AUTO value: update - name: SPRING_PROFILES_ACTIVE value: default - name: SPRING_DATASOURCE_USERNAME valueFrom: secretKeyRef: - name: SPRING_DATASOURCE_PASSWORD valueFrom: secretKeyRef: resources: limits: cpu: '2' memory: 3Gi requests: cpu: 250m memory: 348Mi volumeMounts: - name: storage mountPath: /allure livenessProbe: httpGet: path: /api/result port: http scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 1 periodSeconds: 40 successThreshold: 1 failureThreshold: 3 readinessProbe: httpGet: path: /api/result port: http scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 1 periodSeconds: 40 successThreshold: 1 failureThreshold: 3 startupProbe: httpGet: path: /api/result port: http scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 1 periodSeconds: 25 successThreshold: 1 failureThreshold: 3 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: IfNotPresent securityContext: {} restartPolicy: Always terminationGracePeriodSeconds: 30 dnsPolicy: ClusterFirst serviceAccountName: allure serviceAccount: allure securityContext: {} schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 25% maxSurge: 25% revisionHistoryLimit: 10 progressDeadlineSeconds: 600
During test run generates about 1500 json file
CPU and Memmory limit are not reached
POD turns yellow and we got HTTP 502 error.
No error messages in pod logs, pod just restarts.

UPD
sometimes we see following error
11:55:37.314 [http-nio-8080-exec-9] INFO ru.iopump.qa.allure.service.ResultService [130] - Unzip new entry 'allure/results/fa7f898d-82fe-4f94-bd54-26b53e0be6d8_tmp/allure-results/ae4605fc0fb842a090ecd596cb21aa1f-container.json' 11:55:37.314 [http-nio-8080-exec-9] INFO ru.iopump.qa.allure.service.ResultService [136] - Unzipping successfully finished to 'allure/results/fa7f898d-82fe-4f94-bd54-26b53e0be6d8_tmp' 11:55:37.473 [http-nio-8080-exec-1] ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] [175] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: File system element for parameter 'directory' does not exist: 'allure/results/c65433af-b448-4bf2-beb2-f08cc699cf39_tmp'] with root cause java.lang.IllegalArgumentException: File system element for parameter 'directory' does not exist: 'allure/results/c65433af-b448-4bf2-beb2-f08cc699cf39_tmp'

[kochetkov-ma]

looks like there is a problem with Persistent Volume settings. I don't know what exactly because i didn't use S3 as PV and cannot test this configuration...