Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fgt-tst-pl disk space exhausted #124

Open
joaoluis-pdm opened this issue Oct 19, 2022 · 7 comments
Open

fgt-tst-pl disk space exhausted #124

joaoluis-pdm opened this issue Oct 19, 2022 · 7 comments
Assignees
@joaoluis-pdm
Labels
wontfix This will not be worked on

Comments

@joaoluis-pdm
Copy link
Contributor

joaoluis-pdm commented Oct 19, 2022 •
edited
Loading

Seems to be the overlay2 docker folder:

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
92K    /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
59M    /var/lib/docker/image
128K    /var/lib/docker/network
9.8G    /var/lib/docker/overlay2
16K    /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
...

Does not show up on docker system df... :-(

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         0         7.232GB   7.232GB (100%)
Containers      0         0         0B        0B
Local Volumes   16        0         11.16GB   11.16GB (100%)
Build Cache     0         0         0B        0B

$ docker system df -v
Images space usage:
REPOSITORY                         TAG              IMAGE ID       CREATED        SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
joaoluispdm/pharmaledger-imi-fgt   v0.10.7          40ec353cad55   4 weeks ago    2.412GB   110.6MB       2.301GB       0
api_mah-bayer                      latest           155fcb4d9f8a   5 weeks ago    2.408GB   138.2MB       2.269GB       0
@joaoluis-pdm
Copy link
Contributor Author

joaoluis-pdm commented Oct 19, 2022 •
edited
Loading

fgt-tst-pl migrated to SSD1, and / FS grown from 60GB to 120GB. Details in VMWare.docx

After reboot

# during a restart of all containers

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
92K /var/lib/docker/buildkit
2.3M    /var/lib/docker/containers
59M /var/lib/docker/image
128K    /var/lib/docker/network
26G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
15G /var/lib/docker/volumes

# one hour later

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
92K /var/lib/docker/buildkit
45M /var/lib/docker/containers
59M /var/lib/docker/image
128K    /var/lib/docker/network
40G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
15G /var/lib/docker/volumes

root@fgt-tst-pl:~# date
Wed Oct 19 09:06:03 UTC 2022

overlay2 has grown from 9.8GB to 26GB to 40GB... ???!!!

@joaoluis-pdm joaoluis-pdm self-assigned this Oct 19, 2022
@joaoluis-pdm joaoluis-pdm added the bug Something isn't working label Oct 19, 2022
@joaoluis-pdm
Copy link
Contributor Author

joaoluis-pdm commented Oct 19, 2022 •
edited
Loading

Some tips on overlay2 moby/moby#33775

# du -shc /var/lib/docker/overlay2/*/diff | grep total
12G	total

Not helpful. 12GB would be reasonable ... after all these months... But where are the 40GB ?

pharmaledger@fgt-tst-pl:~$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         3         7.232GB   4.78GB (66%)
Containers      17        17        1.706GB   0B (0%)
Local Volumes   16        15        11.32GB   352.3MB (3%)
Build Cache     0         0         0B        0B

Something smells bad over docker...

@joaoluis-pdm joaoluis-pdm added wontfix This will not be worked on and removed bug Something isn't working labels Oct 19, 2022
@joaoluis-pdm joaoluis-pdm mentioned this issue Oct 19, 2022
Open
@joaoluis-pdm
Copy link
Contributor Author

joaoluis-pdm commented Nov 10, 2022 •
edited
Loading

Detected again on 2022-11-10 10h.

Before any cleanup/restart.

pharmaledger@fgt-tst-pl:~$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G  113G     0 100% /

root@fgt-tst-pl:~# du -sh /var/*
1.9M	/var/backups
104M	/var/cache
4.0K	/var/crash
115G	/var/lib
4.0K	/var/local
0	/var/lock
9.4G	/var/log
4.0K	/var/mail
4.0K	/var/opt
0	/var/run
80K	/var/snap
28K	/var/spool
36K	/var/tmp
20K	/var/www

/var/lib 115GB ? Docker issues ?

root@fgt-tst-pl:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         3         7.232GB   4.78GB (66%)
Containers      17        17        1.706GB   0B (0%)
Local Volumes   16        15        33.14GB   352.3MB (1%)
Build Cache     0         0         0B        0B

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
92K	/var/lib/docker/buildkit
32G	/var/lib/docker/containers
59M	/var/lib/docker/image
128K	/var/lib/docker/network
40G	/var/lib/docker/overlay2
16K	/var/lib/docker/plugins
4.0K	/var/lib/docker/runtimes
4.0K	/var/lib/docker/swarm
4.0K	/var/lib/docker/tmp
4.0K	/var/lib/docker/trust
43G	/var/lib/docker/volumes

We seem to be using 40GB overlay2 + 43GB volumes...

Which ones ?

root@fgt-tst-pl:~# du -sh /var/lib/docker/overlay2/*
288M	/var/lib/docker/overlay2/02595d8cdacea04a2f593e64ab4d27793f3647024c2a5c56954afd14297cbbcc
128K	/var/lib/docker/overlay2/0698b709b7c38e30feecb9fe255ba5e08ac42821548c5670f8ce56e0a8556c77
1.4G	/var/lib/docker/overlay2/09648db2c009b1d77699864451bd6d562b0011dd2c2942ccda67b06210010eaa
2.0G	/var/lib/docker/overlay2/0b09896a383901da53cc5d15621f99509151653328114107b8743fb29b0dff8b
40K	/var/lib/docker/overlay2/0b09896a383901da53cc5d15621f99509151653328114107b8743fb29b0dff8b-init
2.0G	/var/lib/docker/overlay2/0bc88d7bf9f683072ef3c15d4f2370c0359c8ed48ef7eccd6deb7bf77e4e1ca5
40K	/var/lib/docker/overlay2/0bc88d7bf9f683072ef3c15d4f2370c0359c8ed48ef7eccd6deb7bf77e4e1ca5-init
2.0G	/var/lib/docker/overlay2/10b772916195991bb5ef2cb0b1fa051407e83b9a8a64188dd4444da1f22020a8
40K	/var/lib/docker/overlay2/10b772916195991bb5ef2cb0b1fa051407e83b9a8a64188dd4444da1f22020a8-init
17M	/var/lib/docker/overlay2/13e4001d5362c3254ac1c47432c4ca30af08017b5789aa97325068beb4c49500
103M	/var/lib/docker/overlay2/140aec3bf357b0fe38d5296b2226ab1c30cb75b4ed717212b2800c87631d87d5
6.0M	/var/lib/docker/overlay2/1aaf1364256cad9c49550dd78466069a47ed260dc71c2cbf02b46a78f34e5751
2.0G	/var/lib/docker/overlay2/2367181620da0d3e15fd164e32604b309cfc821074f8407767edf59f6beff087
40K	/var/lib/docker/overlay2/2367181620da0d3e15fd164e32604b309cfc821074f8407767edf59f6beff087-init
2.0G	/var/lib/docker/overlay2/2c00280a13676389f03f6bd296e44a218625bd0b64b06fab72607ab605988832
40K	/var/lib/docker/overlay2/2c00280a13676389f03f6bd296e44a218625bd0b64b06fab72607ab605988832-init
2.0G	/var/lib/docker/overlay2/2f41649815d60852794e2ab8c7765fb8f60138f4d9d733094a46f1e2ecd8f5af
40K	/var/lib/docker/overlay2/2f41649815d60852794e2ab8c7765fb8f60138f4d9d733094a46f1e2ecd8f5af-init
1.6G	/var/lib/docker/overlay2/3121128b8a2f6eecbc8146554cfacc43cbdbdae29870909a86af6bd08e3b3b47
2.0G	/var/lib/docker/overlay2/39ff7d2547c7becc02a7e6f1340677b1fd96c6f88cae86bbcf38f44a8399d2e7
40K	/var/lib/docker/overlay2/39ff7d2547c7becc02a7e6f1340677b1fd96c6f88cae86bbcf38f44a8399d2e7-init
56M	/var/lib/docker/overlay2/41dfd4c33f8658beced6b0b384a0be5cd211afa1fea701a0ca742e9738b6458d
40K	/var/lib/docker/overlay2/41dfd4c33f8658beced6b0b384a0be5cd211afa1fea701a0ca742e9738b6458d-init
18M	/var/lib/docker/overlay2/42e6924cf51e722fff586c774024b3d1bb4d7597d5cfdd2a2d077215975c5873
2.0G	/var/lib/docker/overlay2/4526db663a9ae7f48da41858e6194e9aa6eee2db1980fe30286e3bb744135b3d
40K	/var/lib/docker/overlay2/4526db663a9ae7f48da41858e6194e9aa6eee2db1980fe30286e3bb744135b3d-init
18M	/var/lib/docker/overlay2/4eabf59e3e5682697b1895092c084cbfeb2c3f37dd83d16c6594af40accce6aa
44K	/var/lib/docker/overlay2/531e76eefc35f0ac5d37223a952f4fccaf0eb319192c58183c215ed60abe586a
128K	/var/lib/docker/overlay2/55da8263a951fb24d88f89ad1c8b03923945ffb3c0b479df5e1b02bf263b1b2f
36K	/var/lib/docker/overlay2/5ced429f99f7cae503b6cb4563c41dcf112705c37e8e75074143b826689dcc11
2.0G	/var/lib/docker/overlay2/63ac33f69a453c23f86a1260e19cec218ab816899051ff07a1ec938ab5fbb41a
40K	/var/lib/docker/overlay2/63ac33f69a453c23f86a1260e19cec218ab816899051ff07a1ec938ab5fbb41a-init
101M	/var/lib/docker/overlay2/6686ebc9872e2aa7be00a20ae0bec2b7efd4be9c9fd5f930fb0b75087fa98926
40K	/var/lib/docker/overlay2/6686ebc9872e2aa7be00a20ae0bec2b7efd4be9c9fd5f930fb0b75087fa98926-init
24K	/var/lib/docker/overlay2/69a393cfabb5fcab6ee49ffbc8d73ea20e7c680880363035e74c408b4316c5d7
50M	/var/lib/docker/overlay2/6be604d0a97020b790608d73292b2a02199700b5880eea460fa73b08fa4bf68a
1.5G	/var/lib/docker/overlay2/6e801866ddd860ff3e0cbf80ca8981157474c1507ea821fb8ff0b1907e3b13c6
239M	/var/lib/docker/overlay2/6eaa5c0a3e63efc854cd035d2001335e26e2bca7b2275685f5bf3335a5621d4f
4.7M	/var/lib/docker/overlay2/6eeb8f30c0b90712fdc4fdaeafb85b6d38a9620711cd047820604999365e0af6
24K	/var/lib/docker/overlay2/722038ccd1a834384e3bd2f3dff6eed49720bc77c1ba0eee6252075eef2553bd
180K	/var/lib/docker/overlay2/732f00ebe13b66291bff15347812251dce5f0da0db30e0b9d5e69cfd3871c5b7
6.0M	/var/lib/docker/overlay2/73e7447d064f8b518a4d0a9216c0c7adb75750ef0db2fa66cd7ccf5846b73fed
288M	/var/lib/docker/overlay2/8067e32cfbea99ec527a817f2e4a301f0ffa825b8ab0f200a5f7f99f1f1419a7
2.0G	/var/lib/docker/overlay2/85142286c283cff68f3867bf2efe22084f160f9ce509d9ff8e8a6f6eae1b87d2
40K	/var/lib/docker/overlay2/85142286c283cff68f3867bf2efe22084f160f9ce509d9ff8e8a6f6eae1b87d2-init
180K	/var/lib/docker/overlay2/87da5674f4e6eadb442d514874a177509708e6e4ec2b272096f6c0d1945afb21
91M	/var/lib/docker/overlay2/8fb263ae881882ea60dcdc4304d17bbe94b388b05a5aa29d6e086aa5bdab7192
24K	/var/lib/docker/overlay2/a4db6f71dfbba702bccc74664be78067f233adf6005b38305781686dccee5dc5
1.4G	/var/lib/docker/overlay2/a5c3a45eece2b0e23c67e5cda7d233ba926b9dd62fb78f1a2af08afa1bbd1779
1.5G	/var/lib/docker/overlay2/b15de220baab18e7abe375a0851e4e90ede64f432a979847e1652cab486ef24e
2.0G	/var/lib/docker/overlay2/b168e043ea668b15f4dd68104e95a7bf793c96f1c23f9dea54abb055e3a10c60
40K	/var/lib/docker/overlay2/b168e043ea668b15f4dd68104e95a7bf793c96f1c23f9dea54abb055e3a10c60-init
128K	/var/lib/docker/overlay2/b17cddb97b0979f7ec1a67420c425add340ec1ce5299e3c1470d0038b143dce6
1.5G	/var/lib/docker/overlay2/bb3e48a5e82e77a11434aa8e13a680bf82c91b93c932695a855ef3bdd49c2b8b
28K	/var/lib/docker/overlay2/be0d646daedda0c05a6f48a21f82758a012ded2ec525a2d3cbe34865e0d8aef3
2.0G	/var/lib/docker/overlay2/c22f1bde90bca49ebcda988890b220071855eee8b9423dfe5ee12851398fc88b
40K	/var/lib/docker/overlay2/c22f1bde90bca49ebcda988890b220071855eee8b9423dfe5ee12851398fc88b-init
1.9G	/var/lib/docker/overlay2/c35807d874ea3f07e21b40152e48b73a9fdbabb77ad7c23873c9bb4c0463e5e4
40K	/var/lib/docker/overlay2/c35807d874ea3f07e21b40152e48b73a9fdbabb77ad7c23873c9bb4c0463e5e4-init
17M	/var/lib/docker/overlay2/c8698b13ab76cf5d4b62f51d7a3d7f82b8a77aaeb2bbbfa2e8479d34ba74d154
20K	/var/lib/docker/overlay2/d6873530258222f435862d73466a69d97a15a33658e52e1597d10712c4daa6bd
2.0G	/var/lib/docker/overlay2/d97546d294f59fbff03af4aeca122c9c297a4c6ca5e73b1db815c73818ae5a0d
40K	/var/lib/docker/overlay2/d97546d294f59fbff03af4aeca122c9c297a4c6ca5e73b1db815c73818ae5a0d-init
24K	/var/lib/docker/overlay2/d98648f0cbef0adc270cd21463c1e4ccf80727c60ebdff16e2812e7fd94180fa
2.0G	/var/lib/docker/overlay2/e07d0cb4d7ebf5f9833b6c421c83ab9f617e1aa0e480d39872d2ba07e44edc27
40K	/var/lib/docker/overlay2/e07d0cb4d7ebf5f9833b6c421c83ab9f617e1aa0e480d39872d2ba07e44edc27-init
7.6M	/var/lib/docker/overlay2/e0bbad577616111cfcfe21e297c9c51f07834b8bc4605aa57ef2dd174b40b48d
24K	/var/lib/docker/overlay2/e3451e73a9ba2a7465775858a9006d357d6d9e47b01715fd41d675f09cb859ef
36K	/var/lib/docker/overlay2/e541f2e7fe54cecdea1d036b8e287ff1858bc6912c3c1403aad677b3cfcb72a8
24K	/var/lib/docker/overlay2/eb3f43cb7979a9d2675a390b538ee73edca2c5f2eae23aa8e832e2985c084e7e
180K	/var/lib/docker/overlay2/fc4cc08d20fc72f6a0351403aba376fc734c2f849d8d53d316ccb7404e7456b5
28K	/var/lib/docker/overlay2/fe760c33ce6ea5dcd4387487bd0774876cb6664929b3d5dc3644df2ab063cf7c
292K	/var/lib/docker/overlay2/l

root@fgt-tst-pl:~# du -sh /var/lib/docker/volumes/*
58M	/var/lib/docker/volumes/api_mah-bayer_vol
58M	/var/lib/docker/volumes/api_mah-gsk_vol
59M	/var/lib/docker/volumes/api_mah-msd_vol
58M	/var/lib/docker/volumes/api_mah-novo-nordisk_vol
58M	/var/lib/docker/volumes/api_mah-pfizer_vol
58M	/var/lib/docker/volumes/api_mah-roche_vol
58M	/var/lib/docker/volumes/api_mah-sanofi_vol
58M	/var/lib/docker/volumes/api_mah-takeda_vol
58M	/var/lib/docker/volumes/api_pha1_vol
59M	/var/lib/docker/volumes/api_pha2_vol
42G	/var/lib/docker/volumes/api_traceability_vol
58M	/var/lib/docker/volumes/api_whs-takeda_vol
129M	/var/lib/docker/volumes/api_whs-zuellig_vol
58M	/var/lib/docker/volumes/api_whs1_vol
58M	/var/lib/docker/volumes/api_whs2_vol
0	/var/lib/docker/volumes/backingFsBlockDev
409M	/var/lib/docker/volumes/external-volume
32K	/var/lib/docker/volumes/metadata.db

Seems to be the traceability domain that is growing...

root@fgt-tst-pl:~# du -sh  /var/lib/docker/volumes/api_traceability_vol/_data/*
4.0K	/var/lib/docker/volumes/api_traceability_vol/_data/channels
32K	/var/lib/docker/volumes/api_traceability_vol/_data/config
42G	/var/lib/docker/volumes/api_traceability_vol/_data/domains
8.0K	/var/lib/docker/volumes/api_traceability_vol/_data/messaging
4.0K	/var/lib/docker/volumes/api_traceability_vol/_data/mq_JWT_Auth_Seeder
88K	/var/lib/docker/volumes/api_traceability_vol/_data/mqs
4.0K	/var/lib/docker/volumes/api_traceability_vol/_data/notifications
4.0K	/var/lib/docker/volumes/api_traceability_vol/_data/read.me

root@fgt-tst-pl:~# du -sh  /var/lib/docker/volumes/api_traceability_vol/_data/domains/*
28M	/var/lib/docker/volumes/api_traceability_vol/_data/domains/default
42G	/var/lib/docker/volumes/api_traceability_vol/_data/domains/traceability
33M	/var/lib/docker/volumes/api_traceability_vol/_data/domains/vault

root@fgt-tst-pl:~# du -sh  /var/lib/docker/volumes/api_traceability_vol/_data/domains/traceability/*
501M	/var/lib/docker/volumes/api_traceability_vol/_data/domains/traceability/anchors
41G	/var/lib/docker/volumes/api_traceability_vol/_data/domains/traceability/brick-storage

Get some free space.

pharmaledger@fgt-tst-pl:~$ ls -lh images/
total 5.1G
-rw-rw-r-- 1 pharmaledger pharmaledger 5.0G Oct 19 09:15 delete-when-space-needed.empty

@joaoluis-pdm joaoluis-pdm reopened this Nov 10, 2022
@joaoluis-pdm
Copy link
Contributor Author

joaoluis-pdm commented Nov 10, 2022 •
edited
Loading

Logs have been backed to the private sharepoint.

Stopping appers to free 40GB

pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ docker-compose down
pharmaledger@fgt-tst-pl:~/logs$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   70G   43G  63% /
pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ ./bootCompose.sh tst

@joaoluis-pdm
Copy link
Contributor Author

joaoluis-pdm commented Nov 14, 2022 •
edited
Loading

OS upgraded, volumes and images cleared.

pharmaledger@fgt-tst-pl:~$ docker system prune -a
pharmaledger@fgt-tst-pl:~$ docker volume prune

pharmaledger@fgt-tst-pl:~$ dd if=/dev/zero of=~/images/delete5G.when.needed bs=1G count=5
pharmaledger@fgt-tst-pl:~$ dd if=/dev/zero of=~/images/delete5Gb.when.needed bs=1G count=5

Docker re-installed as instructed on https://docs.docker.com/engine/install/ubuntu/#uninstall-docker-engine

@joaoluis-pdm joaoluis-pdm reopened this Sep 27, 2023
@joaoluis-pdm
Copy link
Contributor Author

joaoluis-pdm commented Sep 27, 2023 •
edited
Loading 

pharmaledger@fgt-tst-pl:~$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   87G   26G  77% /
... apt update/upgrade/autoremove 
root@fgt-tst-pl:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               7.8G     0  7.8G   0% /dev
tmpfs                              1.6G  1.7M  1.6G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   87G   26G  77% /
tmpfs                              7.9G     0  7.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2                          974M  209M  698M  24% /boot
/dev/loop5                          56M   56M     0 100% /snap/core18/2785
/dev/loop2                         174M  174M     0 100% /snap/lxd/25112
/dev/loop6                          64M   64M     0 100% /snap/core20/1974
/dev/loop10                         74M   74M     0 100% /snap/core22/858
/dev/loop1                          64M   64M     0 100% /snap/core20/2015
/dev/loop4                          56M   56M     0 100% /snap/core18/2790
/dev/loop8                          41M   41M     0 100% /snap/snapd/19993
/dev/loop3                          74M   74M     0 100% /snap/core22/864
/dev/loop9                         176M  176M     0 100% /snap/lxd/25505
/dev/loop0                          41M   41M     0 100% /snap/snapd/20092
tmpfs                              1.6G     0  1.6G   0% /run/user/1001
overlay                            117G   87G   26G  77% /var/lib/docker/overlay2/a6313b137cfdfa7ce99bd375f27603dd96f1e4542bc9f8b295bf502cef9f020e/merged
overlay                            117G   87G   26G  77% /var/lib/docker/overlay2/86c11e3484d6800fa00c95d81d9c2109b6db1e7514c0b738c1705085731457ce/merged
overlay                            117G   87G   26G  77% /var/lib/docker/overlay2/df683b97e20eea63a255acf31c7e28bae4c7b449469bf521d02b40f9304a89fb/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/f4785884c666706104df5f8140f30fdaf4df1c345bdfdff1887345e966ba8491/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/f0421174fee0b3192b66258fb6a37cf336ff569c61eddf432529721e7e63bea1/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/d5e5a18b7908aae547232c9b14a11bd0c08c5cab8ecf5f57c8ac0de362dff722/mounts/shm
root@fgt-tst-pl:~# reboot

pharmaledger@fgt-tst-pl:~$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   87G   26G  77% /

pharmaledger@fgt-tst-pl:~$ docker logs -f --since=1h fgt-workspace 
error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

Apparently I am hitting docker/for-linux#140

The root@fgt-tst-pl:~# grep -P '\x00' /var/lib/docker/containers/**/*.log gives no results.

Workaround: Using --tail on commands such as

docker log -f --tail 1000 fgt-workspace

Logs are taking some disk space, but should be less than 6GB.

root@fgt-tst-pl:~# ls -lh /var/lib/docker/containers/**/*.log
-rw-r----- 1 root root  20M Sep 27 07:10 /var/lib/docker/containers/090b06f15dcd0be58fbb05912e1a60e19d9fc16bf9b0ea275d15596b327476c0/090b06f15dcd0be58fbb05912e1a60e19d9fc16bf9b0ea275d15596b327476c0-json.log
-rw-r----- 1 root root 396M Sep 27 09:15 /var/lib/docker/containers/226bf4318049e64615ae03622c51509b7a4ea4a67686e216a7b0ddaf899681fc/226bf4318049e64615ae03622c51509b7a4ea4a67686e216a7b0ddaf899681fc-json.log
-rw-r----- 1 root root  20M Sep 27 07:17 /var/lib/docker/containers/2cb0d8f951408b5ea5b942c07000a51071d5554ce90995de47f260d74392b80a/2cb0d8f951408b5ea5b942c07000a51071d5554ce90995de47f260d74392b80a-json.log
-rw-r----- 1 root root 509M Sep 27 09:15 /var/lib/docker/containers/596d2c2565d07388037ecbe91a1062a2d24a4165edfa177d3c4882b2bcd4fbfe/596d2c2565d07388037ecbe91a1062a2d24a4165edfa177d3c4882b2bcd4fbfe-json.log
-rw-r----- 1 root root  20M Sep 27 07:07 /var/lib/docker/containers/738f9547fd4149bcba4719c32c99def8f5b675293bb240d565f231a4202d6fd9/738f9547fd4149bcba4719c32c99def8f5b675293bb240d565f231a4202d6fd9-json.log
-rw-r----- 1 root root  19M Sep 27 06:55 /var/lib/docker/containers/843d2c8bd41716ecbfad407ee949654066980f8a97934bfdd1df5a23b65bbfc6/843d2c8bd41716ecbfad407ee949654066980f8a97934bfdd1df5a23b65bbfc6-json.log
-rw-r----- 1 root root  35M Sep 27 07:00 /var/lib/docker/containers/868627bbf8a26ab2dabf8dcf74aefdc7c24f004ef2116732789610bcef555ace/868627bbf8a26ab2dabf8dcf74aefdc7c24f004ef2116732789610bcef555ace-json.log
-rw-r----- 1 root root  20M Sep 27 07:10 /var/lib/docker/containers/9455722e74eae91299653cadf62e59cda6145c6d8c52290c5ba5227bcbd36cc3/9455722e74eae91299653cadf62e59cda6145c6d8c52290c5ba5227bcbd36cc3-json.log
-rw-r----- 1 root root  20M Sep 27 07:22 /var/lib/docker/containers/989c4ec69a17a9af11316c46d925eceb5396a5d48923521bc71824e58873b6fb/989c4ec69a17a9af11316c46d925eceb5396a5d48923521bc71824e58873b6fb-json.log
-rw-r----- 1 root root  21M Sep 27 07:00 /var/lib/docker/containers/99e64cead7c65d40eb742a88d1c3a9f9af9730068486fe5e0bb0ff3b7cf48302/99e64cead7c65d40eb742a88d1c3a9f9af9730068486fe5e0bb0ff3b7cf48302-json.log
-rw-r----- 1 root root  32M Sep 27 06:42 /var/lib/docker/containers/b5e9b2d8972a1729f98db4ca64174f3b115e2c502b64730606b1a8f0c1c08466/b5e9b2d8972a1729f98db4ca64174f3b115e2c502b64730606b1a8f0c1c08466-json.log
-rw-r----- 1 root root  20M Sep 27 09:16 /var/lib/docker/containers/c999718ebcf935e46e695eb9a9b37bb9ced668676f975613ee6bbdcbe18663f9/c999718ebcf935e46e695eb9a9b37bb9ced668676f975613ee6bbdcbe18663f9-json.log
-rw-r----- 1 root root  760 Sep 27 09:30 /var/lib/docker/containers/d5e5a18b7908aae547232c9b14a11bd0c08c5cab8ecf5f57c8ac0de362dff722/d5e5a18b7908aae547232c9b14a11bd0c08c5cab8ecf5f57c8ac0de362dff722-json.log
-rw-r----- 1 root root  41M Sep 27 07:22 /var/lib/docker/containers/eb0bcd559f11d03821fa176a9782cdb255b8ec856e310f943ed59dbaae84649e/eb0bcd559f11d03821fa176a9782cdb255b8ec856e310f943ed59dbaae84649e-json.log
-rw-r----- 1 root root 4.0G Sep 27 09:53 /var/lib/docker/containers/f0421174fee0b3192b66258fb6a37cf336ff569c61eddf432529721e7e63bea1/f0421174fee0b3192b66258fb6a37cf336ff569c61eddf432529721e7e63bea1-json.log
-rw-r----- 1 root root 811M Sep 27 09:53 /var/lib/docker/containers/f4785884c666706104df5f8140f30fdaf4df1c345bdfdff1887345e966ba8491/f4785884c666706104df5f8140f30fdaf4df1c345bdfdff1887345e966ba8491-json.log
-rw-r----- 1 root root  23M Sep 27 07:10 /var/lib/docker/containers/fcb032a9cc740de91e11f11f1c2bee2ca442b0b40458ecdd99a0556d52ffa621/fcb032a9cc740de91e11f11f1c2bee2ca442b0b40458ecdd99a0556d52ffa621-json.log

root@fgt-tst-pl:~# du -sh /var/lib/docker/containers/
5.9G	/var/lib/docker/containers/
root@fgt-tst-pl:~# docker ps -a
CONTAINER ID   IMAGE                                      COMMAND                  CREATED         STATUS                        PORTS                            NAMES
fcb032a9cc74   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    pha1
226bf4318049   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    pha2
b5e9b2d8972a   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    whs1
843d2c8bd417   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    whs-takeda
eb0bcd559f11   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    whs-zuellig
596d2c2565d0   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    whs2
868627bbf8a2   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-bayer
c999718ebcf9   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-sanofi
99e64cead7c6   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-gsk
2cb0d8f95140   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-takeda
989c4ec69a17   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-roche
738f9547fd41   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-msd
9455722e74ea   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-novo-nordisk
090b06f15dcd   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-pfizer
d5e5a18b7908   traefik:v2.6                               "/entrypoint.sh --lo…"   10 months ago   Up 24 minutes                 80/tcp, 0.0.0.0:8080->8080/tcp   traefik
f0421174fee0   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Up 24 minutes (healthy)       3009/tcp, 8080-8081/tcp          fgt-workspace
f4785884c666   vegardit/traefik-logrotate:latest          "/sbin/tini -- /bin/…"   10 months ago   Up 24 minutes                                                  traefik-logrotate

Attempting to startup as described in #127 (comment)

The startup seems to work.

The excessive use of disk seems in the overlay2 folder.

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
104K	/var/lib/docker/buildkit
5.9G	/var/lib/docker/containers
4.0K	/var/lib/docker/engine-id
58M	/var/lib/docker/image
136K	/var/lib/docker/network
29G	/var/lib/docker/overlay2
16K	/var/lib/docker/plugins
4.0K	/var/lib/docker/runtimes
4.0K	/var/lib/docker/swarm
4.0K	/var/lib/docker/tmp
4.0K	/var/lib/docker/trust
1.3G	/var/lib/docker/volumes

Like @TiagoV-PDMFC said, to diagnose this, the image filesystem should be put in read-only, and we should see what fails.

(Please note in the next comment that overlay2 grew a lot while starting the dockers - builds).

@joaoluis-pdm
Copy link
Contributor Author

joaoluis-pdm commented Sep 27, 2023 •
edited
Loading

Get some extra space meanwhile

root@fgt-tst-pl:~# docker image prune --all
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: joaoluispdm/pharmaledger-imi-fgt:v0.11.2
untagged: joaoluispdm/pharmaledger-imi-fgt@sha256:5df868fe2e3d953a72083cf69ff1c615beb8d67daa56228d3bad571d7b0403f3
deleted: sha256:bf6ddbe7622073b40fd32081fe54889e72736623b195f4405e4b4dcdd6ae0588
deleted: sha256:d7d380fb6caebc8cec7e1909edba6160c48f0a38126a98e4f13b568df877806d
deleted: sha256:f67dc1ac7b52418c9930cdf18fbabca387c798f4360e0ff42a87e36649ab2710
deleted: sha256:b6d25141c84f1291934d196071c055811dd9fdf7c15c2b26020f2067c16865bd
deleted: sha256:5007036c423e808739707dd54f159cbf9f5513e931ad2b551aef6258e95fa4ae
deleted: sha256:8fb23a23af86f09605f9888ad9b1cbe454c60d952223f69437d68c09fd4a3d19
deleted: sha256:14b4415383cec797eb5376cfa7382ef3430b5f504d05e7b56e950c6a986f1a01
deleted: sha256:975ab0f9f82959dd093fb9deab1bc8026150989960e2f1db79d85533d3bf6694
deleted: sha256:56538c10778a0a59a6ff6101893ffd3623b471a6f6be2a2acddc5b1b6a190f44
deleted: sha256:22a2a13a6d015587199a8b604f53e64df2f55e4933a6617ea719b401f3d688b7
untagged: joaoluispdm/pharmaledger-imi-fgt:v0.11.3
untagged: joaoluispdm/pharmaledger-imi-fgt@sha256:31aa14318ff1de2f8cc8ae2cca424f8fee9340cb779f1f8089372a72e7c676e5
deleted: sha256:ac4eee7292783561554318841e64d57e72562e2f330474cfe6c7995ff651f0f8
deleted: sha256:088fe30622359ede939a64cf3352cabeb56f75a869b10979f67f3c5ced203154
deleted: sha256:5b55ebddd2092a05b6b221eef195890f8a0afa038ef344660f3c2835deab9dfd
deleted: sha256:2ac584c79756aa97e22c95498c026fd3fd0b704c1a0385ff45ee52de757c6599
deleted: sha256:aa6b7318cc6c5bebda447e1f343d77a13d8b5b1660e12bd92d47cd08f80cbb53
deleted: sha256:0c29be2b8dc1c063ca62bb3c45538c628a591e4bf4d02c9ce4c44ccc2dc55b41
deleted: sha256:ea03144da2b0832e3dd2f48a95474b9b6c4376410f8b66bab8bd129e007681bf
deleted: sha256:fc09cef5f5babbdf0306204fbbb51623395966d97abc5fce64cf1fcb1d119189
deleted: sha256:149e0bdf13f6b609a8d0bb21827114ac4e04e09992fc2d272cc74dfb8b65eee2
deleted: sha256:ccacc1eb77a64077019006e357ff1f8ccfe2c14731822fe61ab4ebca23735382

Total reclaimed space: 4.618GB

root@fgt-tst-pl:~# docker image prune --all
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: joaoluispdm/pharmaledger-imi-fgt:v0.11.2
untagged: joaoluispdm/pharmaledger-imi-fgt@sha256:5df868fe2e3d953a72083cf69ff1c615beb8d67daa56228d3bad571d7b0403f3
deleted: sha256:bf6ddbe7622073b40fd32081fe54889e72736623b195f4405e4b4dcdd6ae0588
deleted: sha256:d7d380fb6caebc8cec7e1909edba6160c48f0a38126a98e4f13b568df877806d
deleted: sha256:f67dc1ac7b52418c9930cdf18fbabca387c798f4360e0ff42a87e36649ab2710
deleted: sha256:b6d25141c84f1291934d196071c055811dd9fdf7c15c2b26020f2067c16865bd
deleted: sha256:5007036c423e808739707dd54f159cbf9f5513e931ad2b551aef6258e95fa4ae
deleted: sha256:8fb23a23af86f09605f9888ad9b1cbe454c60d952223f69437d68c09fd4a3d19
deleted: sha256:14b4415383cec797eb5376cfa7382ef3430b5f504d05e7b56e950c6a986f1a01
deleted: sha256:975ab0f9f82959dd093fb9deab1bc8026150989960e2f1db79d85533d3bf6694
deleted: sha256:56538c10778a0a59a6ff6101893ffd3623b471a6f6be2a2acddc5b1b6a190f44
deleted: sha256:22a2a13a6d015587199a8b604f53e64df2f55e4933a6617ea719b401f3d688b7
untagged: joaoluispdm/pharmaledger-imi-fgt:v0.11.3
untagged: joaoluispdm/pharmaledger-imi-fgt@sha256:31aa14318ff1de2f8cc8ae2cca424f8fee9340cb779f1f8089372a72e7c676e5
deleted: sha256:ac4eee7292783561554318841e64d57e72562e2f330474cfe6c7995ff651f0f8
deleted: sha256:088fe30622359ede939a64cf3352cabeb56f75a869b10979f67f3c5ced203154
deleted: sha256:5b55ebddd2092a05b6b221eef195890f8a0afa038ef344660f3c2835deab9dfd
deleted: sha256:2ac584c79756aa97e22c95498c026fd3fd0b704c1a0385ff45ee52de757c6599
deleted: sha256:aa6b7318cc6c5bebda447e1f343d77a13d8b5b1660e12bd92d47cd08f80cbb53
deleted: sha256:0c29be2b8dc1c063ca62bb3c45538c628a591e4bf4d02c9ce4c44ccc2dc55b41
deleted: sha256:ea03144da2b0832e3dd2f48a95474b9b6c4376410f8b66bab8bd129e007681bf
deleted: sha256:fc09cef5f5babbdf0306204fbbb51623395966d97abc5fce64cf1fcb1d119189
deleted: sha256:149e0bdf13f6b609a8d0bb21827114ac4e04e09992fc2d272cc74dfb8b65eee2
deleted: sha256:ccacc1eb77a64077019006e357ff1f8ccfe2c14731822fe61ab4ebca23735382

Total reclaimed space: 4.618GB

pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   41G   72G  37% /

pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED         SIZE
joaoluispdm/pharmaledger-imi-fgt   v0.11.4   191e1d1e2c33   10 months ago   2.42GB
vegardit/traefik-logrotate         latest    56c93b36b4af   10 months ago   55.1MB
traefik                            v2.6      22c6901de2be   16 months ago   102MB

Although overlay2 grew 5GB :-( it seems we get more space for a while...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@joaoluis-pdm joaoluis-pdm

Labels
wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@joaoluis-pdm