TST environment did not recover automatically after power-off / power-restore

[joaoluis-pdm]

(The DEV environment, the issue is know PharmaLedger-IMI/blockchain-hlf-workspace#5 , but not the TST, which has the blockchain turned off).

Power seem to have been lost on Dec 7 22:55:05 (due to the floods - https://www.jn.pt/local/noticias/lisboa/lisboa/socorro-condutor-pede-ajuda-em-cima-de-carro-arrastado-pelas-aguas-em-alcantara-15447354.html )

Power seem to have been restored on Dec 8 16:55:11

pharmaledger@fgt-tst-pl:~$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   36G   77G  32% /

pharmaledger@fgt-tst-pl:~$ docker ps -a
CONTAINER ID   IMAGE                                      COMMAND                  CREATED       STATUS                        PORTS                            NAMES
868627bbf8a2   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   2 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          mah-bayer
7e5cbbb59f48   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          pha1
65e5caf53642   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          pha2
9ba91932426f   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          whs-zuellig
e5b79898ef61   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          whs1
c85d168ae41d   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          whs2
e65177128a79   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          whs-takeda
c999718ebcf9   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          mah-sanofi
99e64cead7c6   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          mah-gsk
2cb0d8f95140   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          mah-takeda
989c4ec69a17   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          mah-roche
738f9547fd41   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          mah-msd
9455722e74ea   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          mah-novo-nordisk
090b06f15dcd   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Exited (255) 45 minutes ago   3009/tcp, 8080-8081/tcp          mah-pfizer
d5e5a18b7908   traefik:v2.6                               "/entrypoint.sh --lo…"   3 weeks ago   Up 45 minutes                 80/tcp, 0.0.0.0:8080->8080/tcp   traefik
f0421174fee0   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Up 45 minutes (healthy)       3009/tcp, 8080-8081/tcp          fgt-workspace
f4785884c666   vegardit/traefik-logrotate:latest          "/sbin/tini -- /bin/…"   3 weeks ago   Up 45 minutes                                                  traefik-logrotate

pharmaledger@fgt-tst-pl:~$ ./fgt-workspace/docker/api/saveAllDockerLogs.sh 
# only fgt-workspace is up

pharmaledger@fgt-tst-pl:~$ docker logs --timestamps -f mah-bayer
...
2022-12-07T22:15:23.929705105Z _startMessageListener did:ssi:name:traceability:MAH251339219
# no sign of activity after power restore. Possible docker startup issue ?
pharmaledger@fgt-tst-pl:~$ docker logs --timestamps -f mah-pfizer
...
2022-12-07T21:39:36.578255968Z _startMessageListener did:ssi:name:traceability:MAH135315170
# no sign of activity after power restore. Possible docker startup issue ?

# starting the containers again with docker compose
pharmaledger@fgt-tst-pl:~$ cd fgt-workspace/docker/api/
pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ ./bootCompose.sh tst
+ '[' tst = help -o tst = --help -o tst = -h -o tst = '-?' ']'
+ params=.localhost
+ simple=localhost
+ swagger=.localhost:8080
+ protocol=http
+ environment=local
+ [[ tst = *\d\e\v* ]]
+ [[ tst = *\t\s\t* ]]
+ params=-fgt.pharmaledger.pdmfc.com
+ simple=fgt.pharmaledger.pdmfc.com
+ environment=tst
+ swagger=-fgt.pharmaledger.pdmfc.com
+ protocol=https
+ [[ tst = *\e\t\h* ]]
+ [[ tst = *\b\c* ]]
+ echo 'Launching docker compose in -fgt.pharmaledger.pdmfc.com without ETH'
Launching docker compose in -fgt.pharmaledger.pdmfc.com without ETH
+ SIMPLE=fgt.pharmaledger.pdmfc.com
+ DOMAIN=-fgt.pharmaledger.pdmfc.com
+ SWAGGER=-fgt.pharmaledger.pdmfc.com
+ PROTOCOL=https
+ ENVIRONMENT=tst
+ COMMAND=
+ docker-compose up -d
traefik is up-to-date
fgt-workspace is up-to-date
traefik-logrotate is up-to-date
Starting mah-takeda       ... done
Starting mah-roche        ... done
Starting mah-sanofi       ... done
Starting mah-novo-nordisk ... done
Starting mah-bayer        ... done
Starting mah-msd          ... done
Starting mah-pfizer       ... done
Starting mah-gsk          ... done

# wait for build

[joaoluis-pdm]

The startup went ok.

But the docker issue is not clear.

[joaoluis-pdm]

New powerloss

Power seems to have been lost at Dec 13 05:50:11

Power seems to have been restored at Dec 13 22:18:19

pharmaledger@fgt-tst-pl:~$ docker ps
CONTAINER ID   IMAGE                                      COMMAND                  CREATED       STATUS                   PORTS                            NAMES
d5e5a18b7908   traefik:v2.6                               "/entrypoint.sh --lo…"   3 weeks ago   Up 2 minutes             80/tcp, 0.0.0.0:8080->8080/tcp   traefik
f0421174fee0   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   3 weeks ago   Up 2 minutes (healthy)   3009/tcp, 8080-8081/tcp          fgt-workspace
f4785884c666   vegardit/traefik-logrotate:latest          "/sbin/tini -- /bin/…"   3 weeks ago   Up 2 minutes                                              traefik-logrotate

$ docker logs -f fgt-workspace

# fgt-workspace seems up ok. Probably the others don't start ok because fgt-workspace's apihub is not immediatly available.
# start the rest of the containers

pharmaledger@fgt-tst-pl:~$ cd fgt-workspace/docker/api/
pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ ./bootCompose.sh tst