[BUG][DEPLOY] Uninstall script does not remove all services, replicasets, deployments and olms

[manosnoam]

Using https://github.com/open-cluster-management/deploy/blob/master/multiclusterhub/uninstall.sh
to uninstall ACM, completed with:

$ ./uninstall.sh
Now using project "ocm" on server "https://api.nmanos-aws-devcluster-a.devcluster.openshift.com:6443".
Uninstall stuck... Striping out finalizers from helm releases...
The request is invalid
The request is invalid
The request is invalid
The request is invalid
The request is invalid
The request is invalid
The request is invalid
The request is invalid
The request is invalid
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "certificates.certmanager.k8s.io" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "certificaterequests.certmanager.k8s.io" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "challenges.certmanager.k8s.io" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "clusterissuers.certmanager.k8s.io" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "issuers.certmanager.k8s.io" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "orders.certmanager.k8s.io" not found
Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "cert-manager-webhook-requester" not found
Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "cert-manager-webhook-auth-delegator" not found
Error from server (NotFound): consolelinks.console.openshift.io "acm-console-link" not found
customresourcedefinition.apiextensions.k8s.io "userpreferences.console.open-cluster-management.io" deleted
Error from server (NotFound): oauthclients.oauth.openshift.io "multicloudingress" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "endpointconfigs.multicloud.ibm.com" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "klusterletconfigs.agent.open-cluster-management.io" not found
Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "rcm-controller" not found
Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "rcm-controller" not found
error: the server doesn't have a resource type "policies"
error: the server doesn't have a resource type "policies"
error: the server doesn't have a resource type "policies"
error: the server doesn't have a resource type "policies"
configmap "ingress-controller-leader-ingress-open-cluster-management" deleted
customresourcedefinition.apiextensions.k8s.io "agentclusterinstalls.extensions.hive.openshift.io" deleted
error: resource(s) were provided, but no name was specified
No resources found in hive namespace.
No resources found in hive namespace.
No resources found in hive namespace.
error: the server doesn't have a resource type "hiveconfig"

But later when looking at the ACM namespace I still see several ACM services, replicasets, deployments and olms, which are not in termination process:

$ oc get all,olm -n ocm 

NAME                                                                  READY   STATUS        RESTARTS   AGE
pod/console-chart-d59f8-console-v2-797dff8b66-66d29                   1/1     Terminating   0          7h
pod/console-chart-d59f8-console-v2-797dff8b66-tndpb                   1/1     Terminating   0          7h
pod/multicluster-operators-hub-subscription-84bbd555ff-jb8xb          1/1     Terminating   0          7h2m
pod/multicluster-operators-standalone-subscription-64fc6cfdc4-qvkw8   1/1     Terminating   0          7h2m
pod/multiclusterhub-repo-5b66558845-tlbgg                             1/1     Running       0          7h1m
pod/my-catalog-source-kscwk                                           1/1     Running       0          157m
pod/ocm-controller-8485654c75-46mt2                                   1/1     Running       0          7h
pod/ocm-controller-8485654c75-8qppv                                   1/1     Running       0          7h
pod/ocm-proxyserver-856f674b8b-5b5lh                                  1/1     Running       0          7h
pod/ocm-proxyserver-856f674b8b-fx8gs                                  1/1     Running       0          7h
pod/ocm-webhook-67fc798c69-p5r5g                                      1/1     Running       0          7h1m
pod/ocm-webhook-67fc798c69-smdmb                                      1/1     Running       0          7h1m
pod/search-prod-19fb4-search-api-65c887c77-5qtcn                      1/1     Terminating   0          6h59m
pod/search-prod-19fb4-search-api-65c887c77-q4c22                      1/1     Terminating   0          6h59m
pod/search-redisgraph-0                                               1/1     Terminating   0          6h58m

NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/multiclusterhub-operator-webhook   ClusterIP   172.40.160.186   <none>        443/TCP     7h1m
service/multiclusterhub-repo               ClusterIP   172.40.195.78    <none>        3000/TCP    7h1m
service/my-catalog-source                  ClusterIP   172.40.164.109   <none>        50051/TCP   157m
service/ocm-proxyserver                    ClusterIP   172.40.15.120    <none>        443/TCP     7h
service/ocm-webhook                        ClusterIP   172.40.80.99     <none>        443/TCP     7h1m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/multiclusterhub-repo   1/1     1            1           7h1m
deployment.apps/ocm-controller         2/2     2            2           7h
deployment.apps/ocm-proxyserver        2/2     2            2           7h
deployment.apps/ocm-webhook            2/2     2            2           7h1m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/multiclusterhub-repo-5b66558845   1         1         1       7h1m
replicaset.apps/ocm-controller-8485654c75         2         2         2       7h
replicaset.apps/ocm-proxyserver-856f674b8b        2         2         2       7h
replicaset.apps/ocm-webhook-67fc798c69            2         2         2       7h1m

NAME                                                       IMAGE REPOSITORY                                                                 TAGS     
imagestream.image.openshift.io/acm-operator-bundle-index   image-registry.openshift-image-registry.svc:5000/ocm/acm-operator-bundle-index   v2.4.0   3 

NAME                                          AGE
operatorgroup.operators.coreos.com/my-group   7h2m

NAME                                                            AGE
operator.operators.coreos.com/advanced-cluster-management.ocm   7h2m
operator.operators.coreos.com/submariner.submariner-operator    6h51m

NAME                                                   DISPLAY                  TYPE   PUBLISHER                AGE
catalogsource.operators.coreos.com/my-catalog-source   Testing Catalog Source   grpc   Red Hat Partner (Test)   157m

To Reproduce

1. Install ACM + managed clusters.

2. Use the following script to uninstall ACM:
   https://github.com/open-cluster-management/deploy/blob/master/multiclusterhub/uninstall.sh

3. Run "oc get all,olm -n ocm" (on the ACM namespace).

Expected behavior
All ACM resources should be deleted or in termination process.