Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

f5-bigip-runtime-init error: Error getting token 400 #34

Open
DahlPatric opened this issue Feb 21, 2024 · 9 comments
Open

f5-bigip-runtime-init error: Error getting token 400 #34

DahlPatric opened this issue Feb 21, 2024 · 9 comments

Comments

@DahlPatric
Copy link

Describe the bug

Runtime init script end with "Error getting token 400"
Interface:1.1 seams to be mapped twice could this be related to 400 error message?
There is also some license issues.

Current behavior

2024-02-21T15:23:58.551Z [32141]: info: Resolving parameters
2024-02-21T15:23:58.606Z [32141]: info: Interface:1.2
2024-02-21T15:23:58.607Z [32141]: info: MAC address found for 1.2: 00:17:fa:07:9f:09
2024-02-21T15:23:58.607Z [32141]: info: Local interface 2 MAC address 0017fa079f09 matches Azure network interface 2 MAC address 0017fa079f09
2024-02-21T15:23:58.613Z [32141]: info: Interface:mgmt
2024-02-21T15:23:58.614Z [32141]: info: MAC address found for mgmt: 00:17:fa:07:9e:df
2024-02-21T15:23:58.615Z [32141]: info: Local interface 0 MAC address 0017fa079edf matches Azure network interface 0 MAC address 0017fa079edf
2024-02-21T15:23:58.621Z [32141]: info: Interface:1.1
2024-02-21T15:23:58.621Z [32141]: info: MAC address found for 1.1: 00:17:fa:07:9a:eb
2024-02-21T15:23:58.622Z [32141]: info: Local interface 1 MAC address 0017fa079aeb matches Azure network interface 1 MAC address 0017fa079aeb
2024-02-21T15:23:58.628Z [32141]: info: Interface:1.1
2024-02-21T15:23:58.628Z [32141]: info: MAC address found for 1.1: 00:17:fa:07:9a:eb
2024-02-21T15:23:58.629Z [32141]: info: Local interface 1 MAC address 0017fa079aeb matches Azure network interface 1 MAC address 0017fa079aeb
2024-02-21T15:23:58.758Z [32141]: error: Error getting token 400
2024-02-21T15:23:58.758Z [32141]: info: Sending F5 Teem report for failure case.
2024-02-21T15:23:59.013Z [32141]: warn: Problem with getting data from /mgmt/tm/sys/license endpoint. Leaving regKey with default value
2024-02-21T15:23:59.014Z [32141]: info: {"id":"ba9d9528-e875-994c-95725ff36b09","product":"BIG-IP","cpuCount":8,"diskSize":86016,"memoryInMb":32176,"version":"16.1.4.2","nicCount":3,"platformId":"Z100","hostname":"bigip1","management":"10.45.136.69/26","provisionedModules":{"ltm":"nominal"},"installedPackages":{},"environment":{"pythonVersion":"Python 2.7.5","pythonVersionDetailed":"2.7.5 (default, Dec 1 2023, 09:40:19) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]","nodeVersion":"v6.9.1","libraries":{"ssh":"OpenSSH_7.4p1, OpenSSL 1.0.2u-fips 20 Dec 2019"}}}
2024-02-21T15:23:59.270Z [32141]: error: Device is not licensed yet

Your Environment

cat /config/cloud/secret_id
BigIpSecret/

cat /config/cloud/vault_url
https://f5-kv.vault.azure.cn

cat /config/cloud/license_key
XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

"bigIpPasswordSecretId": {
"value": "https://f5-kv.vault.azure.cn/secrets/BigIpSecret/"
},
"bigIpPasswordSecretValue": {
"value": ""
},

Been switch between using either bigIpPasswordSecretValue Only or as now last time bigIpPasswordSecretId.
What I notice is that if i'm using bigIpPasswordSecretValue Azure Secrets object sill get's created, or is this per design?
@mikeshimkus
Copy link
Collaborator

Interface 1.1 is only mapped once, it is just logging the same information twice.

It appears that you are getting the 400 when trying to make a request to the /mgmt/tm/sys/license endpoint, which indicates the issue is getting the token from BIG-IP REST API, not Azure.

For your last question, yes, it is by design. When you pass bigIpPasswordSecretValue it will create a new secret as documented here: https://github.com/F5Networks/f5-azure-arm-templates-v2/tree/main/examples/failover#prerequisites

@DahlPatric
Copy link
Author

Thanks for response, I'll notice when ARM script is done that I'm not able to logon to F5 with password set for parameter bigIpPasswordSecretValue
With that in mind that must be the reason F5 either can login to end point /mgmt/tm/sys/license.
Obvious I'm missing something in configuration??

This is how I today have it configured.

"bigIpPasswordSecretId": {
"value": ""
},
"bigIpPasswordSecretValue": {
"value": "P@ssw0rd"
},

@mikeshimkus
Copy link
Collaborator

If runtime init did not succeed in applying the DO config for any reason, then you would not be able to login using that password. The getting of the auth token for BIG-IP happens before the DO config, so that would not be the issue. Can you share the entire sanitized output of /var/log/cloud/startup-script.log?

@DahlPatric
Copy link
Author

I'm not able after what's have been configured able to login with my given credential. Sound like a potential reason why accessing license end point. Another with worth to mention is I'm keeping my secret key each time I'm re-deploy other with is fails with object already exist, and object in Azure cant be purged.

Device 1

[azureuser@localhost:NO LICENSE:Standalone] ~ # cat of /var/log/cloud/startup-script.log
cat: of: No such file or directory
2024-02-21T05:12:26.903Z : Starting Custom Script
*   Trying 185.199.109.133...
* Connected to raw.githubusercontent.com (185.199.109.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3045 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*        subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.io
*        start date: Feb 21 00:00:00 2023 GMT
*        expire date: Mar 20 23:59:59 2024 GMT
*        subjectAltName: raw.githubusercontent.com matched
*        issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*        SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* TCP_NODELAY set
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x187f2a0)
} [5 bytes data]
> GET /DahlPatric/f5-azure-china-arm-templates-v2/main/examples/failover/bigip-configurations/runtime-init-conf-3nic-byol-instance01.yaml HTTP/1.1
> Host: raw.githubusercontent.com
> User-Agent: curl/7.47.1
> Accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2.0 200
< cache-control:max-age=300
< content-security-policy:default-src 'none'; style-src 'unsafe-inline'; sandbox
< content-type:text/plain; charset=utf-8
< etag:"b3ad3cedd9eb934d7eff8678f6863d89d29c2ed7f2844e3ce1455f2f48adec3f"
< strict-transport-security:max-age=31536000
< x-content-type-options:nosniff
< x-frame-options:deny
< x-xss-protection:1; mode=block
< x-github-request-id:6C14:3CB7F9:5E123D:6635E7:65D5F6BD
< accept-ranges:bytes
< date:Wed, 21 Feb 2024 13:12:29 GMT
< via:1.1 varnish
< x-served-by:cache-tyo11947-TYO
< x-cache:MISS
< x-cache-hits:0
< x-timer:S1708521150.543692,VS0,VE255
< vary:Authorization,Accept-Encoding,Origin
< access-control-allow-origin:*
< cross-origin-resource-policy:cross-origin
< x-fastly-request-id:ef758906008a7e0aaf9cda6010cebabfb7e4fcc8
< expires:Wed, 21 Feb 2024 13:17:29 GMT
< source-age:0
< content-length:8753
<
{ [803 bytes data]
* Connection #0 to host raw.githubusercontent.com left intact
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 18.67.51.52...
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Connected to cdn.f5.com (18.67.51.52) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [91 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4202 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*        subject: C=US; ST=Washington; L=Seattle; jurisdictionC=US; jurisdictionST=Washington; O=F5, Inc.; businessCategory=Private Organization; serialNumber=601692492; CN=cdn.f5.com
*        start date: Feb 15 19:13:21 2024 GMT
*        expire date: Mar 15 19:13:20 2025 GMT
*        subjectAltName: cdn.f5.com matched
*        issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
*        SSL certificate verify ok.
} [5 bytes data]
> GET /product/cloudsolutions/f5-bigip-runtime-init/v2.0.1/dist/f5-bigip-runtime-init-2.0.1-1.gz.run HTTP/1.1
> Host: cdn.f5.com
> User-Agent: curl/7.47.1
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 10790502
< Connection: keep-alive
< Date: Wed, 21 Feb 2024 07:48:53 GMT
< x-amz-meta-s3cmd-attrs: md5:1fc0dd0d29cd5c80615e1fbad5bd8e55
< Last-Modified: Thu, 21 Dec 2023 23:51:08 GMT
< ETag: "1fc0dd0d29cd5c80615e1fbad5bd8e55"
< Server: AmazonS3
< X-Cache: Hit from cloudfront
< Via: 1.1 3a670abd18e913fcef1bf79a29b36606.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: ICN57-P1
< X-Amz-Cf-Id: mqIaUVV9YUmca5Ukxo4_FjAViIIurPxNqmK33DQaefmgWHezYAY34g==
< Age: 19419
<
{ [5 bytes data]
100 10.2M  100 10.2M    0     0  2195k      0  0:00:04  0:00:04 --:--:-- 2254k
* Connection #0 to host cdn.f5.com left intact
Verifying archive integrity... All good.
Uncompressing F5 BIGIP Runtime Init installation...................
2024-02-21T05:12:34 - HTTP Retry Settings:
2024-02-21T05:12:34 - RETRY: 3
2024-02-21T05:12:34 - RETRY_MAX_TIME: 180
2024-02-21T05:12:34 - MAX_TIME: 5
2024-02-21T05:12:34 - RETRY_DELAY: 60
2024-02-21T05:12:34 - Running RPM install script.
2024-02-21T05:12:34 - Verifying RPM file integrity...
f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm: OK
2024-02-21T05:12:34 - Verifying signature...
2024-02-21T05:12:34 - GPG PUB Key location: https://f5-cft.s3.amazonaws.com/f5-bigip-runtime-init/gpg.key
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3248  100  3248    0     0   2702      0  0:00:01  0:00:01 --:--:--  2704
f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm: rsa sha1 (md5) pgp md5 OK
2024-02-21T05:12:36 - Checking if package is already installed
2024-02-21T05:12:36 - Package is not installed. Preparing for installation.
2024-02-21T05:12:36 - Install location /tmp/f5-bigip-runtime-init does not exist. Creating install location.
2024-02-21T05:12:36 - Install package f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm
./f5-bigip-runtime-init-azure
........ Removed all between
./f5-bigip-runtime-init-azure/src/version
22072 blocks
2024-02-21T05:12:41 - Getting lastest AT metadata at https://cdn.f5.com/product/cloudsolutions/f5-extension-metadata/latest/metadata.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
Warning: Transient problem: timeout Will retry in 60 seconds. 3 retries left.
100 47763  100 47763    0     0  74781      0 --:--:-- --:--:-- --:--:-- 74863
2024-02-21T05:13:47 - Creating command utility.
2024-02-21T05:13:47 - RPM installation is completed.
2024-02-21T13:13:47.989Z [27066]: info: Configuration file: /config/cloud/runtime-init.conf
2024-02-21T13:13:48.006Z [27066]: info: Processing controls parameters
2024-02-21T13:13:48.010Z [27066]: info: F5 Telemetry is disabled.
2024-02-21T13:13:48.013Z [27066]: info: Validating provided declaration
2024-02-21T13:13:48.076Z [27066]: info: Successfully validated declaration
2024-02-21T13:13:48.079Z [27066]: info: Executing custom pre_onboard_enabled commands
2024-02-21T13:13:48.091Z [27066]: info: Executing inline shell command: /usr/bin/setdb provision.extramb 1000 || exit 0
2024-02-21T13:13:48.231Z [27066]: info: Shell command: /usr/bin/setdb provision.extramb 1000 || exit 0 execution completed; response:
2024-02-21T13:13:48.236Z [27066]: info: Executing inline shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0
2024-02-21T13:13:48.364Z [27066]: info: Shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0 execution completed; response:
2024-02-21T13:13:48.413Z [27066]: info: Resolving parameters
2024-02-21T13:13:48.471Z [27066]: info: Interface:mgmt
2024-02-21T13:13:48.475Z [27066]: info: MAC address found for mgmt: 00:17:fa:07:e6:90
2024-02-21T13:13:48.478Z [27066]: info: Local interface 0 MAC address 0017fa07e690 matches Azure network interface 0 MAC address 0017fa07e690
2024-02-21T13:13:48.486Z [27066]: info: Interface:1.1
2024-02-21T13:13:48.489Z [27066]: info: MAC address found for 1.1: 00:17:fa:07:e5:cc
2024-02-21T13:13:48.492Z [27066]: info: Local interface 1 MAC address 0017fa07e5cc matches Azure network interface 1 MAC address 0017fa07e5cc
2024-02-21T13:13:48.497Z [27066]: info: Interface:1.2
2024-02-21T13:13:48.500Z [27066]: info: MAC address found for 1.2: 00:17:fa:07:e6:09
2024-02-21T13:13:48.503Z [27066]: info: Local interface 2 MAC address 0017fa07e609 matches Azure network interface 2 MAC address 0017fa07e609
2024-02-21T13:13:48.510Z [27066]: info: Interface:1.1
2024-02-21T13:13:48.512Z [27066]: info: MAC address found for 1.1: 00:17:fa:07:e5:cc
2024-02-21T13:13:48.515Z [27066]: info: Local interface 1 MAC address 0017fa07e5cc matches Azure network interface 1 MAC address 0017fa07e5cc
2024-02-21T13:13:48.906Z [27066]: error: Error getting token 400
2024-02-21T07:20:50.711Z : Starting Custom Script
*   Trying 185.199.108.133...
* Connected to raw.githubusercontent.com (185.199.108.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 0
* Hostname raw.githubusercontent.com was found in DNS cache
*   Trying 185.199.108.133...
* Connected to raw.githubusercontent.com (185.199.108.133) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 1
*   Trying 185.199.110.133...
* Connected to raw.githubusercontent.com (185.199.110.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3045 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 0
* Hostname raw.githubusercontent.com was found in DNS cache
*   Trying 185.199.110.133...
*   Trying 2606:50c0:8000::154...
* Immediate connect fail for 2606:50c0:8000::154: Network is unreachable
*   Trying 2606:50c0:8001::154...
* Immediate connect fail for 2606:50c0:8001::154: Network is unreachable
*   Trying 2606:50c0:8002::154...
* Immediate connect fail for 2606:50c0:8002::154: Network is unreachable
*   Trying 2606:50c0:8003::154...
* Immediate connect fail for 2606:50c0:8003::154: Network is unreachable
*   Trying 2606:50c0:8000::154...
* Immediate connect fail for 2606:50c0:8000::154: Network is unreachable
*   Trying 2606:50c0:8001::154...
* Immediate connect fail for 2606:50c0:8001::154: Network is unreachable
*   Trying 2606:50c0:8002::154...
* Immediate connect fail for 2606:50c0:8002::154: Network is unreachable
*   Trying 2606:50c0:8003::154...
* Immediate connect fail for 2606:50c0:8003::154: Network is unreachable
*   Trying 2606:50c0:8000::154...
* Immediate connect fail for 2606:50c0:8000::154: Network is unreachable
*   Trying 2606:50c0:8001::154...
* Immediate connect fail for 2606:50c0:8001::154: Network is unreachable
*   Trying 2606:50c0:8002::154...
* Immediate connect fail for 2606:50c0:8002::154: Network is unreachable
*   Trying 2606:50c0:8003::154...
* Immediate connect fail for 2606:50c0:8003::154: Network is unreachable
*   Trying 2606:50c0:8000::154...
* Immediate connect fail for 2606:50c0:8000::154: Network is unreachable
*   Trying 2606:50c0:8001::154...
* Immediate connect fail for 2606:50c0:8001::154: Network is unreachable
*   Trying 2606:50c0:8002::154...
* Immediate connect fail for 2606:50c0:8002::154: Network is unreachable
*   Trying 2606:50c0:8003::154...
* Immediate connect fail for 2606:50c0:8003::154: Network is unreachable
*   Trying 2606:50c0:8000::154...
* Immediate connect fail for 2606:50c0:8000::154: Network is unreachable
*   Trying 2606:50c0:8001::154...
* Immediate connect fail for 2606:50c0:8001::154: Network is unreachable
*   Trying 2606:50c0:8002::154...
* Immediate connect fail for 2606:50c0:8002::154: Network is unreachable
*   Trying 2606:50c0:8003::154...
* Immediate connect fail for 2606:50c0:8003::154: Network is unreachable
*   Trying 2606:50c0:8000::154...
* Immediate connect fail for 2606:50c0:8000::154: Network is unreachable
*   Trying 2606:50c0:8001::154...
* Immediate connect fail for 2606:50c0:8001::154: Network is unreachable
*   Trying 2606:50c0:8002::154...
* Immediate connect fail for 2606:50c0:8002::154: Network is unreachable
*   Trying 2606:50c0:8003::154...
* Immediate connect fail for 2606:50c0:8003::154: Network is unreachable
* Connection timed out after 5000 milliseconds
* Closing connection 1
*   Trying 185.199.109.133...
* Connected to raw.githubusercontent.com (185.199.109.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3045 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*        subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.io
*        start date: Feb 21 00:00:00 2023 GMT
*        expire date: Mar 20 23:59:59 2024 GMT
*        subjectAltName: raw.githubusercontent.com matched
*        issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*        SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* TCP_NODELAY set
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x6b62a0)
} [5 bytes data]
> GET /DahlPatric/f5-azure-china-arm-templates-v2/main/examples/failover/bigip-configurations/runtime-init-conf-3nic-byol-instance01.yaml HTTP/1.1
> Host: raw.githubusercontent.com
> User-Agent: curl/7.47.1
> Accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2.0 200
< cache-control:max-age=300
< content-security-policy:default-src 'none'; style-src 'unsafe-inline'; sandbox
< content-type:text/plain; charset=utf-8
< etag:"b3ad3cedd9eb934d7eff8678f6863d89d29c2ed7f2844e3ce1455f2f48adec3f"
< strict-transport-security:max-age=31536000
< x-content-type-options:nosniff
< x-frame-options:deny
< x-xss-protection:1; mode=block
< x-github-request-id:AF08:3CBD60:6501E7:6DDCAC:65D614FC
< accept-ranges:bytes
< date:Wed, 21 Feb 2024 15:21:34 GMT
< via:1.1 varnish
< x-served-by:cache-tyo11983-TYO
< x-cache:MISS
< x-cache-hits:0
< x-timer:S1708528894.947211,VS0,VE276
< vary:Authorization,Accept-Encoding,Origin
< access-control-allow-origin:*
< cross-origin-resource-policy:cross-origin
< x-fastly-request-id:12b8f4521ccf8bfa67fbea02f1bc8dd390c1d081
< expires:Wed, 21 Feb 2024 15:26:34 GMT
< source-age:0
< content-length:8753
<
{ [802 bytes data]
* Connection #0 to host raw.githubusercontent.com left intact
Verifying archive integrity... All good.
Uncompressing F5 BIGIP Runtime Init installation...................
2024-02-21T07:21:34 - HTTP Retry Settings:
2024-02-21T07:21:34 - RETRY: 3
2024-02-21T07:21:34 - RETRY_MAX_TIME: 180
2024-02-21T07:21:34 - MAX_TIME: 5
2024-02-21T07:21:34 - RETRY_DELAY: 60
2024-02-21T07:21:34 - Running RPM install script.
2024-02-21T07:21:34 - Verifying RPM file integrity...
f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm: OK
2024-02-21T07:21:34 - Verifying signature...
2024-02-21T07:21:34 - GPG PUB Key location: https://f5-cft.s3.amazonaws.com/f5-bigip-runtime-init/gpg.key
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3248  100  3248    0     0   2704      0  0:00:01  0:00:01 --:--:--  2706
f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm: rsa sha1 (md5) pgp md5 OK
2024-02-21T07:21:35 - Checking if package is already installed
2024-02-21T07:21:35 - Package is already installed and utility is created. Exiting with status:0
2024-02-21T15:21:36.013Z [32258]: info: Configuration file: /config/cloud/runtime-init.conf
2024-02-21T15:21:36.029Z [32258]: info: Processing controls parameters
2024-02-21T15:21:36.033Z [32258]: info: F5 Telemetry is disabled.
2024-02-21T15:21:36.035Z [32258]: info: Validating provided declaration
2024-02-21T15:21:36.091Z [32258]: info: Successfully validated declaration
2024-02-21T15:21:36.094Z [32258]: info: Executing custom pre_onboard_enabled commands
2024-02-21T15:21:36.099Z [32258]: info: Executing inline shell command: /usr/bin/setdb provision.extramb 1000 || exit 0
2024-02-21T15:21:36.224Z [32258]: info: Shell command: /usr/bin/setdb provision.extramb 1000 || exit 0 execution completed; response:
2024-02-21T15:21:36.229Z [32258]: info: Executing inline shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0
2024-02-21T15:21:36.357Z [32258]: info: Shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0 execution completed; response:
2024-02-21T15:21:36.418Z [32258]: info: Resolving parameters
2024-02-21T15:21:36.470Z [32258]: info: Interface:1.2
2024-02-21T15:21:36.473Z [32258]: info: MAC address found for 1.2: 00:17:fa:07:e6:09
2024-02-21T15:21:36.476Z [32258]: info: Local interface 2 MAC address 0017fa07e609 matches Azure network interface 2 MAC address 0017fa07e609
2024-02-21T15:21:36.482Z [32258]: info: Interface:1.1
2024-02-21T15:21:36.484Z [32258]: info: MAC address found for 1.1: 00:17:fa:07:e5:cc
2024-02-21T15:21:36.487Z [32258]: info: Local interface 1 MAC address 0017fa07e5cc matches Azure network interface 1 MAC address 0017fa07e5cc
2024-02-21T15:21:36.494Z [32258]: info: Interface:mgmt
2024-02-21T15:21:36.496Z [32258]: info: MAC address found for mgmt: 00:17:fa:07:e6:90
2024-02-21T15:21:36.499Z [32258]: info: Local interface 0 MAC address 0017fa07e690 matches Azure network interface 0 MAC address 0017fa07e690
2024-02-21T15:21:36.507Z [32258]: info: Interface:1.1
2024-02-21T15:21:36.510Z [32258]: info: MAC address found for 1.1: 00:17:fa:07:e5:cc
2024-02-21T15:21:36.513Z [32258]: info: Local interface 1 MAC address 0017fa07e5cc matches Azure network interface 1 MAC address 0017fa07e5cc
2024-02-21T15:21:36.694Z [32258]: error: Error getting token 400
[azureuser@localhost:NO LICENSE:Standalone] ~ #

Device 2

[azureuser@localhost:NO LICENSE:Standalone] ~ # cat of /var/log/cloud/startup-script.log
cat: of: No such file or directory
2024-02-21T05:12:44.097Z : Starting Custom Script
*   Trying 185.199.111.133...
* Connected to raw.githubusercontent.com (185.199.111.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3045 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 0
* Hostname raw.githubusercontent.com was found in DNS cache
*   Trying 185.199.111.133...
* Connected to raw.githubusercontent.com (185.199.111.133) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3045 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*        subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.io
*        start date: Feb 21 00:00:00 2023 GMT
*        expire date: Mar 20 23:59:59 2024 GMT
*        subjectAltName: raw.githubusercontent.com matched
*        issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*        SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* TCP_NODELAY set
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x14012a0)
} [5 bytes data]
> GET /DahlPatric/f5-azure-china-arm-templates-v2/main/examples/failover/bigip-configurations/runtime-init-conf-3nic-byol-instance02.yaml HTTP/1.1
> Host: raw.githubusercontent.com
> User-Agent: curl/7.47.1
> Accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2.0 200
< cache-control:max-age=300
< content-security-policy:default-src 'none'; style-src 'unsafe-inline'; sandbox
< content-type:text/plain; charset=utf-8
< etag:"247cf257a2d02f04038c01017806f17bf4529783cad839e674bea0c6c7f410f0"
< strict-transport-security:max-age=31536000
< x-content-type-options:nosniff
< x-frame-options:deny
< x-xss-protection:1; mode=block
< x-github-request-id:8A8A:A04A4:25C08E:2800DA:65D5F6D4
< accept-ranges:bytes
< date:Wed, 21 Feb 2024 13:12:52 GMT
< via:1.1 varnish
< x-served-by:cache-nrt-rjtf7700052-NRT
< x-cache:MISS
< x-cache-hits:0
< x-timer:S1708521172.136084,VS0,VE346
< vary:Authorization,Accept-Encoding,Origin
< access-control-allow-origin:*
< cross-origin-resource-policy:cross-origin
< x-fastly-request-id:6095c88e25f00bd49a75ed81ad68c347c68ae2af
< expires:Wed, 21 Feb 2024 13:17:52 GMT
< source-age:0
< content-length:8844
<
{ [801 bytes data]
* Connection #1 to host raw.githubusercontent.com left intact
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 18.67.51.52...
* Connected to cdn.f5.com (18.67.51.52) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [91 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4202 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*        subject: C=US; ST=Washington; L=Seattle; jurisdictionC=US; jurisdictionST=Washington; O=F5, Inc.; businessCategory=Private Organization; serialNumber=601692492; CN=cdn.f5.com
*        start date: Feb 15 19:13:21 2024 GMT
*        expire date: Mar 15 19:13:20 2025 GMT
*        subjectAltName: cdn.f5.com matched
*        issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
*        SSL certificate verify ok.
} [5 bytes data]
> GET /product/cloudsolutions/f5-bigip-runtime-init/v2.0.1/dist/f5-bigip-runtime-init-2.0.1-1.gz.run HTTP/1.1
> Host: cdn.f5.com
> User-Agent: curl/7.47.1
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 10790502
< Connection: keep-alive
< Date: Wed, 21 Feb 2024 07:48:53 GMT
< x-amz-meta-s3cmd-attrs: md5:1fc0dd0d29cd5c80615e1fbad5bd8e55
< Last-Modified: Thu, 21 Dec 2023 23:51:08 GMT
< ETag: "1fc0dd0d29cd5c80615e1fbad5bd8e55"
< Server: AmazonS3
< X-Cache: Hit from cloudfront
< Via: 1.1 c9d601fa70fd6862de531284734f9f36.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: ICN57-P1
< X-Amz-Cf-Id: Y-wvZS7Ig7D9VuQ4Vvl_1Sc1cMRy_eYlK_8f3Svw0TsQrZpt5r3fQQ==
< Age: 19440
<
{ [15867 bytes data]
100 10.2M  100 10.2M    0     0  3377k      0  0:00:03  0:00:03 --:--:-- 3378k
* Connection #0 to host cdn.f5.com left intact
Verifying archive integrity... All good.
Uncompressing F5 BIGIP Runtime Init installation...................
2024-02-21T05:12:56 - HTTP Retry Settings:
2024-02-21T05:12:56 - RETRY: 3
2024-02-21T05:12:56 - RETRY_MAX_TIME: 180
2024-02-21T05:12:56 - MAX_TIME: 5
2024-02-21T05:12:56 - RETRY_DELAY: 60
2024-02-21T05:12:56 - Running RPM install script.
2024-02-21T05:12:56 - Verifying RPM file integrity...
f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm: OK
2024-02-21T05:12:56 - Verifying signature...
2024-02-21T05:12:56 - GPG PUB Key location: https://f5-cft.s3.amazonaws.com/f5-bigip-runtime-init/gpg.key
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3248  100  3248    0     0   2129      0  0:00:01  0:00:01 --:--:--  2131
f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm: rsa sha1 (md5) pgp md5 OK
2024-02-21T05:12:58 - Checking if package is already installed
2024-02-21T05:12:58 - Package is not installed. Preparing for installation.
2024-02-21T05:12:58 - Install location /tmp/f5-bigip-runtime-init does not exist. Creating install location.
2024-02-21T05:12:58 - Install package f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm
./f5-bigip-runtime-init-azure
........ Removed all between
./f5-bigip-runtime-init-azure/src/version
22072 blocks
2024-02-21T05:13:02 - Getting lastest AT metadata at https://cdn.f5.com/product/cloudsolutions/f5-extension-metadata/latest/metadata.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
Warning: Transient problem: timeout Will retry in 60 seconds. 3 retries left.
100 47763  100 47763    0     0  99872      0 --:--:-- --:--:-- --:--:-- 99713
2024-02-21T05:14:08 - Creating command utility.
2024-02-21T05:14:08 - RPM installation is completed.
2024-02-21T13:14:08.841Z [27155]: info: Configuration file: /config/cloud/runtime-init.conf
2024-02-21T13:14:08.856Z [27155]: info: Processing controls parameters
2024-02-21T13:14:08.860Z [27155]: info: F5 Telemetry is disabled.
2024-02-21T13:14:08.862Z [27155]: info: Validating provided declaration
2024-02-21T13:14:08.951Z [27155]: info: Successfully validated declaration
2024-02-21T13:14:08.954Z [27155]: info: Executing custom pre_onboard_enabled commands
2024-02-21T13:14:08.964Z [27155]: info: Executing inline shell command: /usr/bin/setdb provision.extramb 1000 || exit 0
2024-02-21T13:14:09.095Z [27155]: info: Shell command: /usr/bin/setdb provision.extramb 1000 || exit 0 execution completed; response:
2024-02-21T13:14:09.100Z [27155]: info: Executing inline shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0
2024-02-21T13:14:09.225Z [27155]: info: Shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0 execution completed; response:
2024-02-21T13:14:09.271Z [27155]: info: Resolving parameters
2024-02-21T13:14:09.327Z [27155]: info: Interface:1.1
2024-02-21T13:14:09.328Z [27155]: info: MAC address found for 1.1: 00:17:fa:07:9a:eb
2024-02-21T13:14:09.329Z [27155]: info: Local interface 1 MAC address 0017fa079aeb matches Azure network interface 1 MAC address 0017fa079aeb
2024-02-21T13:14:09.342Z [27155]: info: Interface:1.1
2024-02-21T13:14:09.343Z [27155]: info: MAC address found for 1.1: 00:17:fa:07:9a:eb
2024-02-21T13:14:09.343Z [27155]: info: Local interface 1 MAC address 0017fa079aeb matches Azure network interface 1 MAC address 0017fa079aeb
2024-02-21T13:14:09.353Z [27155]: info: Interface:1.2
2024-02-21T13:14:09.355Z [27155]: info: MAC address found for 1.2: 00:17:fa:07:9f:09
2024-02-21T13:14:09.358Z [27155]: info: Local interface 2 MAC address 0017fa079f09 matches Azure network interface 2 MAC address 0017fa079f09
2024-02-21T13:14:09.363Z [27155]: info: Interface:mgmt
2024-02-21T13:14:09.366Z [27155]: info: MAC address found for mgmt: 00:17:fa:07:9e:df
2024-02-21T13:14:09.368Z [27155]: info: Local interface 0 MAC address 0017fa079edf matches Azure network interface 0 MAC address 0017fa079edf
2024-02-21T13:14:09.722Z [27155]: error: Error getting token 400
2024-02-21T07:20:55.278Z : Starting Custom Script
*   Trying 185.199.109.133...
* Connected to raw.githubusercontent.com (185.199.109.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3045 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*        subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.io
*        start date: Feb 21 00:00:00 2023 GMT
*        expire date: Mar 20 23:59:59 2024 GMT
*        subjectAltName: raw.githubusercontent.com matched
*        issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*        SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* TCP_NODELAY set
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0xd512a0)
} [5 bytes data]
> GET /DahlPatric/f5-azure-china-arm-templates-v2/main/examples/failover/bigip-configurations/runtime-init-conf-3nic-byol-instance02.yaml HTTP/1.1
> Host: raw.githubusercontent.com
> User-Agent: curl/7.47.1
> Accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2.0 200
< cache-control:max-age=300
< content-security-policy:default-src 'none'; style-src 'unsafe-inline'; sandbox
< content-type:text/plain; charset=utf-8
< etag:"247cf257a2d02f04038c01017806f17bf4529783cad839e674bea0c6c7f410f0"
< strict-transport-security:max-age=31536000
< x-content-type-options:nosniff
< x-frame-options:deny
< x-xss-protection:1; mode=block
< x-github-request-id:B0B2:CEDBC:B5173:C3942:65D614D8
< accept-ranges:bytes
< date:Wed, 21 Feb 2024 15:20:56 GMT
< via:1.1 varnish
< x-served-by:cache-nrt-rjtf7700028-NRT
< x-cache:MISS
< x-cache-hits:0
< x-timer:S1708528857.521019,VS0,VE317
< vary:Authorization,Accept-Encoding,Origin
< access-control-allow-origin:*
< cross-origin-resource-policy:cross-origin
< x-fastly-request-id:1d5ce8ce24034976833642cb5d09c87df35ac38a
< expires:Wed, 21 Feb 2024 15:25:56 GMT
< source-age:0
< content-length:8844
<
{ [5 bytes data]
* Connection #0 to host raw.githubusercontent.com left intact
Verifying archive integrity... All good.
Uncompressing F5 BIGIP Runtime Init installation...................
2024-02-21T07:20:57 - HTTP Retry Settings:
2024-02-21T07:20:57 - RETRY: 3
2024-02-21T07:20:57 - RETRY_MAX_TIME: 180
2024-02-21T07:20:57 - MAX_TIME: 5
2024-02-21T07:20:57 - RETRY_DELAY: 60
2024-02-21T07:20:57 - Running RPM install script.
2024-02-21T07:20:57 - Verifying RPM file integrity...
f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm: OK
2024-02-21T07:20:57 - Verifying signature...
2024-02-21T07:20:57 - GPG PUB Key location: https://f5-cft.s3.amazonaws.com/f5-bigip-runtime-init/gpg.key
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3248  100  3248    0     0   2369      0  0:00:01  0:00:01 --:--:--  2369
f5-bigip-runtime-init-azure-2.0.1-1-signed.noarch.rpm: rsa sha1 (md5) pgp md5 OK
2024-02-21T07:20:58 - Checking if package is already installed
2024-02-21T07:20:58 - Package is already installed and utility is created. Exiting with status:0
2024-02-21T15:20:58.896Z [31990]: info: Configuration file: /config/cloud/runtime-init.conf
2024-02-21T15:20:58.908Z [31990]: info: Processing controls parameters
2024-02-21T15:20:58.910Z [31990]: info: F5 Telemetry is disabled.
2024-02-21T15:20:58.911Z [31990]: info: Validating provided declaration
2024-02-21T15:20:58.964Z [31990]: info: Successfully validated declaration
2024-02-21T15:20:58.965Z [31990]: info: Executing custom pre_onboard_enabled commands
2024-02-21T15:20:58.967Z [31990]: info: Executing inline shell command: /usr/bin/setdb provision.extramb 1000 || exit 0
2024-02-21T15:20:59.085Z [31990]: info: Shell command: /usr/bin/setdb provision.extramb 1000 || exit 0 execution completed; response:
2024-02-21T15:20:59.086Z [31990]: info: Executing inline shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0
2024-02-21T15:20:59.206Z [31990]: info: Shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0 execution completed; response:
2024-02-21T15:20:59.262Z [31990]: info: Resolving parameters
2024-02-21T15:20:59.319Z [31990]: info: Interface:1.2
2024-02-21T15:20:59.320Z [31990]: info: MAC address found for 1.2: 00:17:fa:07:9f:09
2024-02-21T15:20:59.321Z [31990]: info: Local interface 2 MAC address 0017fa079f09 matches Azure network interface 2 MAC address 0017fa079f09
2024-02-21T15:20:59.334Z [31990]: info: Interface:1.1
2024-02-21T15:20:59.335Z [31990]: info: MAC address found for 1.1: 00:17:fa:07:9a:eb
2024-02-21T15:20:59.336Z [31990]: info: Local interface 1 MAC address 0017fa079aeb matches Azure network interface 1 MAC address 0017fa079aeb
2024-02-21T15:20:59.344Z [31990]: info: Interface:1.1
2024-02-21T15:20:59.345Z [31990]: info: MAC address found for 1.1: 00:17:fa:07:9a:eb
2024-02-21T15:20:59.345Z [31990]: info: Local interface 1 MAC address 0017fa079aeb matches Azure network interface 1 MAC address 0017fa079aeb
2024-02-21T15:20:59.353Z [31990]: info: Interface:mgmt
2024-02-21T15:20:59.354Z [31990]: info: MAC address found for mgmt: 00:17:fa:07:9e:df
2024-02-21T15:20:59.354Z [31990]: info: Local interface 0 MAC address 0017fa079edf matches Azure network interface 0 MAC address 0017fa079edf
2024-02-21T15:20:59.487Z [31990]: error: Error getting token 400

[azureuser@localhost:NO LICENSE:Standalone] ~ # cat of /var/log/cloud/bigIpRuntimeInit.log

2024-02-21T14:02:47.983Z [29375]: info: Configuration file: /config/cloud/runtime-init-conf.yaml
2024-02-21T14:02:47.987Z [29375]: error: Attempt to load YAML config failed: Error: ENOENT: no such file or directory, open '/config/cloud/runtime-init-conf.yaml'
2024-02-21T14:02:47.989Z [29375]: error: Attempt to load JSON config failed: Error: ENOENT: no such file or directory, open '/config/cloud/runtime-init-conf.yaml'
2024-02-21T14:02:47.989Z [29375]: error: Provided config file is not valid YAML (1.2 spec) or JSON document. See logs for more details.
2024-02-21T14:02:47.991Z [29375]: error: Provided config file is not valid YAML (1.2 spec) or JSON document. See logs for more details.

@mikeshimkus
Copy link
Collaborator

Hi @DahlPatric, the templates don't really support redeployment very well. If you have a failed deployment, the best bet is to delete the previous deployment and try again. In this case the behavior you're seeing is expected; since the first deployment created the key vault and secret, on the second deployment you would need to provide the existing secret ID, not the secret value.

The two logs you provided seem to be from different runs. The runtime init log shows that the config file failed YAML linting.

Can you completely remove any failed deployments you have, then try again and share the results (including the logs and the template parameters you used)?

@DahlPatric
Copy link
Author

Hi!

Removed all objects except from Key vault secret that I'm now reference to in script.

"bigIpPasswordSecretId": {
      "value": "https://f5-kv.vault.azure.cn/secrets/BigIpSecret"
    },
    "bigIpPasswordSecretValue": {
      "value": ""
    },

[azureuser@localhost:NO LICENSE:Standalone] cloud # ls
custom_commands  license_key  runtime-init.conf  secret_id  telemetry_install_params.tmp  vault_url

Re-run script again and it still fails. Same place as before.
Went inside /config/cloud/runtime-init.conf and change to 'silly' debug mode.

run runtime-init manually:
[azureuser@localhost:NO LICENSE:Standalone] cloud # f5-bigip-runtime-init --config-file /config/cloud/runtime-init.conf

[azureuser@localhost:NO LICENSE:Standalone] cloud # cat  bigIpRuntimeInit_silly.log
2024-02-27T09:21:25.113Z [29182]: silly: F5 Telemetry is enabled.
2024-02-27T09:21:25.113Z [29182]: info: Validating provided declaration
2024-02-27T09:21:25.164Z [29182]: info: Successfully validated declaration
2024-02-27T09:21:25.165Z [29182]: info: Executing custom pre_onboard_enabled commands
2024-02-27T09:21:25.167Z [29182]: info: Executing inline shell command: /usr/bin/setdb provision.extramb 1000 || exit 0
2024-02-27T09:21:25.295Z [29182]: info: Shell command: /usr/bin/setdb provision.extramb 1000 || exit 0 execution completed; response:
2024-02-27T09:21:25.297Z [29182]: info: Executing inline shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0
2024-02-27T09:21:25.416Z [29182]: info: Shell command: /usr/bin/setdb provision.restjavad.extramb 1384 || /usr/bin/setdb restjavad.useextramb true || exit 0 execution completed; response:
2024-02-27T09:21:25.456Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:25.456Z [29182]: info: Resolving parameters
2024-02-27T09:21:25.484Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:25.507Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:25.508Z [29182]: silly: Interface Response:  {"code":200,"body":{"kind":"tm:net:interface:interfacecollectionstate","selfLink":"https://localhost/mgmt/tm/net/interface?ver=16.1.4.2","items":[{"kind":"tm:net:interface:interfacestate","name":"1.1","fullPath":"1.1","generation":91,"selfLink":"https://localhost/mgmt/tm/net/interface/1.1?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":48,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:36:1c","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"},{"kind":"tm:net:interface:interfacestate","name":"1.2","fullPath":"1.2","generation":79,"selfLink":"https://localhost/mgmt/tm/net/interface/1.2?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":64,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:ee","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"},{"kind":"tm:net:interface:interfacestate","name":"mgmt","fullPath":"mgmt","generation":73,"selfLink":"https://localhost/mgmt/tm/net/interface/mgmt?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":32,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:6a","mediaActive":"100TX-FD","mediaFixed":"auto","mediaSfp":"auto","mtu":1500,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"}]},"headers":{"connection":"close","date":"Tue, 27 Feb 2024 09:21:25 GMT","content-type":"application/json;charset=utf-8","allow":"","pragma":"no-cache","cache-control":"no-store, no-cache, must-revalidate","expires":"-1","content-length":"2383","server":"Jetty(9.4.49.v20220914)"}}
2024-02-27T09:21:25.509Z [29182]: info: Interface:1.1
2024-02-27T09:21:25.510Z [29182]: silly: filtered:  [{"kind":"tm:net:interface:interfacestate","name":"1.1","fullPath":"1.1","generation":91,"selfLink":"https://localhost/mgmt/tm/net/interface/1.1?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":48,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:36:1c","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"}]
2024-02-27T09:21:25.510Z [29182]: info: MAC address found for 1.1: 00:17:fa:07:36:1c
2024-02-27T09:21:25.510Z [29182]: info: Local interface 1 MAC address 0017fa07361c matches Azure network interface 1 MAC address 0017fa07361c
2024-02-27T09:21:25.514Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:25.514Z [29182]: silly: Interface Response:  {"code":200,"body":{"kind":"tm:net:interface:interfacecollectionstate","selfLink":"https://localhost/mgmt/tm/net/interface?ver=16.1.4.2","items":[{"kind":"tm:net:interface:interfacestate","name":"1.1","fullPath":"1.1","generation":91,"selfLink":"https://localhost/mgmt/tm/net/interface/1.1?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":48,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:36:1c","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"},{"kind":"tm:net:interface:interfacestate","name":"1.2","fullPath":"1.2","generation":79,"selfLink":"https://localhost/mgmt/tm/net/interface/1.2?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":64,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:ee","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"},{"kind":"tm:net:interface:interfacestate","name":"mgmt","fullPath":"mgmt","generation":73,"selfLink":"https://localhost/mgmt/tm/net/interface/mgmt?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":32,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:6a","mediaActive":"100TX-FD","mediaFixed":"auto","mediaSfp":"auto","mtu":1500,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"}]},"headers":{"connection":"close","date":"Tue, 27 Feb 2024 09:21:25 GMT","content-type":"application/json;charset=utf-8","allow":"","pragma":"no-cache","cache-control":"no-store, no-cache, must-revalidate","expires":"-1","content-length":"2383","server":"Jetty(9.4.49.v20220914)"}}
2024-02-27T09:21:25.515Z [29182]: info: Interface:1.2
2024-02-27T09:21:25.516Z [29182]: silly: filtered:  [{"kind":"tm:net:interface:interfacestate","name":"1.2","fullPath":"1.2","generation":79,"selfLink":"https://localhost/mgmt/tm/net/interface/1.2?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":64,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:ee","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"}]
2024-02-27T09:21:25.517Z [29182]: info: MAC address found for 1.2: 00:17:fa:07:3c:ee
2024-02-27T09:21:25.517Z [29182]: info: Local interface 2 MAC address 0017fa073cee matches Azure network interface 2 MAC address 0017fa073cee
2024-02-27T09:21:25.523Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:25.524Z [29182]: silly: Interface Response:  {"code":200,"body":{"kind":"tm:net:interface:interfacecollectionstate","selfLink":"https://localhost/mgmt/tm/net/interface?ver=16.1.4.2","items":[{"kind":"tm:net:interface:interfacestate","name":"1.1","fullPath":"1.1","generation":91,"selfLink":"https://localhost/mgmt/tm/net/interface/1.1?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":48,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:36:1c","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"},{"kind":"tm:net:interface:interfacestate","name":"1.2","fullPath":"1.2","generation":79,"selfLink":"https://localhost/mgmt/tm/net/interface/1.2?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":64,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:ee","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"},{"kind":"tm:net:interface:interfacestate","name":"mgmt","fullPath":"mgmt","generation":73,"selfLink":"https://localhost/mgmt/tm/net/interface/mgmt?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":32,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:6a","mediaActive":"100TX-FD","mediaFixed":"auto","mediaSfp":"auto","mtu":1500,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"}]},"headers":{"connection":"close","date":"Tue, 27 Feb 2024 09:21:25 GMT","content-type":"application/json;charset=utf-8","allow":"","pragma":"no-cache","cache-control":"no-store, no-cache, must-revalidate","expires":"-1","content-length":"2383","server":"Jetty(9.4.49.v20220914)"}}
2024-02-27T09:21:25.524Z [29182]: info: Interface:mgmt
2024-02-27T09:21:25.525Z [29182]: silly: filtered:  [{"kind":"tm:net:interface:interfacestate","name":"mgmt","fullPath":"mgmt","generation":73,"selfLink":"https://localhost/mgmt/tm/net/interface/mgmt?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":32,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:6a","mediaActive":"100TX-FD","mediaFixed":"auto","mediaSfp":"auto","mtu":1500,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"}]
2024-02-27T09:21:25.526Z [29182]: info: MAC address found for mgmt: 00:17:fa:07:3c:6a
2024-02-27T09:21:25.526Z [29182]: info: Local interface 0 MAC address 0017fa073c6a matches Azure network interface 0 MAC address 0017fa073c6a
2024-02-27T09:21:25.527Z [29182]: silly: ipcalc function resolved first element: 10.45.136.65 of provided IPv4 CIDR
2024-02-27T09:21:25.530Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:25.530Z [29182]: silly: Interface Response:  {"code":200,"body":{"kind":"tm:net:interface:interfacecollectionstate","selfLink":"https://localhost/mgmt/tm/net/interface?ver=16.1.4.2","items":[{"kind":"tm:net:interface:interfacestate","name":"1.1","fullPath":"1.1","generation":91,"selfLink":"https://localhost/mgmt/tm/net/interface/1.1?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":48,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:36:1c","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"},{"kind":"tm:net:interface:interfacestate","name":"1.2","fullPath":"1.2","generation":79,"selfLink":"https://localhost/mgmt/tm/net/interface/1.2?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":64,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:ee","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"},{"kind":"tm:net:interface:interfacestate","name":"mgmt","fullPath":"mgmt","generation":73,"selfLink":"https://localhost/mgmt/tm/net/interface/mgmt?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":32,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:3c:6a","mediaActive":"100TX-FD","mediaFixed":"auto","mediaSfp":"auto","mtu":1500,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"}]},"headers":{"connection":"close","date":"Tue, 27 Feb 2024 09:21:25 GMT","content-type":"application/json;charset=utf-8","allow":"","pragma":"no-cache","cache-control":"no-store, no-cache, must-revalidate","expires":"-1","content-length":"2383","server":"Jetty(9.4.49.v20220914)"}}
2024-02-27T09:21:25.531Z [29182]: info: Interface:1.1
2024-02-27T09:21:25.532Z [29182]: silly: filtered:  [{"kind":"tm:net:interface:interfacestate","name":"1.1","fullPath":"1.1","generation":91,"selfLink":"https://localhost/mgmt/tm/net/interface/1.1?ver=16.1.4.2","bundle":"not-supported","bundleSpeed":"not-supported","enabled":true,"flowControl":"tx-rx","forceGigabitFiber":"disabled","forwardErrorCorrection":"not-supported","ifIndex":48,"lacpPortPriority":32786,"linkTrapsEnabled":"true","lldpAdmin":"txonly","lldpTlvmap":130943,"macAddress":"00:17:fa:07:36:1c","mediaActive":"none","mediaFixed":"10000T-FD","mediaMax":"auto","mediaSfp":"auto","mtu":9198,"portFwdMode":"l3","preferPort":"sfp","qinqEthertype":"0x8100","sflow":{"pollInterval":0,"pollIntervalGlobal":"yes"},"stp":"enabled","stpAutoEdgePort":"enabled","stpEdgePort":"true","stpLinkType":"auto"}]
2024-02-27T09:21:25.532Z [29182]: info: MAC address found for 1.1: 00:17:fa:07:36:1c
2024-02-27T09:21:25.532Z [29182]: info: Local interface 1 MAC address 0017fa07361c matches Azure network interface 1 MAC address 0017fa07361c
2024-02-27T09:21:25.533Z [29182]: silly: ipcalc function resolved first element: 10.45.140.1 of provided IPv4 CIDR
2024-02-27T09:21:25.905Z [29182]: silly: Request response: 400 [object Object]
2024-02-27T09:21:25.907Z [29182]: error: Error getting token 400
2024-02-27T09:21:25.907Z [29182]: info: Sending F5 Teem report for failure case.
2024-02-27T09:21:25.925Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.050Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.068Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.082Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.093Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.106Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.121Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.126Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.178Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.191Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.201Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.202Z [29182]: warn: Problem with getting data from /mgmt/tm/sys/license endpoint. Leaving regKey with default value
2024-02-27T09:21:26.202Z [29182]: info: {"id":"4b502bb0-f355-3545-6e6262277659","product":"BIG-IP","cpuCount":8,"diskSize":86016,"memoryInMb":32176,"version":"16.1.4.2","nicCount":3,"platformId":"Z100","hostname":"bigip1","management":"10.45.136.68/26","provisionedModules":{"ltm":"nominal"},"installedPackages":{},"environment":{"pythonVersion":"Python 2.7.5","pythonVersionDetailed":"2.7.5 (default, Dec  1 2023, 09:40:19) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]","nodeVersion":"v6.9.1","libraries":{"ssh":"OpenSSH_7.4p1, OpenSSL 1.0.2u-fips  20 Dec 2019"}}}
2024-02-27T09:21:26.218Z [29182]: silly: Request response: 200 [object Object]
2024-02-27T09:21:26.221Z [29182]: debug: Telemetry Type: f5-bigip-runtime-init-data
2024-02-27T09:21:26.222Z [29182]: debug: Telemetry Version: 1
2024-02-27T09:21:26.222Z [29182]: debug: F5 TEEM Payload: {"platformDetails":{"platform":"BIG-IP","platformVersion":"16.1.4.2","platformId":"Z100","system":{"cpuCount":8,"memory":32176,"diskSize":86016},"nicCount":3,"modules":{"ltm":"nominal"},"packages":{},"environment":{"pythonVersion":"Python 2.7.5","pythonVersionDetailed":"2.7.5 (default, Dec  1 2023, 09:40:19) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]","nodeVersion":"v6.9.1","libraries":{"ssh":"OpenSSH_7.4p1, OpenSSL 1.0.2u-fips  20 Dec 2019"}}},"templateInfo":{"install":"Error getting token 400","templateName":"bigip.json","templateVersion":"v3.0.0.0","nicCount":3,"cloud":"azure","region":"ChinaNorth3","localization":"en-US"},"product":{"version":"2.0.1","locale":"en-US","installDate":"2024-02-27T09:21:26.221Z","installationId":"4a0beedc-db35-4999-b739-fba18178feb9","installedComponents":{"@f5devcentral/f5-teem":"^1.4.6","ajv":"^6.12.2","axios":"^0.21.4","commander":"^4.1.0","form-data":"^4.0.0","get-user-locale":"^1.4.0","https-proxy-agent":"^3.0.1","jmespath":"^0.15.0","js-yaml":"^3.13.1","mustache":"^4.0.0","netmask":"^2.0.2","uuid":"^8.2.0","winston":"^3.3.3","lodash.where":"^3.1.0"}},"operation":{"clientRequestId":"11a65cf5-6e59-45d1-9630-0f5ae9dac09c","rawCommand":"f5-runtime-init -c /config/cloud/runtime-init.conf","pre_onboard_enabled":{"commands":1},"runtime_params":{"secrets":1,"metadata":4},"vaults":{"aws":0,"azure":1,"gcp":0,"hashicorp":0},"userAgent":"f5-bigip-runtime-init/2.0.1","extension_packages":{"do":"1.41.0","as3":"3.48.0","cf":"2.0.2"},"extension_services":{"do":true,"cf":true},"post_onboard_enabled":{"commands":0,"postHooks":0},"result":"FAILURE","resultSummary":"Error getting token 400","startTime":"2024-02-27T09:21:25.113Z","endTime":"2024-02-27T09:21:25.907Z","installParams":[{"key":"templateName","value":"v3.0.0.0/examples/modules/bigip-standalone/bigip.json"}]}}
2024-02-27T09:21:26.476Z [29182]: error: Device is not licensed yet

@mikeshimkus
Copy link
Collaborator

Can you share the full template parameters? Is bigIpPasswordSecretId the secret created from a previous run? What did you use for bigIpUserAssignManagedIdentity?

These templates have not been tested in Azure China, so the best course of action would be to create an RFE for that. I'll ask someone to reach out to you directly for more information.

@G-gonzalezjimenez
Copy link

Hi @DahlPatric, as Mike mentioned the templates are not tested or supported in Azure China. There is no plan to support this region. Let me know if you have any comments or concerns.

@DahlPatric
Copy link
Author

Can you share the full template parameters? Is bigIpPasswordSecretId the secret created from a previous run? What did you use for bigIpUserAssignManagedIdentity?

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "3.0.0.0",
  "parameters": {
    "allowUsageAnalytics": {
      "value": false
    },
    "artifactLocation": {
      "value": "dev2/examples/"
    },
    "bigIpExternalSelfIp01": {
      "value": "10.45.140.11"
    },
    "bigIpExternalSelfIp02": {
      "value": "10.45.140.12"
    },
    "bigIpExternalSubnetId": {
      "value": "/subscriptions//resourceGroups//providers/Microsoft.Network/virtualNetworks/vnet-ingress-cn3/subnets/frontend-subnet"
    },
    "bigIpExternalVip01": {
      "value": "10.45.140.100"
    },
    "bigIpHostname01": {
      "value": "az-cnn3-ext-010.cloud..net"
    },
    "bigIpHostname02": {
      "value": "az-cnn3-ext-011.cloud..net"
    },
    "BigIpImage": {
      "value": "/subscriptions//resourceGroups//providers/Microsoft.Compute/galleries/network_security_compute_gallery/images/f5-network/versions/6.1.4"
    },
    "bigIpInstanceType": {
      "value": "Standard_D8s_v4"
    },
    "bigIpInternalSelfIp01": {
      "value": "10.45.136.11"
    },
    "bigIpInternalSelfIp02": {
      "value": "10.45.136.12"
    },
    "bigIpInternalSubnetId": {
      "value": "/subscriptions//resourceGroups//providers/Microsoft.Network/virtualNetworks/vnet-ingress-cn3/subnets/backend-subnet"
    },
    "bigIpLicenseKey01": {
      "value": "REMOVED"
    },
    "bigIpLicenseKey02": {
      "value": "REMOVED"
    },
    "bigIpMgmtAddress01": {
      "value": "10.45.136.68"
    },
    "bigIpMgmtAddress02": {
      "value": "10.45.136.69"
    },
    "bigIpMgmtSubnetId": {
      "value": "/subscriptions//resourceGroups//providers/Microsoft.Network/virtualNetworks/vnet-ingress-cn3/subnets/management-subnet"
    },
    "bigIpPasswordSecretId": {
      "value": "https://f5-kv.vault.azure.cn/secrets/BigIpSecret"
    },
    "bigIpPasswordSecretValue": {
      "value": ""
    },
    "bigIpPeerAddr": {
      "value": "10.45.140.11"
    },
    "bigIpRuntimeInitConfig01": {
      "value": "https://raw.githubusercontent.com/DahlPatric/f5-azure-china-arm-templates-v2/main/examples/failover/bigip-configurations/runtime-init-conf-3nic-byol-instance01.yaml"
    },
    "bigIpRuntimeInitConfig02": {
      "value": "https://raw.githubusercontent.com/DahlPatric/f5-azure-china-arm-templates-v2/main/examples/failover/bigip-configurations/runtime-init-conf-3nic-byol-instance02.yaml"
    },
    "bigIpRuntimeInitPackageUrl": {
      "value": "https://cdn.f5.com/product/cloudsolutions/f5-bigip-runtime-init/v2.0.1/dist/f5-bigip-runtime-init-2.0.1-1.gz.run"
    },
    "bigIpUserAssignManagedIdentity": {
      "value": ""
    },
    "cfeStorageAccountName": {
      "value": "stcfe"
    },
    "cfeTag": {
      "value": "bigip_high_availability_solution"
    },
    "numNics": {
      "value": 3
    },
    "provisionPublicIpMgmt": {
      "value": false
    },
    "provisionServicePublicIp": {
      "value": false
    },
    "restrictedSrcAddressApp": {
      "value": "0.0.0.0/0"
    },
    "restrictedSrcAddressMgmt": {
      "value": "10.0.0.0/8"
    },
    "sshKey": {
      "value": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0kPei1fGudsI71Rt7ZZlSVjGgKLovfNP96mo/t7AxHJ/P0C2TKr5iPMITx4xPx9ehi+IjyDL0CKoiCNZxTgcqN72ccd4mRt0Tvu+cMoAshm8wIrR4HDYRuA4R/CPDe6HZ58VJad46vx3MpqsX+c6OcFDaCJIyNUNjFwc2d8KDb1n1FqTzIJQvVYnGExIw9Xrb/qM5Lj5ZeIAPfzD1Csa6EQk+ubksYMQTCKhbpIUzTPzKtwXLBj/4gkZsUl5HmV3ql/cnK4Ghk= generated-by-azure"
	  },
    "tagValues": {
      "value": {
        "application": "BigIP",
        "cost": "F5",
        "environment": "Production",
        "group": "Network-security",
        "owner": "Network-security"
      }
    },
    "templateBaseUrl": {
      "value": "https://raw.githubusercontent.com/DahlPatric/f5-azure-china-arm-templates-v2/"
    },
    "uniqueString": {
      "value": "f5"
    },
    "useAvailabilityZones": {
      "value": true
    }
  }
}

@G-gonzalezjimenez
One of F5 SME in Sweden did try to use templates in West Europa and suffer same issue, that why I'm not thinking this actually is related to Azure China. But one difference was that one of his node actually got configured with bigip-runtime and the other not. My speculation is that there is something during the fetching/configuring of credentials, and that result in not be able to access license endpoint.

These templates have not been tested in Azure China, so the best course of action would be to create an RFE for that. I'll ask someone to reach out to you directly for more information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mikeshimkus @G-gonzalezjimenez @DahlPatric