AWS credential load error #2893

Zan-L · 2024-09-22T05:35:22Z

Environment

Delta-rs version: 0.20.0

Binding: Python

Environment:

Cloud provider: AWS
OS: Linux
Other:

Bug

What happened: Calling DeltaTable(path) results in OSError: Operation not supported: an error occurred while loading credentials

What you expected to happen: The same code runs without problems in 0.19.2

How to reproduce it:

More details:
File "/usr/local/lib/python3.12/site-packages/deltalake/table.py", line 412, in init
self._table = RawDeltaTable(
^^^^^^^^^^^^^^

The text was updated successfully, but these errors were encountered:

ion-elgreco · 2024-09-22T06:01:54Z

You gotta give more info on the actual type of credentials you are passing..

rtyler · 2024-09-22T12:51:51Z

Yes please more details! I did some refactoring around AWS credential loading in 0.20.0 to fix a number of long-stanging bugs and utilize the AWS SDK for Rust more effectively, I'm quite curious what the problem here might me 😄

Zan-L · 2024-09-22T14:01:31Z

Sorry I really couldn't provide anything more - I never need to pass any credentials because I only write to the same AWS account as the ECS tasks that the script runs on.

Zan-L · 2024-09-22T14:47:01Z

Some other info that may help:

path is a Path object (upath.UPath)
I just did a test in my local (Windows) with AWS environment variables (AWS_ACCESS_KEY_ID, etc.) and it worked well with both str path and UPath path.
In the ECS script, I did write_deltalake() before DeltaTable(path) and somehow it still worked. Maybe the difference between how these two functions retrieve credentials can help narrow down the cause.
Instead of having environment variables like AWS_ACCESS_KEY_ID, ECS tasks got AWS_CONTAINER_CREDENTIALS_FULL_URI (in the case of Fargate) instead and AWS SDK are configured to automatically query this endpoint to obtain credentials.

whistlebee · 2024-09-25T02:08:09Z

I'm facing a similar issue also running on ECS. Writing to S3 with the DynamoDB locking provider. I see this from the logs:

[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))

jcran · 2024-09-30T01:14:21Z

Also hit this today while running 0.20.0 on ECS. Specifically "OSError: Operation not supported: an error occurred while loading credentials"

Code that triggers the error:

delta_table = DeltaTable(full_table_path, storage_options=delta_s3_options())

here's the ECS task role permissions

s3 (purposefully broad for now):

		{
			"Action": [
				"s3:*"
			],
			"Effect": "Allow",
			"Resource": "*"
		},

dynamodb:

		{
			"Action": [
				"dynamodb:GetItem",
				"dynamodb:Query",
				"dynamodb:PutItem",
				"dynamodb:UpdateItem",
				"dynamodb:DeleteItem"
			],
			"Effect": "Allow",
			"Resource": "*"
		},

passing these storage_options in prod.

            "AWS_REGION": AWS_REGION,
            "AWS_S3_LOCKING_PROVIDER": "dynamodb",
            "DELTA_DYNAMO_TABLE_NAME": DYNAMODB_DELTA_LOCK_TABLE

Note that i don't hit this in dev, using storage_options:

            "AWS_ACCESS_KEY_ID": MINIO_USER,
            "AWS_SECRET_ACCESS_KEY": MINIO_PASS,
            "AWS_S3_ALLOW_UNSAFE_RENAME": "true",
            "endpoint": S3_URL,
            "AWS_REGION": "",
            "allow_http": "true"

UPDATE: Confirmed still an issue in 0.20.1, and no issue in 0.19.2

hbaslana · 2024-10-07T09:16:15Z

I am experiencing similar issues with Delta Lake when upgrading from version 0.19.2 to 0.20.0 or higher. The configuration that worked perfectly in version 0.19.2 now fails with the following error messages:

Error Messages

[2024-10-07T07:16:18Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
[2024-10-07T07:16:19Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
[2024-10-07T07:16:20Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
[2024-10-07T07:16:21Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
[2024-10-07T07:16:21Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Host is down (os error 64): tcp connect error: Host is down (os error 64): Host is down (os error 64) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 64, kind: Uncategorized, message: "Host is down" })), connection: Unknown } }) }))
[2024-10-07T07:16:21Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Host is down (os error 64): tcp connect error: Host is down (os error 64): Host is down (os error 64) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 64, kind: Uncategorized, message: "Host is down" })), connection: Unknown } }) }))
python3.11/site-packages/deltalake/table.py", line 412, in __init__
    self._table = RawDeltaTable(
                  ^^^^^^^^^^^^^^
OSError: Operation not supported: the credential provider was not enabled
Configuration

Below is the configuration that works for version 0.19.2 but fails for version 0.20.0 and above:

{
  "access_key_id": "ACCESS_KEY",
  "secret_access_key": "SECRET_KEY",
  "endpoint_url": "PURE_STORAGE_S3_ENDPOINT",
  "region": "us-east-1",
  "bucket_name": "MY_BUCKET",
  "default_region": "us-east-1",
  "allow_http": "true",
  "aws_s3_allow_unsafe_rename": "true",
  "copy_if_not_exists": "header: cf-copy-destination-if-none-match: *"
}

Steps to Reproduce
Use the above configuration with Delta Lake version 0.19.2. It works without any issues.
Upgrade to Delta Lake version 0.20.0 or higher.
Attempt to use the same configuration, which results in the errors mentioned above.
Expected Behavior
The configuration should work seamlessly with Delta Lake version 0.20.0 and above, as it does with version 0.19.2.

Actual Behavior
The system fails to load the region from IMDS and throws multiple timeout and connection errors, eventually leading to an OSError indicating that the credential provider was not enabled.

Additional Information
Python Version: 3.11
Delta Lake Version: 0.20.0 and above
Storage Backend: Pure Storage FlashBlade - S3

Updated

Workaround:
I found a workaround for this issue by providing AWS credentials and configuration via environment variables. This approach allows me to bypass the IMDS connection problems. Here’s how I set the environment variables:

import os

os.environ['AWS_ACCESS_KEY_ID'] = 'ACCESS_KEY'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'SECRET_KEY'
os.environ['AWS_REGION'] = 'us-east-1'
os.environ['AWS_ENDPOINT_URL'] = 'PURE_STORAGE_S3_ENDPOINT'
write_deltalake(...)

After setting these environment variables, I can successfully use Delta Lake version 0.20.0 and 0.20.1 without encountering the previous errors.

rtyler · 2024-10-07T21:08:03Z

Thanks for the thorough report @hbaslana ! I believe that we may have a fix for this coming from #2924 which I merged earlier today.

If you're up for re-doing your test with this Python wheel from CI: deltalake-0.20.2%2Bf9436cba-cp38-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl that would be very helpful (you can also just build main too)

hbaslana · 2024-10-08T04:58:03Z

Thanks for the thorough report @hbaslana ! I believe that we may have a fix for this coming from #2924 which I merged earlier today.

If you're up for re-doing your test with this Python wheel from CI: deltalake-0.20.2%2Bf9436cba-cp38-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl that would be very helpful (you can also just build main too)

Thanks for the update. However, the same behavior persists. I found out that when I supply AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY through environment variables, it successfully writes, albeit with warnings. It seems to get other information, such as ENDPOINT_URL, from storage_options, but the key/secret is not loading.
[2024-10-08T04:50:17Z WARN aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))

The warning disappears when the environment variable is provided:

os.environ['AWS_REGION'] = 'us-east-1'

rtyler · 2024-10-08T13:18:28Z

Running through a manual test case with RUST_LOG=debug in the environment I'm getting the following printed out:

[2024-10-08T13:15:37Z DEBUG deltalake_aws] S3LogStoreFactory has been asked to create a LogStore where the underlying store has copy-if-not-exists enabled - no locking provider required
[2024-10-08T13:15:37Z DEBUG aws_config::meta::credentials::chain] load_credentials; provider=StorageOptions
[2024-10-08T13:15:37Z DEBUG deltalake_aws::credentials] Attempting to pull credentials from `StorageOptions`
[2024-10-08T13:15:37Z DEBUG aws_config::meta::credentials::chain] provider in chain did not provide credentials provider=StorageOptions context=the credential provider was not enabled: access key not in StorageOptions (CredentialsNotLoaded
(CredentialsNotLoaded { source: Some("access key not in StorageOptions") }))

This is with storage_options set to:

{'access_key_id': 'deltalake', 'secret_access_key': 'weloverust', 'endpoint_url': 'https://localhost:4566', 'region': 'us-east-1', 'bucket_name': 'MY_BUCKET', 'default_region': 'us-east-1', 'allow_http': 'true', 'aws_s3_allow_unsafe_rename': 'true', 'copy_if_not_exists': 'header: cf-copy-destination-if-none-match: *', 'access_key': 'deltalake'}

which is running against our localstack defined in docker-compose.yml

I'm curious if you're seeing something similar @hbaslana , either way, I'm going to see if I can nail down where this hiccup might be happening in the main branch 🕵️

This change addresses some of the problems tacked onto delta-io#2893 but does not address the concern with ECS specifically. It does however improve the handling of `storage_options` since the improved credential provider code was introduced

rtyler · 2024-10-08T14:29:25Z

This error from @whistlebee is interesting!

I'm facing a similar issue also running on ECS. Writing to S3 with the DynamoDB locking provider. I see this from the logs:

[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))

The endpoint that ECS is configured to use is rate-limiting (HTTP 429) the requests for credentials, which is very interesting because the code that is doing that is ... provided by AWS 😆

For this issue and the details provided by @jcran I am going to have to do some hacking to get a test set up working correctly since we're not regularly able to run CI with ECS-based containers 🤔 🕵️

That said, I have a build brewing for @hbaslana to test which I believe addresses their issue (see also #2931)

This change addresses some of the problems tacked onto #2893 but does not address the concern with ECS specifically. It does however improve the handling of `storage_options` since the improved credential provider code was introduced

hbaslana · 2024-10-09T05:17:50Z

This error from @whistlebee is interesting!

I'm facing a similar issue also running on ECS. Writing to S3 with the DynamoDB locking provider. I see this from the logs:

[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=EcsContainer error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))
[2024-09-24T16:18:27Z WARN  aws_config::meta::credentials::chain] provider failed to provide credentials provider=DefaultChain error=an error occurred while loading credentials: Non-success status from HTTP credential provider: StatusCode(429) (ProviderError(ProviderError { source: "Non-success status from HTTP credential provider: StatusCode(429)" }))

The endpoint that ECS is configured to use is rate-limiting (HTTP 429) the requests for credentials, which is very interesting because the code that is doing that is ... provided by AWS 😆

For this issue and the details provided by @jcran I am going to have to do some hacking to get a test set up working correctly since we're not regularly able to run CI with ECS-based containers 🤔 🕵️

That said, I have a build brewing for @hbaslana to test which I believe addresses their issue (see also #2931)

Great work!! @rtyler Latest #2931 resolved write issue, now it keeps credentials from storage options however still not getting region, what's the correct way to provide region through storage_options ? maybe it is key issue similar to credentials.
os.environ['AWS_REGION'] = 'us-east-1' <- resolves warnings

hbaslana · 2024-10-09T07:01:20Z

what about setting region similar to your approach
aws/src/storage.rs

impl S3ObjectStoreFactory {
    fn with_env_s3(&self, options: &StorageOptions) -> StorageOptions {
        let mut options = StorageOptions(
            options
                .0
                .clone()
                .into_iter()
                .map(|(k, v)| {
                    if let Ok(config_key) = AmazonS3ConfigKey::from_str(&k.to_ascii_lowercase()) {
                        (config_key.as_ref().to_string(), v)
                    } else {
                        (k, v)
                    }
                })
                .collect(),
        );

        for (os_key, os_value) in std::env::vars_os() {
            if let (Some(key), Some(value)) = (os_key.to_str(), os_value.to_str()) {
                if let Ok(config_key) = AmazonS3ConfigKey::from_str(&key.to_ascii_lowercase()) {
                    if !options.0.contains_key(config_key.as_ref()) {
                        options
                            .0
                            .insert(config_key.as_ref().to_string(), value.to_string());
                    }
                }
            }
        }

        let config_keys: HashMap<AmazonS3ConfigKey, String> =
            HashMap::from_iter(options.0.iter().filter_map(|(k, v)| {
                match AmazonS3ConfigKey::from_str(&k.to_lowercase()) {
                    Ok(k) => Some((k, v.into())),
                    Err(_) => None,
                }
            }));

        if std::env::var(constants::AWS_REGION).is_err() {
            if let Some(region) = config_keys.get(&AmazonS3ConfigKey::Region).cloned() {
                std::env::set_var(constants::AWS_REGION, region);
            }
        }
        
        options
    }
}

Zan-L added the bug Something isn't working label Sep 22, 2024

rtyler added the binding/python Issues for the Python package label Sep 22, 2024

rtyler self-assigned this Sep 22, 2024

rtyler added the storage/aws AWS S3 storage related label Sep 22, 2024

rtyler mentioned this issue Oct 8, 2024

fix: properly handle the different flavors of storage options keys #2931

Merged

rtyler added this to the Rust v1.0.0 milestone Oct 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS credential load error #2893

AWS credential load error #2893

Zan-L commented Sep 22, 2024

ion-elgreco commented Sep 22, 2024

rtyler commented Sep 22, 2024

Zan-L commented Sep 22, 2024

Zan-L commented Sep 22, 2024 •

edited

Loading

whistlebee commented Sep 25, 2024 •

edited

Loading

jcran commented Sep 30, 2024 •

edited

Loading

hbaslana commented Oct 7, 2024 •

edited

Loading

rtyler commented Oct 7, 2024

hbaslana commented Oct 8, 2024

rtyler commented Oct 8, 2024

rtyler commented Oct 8, 2024

hbaslana commented Oct 9, 2024 •

edited

Loading

hbaslana commented Oct 9, 2024

AWS credential load error #2893

AWS credential load error #2893

Comments

Zan-L commented Sep 22, 2024

Environment

Bug

ion-elgreco commented Sep 22, 2024

rtyler commented Sep 22, 2024

Zan-L commented Sep 22, 2024

Zan-L commented Sep 22, 2024 • edited Loading

whistlebee commented Sep 25, 2024 • edited Loading

jcran commented Sep 30, 2024 • edited Loading

hbaslana commented Oct 7, 2024 • edited Loading

Updated

rtyler commented Oct 7, 2024

hbaslana commented Oct 8, 2024

rtyler commented Oct 8, 2024

rtyler commented Oct 8, 2024

hbaslana commented Oct 9, 2024 • edited Loading

hbaslana commented Oct 9, 2024

Zan-L commented Sep 22, 2024 •

edited

Loading

whistlebee commented Sep 25, 2024 •

edited

Loading

jcran commented Sep 30, 2024 •

edited

Loading

hbaslana commented Oct 7, 2024 •

edited

Loading

hbaslana commented Oct 9, 2024 •

edited

Loading