Failure to connect to Kinesis: Failed to init kvs producer. Error: Unable to create Rotating Credential provider. #976
-
|
I get this log after testing the container locally and publishing it to an ECS cluster. Without Credentials: Failed to init kvs producer. Error: Unable to create Rotating Credential provider. Error status: 0x15000022With Credentials: describeStreamCurlHandler(): DescribeStream API response: {"message":"The security token included in the request is invalid."}ThoughtsIt appears that even though there is an IAM role attached to the task definition, that the contianer needs generated credentials. I don't necessarily understand why that is the case... but that is what it seems. What I have triedI have tried the following:
ENV AWS_ACCESS_KEY_ID=
ENV AWS_SECRET_ACCESS_KEY=
ENV AWS_SESSION_TOKEN=try:
response = boto3.session.Session().get_credentials()
# Get the temporary credentials
credentials = response
os.environ["AWS_ACCESS_KEY_ID"] = credentials.access_key
os.environ["AWS_SECRET_ACCESS_KEY"] = credentials.secret_key
os.environ["AWS_SESSION_TOKEN"] = credentials.token
print("Credentials Set.")
except Exception as E:
raise ValueError(f"Token Generation Failed: {E}")Any thoughts would be appreciated. I am not sure why tokens would be stale since the code is grabbing credentials from the IAM role attached to the ECS resource itself. I would suggest that these environment variables are actually optional and that an IAM role itself would just work Tagging @niyatim23 for visibility - you helped me work through my previous issue! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 15 replies
-
|
Hi @Alex-Wenner-FHR, can you please share the permissions you are giving to the IAM role? The error code |
Beta Was this translation helpful? Give feedback.
-
|
The main difference here, is that locally I am setting these keys statically: ENV AWS_ACCESS_KEY_ID=<ACCESS>
ENV AWS_SECRET_ACCESS_KEY=<SECRET>
ENV AWS_SESSION_TOKEN=<SESSION>as where in the cloud, this is not ideal. we are trying to do so dynamically based off the role attached to the resources itself so the dockerfile is like seen above ENV AWS_ACCESS_KEY_ID=
ENV AWS_SECRET_ACCESS_KEY=
ENV AWS_SESSION_TOKEN=and then set by this script, on container boot (prior to the gstreamer command) try:
response = boto3.session.Session().get_credentials()
# Get the temporary credentials
credentials = response
os.environ["AWS_ACCESS_KEY_ID"] = credentials.access_key
os.environ["AWS_SECRET_ACCESS_KEY"] = credentials.secret_key
os.environ["AWS_SESSION_TOKEN"] = credentials.token
print("Credentials Set.")
except Exception as E:
raise ValueError(f"Token Generation Failed: {E}") |
Beta Was this translation helpful? Give feedback.
-
|
By credential provider, I mean any of these. When do these credentials expire? Does the SDK upload to KVS until the credentials expire? |
Beta Was this translation helpful? Give feedback.
-
|
I mean I guess, static credential provider. No, it does not upload anything at all... That is why I am confused. Just to unpack this a bit further, why is a credential provider needed if this container is deployed on an AWS resource that has an IAM role attached to it, that grants all necessary permissions? |
Beta Was this translation helpful? Give feedback.
-
|
It feels like a credential provider would be needed assuming this container is not directly in contact with an IAM role. Like locally. If I were to run this container locally it makes sense to me to have credentials generated and share them to the container. But I am very confused as to why you have to have credentials while running in the cloud, with a grantable IAM role attached. Can you explain this to me? Maybe I am just misunderstanding this process and that is what is happening. |
Beta Was this translation helpful? Give feedback.
-
|
My biggest caveat here, I cannot hardcode AWS Keys & Secrets as ENV variables in the dockerfile. The py script above is me trying to grab them and set them discreetly. Any thoughts would be much appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
The problem doesnt seem to be with the credentials to start with. Based on the status code, it seems that the SDK is unable to access the file to read the credentials from. Ensure the file has read permissions for the SDK to parse the file for credentials. |
Beta Was this translation helpful? Give feedback.
-
|
@disa6302 thanks for the follow up! There is no credentials file. I am making use of the AWS env vars which is not working... But my big question, is if this container is deployed to an AWS resource that has an IAM role attached, with all the access it needs why is it needed to have Access Key, Secret Key, and Session Token? Wouldn't the IAM role be able to facilitate the federation for that resource? |
Beta Was this translation helpful? Give feedback.
-
|
The SDK supports 3 modes of credential providers at the moment:
If you are running it within ECS, I am guessing the credentials are being vended using AWS_CONTAINER_CREDENTIALS_FULL_URI. This is not supported in the SDK at the moment. So there needs to be an out of band way to vend these credentials to the SDK. Let me know if I have misunderstood your setup and I can clarify better. |
Beta Was this translation helpful? Give feedback.
-
|
@disa6302 okay got it. This is helpful... after some internal conversations, I think we can make option 1 work through static credentials. Is it on your team's roadmap to make this container/sdk work on other platforms such as ECS, EC2, or Lambda through the use of assuming IAM roles? It seems like this would be the most ideal approach to granting said container access. |
Beta Was this translation helpful? Give feedback.
-
|
We are looking to incorporate it, but no timelines yet. I will update this discussion once we have some update. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, thank you! Fantastic! I apologize... I kind of came into this issue/discussion assuming that IAM roles just should work. I will mark this as answered. Thanks for the help @disa6302 & @niyatim23 |
Beta Was this translation helpful? Give feedback.
@Alex-Wenner-FHR ,
The problem doesnt seem to be with the credentials to start with. Based on the status code, it seems that the SDK is unable to access the file to read the credentials from. Ensure the file has read permissions for the SDK to parse the file for credentials.