I'm trying to collect logs from SQS queue and foreword it to to splunk. When I did that using KEY ID and Private Key it worked without any issues. I now need to achieve that using the assume role feature of Vector.
[sources.demo]
type = "aws_sqs"
queue_url = "https://queue.amazonaws.com/accountID/QueueName"
auth.assume_role ="arn:aws:iam::AccountID:role/RoleName"
[sinks.tosplunk]
type = "splunk_hec_logs"
inputs = [ "demo" ]
endpoint = "http://172.1.20.147:8088"
default_token="32fff858-83b8-436a-89b2-8c4384071437"
encoding.codec = "json"
When running the Vector I receive the following:
2023-09-06T14:18:21.437145Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-09-06T14:18:21.437189Z WARN vector::app: DEPRECATED The openssl legacy provider provides algorithms and key sizes no longer recommended for use. Set `--openssl-legacy-provider=false` or `VECTOR_OPENSSL_LEGACY_PROVIDER=false` to disable. See https://vector.dev/highlights/2023-08-15-0-32-0-upgrade-guide/#legacy-openssl for details.
2023-09-06T14:18:21.437846Z INFO vector::app: Loaded openssl provider. provider="legacy"
2023-09-06T14:18:21.438050Z INFO vector::app: Loaded openssl provider. provider="default"
2023-09-06T14:18:21.438540Z INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/aws/auth.rs:256:60
I have enabled backtrace that returned the following:
0: 0x55db8f7602b4 - std::backtrace_rs::backtrace::libunwind::trace::h782cc21a5acaf6cb
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x55db8f7602b4 - std::backtrace_rs::backtrace::trace_unsynchronized::hc579eb24ab204515
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x55db8f7602b4 - std::sys_common::backtrace::_print_fmt::h7223525cfdbacda2
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:65:5
3: 0x55db8f7602b4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbd7d55b7108d2ab8
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:44:22
4: 0x55db8ede713f - core::fmt::rt::Argument::fmt::hb4f4a02b9bd9dd49
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/fmt/rt.rs:138:9
5: 0x55db8ede713f - core::fmt::write::h6d54cd7c9e155ec5
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/fmt/mod.rs:1094:21
6: 0x55db8f72ab40 - std::io::Write::write_fmt::h6a453a71c692f63b
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/io/mod.rs:1713:15
7: 0x55db8f764d6f - std::sys_common::backtrace::_print::h1cbaa8b42678f928
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:47:5
8: 0x55db8f764d6f - std::sys_common::backtrace::print::h4ddf81241a51b337
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:34:9
9: 0x55db8f7645a7 - std::panicking::default_hook::{{closure}}::hff91f1f484ade5cd
10: 0x55db8f765523 - std::panicking::default_hook::h21f14afd59f7aef9
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:288:9
11: 0x55db8f765523 - std::panicking::rust_panic_with_hook::h45f66047b14c555c
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:705:13
12: 0x55db8f764fd3 - std::panicking::begin_panic_handler::{{closure}}::h49d1a88ef0908eb4
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:595:13
13: 0x55db8f764f66 - std::sys_common::backtrace::__rust_end_short_backtrace::hccebf9e57f8cc425
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:151:18
14: 0x55db8f764f51 - rust_begin_unwind
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:593:5
15: 0x55db8e72efb2 - core::panicking::panic_fmt::h54ec9d0e3180a83d
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:67:14
16: 0x55db8e72f182 - core::panicking::panic::h65157a6ac7f1357a
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:117:5
17: 0x55db8fb6e28a - vector::aws::auth::AwsAuthentication::credentials_provider::{{closure}}::h7ca136e78d5a7c77
18: 0x55db8fd9d041 - vector::aws::create_client::{{closure}}::h878e52754b2ef708
19: 0x55db90829ecc - <vector::sources::aws_sqs::config::AwsSqsConfig as vector::config::source::SourceConfig>::build::{{closure}}::had640ee706597d05
20: 0x55db91b2976a - vector::topology::builder::Builder::build_sources::{{closure}}::h4d306bad405738cc
21: 0x55db91b218b0 - vector::topology::builder::build_pieces::{{closure}}::hb29330fad6ba6122
22: 0x55db91af6748 - vector::topology::build_or_log_errors::{{closure}}::h2a2386d28b9e0dec
23: 0x55db91bd5ba4 - vector::app::ApplicationConfig::from_opts::{{closure}}::h89c8e606291e878d
24: 0x55db91bcad01 - vector::app::Application::prepare_from_opts::h316d064e2cbccb73
25: 0x55db91ac7d7c - vector::app::Application::run::h0b91824e891723ed
26: 0x55db8e8213f9 - vector::main::h265a7f2e6229448d
27: 0x55db8e820023 - std::sys_common::backtrace::__rust_begin_short_backtrace::h697adb6bc2265b38
28: 0x55db8e821f9b - main
29: 0x7f4b88bd613a - __libc_start_main
30: 0x55db8e81ff2a - <unknown>
31: 0x0 - <unknown>
I have tried adding:
auth.region= "us-east-1"
But it didn't change anything.
I have made sure the role that is assigned to the EC2 has permission to assume the IAM role I put in, I have also queried the AWS metadata service just to make sure that the correct role is assigned successfully.
When I remove the line of auth.assume_role it stop crashing and just returned with a generic error about not being able to retrieve the logs.
I believe that the issue is with the auth.assume_role line but I can't put my finger on what is the exact issue.
Try to upgrade vector to version 0.33.0. There were some fixes in auth part where you have the error: https://github.com/vectordotdev/vector/commit/fd0ccd558715fc1e964755d2ebf7c9ca9fa1d7ee