Secure OCI Vault Integration for Fluent Bit

Securing sensitive credentials in modern cloud infrastructures is a top priority for system administrators and DevOps engineers. In this guide, we will walk through a production-ready integration of Oracle Cloud Infrastructure (OCI) Vault with Fluent Bit to securely handle streaming credentials. By leveraging Instance Principals, we can completely eliminate the need for hardcoded API keys, configuration files with plaintext secrets, or static passwords residing on persistent disk storage.

1. OCI Identity Setup (The Foundation)

To allow the Compute instance to communicate with the Vault service securely without local credentials, we configure the OCI Identity and Access Management (IAM) layer.

Step 1: Create a Dynamic Group

• Name: LogShipperGroup
• Rule: Any {instance.id = 'ocid1.instance.oc1.me-dubai-1.amaaaaaaa7exampleid'}

Step 2: Create an IAM Policy

• Statement: Allow dynamic-group LogShipperGroup to read secret-bundles in compartment Your_Production_Compartment

2. The Secure Wrapper Script

Install the OCI CLI to facilitate the secret retrieval process on the host instance.

sudo yum install python3-pip
sudo pip3 install oci-cli

The following wrapper script fetches the secret directly into memory (RAM) and immediately launches the Fluent Bit process, ensuring the password never touches the persistent storage in plain text.

1. Create the script file: sudo nano /usr/local/bin/fluent-bit-wrapper.sh
2. Paste the configuration:

#!/bin/bash
# Fetching secret from OCI Vault using Instance Principal
KAFKA_PWD=$(/usr/local/bin/oci secrets secret-bundle get --auth instance_principal \
--secret-id ocid1.vaultsecret.oc1.me-dubai-1.amaaaaaaaxxxxxxxxxxx \
--query "data.\"secret-bundle-content\".content" --raw-output | /usr/bin/base64 -d)

# Safety check: if the fetch failed, stop execution
if [ -z "$KAFKA_PWD" ]; then
    echo "Error: KAFKA_PWD is empty. OCI fetch failed."
    exit 1
fi

export KAFKA_PWD

# Start Fluent Bit and replace the shell process
exec /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf

3. Set executable permissions: sudo chmod +x /usr/local/bin/fluent-bit-wrapper.sh

3. Fluent Bit Configuration

Update your main configuration file to reference the inline ${KAFKA_PWD} environment variable, which will be populated dynamically from the wrapper script at execution time.

File path: /etc/fluent-bit/fluent-bit.conf

[SERVICE]
    Flush        1
    Daemon        Off
    Log_Level     info
    HTTP_Server   On
    HTTP_Port     2020

[INPUT]
    Name                 kafka
    Tag                  oci_logs
    Brokers              c3lqj3t6cjoa.streaming.me-dubai-1.oci.oraclecloud.com:9092
    Topics               LexPrivateStream
    Group_Id             siem-tester-v3
    Format               json
    rdkafka.security.protocol SASL_SSL
    rdkafka.sasl.mechanism    PLAIN
    rdkafka.sasl.username     <tenancy-name>/<user-name>/ocid1.streampool.oc1.me-dubai-1.amaaaaaaaxxxxxx
    rdkafka.sasl.password    ${KAFKA_PWD}
    rdkafka.auto.offset.reset  smallest

[FILTER]
    Name         nest
    Match        oci_logs
    Operation    lift
    Nested_under payload

[OUTPUT]
    Name                syslog
    Match               oci_logs
    Host                <siem-ip>
    Port                514
    Mode                tcp
    Syslog_Format       rfc5424
    syslog_appname_key  type
    syslog_hostname_key source
    syslog_message_key  data

4. Systemd Service Override

Direct Systemd to utilize the secure wrapper script instead of executing the default binary directly.

1. Open Systemd override editor: sudo systemctl edit fluent-bit
2. Paste the following configuration:

[Service]
# Clear the existing ExecStart
ExecStart=
# Run our secure wrapper
ExecStart=/usr/local/bin/fluent-bit-wrapper.sh

# Security Hardening
ProtectSystem=full
NoNewPrivileges=yes

5. Deployment & Verification

Apply the configuration changes, restart the service, and verify that the environment variable is correctly injected into the process memory space.

# 1. Reload Systemd configuration
sudo systemctl daemon-reload

# 2. Reset any previous failure counts
sudo systemctl reset-failed fluent-bit

# 3. Start the service
sudo systemctl restart fluent-bit

# 4. Final verification of the environment variable in the running process
sudo strings /proc/$(pgrep fluent-bit)/environ | grep KAFKA_PWD

# 5. Check metrics for data ingestion status
curl -s http://localhost:2020/api/v1/metrics | jq '.input."kafka.0"'

By using this approach, your infrastructure stays secure and compliant. Passwords and credentials never touch physical disk drives in plaintext, reducing the risk of accidental exposure during system diagnostics or unauthorized access to configuration backups.