OCI Streaming Service – Kafka Client Behavior

Oracle Cloud Infrastructure (OCI) Streaming Service provides an enterprise-grade, real-time message streaming platform. Because it offers Kafka-compatible endpoints, developers often expect standard open-source Kafka clients to work seamlessly out of the box. However, depending on whether you route traffic through a public or private virtual cloud network (VCN) endpoint, client behavior can vary significantly. This guide explores the compatibility differences between Python and Java Kafka clients when interacting with OCI Streaming endpoints.

Overview

Oracle OCI Streaming Service provides Kafka-compatible endpoints that can be accessed via two primary methods:

• Public Endpoint
• Private Endpoint (VCN)

While OCI Streaming is Kafka-API compatible, client behavior differs depending on endpoint type and client implementation. This document records tested behavior, working approaches, and known limitations based on real troubleshooting scenarios.

Summary of Findings

Key Conclusion

OCI Streaming Private Endpoints work reliably with Java Kafka clients. Python (librdkafka-based) clients work on Public Endpoints but are unreliable on Private Endpoints.

This behavior is not a network, DNS, NSG, or authentication issue. It is a client compatibility limitation inherent to how the underlying libraries handle connection pooling and protocol negotiations.

Environment Assumptions

• OCI Streaming Service (Kafka-compatible)
• Stream Pool configured with:
  • Public Endpoint or
  • Private Endpoint in VCN
• VM running inside OCI
• Valid OCI Auth Token
• Target Stream already exists

Python Client – Public Endpoint (Works)

Use Case

• Testing
• Development
• Non-private access
• When Public Endpoint usage is acceptable under your security policy

Requirements

pip install confluent-kafka certifi

Python Consumer Example (Public Endpoint)

from confluent_kafka import Consumer
import certifi

conf = {
    'bootstrap.servers': 'STREAM_POOL_PUBLIC_ENDPOINT:9092',

    'security.protocol': 'SASL_SSL',
    'ssl.ca.location': certifi.where(),
    'sasl.mechanism': 'PLAIN',

    'sasl.username': 'TENANCY/Default/USER_EMAIL/STREAM_POOL_OCID',
    'sasl.password': 'OCI_AUTH_TOKEN',

    'group.id': 'python-public-consumer',
    'auto.offset.reset': 'earliest',
}

consumer = Consumer(conf)
consumer.subscribe(['stream-name'])

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            print(msg.error())
            continue
        print(msg.value().decode())
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

Observed Behavior

• Stable connection
• Messages received
• No unexpected disconnects

Python Client – Private Endpoint (Does NOT Work Reliably)

Observed Symptoms

• TLS handshake succeeds
• SASL authentication succeeds
• Metadata and group coordination succeed
• Broker disconnects after ~3 seconds:

  Disconnected (after 3000ms in state UP)
  

• This occurrence persists when using:
  • subscribe()
  • assign()
  • Different librdkafka versions
  • Various Kafka tuning parameters

Root Cause

librdkafka-based clients are not reliable with OCI Streaming Private Endpoints. The public endpoint uses an OCI-managed gateway that shields and masks protocol differences. Conversely, the private endpoint connects directly to the internal brokers, where non-Java client implementations run into compatibility issues.

Recommendation

Do not use Python Kafka clients with OCI Streaming Private Endpoints.

Java Client – Private Endpoint (Recommended & Supported)

Why Java Works

• Oracle officially supports and prioritizes Java Kafka clients.
• The private endpoint infrastructure is fully validated against Java client implementations.
• Stable consumer group handling with no unexpected timeouts.

Java Method 1: Kafka Console Consumer (Simplest)

For quick testing or command-line consumption, you can run the standard consumer utilities provided by Apache Kafka.

Prerequisites

sudo yum install -y java-11-openjdk

Download Kafka (Java tools)

If the direct version link fails, use the Apache archive mirror:

wget https://downloads.apache.org/kafka/3.8.1/kafka_2.13-3.8.1.tgz
tar -xzf kafka_2.13-3.8.1.tgz
cd kafka_2.13-3.8.1

Create JAAS Config

Create a file named oci-jaas.conf:

KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="TENANCY/Default/USER_EMAIL/STREAM_POOL_OCID"
  password="OCI_AUTH_TOKEN";
};

Export the configuration location environment variable:

export KAFKA_OPTS="-Djava.security.auth.login.config=$PWD/oci-jaas.conf"

Consumer Properties

Create a file named consumer.properties:

bootstrap.servers=STREAM_POOL_PRIVATE_ENDPOINT:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
auto.offset.reset=earliest

Run the Consumer

bin/kafka-console-consumer.sh \
  --bootstrap-server STREAM_POOL_PRIVATE_ENDPOINT:9092 \
  --topic stream-name \
  --consumer.config consumer.properties

Observed Behavior

• Stable connection
• Consistent message delivery
• No sudden disconnects

Conclusion

When architecture planning for OCI Streaming Service, base your client SDK choices on your networking topology:

• Use Python (or other librdkafka-based SDKs) when: Public endpoints are acceptable, or for rapid development and local testing.
• Use Java when: Private endpoints inside a VCN are required, or for production workloads, strict compliance frameworks, and enterprise-grade data pipelines.