Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
Learn more about Pulsar at https://pulsar.apache.org
- Horizontally scalable (Millions of independent topics and millions of messages published per second)
- Strong ordering and consistency guarantees
- Low latency durable storage
- Topic and queue semantics
- Load balancer
- Designed for being deployed as a hosted service:
- Multi-tenant
- Authentication
- Authorization
- Quotas
- Support mixing very different workloads
- Optional hardware isolation
- Keeps track of consumer cursor position
- REST API for provisioning, admin and stats
- Geo replication
- Transparent handling of partitioned topics
- Transparent batching of messages
This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for components in the Pulsar ecosystem, including connectors, adapters, and other language clients.
- Pulsar Connectors (bundled as pulsar-io)
- Pulsar Translation
- Pulsar SQL (Pulsar Presto Connector) (bundled as pulsar-sql)
- Ruby Client
- pulsar ver > 2.10 and master branch
Components | Java Version |
---|---|
Broker | 17 |
Functions / IO | 17 |
CLI | 17 |
Java Client | 8 or 11 or 17 |
- 2.8 <= pulsar ver <= 2.10
Components | Java Version |
---|---|
Broker | 11 |
Functions / IO | 11 |
CLI | 8 or 11 |
Java Client | 8 or 11 |
- pulsar ver < 2.8
Components | Java Version |
---|---|
All | 8 or 11 |
-
JDK
Pulsar Version JDK Version master and 2.11 + JDK 17 2.8 / 2.9 / 2.10 JDK 11 2.7 - JDK 8 -
Maven 3.6.1+
-
zip
Note:
This project includes a Maven Wrapper that can be used instead of a system-installed Maven. Use it by replacing
mvn
by./mvnw
on Linux andmvnw.cmd
on Windows in the commands below.It's better to use CMD rather than Powershell on Windows. Because maven will activate the
windows
profile which runsrename-netty-native-libs.cmd
.
Compile and install:
$ mvn install -DskipTests
Compile and install individual module
$ mvn -pl module-name (e.g: pulsar-broker) install -DskipTests
mvn install -Pcore-modules,-main -DskipTests
Run Unit Tests:
$ mvn test
Run Individual Unit Test:
$ mvn -pl module-name (e.g: pulsar-client) test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest)
Run Selected Test packages:
$ mvn test -pl module-name (for example, pulsar-broker) -Dinclude=org/apache/pulsar/**/*.java
Start standalone Pulsar service:
$ bin/pulsar standalone
Check https://pulsar.apache.org for documentation and examples.
Read https://pulsar.apache.org/contribute/setup-ide for setting up IntelliJ IDEA or Eclipse for developing Pulsar.
from https://github.com/apache/pulsar
-------------------------------------------------
Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar。
https://streamnative.io/docs/kop
Note:
KoP is now archived. It's recommended to try KSN (Kafka on StreamNative), see https://docs.streamnative.io/docs/kafka-on-cloud
KoP (Kafka on Pulsar) brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the KoP protocol handler to your existing Pulsar cluster, you can migrate your existing Kafka applications and services to Pulsar without modifying the code. This enables Kafka applications to leverage Pulsar’s powerful features, such as:
- Streamlined operations with enterprise-grade multi-tenancy
- Simplified operations with a rebalance-free architecture
- Infinite event stream retention with Apache BookKeeper and tiered storage
- Serverless event processing with Pulsar Functions
KoP, implemented as a Pulsar protocol handler plugin with the protocol name "kafka", is loaded when Pulsar broker starts. It helps reduce the barriers for people adopting Pulsar to achieve their business success by providing a native Kafka protocol support on Apache Pulsar. By integrating the two popular event streaming ecosystems, KoP unlocks new use cases. You can leverage advantages from each ecosystem and build a truly unified event streaming platform with Apache Pulsar to accelerate the development of real-time applications and services.
KoP implements the Kafka wire protocol on Pulsar by leveraging the existing components (such as topic discovery, the distributed log library - ManagedLedger, cursors and so on) that Pulsar already has.
The following figure illustrates how the Kafka-on-Pulsar protocol handler is implemented within Pulsar.
The version of KoP x.y.z.m
conforms to Pulsar x.y.z
, while m
is the patch version number. KoP might also be compatible with older patched versions, but it's not guaranteed. See upgrade.md for details.
KoP is compatible with Kafka clients 0.9 or higher. For Kafka clients 3.2.0 or higher, you have to add the following configurations in KoP because of KIP-679.
kafkaTransactionCoordinatorEnabled=true
brokerDeduplicationEnabled=true
You can configure and manage KoP based on your requirements. Check the following guides for more details.
- Quick Start
- Configure KoP
- Monitor KoP
- Upgrade
- Secure KoP
- Schema Registry
- Implementation: How to converse Pulsar and Kafka
No comments:
Post a Comment