Pulsar

 

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.

Learn more about Pulsar at https://pulsar.apache.org

Main features

• Horizontally scalable (Millions of independent topics and millions of messages published per second)
• Strong ordering and consistency guarantees
• Low latency durable storage
• Topic and queue semantics
• Load balancer
• Designed for being deployed as a hosted service:
  • Multi-tenant
  • Authentication
  • Authorization
  • Quotas
  • Support mixing very different workloads
  • Optional hardware isolation
• Keeps track of consumer cursor position
• REST API for provisioning, admin and stats
• Geo replication
• Transparent handling of partitioned topics
• Transparent batching of messages

Repositories

This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for components in the Pulsar ecosystem, including connectors, adapters, and other language clients.

• Pulsar Core

Helm Chart

• Pulsar Helm Chart

Ecosystem

• Pulsar Adapters

Clients

• .NET/C# Client
• C++ Client
• Go Client
• NodeJS Client
• Python Client
• Reactive Java Client

Dashboard & Management Tools

• Pulsar Manager

Website

• Pulsar Site

CI/CD

• Pulsar CI

Archived/Halted

• Pulsar Connectors (bundled as pulsar-io)
• Pulsar Translation
• Pulsar SQL (Pulsar Presto Connector) (bundled as pulsar-sql)
• Ruby Client

Pulsar Runtime Java Version Recommendation

• pulsar ver > 2.10 and master branch

Components	Java Version
Broker	17
Functions / IO	17
CLI	17
Java Client	8 or 11 or 17

• 2.8 <= pulsar ver <= 2.10

Components	Java Version
Broker	11
Functions / IO	11
CLI	8 or 11
Java Client	8 or 11

• pulsar ver < 2.8

Components	Java Version
All	8 or 11

Build Pulsar

Requirements

• JDK

  Pulsar Version	JDK Version
  master and 2.11 +	JDK 17
  2.8 / 2.9 / 2.10	JDK 11
  2.7 -	JDK 8

• Maven 3.6.1+

• zip

Note:

This project includes a Maven Wrapper that can be used instead of a system-installed Maven. Use it by replacing mvn by ./mvnw on Linux and mvnw.cmd on Windows in the commands below.

It's better to use CMD rather than Powershell on Windows. Because maven will activate the windows profile which runs rename-netty-native-libs.cmd.

Build

Compile and install:

$ mvn install -DskipTests

Compile and install individual module

$ mvn -pl module-name (e.g: pulsar-broker) install -DskipTests

Minimal build (This skips most of external connectors and tiered storage handlers)

mvn install -Pcore-modules,-main -DskipTests

Run Unit Tests:

$ mvn test

Run Individual Unit Test:

$ mvn -pl module-name (e.g: pulsar-client) test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest)

Run Selected Test packages:

$ mvn test -pl module-name (for example, pulsar-broker) -Dinclude=org/apache/pulsar/**/*.java

Start standalone Pulsar service:

$ bin/pulsar standalone

Check https://pulsar.apache.org for documentation and examples.

Setting up your IDE

Read https://pulsar.apache.org/contribute/setup-ide for setting up IntelliJ IDEA or Eclipse for developing Pulsar.

from https://github.com/apache/pulsar

-------------------------------------------------

Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar。

https://streamnative.io/docs/kop

Kafka-on-Pulsar (KoP)

Note:

KoP is now archived. It's recommended to try KSN (Kafka on StreamNative), see https://docs.streamnative.io/docs/kafka-on-cloud

KoP (Kafka on Pulsar) brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the KoP protocol handler to your existing Pulsar cluster, you can migrate your existing Kafka applications and services to Pulsar without modifying the code. This enables Kafka applications to leverage Pulsar’s powerful features, such as:

• Streamlined operations with enterprise-grade multi-tenancy
• Simplified operations with a rebalance-free architecture
• Infinite event stream retention with Apache BookKeeper and tiered storage
• Serverless event processing with Pulsar Functions

KoP, implemented as a Pulsar protocol handler plugin with the protocol name "kafka", is loaded when Pulsar broker starts. It helps reduce the barriers for people adopting Pulsar to achieve their business success by providing a native Kafka protocol support on Apache Pulsar. By integrating the two popular event streaming ecosystems, KoP unlocks new use cases. You can leverage advantages from each ecosystem and build a truly unified event streaming platform with Apache Pulsar to accelerate the development of real-time applications and services.

KoP implements the Kafka wire protocol on Pulsar by leveraging the existing components (such as topic discovery, the distributed log library - ManagedLedger, cursors and so on) that Pulsar already has.

The following figure illustrates how the Kafka-on-Pulsar protocol handler is implemented within Pulsar.

Version compatibility

The version of KoP x.y.z.m conforms to Pulsar x.y.z, while m is the patch version number. KoP might also be compatible with older patched versions, but it's not guaranteed. See upgrade.md for details.

KoP is compatible with Kafka clients 0.9 or higher. For Kafka clients 3.2.0 or higher, you have to add the following configurations in KoP because of KIP-679.

kafkaTransactionCoordinatorEnabled=true
brokerDeduplicationEnabled=true

How to use KoP

You can configure and manage KoP based on your requirements. Check the following guides for more details.

• Quick Start
• Configure KoP
• Monitor KoP
• Upgrade
• Secure KoP
• Schema Registry
• Implementation: How to converse Pulsar and Kafka

from https://github.com/streamnative/kop

-----------

 

 

bookkeeper

 Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads

bookkeeper.apache.org/

Apache BookKeeper is a scalable, fault-tolerant and low latency storage service optimized for append-only workloads.

It is suitable for being used in following scenarios:

• WAL (Write-Ahead-Logging), e.g. HDFS NameNode, Pravega.
• Message Store, e.g. Apache Pulsar.
• Offset/Cursor Store, e.g. Apache Pulsar.
• Object/Blob Store, e.g. storing state machine snapshots.

Get Started

• Checkout the project website.
• Concepts: Start with the basic concepts of Apache BookKeeper. This will help you to fully understand the other parts of the documentation.
• Follow the Installation guide to set up BookKeeper.

Documentation

Please visit the Documentation from the project website for more information.

Get In Touch

Report a Bug

For filing bugs, suggesting improvements, or requesting new features, help us out by opening a GitHub issue.

Need Help?

Subscribe or mail the user@bookkeeper.apache.org list - Ask questions, find answers, and also help other users.

Subscribe or mail the dev@bookkeeper.apache.org list - Join development discussions, propose new ideas and connect with contributors.

Join us on Slack - This is the most immediate way to connect with Apache BookKeeper committers and contributors.

from https://github.com/apache/bookkeeper

------

BookKeeper installation

You can install BookKeeper either by downloading a GZipped tarball package, using the Docker image or cloning the BookKeeper repository.

Requirements

• Unix environment
• Java Development Kit 1.8 or later

Download

You can download Apache BookKeeper releases from the Download page.

Clone

To build BookKeeper from source, clone the repository from the GitHub mirror:

$ git clone https://github.com/apache/bookkeeper

Build using Maven

Once you have the BookKeeper on your local machine, either by downloading or cloning it, you can then build BookKeeper from source using Maven:

$ mvn package

Since 4.8.0, bookkeeper introduces table service. If you would like to build and tryout table service, you can build it with stream profile.

$ mvn package -Dstream

You can skip tests by adding the -DskipTests flag when running mvn package.

Useful Maven commands

Some other useful Maven commands beyond mvn package:

Command	Action
mvn clean	Removes build artifacts
mvn compile	Compiles JAR files from Java sources
mvn compile spotbugs:spotbugs	Compile using the Maven SpotBugs plugin
mvn install	Install the BookKeeper JAR locally in your local Maven cache (usually in the ~/.m2 directory)
mvn deploy	Deploy the BookKeeper JAR to the Maven repo (if you have the proper credentials)
mvn verify	Performs a wide variety of verification and validation tasks
mvn apache-rat:check	Run Maven using the Apache Rat plugin
mvn compile javadoc:aggregate	Build Javadocs locally
mvn -am -pl bookkeeper-dist/server package	Build a server distribution using the Maven Assembly plugin

You can enable table service by adding the -Dstream flag when running above commands.

from https://bookkeeper.apache.org/docs/getting-started/installation/