PythonWebConference_ Cloud Native Apache Pulsar Development 202 with Python

Cloud Native Apache
Pulsar Development 202
with Python
Tim Spann
Principal Developer Advocate

2
https://2023.pythonwebconf.com/presentations/apache-pulsar
-development-202-with-python
Developers want to develop real-time applications that can turn raw text data into analyzed sentences and smart sentiment? Have you heard
of real-time analytics, let's leverage your Python skills to make it happen now.
In this talk I will show developers how to develop real-time applications that use Pulsar functions to turn live event data into live NLP results
and sentiment analyzed output.
We will walk through how to setup various scenarios to feed live data from apps and webapps to Apache Pulsar for real-time analytics and
NLP.
At the end of the talk, developers will be able to use Python for real-time NLP analytics and have gained insight on how and when to use
various streaming protocols, platforms, libraries and systems including Apache Pulsar, Apache Kafka, MQTT, Websockets, AMQP and Apache
Spark.

FLiPN-FLaNK Stack
Tim Spann
@PaasDev // Blog: www.datainmotion.dev
Principal Developer Advocate.
Princeton Future of Data Meetup.
ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC
https://github.com/tspannhw/EverythingApacheNiFi
https://medium.com/@tspann
Apache NiFi x Apache Kafka x Apache Flink

© 2023 Cloudera, Inc. All rights reserved. 5
FLiP Stack Weekly
This week in Apache Flink, Apache Kafka, Apache
NiFi, Apache Pulsar, Apache Iceberg, Python and
Open Source friends.
https://bit.ly/32dAJft

6
Building
Real-Time
Requires a Team

Messages - the Basic Unit of Apache Pulsar
7
Component Description
Value / data payload The data carried by the message. All Pulsar messages contain raw bytes, although message data
can also conform to data schemas.
Key Messages are optionally tagged with keys, used in partitioning and also is useful for things like
topic compaction.
Properties An optional key/value map of user-defined properties.
Producer name The name of the producer who produces the message. If you do not specify a producer name, the
default name is used.
Sequence ID Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the
message is its order in that sequence.

Integrated Schema Registry
Schema Registry
schema-1 (value=Avro/Protobuf/JSON) schema-2
(value=Avro/Protobuf/JSON)
schema-3
(value=Avro/Protobuf/JSON)
Schema
Data
ID
Local Cache
for Schemas
+
Schema
Data
ID +
Local Cache
for Schemas
Send schema-1
(value=Avro/Protobuf/JSON) data
serialized per schema ID
Send (register)
schema (if not in
local cache)
Read schema-1
(value=Avro/Protobuf/JSON) data
deserialized per schema ID
Get schema by ID (if
not in local cache)
Producers Consumers
8

The FLiPN Kitten crosses the stream
4 ways with Apache Pulsar
9

Data Offloaders
(Tiered Storage)
Client Libraries
Apache Pulsar Ecosystem
hub.streamnative.io
Connectors
(Sources & Sinks)
Protocol Handlers
Pulsar Functions
(Lightweight Stream
Processing)
Processing Engines
… and more!
… and more!

12
Pulsar Functions
● Consume messages from one or
more Pulsar topics.
● Apply user-supplied processing
logic to each message.
● Publish the results of the
computation to another topic.
● Support multiple programming
languages (Java, Python, Go)
● Can leverage 3rd-party libraries to
support the execution of ML
models on the edge.

from pulsar import Function
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
import json
class Chat(Function):
def __init__(self):
pass
def process(self, input, context):
fields = json.loads(input)
sid = SentimentIntensityAnalyzer()
ss = sid.polarity_scores(fields["comment"])
row = { }
row['id'] = str(msg_id)
if ss['compound'] < 0.00:
row['sentiment'] = 'Negative'
else:
row['sentiment'] = 'Positive'
row['comment'] = str(fields["comment"])
json_string = json.dumps(row)
return json_string
Entire Function
ML Function
13

Install Python Pulsar Client
pip3 install pulsar-client=='2.11.0[all]'
# Depending on Platform May Need C++ Client Built
For Python on Pulsar on Pi https://github.com/tspannhw/PulsarOnRaspberryPi
https://pulsar.apache.org/docs/en/client-libraries-python/
14
https://github.com/tspannhw/pulsar-sentiment-function

Building a Python Producer
import pulsar
client = pulsar.Client('pulsar://localhost:6650')
producer
client.create_producer('persistent://conf/ete/first')
producer.send(('Simple Text Message').encode('utf-8'))
client.close()
15

python3 prod.py -su pulsar+ssl://name1.name2.snio.cloud:6651 -t
persistent://public/default/pyth --auth-params
'{"issuer_url":"https://auth.streamnative.cloud", "private_key":"my.json",
"audience":"urn:sn:pulsar:name:myclustr"}'
from pulsar import Client, AuthenticationOauth2
parse = argparse.ArgumentParser(prog=prod.py')
parse.add_argument('-su', '--service-url', dest='service_url', type=str,
required=True)
args = parse.parse_args()
client = pulsar.Client(args.service_url,
authentication=AuthenticationOauth2(args.auth_params))
https://github.com/streamnative/examples/blob/master/cloud/python/OAuth2Producer.py
https://github.com/tspannhw/FLiP-Pi-BreakoutGarden
Producer with OAuth to Cloud
16

import pulsar
from pulsar.schema import *
from pulsar.schema import AvroSchema
class thermal(Record):
uuid = String()
client = pulsar.Client('pulsar://pulsar1:6650')
thermalschema = AvroSchema(thermal)
producer =
client.create_producer(topic='persistent://public/default/pi-thermal-avro',
schema=thermalschema,properties={"producer-name": "thrm" })
thermalRec = thermal()
thermalRec.uuid = "unique-name"
producer.send(thermalRec,partition_key=uniqueid)
https://github.com/tspannhw/FLiP-Pi-Thermal
Example Avro Schema Usage

import pulsar
from pulsar.schema import JsonSchema
class weather(Record):
uuid = String()
client = pulsar.Client('pulsar://pulsar1:6650')
wsc = JsonSchema(thermal)
producer =
client.create_producer(topic='persistent://public/default/wthr,schema=wsc,pro
perties={"producer-name": "wthr" })
weatherRec = weather()
weatherRec.uuid = "unique-name"
producer.send(weatherRec,partition_key=uniqueid)
https://github.com/tspannhw/FLiP-Pi-Weather
https://github.com/tspannhw/FLiP-PulsarDevPython101
Example JSON Schema Usage

import pulsar
consumer =
client.subscribe('persistent://conf/ete/first',subscription_name='mine')
while True:
msg = consumer.receive()
print("Received message: '%s'" % msg.data())
consumer.acknowledge(msg)
client.close()
Building a Python Consumer
19

from pyscylla.cluster import Cluster
from pyscylla.session import Session
from pyscylla.schema import Model, columns
from pyscylla.query import EQ
from pyschema import Schema, fields
import json
class User(Model):
id = columns.UUID(primary_key=True)
name = columns.Text()
email = columns.Text()
class UserSchema(Schema):
id = fields.UUID()
name = fields.String()
email = fields.String()
ChatGPT Built a Pulsar Function
20

class Function(pulsar_function.PulsarFunction):
def process(self, json_message, context):
# Connect to the Scylla database
cluster = Cluster(["127.0.0.1"])
session = cluster.connect()
# Parse the JSON message using the JSON schema
user_json = json.loads(json_message)
user_data = UserSchema().loads(user_json)
# Look up the user in the Scylla database
user = User.objects(session).get(id=EQ(user_data["id"]))
# Update the JSON object with the user data from the database
user_json["name"] = user.name
user_json["email"] = user.email
return json.dumps(user_json)
ChatGPT Built a Pulsar Function 2
21

pip3 install paho-mqtt
import paho.mqtt.client as mqtt
client = mqtt.Client("rpi4-iot")
row = { }
row['gasKO'] = str(readings)
json_string = json_string.strip()
client.connect("pulsar-server.com", 1883, 180)
client.publish("persistent://public/default/mqtt-2",
payload=json_string,qos=0,retain=True)
https://www.slideshare.net/bunkertor/data-minutes-2-apache-pulsar-with-mqtt-for-edge-computing-lightning-2022
Sending MQTT Messages

pip3 install websocket-client
import websocket, base64, json
topic = 'ws://server:8080/ws/v2/producer/persistent/public/default/topic1'
ws = websocket.create_connection(topic)
message = "Hello Python Web Conference"
message_bytes = message.encode('ascii')
base64_bytes = base64.b64encode(message_bytes)
base64_message = base64_bytes.decode('ascii')
ws.send(json.dumps({'payload' : base64_message,'properties': {'device' :
'macbook'},'context' : 5}))
response = json.loads(ws.recv())
https://pulsar.apache.org/docs/en/client-libraries-websocket/
https://github.com/tspannhw/FLiP-IoT/blob/main/wspulsar.py
https://github.com/tspannhw/FLiP-IoT/blob/main/wsreader.py
Sending Websocket Messages

pip3 install kafka-python
from kafka import KafkaProducer
from kafka.errors import KafkaError
row = { }
row['gasKO'] = str(readings)
json_string = json_string.strip()
producer = KafkaProducer(bootstrap_servers='pulsar1:9092',retries=3)
producer.send('topic-kafka-1', json.dumps(row).encode('utf-8'))
producer.flush()
https://github.com/streamnative/kop
https://docs.streamnative.io/platform/v1.0.0/concepts/kop-concepts
Sending Kafka Messages

bin/pulsar-admin functions create --auto-ack true --py py/src/sentiment.py
--classname "sentiment.Chat" --inputs "persistent://public/default/chat"
--log-topic "persistent://public/default/logs" --name Chat --output
"persistent://public/default/chatresult"
https://github.com/tspannhw/pulsar-pychat-function
DevOps: Deploying Functions
25

26
https://dev.to/tspannhw/consuming-streaming-stocks-data-with-python-websock
ets-and-pulsar-1h5a

27
Example Walk Through
import pulsar
class Stock (Record):
symbol = String()
ts = Float()
currentts = Float()
volume = Float()
price = Float()
tradeconditions = String()
uuid = String()

28
producer = client.create_producer(topic='persistent://public/default/stocks'
,schema=JsonSchema(Stock),properties={"producer-name":
"py-stocks","producer-id": "pystocks1" })

29
uuid_key =
'{0}_{1}'.format(strftime("%Y%m%d%H%M%S",gmtime()),uuid.uuid4())
stockRecord = Stock()
stockRecord.symbol = stockitem['s']
stockRecord.ts = float(stockitem['t'])
stockRecord.currentts = float(strftime("%Y%m%d%H%M%S",gmtime()))
stockRecord.volume = float(stockitem['v'])
stockRecord.price = float(stockitem['p'])
stockRecord.tradeconditions = ','.join(stockitem['c'])
stockRecord.uuid = uuid_key
if ( stockitem['s'] != '' ):
producer.send(stockRecord,partition_key=str(uuid_key))

31
Example Web Page
https://github.com/tspannhw/pulsar-transit-function
<link rel="stylesheet" type="text/css"
href="https://cdn.datatables.net/1.11.3/css/jquery.dataTables.min.css">
<script type="text/javascript" language="javascript"
src="https://code.jquery.com/jquery-3.5.1.js">
</script>
<script type="text/javascript" language="javascript"
src="https://cdn.datatables.net/1.11.3/js/jquery.dataTables.min.js">
</script>

32
<table id="example" class="display" style="width:100%">
<thead>
<tr>
<th><b>Key</b></th>
<th><b>Publish Time</b></th>
<th><b>Msg</b></th>
<th><b>Latitude</b></th>
<th><b>Longitude</b></th>
<th><b>Pub Date</b></th>
<th><b>System Time</b></th> </tr> </thead>
<tfoot>
<tr>
<th><b>Key</b></th>
<th><b>Publish Time</b></th>
<th><b>Msg</b></th>
<th><b>Latitude</b></th>
<th><b>Longitude</b></th>
<th><b>Pub Date</b></th>
<th><b>System Time</b></th>
</tr>
</tfoot>
</table>

33
$(document).ready(function() {
var t = $('#example').DataTable();
var wsUri =
"ws://pulsar1:8080/ws/v2/consumer/persistent/public/default/trans
com/tc-reader?subscriptionType=Shared";
websocket = new WebSocket(wsUri);
websocket.onopen = function(evt) {
console.log('open');
};
websocket.onerror = function(evt) {console.log('ERR', evt)};

34
websocket.onmessage = function(evt) {
var dataPoints = JSON.parse(evt.data);
if ( dataPoints === undefined || dataPoints == null || dataPoints.payload === undefined || dataPoints.payload
== null ) {
return;
}
if (websocket.readyState === WebSocket.OPEN) {
websocket.send("{"messageId": "" + dataPoints.messageId + ""}");
}
if (IsJsonString(atob(dataPoints.payload))) {
var pulsarMessage = JSON.parse(atob(dataPoints.payload));
if ( pulsarMessage === undefined || pulsarMessage == null ) {
return;
}
t.row.add( [ dataPoints.key, dataPoints.publishTime, pulsarMessage.title,
pulsarMessage.latitude, pulsarMessage.longitude, pulsarMessage.pubDate,
pulsarMessage.ts] ).draw(true );
}
};
} );
https://medium.com/@tspann/transit-watch-real-time-feeds-d98ff62b3bbb

Apache NiFi with Python Custom Processors
Python as a 1st class citizen

36
https://github.com/tspannhw https://www.datainmotion.dev/
https://www.meetup.com/futureofdata-princeton/

PythonWebConference_ Cloud Native Apache Pulsar Development 202 with Python

More Related Content

Similar to PythonWebConference_ Cloud Native Apache Pulsar Development 202 with Python

More from Timothy Spann

Recently uploaded

PythonWebConference_ Cloud Native Apache Pulsar Development 202 with Python