Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Intro to MQTT
Pietro Manzoni
Universitat Politecnica de Valencia (UPV)
Valencia - SPAIN
pmanzoni@disca.upv.es
1
http://bit.ly/ictp2019-mqtt
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Universitat Politècnica de València
¡ The Universitat Politècnica de València (UPV) is a
Spanish public educational institution founded in
1968.
¡ Its academic community comprises 36.823 students,
almost 2.661 lecturers and researchers, and 1.422
administration and services professionals.
¡ The Vera Campus covers around 840.000 m2 and is
almost 2 km long. It is a pedestrian campus with over
123.000 m2 of green areas.
¡ UPV is composed of 10 schools, 3 faculties and 2
higher polytechnic schools.
2
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
From “byte streams” to “messages”
¡The “old” vision of data
communication was based
on reliable byte
streams, i.e., TCP
¡Nowadays messages
interchange is becoming
more common
£E.g., Twitter, Whatsapp,
Instagram, Snapchat,
Facebook,...
¡Actually is not that new…
£emails: SMTP+MIME,
£FTP,
3
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Ways to interchange “messages”
4
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Request/response approach
¡ REST: Representational State Transfer
¡ Widely used; based on HTTP
¡ Lighter version: CoAP (Constrained Application Protocol)
5
Server
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Pub/sub approach
¡ Publish/Subscriber
o aka: producer/consumer
¡ Various protocols:
o MQTT, AMQP, XMPP (was Jabber)
¡ Growing technique
o E.g., https://cloud.google.com/iot/docs/how-tos/mqtt-bridge
6
Publisher 1 Publisher 2
Subscriber 1 Subscriber 2 Subscriber 3
Broker
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Pub/sub approach
¡ Pub/Sub separate a client, who is sending a message about a
specific topic, called publisher, from another client (or more
clients), who is receiving the message, called subscriber.
¡ There is a third component, called broker, which is known by
both the publisher and subscriber, which filters all incoming
messages and distributes them accordingly.
7
HiveMQ©
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
An example
8
Source: https://zoetrope.io/tech-blog/brief-practical-introduction-mqtt-protocol-and-its-application-iot
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Intro to MQTT
¡ Fundamental concepts
9
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Message Queuing Telemetry Transport
¡ A lightweight publish-subscribe protocol that can run on
embedded devices and mobile platforms è http://mqtt.org/
o Low power usage.
o Binary compressed headers
o Maximum message size of 256MB
• not really designed for sending large amounts of data
• better at a high volume of low size messages.
¡ Documentation sources:
o The MQTT community wiki:
• https://github.com/mqtt/mqtt.github.io/wiki
o A very good tutorial:
• http://www.hivemq.com/mqtt-essentials/
10
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Some details about versions
¡ MQTT 3.1.1 is the current version of the protocol.
o Standard document here:
• http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html
o October 29th 2014: MQTT was officially approved as OASIS Standard.
• https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=mqtt
¡ MQTT v5.0 is the successor of MQTT 3.1.1
o Current status: Committee Specification 02 (15 May 2018)
• http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html
o Not backward compatible; too many new things are introduced so
existing implementations have to be revisited, for example:
• Enhancements for scalability and large scale systems in respect to setups
with 1000s and millions of devices.
• Improved error reporting (Reason Code & Reason String)
• Performance improvements and improved support for small clients
o https://www.youtube.com/watch?time_continue=3&v=YIpesv_bJgU
11
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT works on top of…
¡ mainly of TCP
o There is also the closely related MQTT for Sensor Networks (MQTT-SN)
where TCP is replaced by UDP à TCP stack is too complex for WSN
¡ websockets can be used, too!
o Websockets allows you to receive MQTT data directly into a web browser.
¡ Both, TCP & websockets can work on top of “Transport Layer
Security (TLS)” (and its predecessor, Secure Sockets Layer (SSL))
12
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Publish/subscribe interactions sequence
13
Publisher Subscriber
BROKER
connect
connect ACK
connect
connect ACK
subscribe (topic)
subscribe ACK
publish (topic, data)
publish (topic, data)
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Topics
¡ MQTT Topics are structured in a hierarchy similar to folders and files in a file
system using the forward slash ( / ) as a delimiter.
¡ Allow to create a user friendly and self descriptive naming structures
¡ Topic names are:
o Case sensitive
o use UTF-8 strings.
o Must consist of at least one character to be valid.
¡ Except for the $SYS topic there is no default or standard topic structure.
14
Special $SYS/ topics
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Topics wildcards
¡ Topic subscriptions can have wildcards. These enable nodes to
subscribe to groups of topics that don’t exist yet, allowing greater
flexibility in the network’s messaging structure.
o ‘+’ matches anything at a given tree level
o ‘#’ matches a whole sub-tree
¡ Examples:
o Subscribing to topic house/# covers:
ü house/room1/main-light
ü house/room1/alarm
ü house/garage/main-light
ü house/main-door
o Subscribing to topic house/+/main-light covers:
ü house/room1/main-light
ü house/room2/main-light
ü house/garage/main-light
o but doesn’t cover
ü house/room1/side-light
ü house/room2/side-light
15
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Quality of Service (QoS)
¡ Messages are published with a Quality of Service (QoS) level, which specifies
delivery requirements.
¡ A QoS 0 (“at most once”) message is fire-and-forget.
o For example, a notification from a doorbell may only matter when
immediately delivered.
¡ With QoS 1 (“at least once”), the broker stores messages on disk and retries
until clients have acknowledged their delivery.
o (Possibly with duplicates.) It’s usually worth ensuring error messages are
delivered, even with a delay.
¡ QoS 2 (“exactly once”) messages have a second acknowledgement round-
trip, to ensure that non-idempotent messages can be delivered exactly
once.
16
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Retained Messages!!!
¡ A retained message is a normal MQTT message with the
retained flag set to true. The broker will store the last retained
message and the corresponding QoS for that topic
o Each client that subscribes to a topic pattern, which matches the topic of
the retained message, will receive the message immediately after
subscribing.
o For each topic only one retained message will be stored by the broker.
¡ Retained messages can help newly subscribed clients to get a
status update immediately after subscribing to a topic and
don’t have to wait until a publishing clients send the next
update.
o In other words a retained message on a topic is the last known good
value, because it doesn’t have to be the last value, but it certainly is the
last message with the retained flag set to true.
17
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Intro to MQTT
¡ Brokers and clients
18
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
BROKER
Creating a broker
19
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Available MQTT brokers
¡ The most widely used are:
o http://mosquitto.org/
• man page: https://mosquitto.org/man/mosquitto-8.html
o http://www.hivemq.com/
• The standard trial version only supports 25 connections.
¡ And also:
o https://www.rabbitmq.com/mqtt.html
o http://activemq.apache.org/mqtt.html
¡ A quite complete list can be found here:
o https://github.com/mqtt/mqtt.github.io/wiki/servers
20
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Installing Mosquitto on a Raspberry Pi
¡ It takes only a few seconds to install a Mosquitto broker on a
Raspberry. You need to execute the following steps:
sudo apt-get update
sudo apt-get install mosquitto mosquitto-clients
¡ Installation guidelines with websockets
https://gist.github.com/smoofit/dafa493aec8d41ea057370dbfde3f3fc
¡ Managing the broker:
o To start and stop its execution use:
sudo /etc/init.d/mosquitto start/stop
o Verbose mode:
sudo mosquitto –v
o To check if the broker is running you can use the command:
sudo netstat -tanlp | grep 1883
• note: "-tanlp" stands for: tcp, all, numeric, listening, program
21
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Cloud based MQTT brokers: CloudMQTT
22
https://www.cloudmqtt.com/ è based on Mosquitto
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Cloud based brokers: flespi
23
https://flespi.com/mqtt-broker
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Cloud based brokers: flespi
24
I1RKMMIUJppLdlQoSgAQ8MvJPyNV9R2HIJgijO1S1gt5rajaeIOaiaKWwlHt2z1z
https://flespi.io/#/panel/mqttboard
https://flespi.com/mqtt-api
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Open brokers (“Sandboxes”)
¡ TCP based:
o https://iot.eclipse.org/getting-started/#sandboxes
• Hostname: iot.eclipse.org
o http://test.mosquitto.org/
• Hostname: test.mosquitto.org
o https://www.hivemq.com/mqtt-demo/
• Hostname: broker.hivemq.com
• http://www.mqtt-dashboard.com/
o Ports:
• standard: 1883
• encrypted: 8883 (TLS v1.2, v1.1 or v1.0 with x509 certificates)
¡ Websockets based:
o broker.mqttdashboard.com port: 8000
o test.mosquitto.org port: 8080
o broker.hivemq.com port: 8000
¡ https://github.com/mqtt/mqtt.github.io/wiki/public_brokers
25
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
BROKER
Creating clients
26
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Clients for testing
¡ The Mosquitto broker comes with a couple of useful commands
to quickly publish and subscribe to some topic.
¡ Their basic syntax is the following.
o mosquitto_sub -h HOSTNAME -t TOPIC
o mosquitto_pub -h HOSTNAME -t TOPIC -m MSG
¡ More information can be found:
o https://mosquitto.org/man/mosquitto_sub-1.html
o https://mosquitto.org/man/mosquitto_pub-1.html
27
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT clients: iOS
28
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT clients: Android
29
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT websocket clients
30
http://test.mosquitto.org/ws.html http://mitsuruog.github.io/what-mqtt/
http://www.hivemq.com/demos/websocket-client/
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Intro to MQTT
¡ Time for some exercise: Lab 0
31
https://bit.ly/ictp2019-lab0
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Intro to MQTT
¡ More time for some demo/exercise: Lab 0.1
32
https://www.raspberrypi.org/products/sense-hat/
Broker address:
192.168.XX.XX
port: 9001
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Intro to MQTT
¡ Clients in Python
33
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT clients: Python vs Micropython
¡ The MQTT available versions for Python and MicroPython are
slightly different.
¡ MicroPython is intended for constrained environments, in
particular, microcontrollers, which have orders of magnitude
less performance and memory than "desktop" systems on
which Python3
¡ Basically remember that, when using the LoPy you have to use
the MicroPython version of MQTT
¡ In the following we will see information about both cases.
34
vs.
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Eclipse Paho Python
¡ Eclipse Paho Python (originally the mosquitto Python client)
o http://www.eclipse.org/paho/
¡ Documentation: https://pypi.org/project/paho-mqtt/
o or: http://www.eclipse.org/paho/clients/python/docs/
¡ Source: https://github.com/eclipse/paho.mqtt.python
35
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Paho MQTT Python client: general usage flow
The general usage flow is as follows:
¡ Create a client instance
¡ Connect to a broker using one of the connect*() functions
¡ Call one of the loop*() functions to maintain network traffic
flow with the broker
¡ Use subscribe() to subscribe to a topic and receive messages
¡ Use publish() to publish messages to the broker
¡ Use disconnect() to disconnect from the broker
36
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
# File: sisub.py
import paho.mqtt.client as mqtt
THE_BROKER = "iot.eclipse.org"
THE_TOPIC = "$SYS/#"
CLIENT_ID = ""
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
print("Connected to ", client._host, "port: ", client._port)
print("Flags: ", flags, "returned code: ", rc)
client.subscribe(THE_TOPIC, qos=0)
# The callback for when a message is received from the server.
def on_message(client, userdata, msg):
print("sisub: msg received with topic: {} and payload: {}".format(msg.topic, str(msg.payload)))
client = mqtt.Client(client_id=CLIENT_ID,
clean_session=True,
userdata=None,
protocol=mqtt.MQTTv311,
transport="tcp")
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set(None, password=None)
client.connect(THE_BROKER, port=1883, keepalive=60)
# Blocking call that processes network traffic, dispatches callbacks and handles reconnecting.
client.loop_forever()
Example 1: a simple subscriber
37
More on this later
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Connected to iot.eclipse.org port: 1883
Flags: {'session present': 0} returned code: 0
sisub: msg received with topic: $SYS/broker/version and payload: b'mosquitto version 1.4.15'
sisub: msg received with topic: $SYS/broker/timestamp and payload: b'2018-04-11 '
sisub: msg received with topic: $SYS/broker/clients/total and payload: b'162523'
sisub: msg received with topic: $SYS/broker/clients/active and payload: b'4103'
sisub: msg received with topic: $SYS/broker/clients/inactive and payload: b'158420'
sisub: msg received with topic: $SYS/broker/clients/maximum and payload: b'162524'
sisub: msg received with topic: $SYS/broker/clients/disconnected and payload: b'158420'
sisub: msg received with topic: $SYS/broker/clients/connected and payload: b'4103'
sisub: msg received with topic: $SYS/broker/clients/expired and payload: b'0'
sisub: msg received with topic: $SYS/broker/messages/received and payload: b'1171291305'
sisub: msg received with topic: $SYS/broker/messages/sent and payload: b'6271921352'
sisub: msg received with topic: $SYS/broker/messages/stored and payload: b'1380714’ …
…
Example 1: output
38
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Paho MQTT Python client: connect
¡ connect(host, port=1883, keepalive=60, bind_address=””)
¡ The broker acknowledgement will generate a callback
(on_connect).
¡ Return Codes:
o 0: Connection successful
o 1: Connection refused – incorrect protocol version
o 2: Connection refused – invalid client identifier
o 3: Connection refused – server unavailable
o 4: Connection refused – bad username or password
o 5: Connection refused – not authorised
o 6-255: Currently unused.
39
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Paho MQTT Python client: pub/sub
subscribe(topic, qos=0)
o e.g., subscribe("my/topic", 2)
o E.g., subscribe([("my/topic", 0), ("another/topic", 2)])
o on_message(client, userdata, message) Called when a message has been
received on a topic that the client subscribes to.
publish(topic, payload=None, qos=0, retain=False)
40
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Paho MQTT Python client: Network loop
41
What are network loops for?
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Paho MQTT Python client: Network loop
loop_forever()
¡ This is a blocking form of the network loop and will not return until
the client calls disconnect(). It automatically handles reconnecting.
loop_start() / loop_stop()
¡ These functions implement a threaded interface to the network loop.
o Calling loop_start() once, before or after connect(), runs a thread in the
background to call loop() automatically. This frees up the main thread for other
work that may be blocking.
o Call loop_stop() to stop the background thread.
loop(timeout=1.0)
¡ Call regularly to process network events.
o This call waits in select() until the network socket is available for reading or
writing, if appropriate, then handles the incoming/outgoing data.
o This function blocks for up to timeout seconds.
42
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
import sys
import time
import paho.mqtt.client as mqtt
THE_BROKER = "test.mosquitto.org"
THE_TOPIC = "$SYS/broker/load/bytes/#"
def on_connect(mqttc, obj, flags, rc):
print("Connected to ", mqttc._host, "port: ", mqttc._port)
mqttc.subscribe(THE_TOPIC, 0)
def on_message(mqttc, obj, msg):
global msg_counter
print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))
msg_counter+=1
def on_subscribe(mqttc, obj, mid, granted_qos):
print("Subscribed: ", mid, "granted QoS: ", granted_qos)
mqttc = mqtt.Client()
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.connect(THE_BROKER, keepalive=60)
msg_counter = 0
mqttc.loop_start()
while msg_counter < 10:
time.sleep(0.1)
mqttc.loop_stop()
print msg_counter
Example 2: subscriber with loop_start/loop_stop
43
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Example 3: very basic periodic producer
44
import random
import time
import paho.mqtt.client as mqtt
THE_BROKER = "test.mosquitto.org"
THE_TOPIC = "PMtest/rndvalue"
CLIENT_ID = ""
# The callback for when the client receives a CONNACK response
from the server.
def on_connect(client, userdata, flags, rc):
print("Connected to ", client._host, "port: ", client._port)
print("Flags: ", flags, "returned code: ", rc)
# The callback for when a message is published.
def on_publish(client, userdata, mid):
print("sipub: msg published (mid={})".format(mid))
client = mqtt.Client(client_id=CLIENT_ID,
clean_session=True,
userdata=None,
protocol=mqtt.MQTTv311,
transport="tcp")
client.on_connect = on_connect
client.on_publish = on_publish
client.username_pw_set(None, password=None)
client.connect(THE_BROKER, port=1883, keepalive=60)
client.loop_start()
while True:
msg_to_be_sent = random.randint(0, 100)
client.publish(THE_TOPIC,
payload=msg_to_be_sent,
qos=0,
retain=False)
time.sleep(5)
client.loop_stop()
Generates a new data every 5 secs
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Example 3: output
45
Output obtained with a
modified version of Example1.
Which parts of that code
had to be modified?
Python 3.6.1 (default, Dec 2015, 13:05:11)
[GCC 4.8.2] on linux
sipub: msg published (mid=1)
Connected to test.mosquitto.org port: 1883
Flags: {'session present': 0} returned code: 0
sipub: msg published (mid=2)
sipub: msg published (mid=3)
sipub: msg published (mid=4)
sipub: msg published (mid=5)
sipub: msg published (mid=6)
sipub: msg published (mid=7)
Python 3.6.1 (default, Dec 2015, 13:05:11)
[GCC 4.8.2] on linux
Connected to test.mosquitto.org port: 1883
Flags: {'session present': 0} returned code: 0
sisub: msg received with topic: PMtest/rndvalue and payload: b'11'
sisub: msg received with topic: PMtest/rndvalue and payload: b'14'
sisub: msg received with topic: PMtest/rndvalue and payload: b'31'
sisub: msg received with topic: PMtest/rndvalue and payload: b'27'
sisub: msg received with topic: PMtest/rndvalue and payload: b'60'
sisub: msg received with topic: PMtest/rndvalue and payload: b'70'
sisub: msg received with topic: PMtest/rndvalue and payload: b'60'
sisub: msg received with topic: PMtest/rndvalue and payload: b'66'
sisub: msg received with topic: PMtest/rndvalue and payload: b'45'
sisub: msg received with topic: PMtest/rndvalue and payload: b'56'
sisub: msg received with topic: PMtest/rndvalue and payload: b'37’
…
Producer output
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Example 4: Pub/Sub with JSON
46
…
mqttc.loop_start()
while True:
# Getting the data
the_time = time.strftime("%H:%M:%S")
the_value = random.randint(1,100)
the_msg={'Sensor': 1, 'C_F': 'C',
'Value': the_value, 'Time': the_time}
the_msg_str = json.dumps(the_msg)
mqttc.publish(THE_TOPIC, the_msg_str)
time.sleep(5)
mqttc.loop_stop()
…
# The callback for when a PUBLISH message is received
from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
themsg = json.loads(str(msg.payload))
print("Sensor "+str(themsg['Sensor'])+" got value "+
str(themsg['Value'])+" "+themsg['C_F']+
" at time "+str(themsg['Time']))
…
Producer
Consumer
paho-code:pietro$ python example4-cons.py
Connected with result code 0
PMtest/jsonvalue {"Time": "12:19:30", "Sensor": 1, "Value": 33, "C_F": "C"}
Sensor 1 got value 33 C at time 12:19:30
PMtest/jsonvalue {"Time": "12:19:35", "Sensor": 1, "Value": 11, "C_F": "C"}
Sensor 1 got value 11 C at time 12:19:35
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT with MicroPython
¡ Import the library
from mqtt import MQTTClient
¡ Creating a client:
MQTTclient(client_id, server, port=0, user=None, password=None,
keepalive=0, ssl=False, ssl_params={})
e.g., client = MQTTClient("dev_id", "10.1.1.101", 1883)
¡ The various calls:
• connect(clean_session=True):
• publish(topic, msg, retain=False, qos=0):
• subscribe(topic, qos=0):
• set_callback(self, f):
¡ wait_msg():
o Wait for a single incoming MQTT message and process it. Subscribed messages are
delivered to a callback previously set by .set_callback() method. Other (internal) MQTT
messages processed internally.
¡ check_msg():
o Checks whether a pending message from server is available. If not, returns
immediately with None. Otherwise, does the same processing as wait_msg.
47
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MicroPython: a simple publisher
48
# file: mp_sipub.py
from mqtt import MQTTClient
import pycom
import sys
import time
import ufun
wifi_ssid = 'THE_NAME_OF_THE_AP'
wifi_passwd = ''
THE_BROKER = "iot.eclipse.org"
THE_TOPIC = "test/SRM2018"
CLIENT_ID = ""
def settimeout(duration):
pass
def get_data_from_sensor(sensor_id="RAND"):
if sensor_id == "RAND":
return ufun.random_in_range()
### if __name__ == "__main__":
ufun.connect_to_wifi(wifi_ssid, wifi_passwd)
client = MQTTClient(CLIENT_ID, THE_BROKER, 1883)
print ("Connecting to broker: " + THE_BROKER)
try:
client.connect()
except OSError:
print ("Cannot connect to broker: " + THE_BROKER)
sys.exit()
print ("Connected to broker: " + THE_BROKER)
print('Sending messages...')
while True:
# creating the data
the_data = get_data_from_sensor()
# publishing the data
client.publish(THE_TOPIC, str(the_data))
print("Published message with value: {}".format(the_data))
time.sleep(1)
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MicroPython: a simple subscriber
49
# file: mp_sisub.py
from mqtt import MQTTClient
import pycom
import sys
import time
import ufun
wifi_ssid = 'THE_NAME_OF_THE_AP'
wifi_passwd = ''
THE_BROKER = "iot.eclipse.org"
THE_TOPIC = "test/SRM2018"
CLIENT_ID = ""
def settimeout(duration):
pass
def on_message(topic, msg):
print("Received msg: ", str(msg),
"with topic: ", str(topic))
### if __name__ == "__main__":
ufun.connect_to_wifi(wifi_ssid, wifi_passwd)
client = MQTTClient(CLIENT_ID, THE_BROKER, 1883)
client.set_callback(on_message)
print ("Connecting to broker: " + THE_BROKER)
try:
client.connect()
except OSError:
print ("Cannot connect to broker: " + THE_BROKER)
sys.exit()
print ("Connected to broker: " + THE_BROKER)
client.subscribe(THE_TOPIC)
print('Waiting messages...')
while 1:
client.check_msg()
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Intro to MQTT
¡ Some final details
50
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT Keep alive
¡ The keep alive functionality assures that the connection is still
open and both broker and client are connected to one another.
¡ The client specifies a time interval in seconds and
communicates it to the broker during the establishment of the
connection.
o The interval is the longest possible period of time which broker and client
can endure without sending a message.
o If the broker doesn’t receive a PINGREQ or any other packet from a
particular client, it will close the connection and send out the last will and
testament message (if the client had specified one).
¡ Good to Know
o The MQTT client is responsible of setting the right keep alive value.
o The maximum keep alive is 18h 12min 15 sec.
o If the keep alive interval is set to 0, the keep alive mechanism is
deactivated.
51
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
“Will” message
¡ When clients connect, they can specify an optional “will”
message, to be delivered if they are unexpectedly disconnected
from the network.
o (In the absence of other activity, a 2-byte ping message is sent to clients
at a configurable interval.)
¡ This “last will and testament” can be used to notify other parts
of the system that a node has gone down.
52
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Persistent session
¡ A persistent session saves all information relevant for the client on the
broker. The session is identified by the clientId provided by the client
on connection establishment
¡ So what will be stored in the session?
o Existence of a session, even if there are no subscriptions
o All subscriptions
o All messages in a Quality of Service (QoS) 1 or 2 flow, which are not confirmed by the
client
o All new QoS 1 or 2 messages, which the client missed while it was offline
o All received QoS 2 messages, which are not yet confirmed to the client
o That means even if the client is offline all the above will be stored by the broker and
are available right after the client reconnects.
¡ Persistent session on the client side
o Similar to the broker, each MQTT client must store a persistent session too. So when
a client requests the server to hold session data, it also has the responsibility to hold
some information by itself:
o All messages in a QoS 1 or 2 flow, which are not confirmed by the broker
o All received QoS 2 messages, which are not yet confirmed to the broker
53
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Topics best practices
¡ First of all:
o Don’t use a leading forward slash
o Don’t use spaces in a topic
o Use only ASCII characters, avoid non printable characters
¡ Then, try to..
o Keep the topic short and concise
o Use specific topics, instead of general ones
o Don’t forget extensibility
¡ Finally, be careful and don’t subscribe to #
54
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
A few words on security
¡ MQTT has the option for Transport Layer Security
(TLS) encryption.
¡ MQTT also provides username/password
authentication with the broker.
o Note that the password is transmitted in clear text. Thus, be sure to use
TLS encryption if you are using authentication.
55
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
A few words on security
56
Smart homes can be easily hacked via
unsecured MQTT servers
https://www.helpnetsecurity.com/2018/08/20/unsecured-mqtt-servers/
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT vs REST
¡ Can they really be compared?!?!?
o MQTT was created basically as a lightweight messaging protocol for
lightweight communication between devices and computer systems
o REST stands on the shoulders of the almighty HTTP
¡ So it’s better to understand their weak and strong points and
build a system taking the best of both worlds… if required
57
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
REST advantages
¡ It is always independent of the type of platform or languages
o The only thing is that it is indispensable that the responses to the requests
should always take place in the language used for the information
exchange, normally XML or JSON.
¡ It is stateless è This allows for scalability, by adding additional
server nodes behind a load balancer
o No state can be stored on servers: “keep the application state on the
client.”
o All messages exchanged between client and server have all the context
needed to know what to do with the message.
58
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
REST disadvantages
¡ Today’s real world embedded devices for IoT usually lacks the
ability to handle high-level protocols like HTTP and they may
be served better by lightweight binary protocols.
¡ It is PULL based. This poses a problem when services depend
on being up to date with data they don’t own and manage.
o Being up to date requires polling, which quickly add up in a system with
enough interconnected services.
o Pull style can produce heavy unnecessary workloads and bandwidth
consumption due to for example a request/response polling-based
monitoring & control systems
¡ It is based on one-to-one interactions
59
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Advantages of MQTT
¡ Push based: no need to continuously look for updates
¡ It has built-in function useful for reliable behavior in an
unreliable or intermittently connected wireless environments.
1. “last will & testament” so all apps know immediately if a client
disconnects ungracefully,
2. “retained message” so any user re-connecting immediately gets the very
latest information, etc.
¡ Useful for one-to-many, many-to-many applications
¡ Small memory footprint protocol, with reduced use of battery
60
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
Energy usage: some number
61
cost of ‘maintaining’ that connection (in % Battery / Hour):
amount of power taken to establish the initial connection to the server:
3G – 240s Keep Alive – % Battery Used Creating and Maintaining a Connection
you’d save ~4.1% battery per day just
by using MQTT over HTTPS to
maintain an open stable connection.
http://stephendnicholas.com/posts/power-profiling-mqtt-vs-https
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019
MQTT disadvantages
¡ If the broker fails…
¡ Does not define a standard client API, so application developers
have to select the best fit.
¡ Does not include many features that are common in Enterprise
Messaging Systems like:
o expiration, timestamp, priority, custom message headers, …
¡ Does not have a point-to-point (aka queues) messaging pattern
o Point to Point or One to One means that there can be more than one
consumer listening on a queue but only one of them will be get the
message
¡ Maximum message size 256MB
62
Workshop on Rapid Prototyping of IoT for Science (smr3268) – January 2019

MQTT_v2 protocol for IOT based applications

  • 1.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Intro to MQTT Pietro Manzoni Universitat Politecnica de Valencia (UPV) Valencia - SPAIN pmanzoni@disca.upv.es 1 http://bit.ly/ictp2019-mqtt
  • 2.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Universitat Politècnica de València ¡ The Universitat Politècnica de València (UPV) is a Spanish public educational institution founded in 1968. ¡ Its academic community comprises 36.823 students, almost 2.661 lecturers and researchers, and 1.422 administration and services professionals. ¡ The Vera Campus covers around 840.000 m2 and is almost 2 km long. It is a pedestrian campus with over 123.000 m2 of green areas. ¡ UPV is composed of 10 schools, 3 faculties and 2 higher polytechnic schools. 2
  • 3.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 From “byte streams” to “messages” ¡The “old” vision of data communication was based on reliable byte streams, i.e., TCP ¡Nowadays messages interchange is becoming more common £E.g., Twitter, Whatsapp, Instagram, Snapchat, Facebook,... ¡Actually is not that new… £emails: SMTP+MIME, £FTP, 3
  • 4.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Ways to interchange “messages” 4
  • 5.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Request/response approach ¡ REST: Representational State Transfer ¡ Widely used; based on HTTP ¡ Lighter version: CoAP (Constrained Application Protocol) 5 Server
  • 6.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Pub/sub approach ¡ Publish/Subscriber o aka: producer/consumer ¡ Various protocols: o MQTT, AMQP, XMPP (was Jabber) ¡ Growing technique o E.g., https://cloud.google.com/iot/docs/how-tos/mqtt-bridge 6 Publisher 1 Publisher 2 Subscriber 1 Subscriber 2 Subscriber 3 Broker
  • 7.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Pub/sub approach ¡ Pub/Sub separate a client, who is sending a message about a specific topic, called publisher, from another client (or more clients), who is receiving the message, called subscriber. ¡ There is a third component, called broker, which is known by both the publisher and subscriber, which filters all incoming messages and distributes them accordingly. 7 HiveMQ©
  • 8.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 An example 8 Source: https://zoetrope.io/tech-blog/brief-practical-introduction-mqtt-protocol-and-its-application-iot
  • 9.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Intro to MQTT ¡ Fundamental concepts 9
  • 10.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Message Queuing Telemetry Transport ¡ A lightweight publish-subscribe protocol that can run on embedded devices and mobile platforms è http://mqtt.org/ o Low power usage. o Binary compressed headers o Maximum message size of 256MB • not really designed for sending large amounts of data • better at a high volume of low size messages. ¡ Documentation sources: o The MQTT community wiki: • https://github.com/mqtt/mqtt.github.io/wiki o A very good tutorial: • http://www.hivemq.com/mqtt-essentials/ 10
  • 11.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Some details about versions ¡ MQTT 3.1.1 is the current version of the protocol. o Standard document here: • http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html o October 29th 2014: MQTT was officially approved as OASIS Standard. • https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=mqtt ¡ MQTT v5.0 is the successor of MQTT 3.1.1 o Current status: Committee Specification 02 (15 May 2018) • http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html o Not backward compatible; too many new things are introduced so existing implementations have to be revisited, for example: • Enhancements for scalability and large scale systems in respect to setups with 1000s and millions of devices. • Improved error reporting (Reason Code & Reason String) • Performance improvements and improved support for small clients o https://www.youtube.com/watch?time_continue=3&v=YIpesv_bJgU 11
  • 12.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT works on top of… ¡ mainly of TCP o There is also the closely related MQTT for Sensor Networks (MQTT-SN) where TCP is replaced by UDP à TCP stack is too complex for WSN ¡ websockets can be used, too! o Websockets allows you to receive MQTT data directly into a web browser. ¡ Both, TCP & websockets can work on top of “Transport Layer Security (TLS)” (and its predecessor, Secure Sockets Layer (SSL)) 12
  • 13.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Publish/subscribe interactions sequence 13 Publisher Subscriber BROKER connect connect ACK connect connect ACK subscribe (topic) subscribe ACK publish (topic, data) publish (topic, data)
  • 14.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Topics ¡ MQTT Topics are structured in a hierarchy similar to folders and files in a file system using the forward slash ( / ) as a delimiter. ¡ Allow to create a user friendly and self descriptive naming structures ¡ Topic names are: o Case sensitive o use UTF-8 strings. o Must consist of at least one character to be valid. ¡ Except for the $SYS topic there is no default or standard topic structure. 14 Special $SYS/ topics
  • 15.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Topics wildcards ¡ Topic subscriptions can have wildcards. These enable nodes to subscribe to groups of topics that don’t exist yet, allowing greater flexibility in the network’s messaging structure. o ‘+’ matches anything at a given tree level o ‘#’ matches a whole sub-tree ¡ Examples: o Subscribing to topic house/# covers: ü house/room1/main-light ü house/room1/alarm ü house/garage/main-light ü house/main-door o Subscribing to topic house/+/main-light covers: ü house/room1/main-light ü house/room2/main-light ü house/garage/main-light o but doesn’t cover ü house/room1/side-light ü house/room2/side-light 15
  • 16.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Quality of Service (QoS) ¡ Messages are published with a Quality of Service (QoS) level, which specifies delivery requirements. ¡ A QoS 0 (“at most once”) message is fire-and-forget. o For example, a notification from a doorbell may only matter when immediately delivered. ¡ With QoS 1 (“at least once”), the broker stores messages on disk and retries until clients have acknowledged their delivery. o (Possibly with duplicates.) It’s usually worth ensuring error messages are delivered, even with a delay. ¡ QoS 2 (“exactly once”) messages have a second acknowledgement round- trip, to ensure that non-idempotent messages can be delivered exactly once. 16
  • 17.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Retained Messages!!! ¡ A retained message is a normal MQTT message with the retained flag set to true. The broker will store the last retained message and the corresponding QoS for that topic o Each client that subscribes to a topic pattern, which matches the topic of the retained message, will receive the message immediately after subscribing. o For each topic only one retained message will be stored by the broker. ¡ Retained messages can help newly subscribed clients to get a status update immediately after subscribing to a topic and don’t have to wait until a publishing clients send the next update. o In other words a retained message on a topic is the last known good value, because it doesn’t have to be the last value, but it certainly is the last message with the retained flag set to true. 17
  • 18.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Intro to MQTT ¡ Brokers and clients 18
  • 19.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 BROKER Creating a broker 19
  • 20.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Available MQTT brokers ¡ The most widely used are: o http://mosquitto.org/ • man page: https://mosquitto.org/man/mosquitto-8.html o http://www.hivemq.com/ • The standard trial version only supports 25 connections. ¡ And also: o https://www.rabbitmq.com/mqtt.html o http://activemq.apache.org/mqtt.html ¡ A quite complete list can be found here: o https://github.com/mqtt/mqtt.github.io/wiki/servers 20
  • 21.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Installing Mosquitto on a Raspberry Pi ¡ It takes only a few seconds to install a Mosquitto broker on a Raspberry. You need to execute the following steps: sudo apt-get update sudo apt-get install mosquitto mosquitto-clients ¡ Installation guidelines with websockets https://gist.github.com/smoofit/dafa493aec8d41ea057370dbfde3f3fc ¡ Managing the broker: o To start and stop its execution use: sudo /etc/init.d/mosquitto start/stop o Verbose mode: sudo mosquitto –v o To check if the broker is running you can use the command: sudo netstat -tanlp | grep 1883 • note: "-tanlp" stands for: tcp, all, numeric, listening, program 21
  • 22.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Cloud based MQTT brokers: CloudMQTT 22 https://www.cloudmqtt.com/ è based on Mosquitto
  • 23.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Cloud based brokers: flespi 23 https://flespi.com/mqtt-broker
  • 24.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Cloud based brokers: flespi 24 I1RKMMIUJppLdlQoSgAQ8MvJPyNV9R2HIJgijO1S1gt5rajaeIOaiaKWwlHt2z1z https://flespi.io/#/panel/mqttboard https://flespi.com/mqtt-api
  • 25.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Open brokers (“Sandboxes”) ¡ TCP based: o https://iot.eclipse.org/getting-started/#sandboxes • Hostname: iot.eclipse.org o http://test.mosquitto.org/ • Hostname: test.mosquitto.org o https://www.hivemq.com/mqtt-demo/ • Hostname: broker.hivemq.com • http://www.mqtt-dashboard.com/ o Ports: • standard: 1883 • encrypted: 8883 (TLS v1.2, v1.1 or v1.0 with x509 certificates) ¡ Websockets based: o broker.mqttdashboard.com port: 8000 o test.mosquitto.org port: 8080 o broker.hivemq.com port: 8000 ¡ https://github.com/mqtt/mqtt.github.io/wiki/public_brokers 25
  • 26.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 BROKER Creating clients 26
  • 27.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Clients for testing ¡ The Mosquitto broker comes with a couple of useful commands to quickly publish and subscribe to some topic. ¡ Their basic syntax is the following. o mosquitto_sub -h HOSTNAME -t TOPIC o mosquitto_pub -h HOSTNAME -t TOPIC -m MSG ¡ More information can be found: o https://mosquitto.org/man/mosquitto_sub-1.html o https://mosquitto.org/man/mosquitto_pub-1.html 27
  • 28.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT clients: iOS 28
  • 29.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT clients: Android 29
  • 30.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT websocket clients 30 http://test.mosquitto.org/ws.html http://mitsuruog.github.io/what-mqtt/ http://www.hivemq.com/demos/websocket-client/
  • 31.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Intro to MQTT ¡ Time for some exercise: Lab 0 31 https://bit.ly/ictp2019-lab0
  • 32.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Intro to MQTT ¡ More time for some demo/exercise: Lab 0.1 32 https://www.raspberrypi.org/products/sense-hat/ Broker address: 192.168.XX.XX port: 9001
  • 33.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Intro to MQTT ¡ Clients in Python 33
  • 34.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT clients: Python vs Micropython ¡ The MQTT available versions for Python and MicroPython are slightly different. ¡ MicroPython is intended for constrained environments, in particular, microcontrollers, which have orders of magnitude less performance and memory than "desktop" systems on which Python3 ¡ Basically remember that, when using the LoPy you have to use the MicroPython version of MQTT ¡ In the following we will see information about both cases. 34 vs.
  • 35.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Eclipse Paho Python ¡ Eclipse Paho Python (originally the mosquitto Python client) o http://www.eclipse.org/paho/ ¡ Documentation: https://pypi.org/project/paho-mqtt/ o or: http://www.eclipse.org/paho/clients/python/docs/ ¡ Source: https://github.com/eclipse/paho.mqtt.python 35
  • 36.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Paho MQTT Python client: general usage flow The general usage flow is as follows: ¡ Create a client instance ¡ Connect to a broker using one of the connect*() functions ¡ Call one of the loop*() functions to maintain network traffic flow with the broker ¡ Use subscribe() to subscribe to a topic and receive messages ¡ Use publish() to publish messages to the broker ¡ Use disconnect() to disconnect from the broker 36
  • 37.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 # File: sisub.py import paho.mqtt.client as mqtt THE_BROKER = "iot.eclipse.org" THE_TOPIC = "$SYS/#" CLIENT_ID = "" # The callback for when the client receives a CONNACK response from the server. def on_connect(client, userdata, flags, rc): print("Connected to ", client._host, "port: ", client._port) print("Flags: ", flags, "returned code: ", rc) client.subscribe(THE_TOPIC, qos=0) # The callback for when a message is received from the server. def on_message(client, userdata, msg): print("sisub: msg received with topic: {} and payload: {}".format(msg.topic, str(msg.payload))) client = mqtt.Client(client_id=CLIENT_ID, clean_session=True, userdata=None, protocol=mqtt.MQTTv311, transport="tcp") client.on_connect = on_connect client.on_message = on_message client.username_pw_set(None, password=None) client.connect(THE_BROKER, port=1883, keepalive=60) # Blocking call that processes network traffic, dispatches callbacks and handles reconnecting. client.loop_forever() Example 1: a simple subscriber 37 More on this later
  • 38.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Connected to iot.eclipse.org port: 1883 Flags: {'session present': 0} returned code: 0 sisub: msg received with topic: $SYS/broker/version and payload: b'mosquitto version 1.4.15' sisub: msg received with topic: $SYS/broker/timestamp and payload: b'2018-04-11 ' sisub: msg received with topic: $SYS/broker/clients/total and payload: b'162523' sisub: msg received with topic: $SYS/broker/clients/active and payload: b'4103' sisub: msg received with topic: $SYS/broker/clients/inactive and payload: b'158420' sisub: msg received with topic: $SYS/broker/clients/maximum and payload: b'162524' sisub: msg received with topic: $SYS/broker/clients/disconnected and payload: b'158420' sisub: msg received with topic: $SYS/broker/clients/connected and payload: b'4103' sisub: msg received with topic: $SYS/broker/clients/expired and payload: b'0' sisub: msg received with topic: $SYS/broker/messages/received and payload: b'1171291305' sisub: msg received with topic: $SYS/broker/messages/sent and payload: b'6271921352' sisub: msg received with topic: $SYS/broker/messages/stored and payload: b'1380714’ … … Example 1: output 38
  • 39.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Paho MQTT Python client: connect ¡ connect(host, port=1883, keepalive=60, bind_address=””) ¡ The broker acknowledgement will generate a callback (on_connect). ¡ Return Codes: o 0: Connection successful o 1: Connection refused – incorrect protocol version o 2: Connection refused – invalid client identifier o 3: Connection refused – server unavailable o 4: Connection refused – bad username or password o 5: Connection refused – not authorised o 6-255: Currently unused. 39
  • 40.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Paho MQTT Python client: pub/sub subscribe(topic, qos=0) o e.g., subscribe("my/topic", 2) o E.g., subscribe([("my/topic", 0), ("another/topic", 2)]) o on_message(client, userdata, message) Called when a message has been received on a topic that the client subscribes to. publish(topic, payload=None, qos=0, retain=False) 40
  • 41.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Paho MQTT Python client: Network loop 41 What are network loops for?
  • 42.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Paho MQTT Python client: Network loop loop_forever() ¡ This is a blocking form of the network loop and will not return until the client calls disconnect(). It automatically handles reconnecting. loop_start() / loop_stop() ¡ These functions implement a threaded interface to the network loop. o Calling loop_start() once, before or after connect(), runs a thread in the background to call loop() automatically. This frees up the main thread for other work that may be blocking. o Call loop_stop() to stop the background thread. loop(timeout=1.0) ¡ Call regularly to process network events. o This call waits in select() until the network socket is available for reading or writing, if appropriate, then handles the incoming/outgoing data. o This function blocks for up to timeout seconds. 42
  • 43.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 import sys import time import paho.mqtt.client as mqtt THE_BROKER = "test.mosquitto.org" THE_TOPIC = "$SYS/broker/load/bytes/#" def on_connect(mqttc, obj, flags, rc): print("Connected to ", mqttc._host, "port: ", mqttc._port) mqttc.subscribe(THE_TOPIC, 0) def on_message(mqttc, obj, msg): global msg_counter print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload)) msg_counter+=1 def on_subscribe(mqttc, obj, mid, granted_qos): print("Subscribed: ", mid, "granted QoS: ", granted_qos) mqttc = mqtt.Client() mqttc.on_message = on_message mqttc.on_connect = on_connect mqttc.on_subscribe = on_subscribe mqttc.connect(THE_BROKER, keepalive=60) msg_counter = 0 mqttc.loop_start() while msg_counter < 10: time.sleep(0.1) mqttc.loop_stop() print msg_counter Example 2: subscriber with loop_start/loop_stop 43
  • 44.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Example 3: very basic periodic producer 44 import random import time import paho.mqtt.client as mqtt THE_BROKER = "test.mosquitto.org" THE_TOPIC = "PMtest/rndvalue" CLIENT_ID = "" # The callback for when the client receives a CONNACK response from the server. def on_connect(client, userdata, flags, rc): print("Connected to ", client._host, "port: ", client._port) print("Flags: ", flags, "returned code: ", rc) # The callback for when a message is published. def on_publish(client, userdata, mid): print("sipub: msg published (mid={})".format(mid)) client = mqtt.Client(client_id=CLIENT_ID, clean_session=True, userdata=None, protocol=mqtt.MQTTv311, transport="tcp") client.on_connect = on_connect client.on_publish = on_publish client.username_pw_set(None, password=None) client.connect(THE_BROKER, port=1883, keepalive=60) client.loop_start() while True: msg_to_be_sent = random.randint(0, 100) client.publish(THE_TOPIC, payload=msg_to_be_sent, qos=0, retain=False) time.sleep(5) client.loop_stop() Generates a new data every 5 secs
  • 45.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Example 3: output 45 Output obtained with a modified version of Example1. Which parts of that code had to be modified? Python 3.6.1 (default, Dec 2015, 13:05:11) [GCC 4.8.2] on linux sipub: msg published (mid=1) Connected to test.mosquitto.org port: 1883 Flags: {'session present': 0} returned code: 0 sipub: msg published (mid=2) sipub: msg published (mid=3) sipub: msg published (mid=4) sipub: msg published (mid=5) sipub: msg published (mid=6) sipub: msg published (mid=7) Python 3.6.1 (default, Dec 2015, 13:05:11) [GCC 4.8.2] on linux Connected to test.mosquitto.org port: 1883 Flags: {'session present': 0} returned code: 0 sisub: msg received with topic: PMtest/rndvalue and payload: b'11' sisub: msg received with topic: PMtest/rndvalue and payload: b'14' sisub: msg received with topic: PMtest/rndvalue and payload: b'31' sisub: msg received with topic: PMtest/rndvalue and payload: b'27' sisub: msg received with topic: PMtest/rndvalue and payload: b'60' sisub: msg received with topic: PMtest/rndvalue and payload: b'70' sisub: msg received with topic: PMtest/rndvalue and payload: b'60' sisub: msg received with topic: PMtest/rndvalue and payload: b'66' sisub: msg received with topic: PMtest/rndvalue and payload: b'45' sisub: msg received with topic: PMtest/rndvalue and payload: b'56' sisub: msg received with topic: PMtest/rndvalue and payload: b'37’ … Producer output
  • 46.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Example 4: Pub/Sub with JSON 46 … mqttc.loop_start() while True: # Getting the data the_time = time.strftime("%H:%M:%S") the_value = random.randint(1,100) the_msg={'Sensor': 1, 'C_F': 'C', 'Value': the_value, 'Time': the_time} the_msg_str = json.dumps(the_msg) mqttc.publish(THE_TOPIC, the_msg_str) time.sleep(5) mqttc.loop_stop() … # The callback for when a PUBLISH message is received from the server. def on_message(client, userdata, msg): print(msg.topic+" "+str(msg.payload)) themsg = json.loads(str(msg.payload)) print("Sensor "+str(themsg['Sensor'])+" got value "+ str(themsg['Value'])+" "+themsg['C_F']+ " at time "+str(themsg['Time'])) … Producer Consumer paho-code:pietro$ python example4-cons.py Connected with result code 0 PMtest/jsonvalue {"Time": "12:19:30", "Sensor": 1, "Value": 33, "C_F": "C"} Sensor 1 got value 33 C at time 12:19:30 PMtest/jsonvalue {"Time": "12:19:35", "Sensor": 1, "Value": 11, "C_F": "C"} Sensor 1 got value 11 C at time 12:19:35
  • 47.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT with MicroPython ¡ Import the library from mqtt import MQTTClient ¡ Creating a client: MQTTclient(client_id, server, port=0, user=None, password=None, keepalive=0, ssl=False, ssl_params={}) e.g., client = MQTTClient("dev_id", "10.1.1.101", 1883) ¡ The various calls: • connect(clean_session=True): • publish(topic, msg, retain=False, qos=0): • subscribe(topic, qos=0): • set_callback(self, f): ¡ wait_msg(): o Wait for a single incoming MQTT message and process it. Subscribed messages are delivered to a callback previously set by .set_callback() method. Other (internal) MQTT messages processed internally. ¡ check_msg(): o Checks whether a pending message from server is available. If not, returns immediately with None. Otherwise, does the same processing as wait_msg. 47
  • 48.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MicroPython: a simple publisher 48 # file: mp_sipub.py from mqtt import MQTTClient import pycom import sys import time import ufun wifi_ssid = 'THE_NAME_OF_THE_AP' wifi_passwd = '' THE_BROKER = "iot.eclipse.org" THE_TOPIC = "test/SRM2018" CLIENT_ID = "" def settimeout(duration): pass def get_data_from_sensor(sensor_id="RAND"): if sensor_id == "RAND": return ufun.random_in_range() ### if __name__ == "__main__": ufun.connect_to_wifi(wifi_ssid, wifi_passwd) client = MQTTClient(CLIENT_ID, THE_BROKER, 1883) print ("Connecting to broker: " + THE_BROKER) try: client.connect() except OSError: print ("Cannot connect to broker: " + THE_BROKER) sys.exit() print ("Connected to broker: " + THE_BROKER) print('Sending messages...') while True: # creating the data the_data = get_data_from_sensor() # publishing the data client.publish(THE_TOPIC, str(the_data)) print("Published message with value: {}".format(the_data)) time.sleep(1)
  • 49.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MicroPython: a simple subscriber 49 # file: mp_sisub.py from mqtt import MQTTClient import pycom import sys import time import ufun wifi_ssid = 'THE_NAME_OF_THE_AP' wifi_passwd = '' THE_BROKER = "iot.eclipse.org" THE_TOPIC = "test/SRM2018" CLIENT_ID = "" def settimeout(duration): pass def on_message(topic, msg): print("Received msg: ", str(msg), "with topic: ", str(topic)) ### if __name__ == "__main__": ufun.connect_to_wifi(wifi_ssid, wifi_passwd) client = MQTTClient(CLIENT_ID, THE_BROKER, 1883) client.set_callback(on_message) print ("Connecting to broker: " + THE_BROKER) try: client.connect() except OSError: print ("Cannot connect to broker: " + THE_BROKER) sys.exit() print ("Connected to broker: " + THE_BROKER) client.subscribe(THE_TOPIC) print('Waiting messages...') while 1: client.check_msg()
  • 50.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Intro to MQTT ¡ Some final details 50
  • 51.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT Keep alive ¡ The keep alive functionality assures that the connection is still open and both broker and client are connected to one another. ¡ The client specifies a time interval in seconds and communicates it to the broker during the establishment of the connection. o The interval is the longest possible period of time which broker and client can endure without sending a message. o If the broker doesn’t receive a PINGREQ or any other packet from a particular client, it will close the connection and send out the last will and testament message (if the client had specified one). ¡ Good to Know o The MQTT client is responsible of setting the right keep alive value. o The maximum keep alive is 18h 12min 15 sec. o If the keep alive interval is set to 0, the keep alive mechanism is deactivated. 51
  • 52.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 “Will” message ¡ When clients connect, they can specify an optional “will” message, to be delivered if they are unexpectedly disconnected from the network. o (In the absence of other activity, a 2-byte ping message is sent to clients at a configurable interval.) ¡ This “last will and testament” can be used to notify other parts of the system that a node has gone down. 52
  • 53.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Persistent session ¡ A persistent session saves all information relevant for the client on the broker. The session is identified by the clientId provided by the client on connection establishment ¡ So what will be stored in the session? o Existence of a session, even if there are no subscriptions o All subscriptions o All messages in a Quality of Service (QoS) 1 or 2 flow, which are not confirmed by the client o All new QoS 1 or 2 messages, which the client missed while it was offline o All received QoS 2 messages, which are not yet confirmed to the client o That means even if the client is offline all the above will be stored by the broker and are available right after the client reconnects. ¡ Persistent session on the client side o Similar to the broker, each MQTT client must store a persistent session too. So when a client requests the server to hold session data, it also has the responsibility to hold some information by itself: o All messages in a QoS 1 or 2 flow, which are not confirmed by the broker o All received QoS 2 messages, which are not yet confirmed to the broker 53
  • 54.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Topics best practices ¡ First of all: o Don’t use a leading forward slash o Don’t use spaces in a topic o Use only ASCII characters, avoid non printable characters ¡ Then, try to.. o Keep the topic short and concise o Use specific topics, instead of general ones o Don’t forget extensibility ¡ Finally, be careful and don’t subscribe to # 54
  • 55.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 A few words on security ¡ MQTT has the option for Transport Layer Security (TLS) encryption. ¡ MQTT also provides username/password authentication with the broker. o Note that the password is transmitted in clear text. Thus, be sure to use TLS encryption if you are using authentication. 55
  • 56.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 A few words on security 56 Smart homes can be easily hacked via unsecured MQTT servers https://www.helpnetsecurity.com/2018/08/20/unsecured-mqtt-servers/
  • 57.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT vs REST ¡ Can they really be compared?!?!? o MQTT was created basically as a lightweight messaging protocol for lightweight communication between devices and computer systems o REST stands on the shoulders of the almighty HTTP ¡ So it’s better to understand their weak and strong points and build a system taking the best of both worlds… if required 57
  • 58.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 REST advantages ¡ It is always independent of the type of platform or languages o The only thing is that it is indispensable that the responses to the requests should always take place in the language used for the information exchange, normally XML or JSON. ¡ It is stateless è This allows for scalability, by adding additional server nodes behind a load balancer o No state can be stored on servers: “keep the application state on the client.” o All messages exchanged between client and server have all the context needed to know what to do with the message. 58
  • 59.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 REST disadvantages ¡ Today’s real world embedded devices for IoT usually lacks the ability to handle high-level protocols like HTTP and they may be served better by lightweight binary protocols. ¡ It is PULL based. This poses a problem when services depend on being up to date with data they don’t own and manage. o Being up to date requires polling, which quickly add up in a system with enough interconnected services. o Pull style can produce heavy unnecessary workloads and bandwidth consumption due to for example a request/response polling-based monitoring & control systems ¡ It is based on one-to-one interactions 59
  • 60.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Advantages of MQTT ¡ Push based: no need to continuously look for updates ¡ It has built-in function useful for reliable behavior in an unreliable or intermittently connected wireless environments. 1. “last will & testament” so all apps know immediately if a client disconnects ungracefully, 2. “retained message” so any user re-connecting immediately gets the very latest information, etc. ¡ Useful for one-to-many, many-to-many applications ¡ Small memory footprint protocol, with reduced use of battery 60
  • 61.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 Energy usage: some number 61 cost of ‘maintaining’ that connection (in % Battery / Hour): amount of power taken to establish the initial connection to the server: 3G – 240s Keep Alive – % Battery Used Creating and Maintaining a Connection you’d save ~4.1% battery per day just by using MQTT over HTTPS to maintain an open stable connection. http://stephendnicholas.com/posts/power-profiling-mqtt-vs-https
  • 62.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019 MQTT disadvantages ¡ If the broker fails… ¡ Does not define a standard client API, so application developers have to select the best fit. ¡ Does not include many features that are common in Enterprise Messaging Systems like: o expiration, timestamp, priority, custom message headers, … ¡ Does not have a point-to-point (aka queues) messaging pattern o Point to Point or One to One means that there can be more than one consumer listening on a queue but only one of them will be get the message ¡ Maximum message size 256MB 62
  • 63.
    Workshop on RapidPrototyping of IoT for Science (smr3268) – January 2019