os distributed system theoretical foundation

Theoretical Foundations
▸Clocks
▹Logical clocks (Lamport)
▹Partially ordered clocks (Fidge)
▸Causally ordered message delivery
▹Broadcast-based (Birman et al.)
▹Unicast-based (Raynal, Schiper & Toueg)
▸System states
▹Global snapshots (Chandy & Lamport)
▹Termination detection (Huang)
Theoretical Foundations
1

Activity
▸Processes are either active or passive
▸Active:
▹processing a computation or application
▹Only active processes send messages
▹May become passive at any time
▸Passive:
▹not processing a computation or application
▹Reactivated only upon receiving a message
Theoretical Foundations > Activity
2

Termination
▸A computation or application is said to be
terminated iff all processes are idle and there
are no messages in transit
Theoretical Foundations > Termination
3

Huang’s Protocol
▸Assumptions:
▹Control process initiates the computation
▹Communication channels exist between every
process pair (i.e., a complete network topology)
▹Message delay is arbitrary but finite
(i.e., a lossless network)
▹No process failures
Theoretical Foundations > Termination > Huang’s Protocol
4

Huang’s Protocol
▸Rule 1: Sending an application message
When a process p sends a message m, it divides it
weight Wp into W1 and W2. It keeps W1 and sends
W2 along with the message.
send(m, W2);
Wp = W1;
5

Huang’s Protocol
▸Rule 2: Receiving an application message
When process q receives a message, it adds the
message’s weight Wm to its own weight Wq.
If q is idle, q is reactivated.
receive(m, Wm);
Wq = Wq + Wm;
6

Huang’s Protocol
▸Rule 3: Becoming idle
When process q becomes idle, it sends its weight
Wq along with a control message to the control
process c.
send(m, Wq);
idle();
7

Huang’s Protocol
▸Rule 4: Monitoring
When the control process c receives a control
message, it adds the message’s weight Wm to its
own weight Wc. If Wc equals 1, the computation
has terminated.
receive(m, Wm);
Wc = Wc + Wm;
if(Wc == 1) termination = true;
8

▸Can represent W as a binary number in an
unbounded array of bits with the binary point
before the first bit
▹(i.e., “100” has a binary value of 0.100… or a
decimal value of 0.5)
▸The control process c starts by sending a
message with Wm = “1” and sets Wc = “1”
▸Termination occurs when Wc rolls over
(i.e., the bit array Wc requires a new leading bit)
Binary Representation
9

▸Evenly divide the binary array into window slots
▸Every weight W is represented by (…)i, where (…) is
a window of bits and i is the slot number
▸Let H be the maximum array size (e.g., H = 16 bits)
▸Let h be the window size (e.g., h = 4 bits)
▸Since H = 4h, we need 2 bits (because 22 = 4) to
represent the slot number
▸Hence, only 6 bits is required to represent a slot
(e.g., 0000 0000 0000 0001 = 0001|11)
Space-Efficient Representation
10

▸ When a process p has weight Wp = (0…01)i and needs to
send a message, it slides the window right one slot and
divides Wp into W1 and W2
▸ Example:
▹Wp = 0001|01 (i.e., 0000 0001)
▹p sets W1 to 1000|10 (i.e., 0000 0000 1)
▹p sets W2 to 1000|10 (i.e., 0000 0000 1)
▸ When a process p receives a message with weight Wm
and the sum of Wp and Wm do not fit in one window slot,
p forwards the smaller weight to the control process
▸ Hence, only the weight of the control (WC) needs 16 bits
Space-Efficient Scheme
11

▸When a process p needs to send a message and
has weight Wp = (0…01)i, where i is the last
window slot, p must first send a weight request
to the control with its weight attached
▸When the control process c receives a weight
request of form (0…01)i, it will send a weight
supply message to the requesting process of the
form (0…01)j, where j < i
Space-Efficient Scheme
12

▸Does the computation terminate at c4?
▸There must be at least one other active process
Termination Detection Exercise
13
C
D
E
1
0011
0011
01
01
00011
00011
00011
00011
00011
11 11011 1111
011
0001
control
m35 m36
0
0
0
c4
No, WC != 1

Resource Allocation
▸Distributed mutual exclusion
▹Permission-based (Ricart & Agrawala)
▹Quorum-based (Maekawa)
▹Token-based (Raymond)
Resource Allocation
14

Distributed Mutual Exclusion:
Permission-based Algorithms
CS/CE/TE 6378
Advanced Operating Systems

Distributed Mutual Exclusion
▸Mutual Exclusion:
▹a resource granted to a process must be released
before it can be granted to another process
▸Conditions for Distributed Mutual Exclusion:
1. Requests for a resource should be granted in the
order in which they were made
2. Every granted resource will eventually be released,
to ensure every request will eventually be granted
Resource Allocation > Distributed Mutual Exclusion
16

Permission-Based Algorithms
▸Algorithms that ensure mutual exclusion by
obtaining permission from every process in the
system
▸Whenever a process requests a resource, it must
request permission from every other process in
the system
▸Two algorithms:
▹Lamport (1978)
▹Ricart & Agrawala (1981)
Resource Allocation > Distributed Mutual Exclusion > Permission-Based
17

Lamport Algorithm
▸Assumptions:
▹Channels causally order message (i.e., FIFO)
▹Every message is eventually received
(i.e., lossless network)
▹Processes do not fail
▹Every process has its own request queue RQ, which is
never seen by other processes
▹Scalar logical clocks are used for timestamps
▹A total ordering relation is used to determine priority
Resource Allocation > Distributed Mutual Exclusion > Permission-Based > Lamport
18

Lamport Algorithm
▸Rule 1
To request a resource, process i sends a request
message Tm:i to every other process, where Tm is
the timestamp of the message. Process i then
stores a copy of the message in its request
queue.
19

Lamport Example
▸Total ordering: C ≺ D ≺ E
20
C
D
E
RQC = { }
RQD = { }
RQE = { }

Lamport Example
▸D requests the resource by sending request
messages to C and E
21
C
D
E
1
RQC = { }
RQD = { }
RQE = { }
req(1:D)
req(1:D)

Lamport Example
▸D then stores its own request
22
C
D
E
1
RQC = { }
RQD = { 1:D
}
RQE = { }
req(1:D)
req(1:D)

Lamport Example
▸C requests the resource by sending request
messages to D and E
23
C
D
E
1
1
RQC = { }
RQD = { 1:D
}
RQE = { }
req(1:D)
req(1:D)
req(1:C)

Lamport Example
▸C then stores its own request
24
C
D
E
1
1
RQC = { 1:C }
RQD = { 1:D
}
RQE = { }
req(1:D)
req(1:D)
req(1:C)

Lamport Algorithm
▸Rule 2
When process j receives a request message Tm:i,
it places it in its request queue and sends a
timestamped acknowledgment back to process i.
25

Lamport Example
26
C
D
E
1
1
RQC = { 1:C }
RQD = { 1:D
}
RQE = { }
req(1:D)
req(1:D)
req(1:C)

Lamport Example
▸E places D’s request in its queue and sends an
acknowledgment back
27
C
D
E
1
1
2 3
RQC = { 1:C }
RQD = { 1:D
}
RQE = { 1:D }
req(1:D)
req(1:D)
req(1:C)
ack(3:E)

Lamport Example
▸D places C’s request in its queue and sends an
acknowledgment back
28
C
D
E
1
1
2 3
2 3
RQC = { 1:C }
RQD = { 1:C,
1:D }
RQE = { 1:D }
req(1:D)
req(1:D)
req(1:C)
ack(3:D)
ack(3:E)

Lamport Example
▸C places D’s request in its queue and sends an
acknowledgment back
29
C
D
E
1 2 3
1
2 3
2 3
RQE = { 1:D }
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E)
RQD = { 1:C,
1:D }
RQC = { 1:C,
1:D }

Lamport Algorithm
▸Rule 5
Process i is granted the resource when the
following conditions are satisfied:
1. There is a request Tm:i in its request queue that is
ordered before any other request by the relation ⇒
2. Process i has received a message from every other
process with a timestamp later than Tm
30

Lamport Example
31
C
D
E
1 2 3
1
2 3
2 3
RQE = { 1:D }
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E)
RQD = { 1:C,
1:D }
RQC = { 1:C,
1:D }

Lamport Example
▸D isn’t granted the resource since 1:C ⇒ 1:D
32
C
D
E
1 2 3
1
2 3
2 3
4
RQE = { 1:D }
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E)
RQC = { 1:C,
1:D }
RQD = { 1:C,
1:D }

Lamport Example
▸C isn’t granted the resource since it has not
received a message from E
33
C
D
E
1 2 3 4
1
2 3
2 3
4
RQE = { 1:D }
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E)
RQC = { 1:C,
1:D }
RQD = { 1:C,
1:D }

Lamport Example
▸E places C’s request in its queue and sends an
acknowledgment back
34
C
D
E
1 2 3 4
1
2 3
2 3
4
4 5
RQE = { 1:C, 1:D }
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQC = { 1:C,
1:D }
RQD = { 1:C,
1:D }

Lamport Example
▸D isn’t granted the resource since 1:C ⇒ 1:D
35
C
D
E
1 2 3 4
1
2 3
2 3
4
4 5
5
RQE = { 1:C, 1:D }
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQC = { 1:C,
1:D }
RQD = { 1:C,
1:D }

Lamport Example
▸C is granted the resource since 1:C ⇒ 1:D and it
has received more recent messages from D and E
36
C
D
E
1 2 3 4 6
1
2 3
2 3
4
4 5
5
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQC = { 1:C,
1:D }
RQD = { 1:C,
1:D }
RQE = { 1:C, 1:D }

Lamport Algorithm
▸Rule 3
To release a resource, process i removes any Tm:i
requests from its request queue and sends a
timestamped release message to every other
process.
37

Lamport Example
▸C is finished with resource
38
C
D
E
1 2 3 4 6
1
2 3
2 3
4
4 5
5
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQC = { 1:C,
1:D }
RQD = { 1:C,
1:D }
RQE = { 1:C, 1:D }
7

Lamport Example
▸C removes its request from its own queue
39
C
D
E
1 2 3 4 6
1
2 3
2 3
4
4 5
5
req(1:D)
req(1:D)
req(1:C) ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQC = { 1:D }
RQD = { 1:C,
1:D }
RQE = { 1:C, 1:D }
7

Lamport Example
▸C releases the resource by sending release
messages to D and E
40
C
D
E
1 2 3 4 6 7
1
2 3
2 3
4
4 5
5
req(1:D)
req(1:D)
req(1:C) rel(7:C)
ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQD = { 1:C,
1:D }
RQE = { 1:C, 1:D }
RQC = { 1:D }

Lamport Algorithm
▸Rule 4
When process j receives a release message from
process i, it removes any Tm:i requests from its
request queue.
41

Lamport Example
42
C
D
E
1 2 3 4 6 7
1
2 3
2 3
4
4 5
5
req(1:D)
req(1:D)
req(1:C) rel(7:C)
ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQD = { 1:C,
1:D }
RQE = { 1:C, 1:D }
RQC = { 1:D }

Lamport Example
▸E removes C’s request
43
C
D
E
1 2 3 4 6 7
1
2 3
2 3
4
4 5
5
8
req(1:D)
req(1:D)
req(1:C) rel(7:C)
ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQD = { 1:C,
1:D }
RQE = { 1:D }
RQC = { 1:D }

Lamport Example
▸D removes C’s request
44
C
D
E
1 2 3 4 6 7
1
2 3
2 3
4
4 5
5
8
8
req(1:D)
req(1:D)
req(1:C) rel(7:C)
ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQD = { 1:D
}
RQC = { 1:D }
RQE = { 1:D }

Lamport Example
▸D is granted the resource since it has received
more recent messages than 1:D from C and E
45
C
D
E
1 2 3 4 6 7
1
2 3
2 3
4
4 5
5
8
8
req(1:D)
req(1:D)
req(1:C) rel(7:C)
ack(3:C)
ack(3:D)
ack(3:E) ack(5:E)
RQC = { 1:D }
RQE = { 1:D }
RQD = { 1:D
}

Complexity of Lamport Algorithm
▸ Parameters:
▹N: number of processes in the system
▹T: message transmission time
▹E: critical section execution time
▸ Message complexity: 3(N – 1)
▹N – 1 request messages
▹N – 1 acknowledgement messages
▹N – 1 release messages
▸ Response time (under light competition): 2T + E
▹Transmission time for request messages
▹Transmission time for acknowledgement messages
▹Execution time of current critical section
46

Ricart & Agrawala Algorithm
▸Assumptions:
▹Every message is eventually received but not necessarily
in order sent (i.e., lossless network)
▹Processes do not fail
▹Every request is sequenced based on the highest
sequence number seen before by that process
▹Every process is numbered to determine priority in the
event of equivalent sequence numbers
(lower the number, higher the priority)
▹Each rule is implemented in a separate thread with a
shared semaphore
Resource Allocation > Distributed Mutual Exclusion > Permission-Based > Ricart & Agrawala
47

▸Rule 1: Mutual Exclusion Invocation
requesting_critical_section = true;
sequence_no = highest_sequence_no + 1;
outstanding_replies = N – 1;
for(i = 0; i < N; i++) {
if(i != me) send(REQUEST(sequence_no, me), i);
}
while(outstanding_replies > 0) { }
critical_section();
requesting_critical_section = false;
for(i = 0; i < N; i++) {
if(reply_deferred[i]) {
reply_deferred[i] = false;
send(REPLY, i);
}
}
48

Ricart & Agrawala Example
▸Node numbering (for priority):
▹P = 0, Q = 1, R = 2
49
P
Q
R

▸R makes request (requesting_critical_section = true)
▸R’s sequence_no = highest_sequence_no + 1 = 0 + 1 = 1
▸R’s outstanding_replies = N – 1 = 3 – 1 = 2
50
P
Q
R

▸R sends REQUEST(sequence_no, me) messages to
P and Q
51
P
Q
R
req(1,2)

▸Q makes request (requesting_critical_section = true)
▸Q’s sequence_no = highest_sequence_no + 1 = 0 + 1 = 1
▸Q’s outstanding_replies = N – 1 = 3 – 1 = 2
52
P
Q
R
req(1,2)

▸Q sends REQUEST(sequence_no, me) messages to
P and R
53
P
Q
R
req(1,1)
req(1,1)
req(1,2)

▸Rule 2: Receiving Requests(j, k)
highest_sequence_no = max(highest_sequence_no, j);
defer = requesting_critical_section &&
((j > sequence_no) ||(j == sequence_no && k >
me));
if(defer) {
reply_deferred[k] = true;
}
else {
send(REPLY, k);
}
54

▸P receives Q’s REQUEST(1,1) message
▸ highest_sequence_no = max(highest_sequence_no, j)
= max(0, 1) = 1
55
P
Q
R
req(1,1)
req(1,1)
req(1,2)

▸ defer = requesting_critical_section &&
((j > sequence_no) ||(j == sequence_no && k > me))
= false && … = false
▸P sends REPLY message to Q
56
P
Q
R
reply
req(1,1)
req(1,1)
req(1,2)

▸Q receives R’s REQUEST(1,2) message
= max(0, 1) = 1
57
P
Q
R
reply
req(1,1)
req(1,1)
req(1,2)

= true && ((1 > 1) || (1 == 1 && 2 > 1)) = true
▸Q defers R’s request (reply_deferred[k] = true)
58
P
Q
R
reply
req(1,1)
req(1,1)
req(1,2)

▸R receives Q’s REQUEST(1,1) message
= max(0, 1) = 1
59
P
Q
R
reply
req(1,1)
req(1,1)
req(1,2)

= true && ((1 > 1) || (1 == 1 && 1 > 2)) = false
▸R sends REPLY message to Q
60
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(1,2)

▸P makes request (requesting_critical_section = true)
▸P’s sequence_no = highest_sequence_no + 1 = 1 + 1 = 2
▸P’s outstanding_replies = N – 1 = 3 – 1 = 2
61
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(1,2)

▸P sends REQUEST(sequence_no, me) messages to
Q and R
62
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0)
req(1,2)

▸Rule 3: Receiving Replies
outstanding_replies--;
63

▸Q receives R’s REPLY message
▸Q’s outstanding_replies = 2 – 1 = 1
64
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0)
req(1,2)

▸Q receives P’s REPLY message
▸Q’s outstanding_replies = 1 – 1 = 0
65
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0)
req(1,2)

for(i = 0; i < N; i++) {
}
critical_section();
for(i = 0; i < N; i++) {
send(REPLY, i);
}
}
66

▸Q enters it’s critical_section()
67
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0)
req(1,2)

▸Q receives P’s REQUEST(2,0) message
= max(1, 2) = 2
68
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0)
req(1,2)

= true && ((2 > 1) || (2 == 1 && 0 > 1)) = true
▸Q defers P’s request (reply_deferred[k] = true)
69
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0)
req(1,2)

▸R receives P’s REQUEST(2,0) message
= max(1, 2) = 2
70
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0)
req(1,2)

= true && ((2 > 1) || (2 == 1 && 0 > 2)) = true
▸R defers P’s request (reply_deferred[k] = true)
71
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0)
req(1,2)

for(i = 0; i < N; i++) {
}
critical_section();
for(i = 0; i < N; i++) {
send(REPLY, i);
}
}
72

▸Q exits critical section
▸Since reply_deferred[0], Q sends REPLY message to P
▸Since reply_deferred[2], Q sends REPLY message to R
73
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0) reply
reply
req(1,2)

▸P and R receive Q’s REPLY messages
▸P’s outstanding_replies = 2 – 1 = 1
▸R’s outstanding_replies = 2 – 1 = 1
74
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0) reply
reply
req(1,2)

▸P receives R’s REQUEST(1,2) message
= max(2, 1) = 2
75
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0) reply
reply
req(1,2)

= true && ((1 > 2) || (1 == 2 && 2 > 0)) = false
▸P sends REPLY message to R
76
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0) reply
reply
reply
req(1,2)

▸R receives P’s REPLY message
▸R’s outstanding_replies = 1 – 1 = 0
▸R enters it’s critical_section()
77
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0) reply
reply
reply
req(1,2)

▸R exits critical section
▸Since reply_deferred[0], R sends REPLY message to P
78
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0) reply
reply
reply
reply
req(1,2)

▸P receives R’s REPLY message
▸P’s outstanding_replies = 1 – 1 = 0
▸P enters it’s critical_section()
79
P
Q
R
reply
reply
req(1,1)
req(1,1)
req(2,0) reply
reply
reply
reply
req(1,2)

Complexity of Ricart & Agrawala
▸Message complexity: 2(N – 1)
▹N – 1 request messages
▹N – 1 reply messages
▸Response time (under light competition): 2T + E
▹Transmission time for request messages
▹Transmission time for reply messages
▹Execution time of current critical section
80

Deadlock and Starvation
▸Deadlock:
▹Occurs when no process in the system is in its critical
section and no requesting process can ever proceed to
its own critical section
▸Starvation:
▹Occurs when one process must wait indefinitely to
enter its critical section even though other nodes are
entering and exiting their own critical sections
81

Upcoming Classes
▸Quorum-based Distributed Mutual Exclusion
▸Token-based Distributed Mutual Exclusion
82

os distributed system theoretical foundation

More Related Content

Similar to os distributed system theoretical foundation

More from Anonymous9etQKwW

Recently uploaded

os distributed system theoretical foundation