machinelearning project

machinelearning project

• 1.
  000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 Subspace Clustering withMissing Data Lianli Liu Dejiao Zhang Jiabei Zheng Abstract 1 Subspace clustering with missing data can be seen as the combination of subspace clustering and low rank matrix completion, which is essentially equivalent to high-rank matrix completion under the assumption that columns of the matrix X ∈ Rd×N belong to a union of subspaces. It’s a challenging problem, both in terms of computation and inference. In this report, we study two efficient algorithms proposed for solving this problem, the EM-type algorithm [1] and k-means form algorithm – k-GROUSE[2], and implement the two algorithms on both simulated and real dataset. Besides that we will also give a brief description of recently developed theorem [1] on the sampling complexity for subspace clustering, which is a great improvement over the previously existing theoretic analysis [3] that requires impractically sample size, e.g. the number of observed vectors per subspace should be super-polynomial in the dimension d. 1 Problem Statement Consider a matrix X ∈ Rd×N , with entries missing at random, whose columns lie in the union of at most K unknown subspaces of Rd , each has dimension at most rk < d. The goal of subspace clustering is to infer the underlying K subspaces from the partially observed matrix XΩ, where Ω denote the indexes of the observed entries, and to cluster the columns of XΩ into groups that lie in the same subspaces. 2 Sampling Complexity for Generic Subspace Clustering and EM Algorithm 2.1 Sampling Complexity [1] shows that subspace clustering is possible without impractically large sample size as that required by previous work [3]. The main assumptions used in the theoretic analysis of [1] is different from those arising from traditional Low- Rank Matrix Completion, e.g. [3]. It assumes the columns of X are drawn from a non-atomic distribution supported on a union of low-dimensional generic subspaces, and entries in the columns are missing uniformly at random. Before stating the main result in [1], we provide the following definition first. Definition 1. We denote the set of d × n matrices with rank r by M(r, d × n). A generic (d × n) matrix of rank r is a continuous M(r, d × n) valued random variable. We say a subspace S is generic if a matrix whose columns are drawn i.i.d according to a non-atomic distribution with support on S is generic a.s. The key assumptions of this method is following: • A1. The columns of the d × N data matrix X are drawn according to a non-atomic distribution with support on the union of at most K generic subspaces. The subspaces, denoted by S = {Sk|k = 0, . . . , K − 1}, each has rank exactly r < d. • A2. The probability that a column is drawn from subspace k is ρk. Let ρ∗ be the bound on mink{ρk}. 1 Authors by alphabetical order 1
• 2.
  052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 • A3. Weobserve X only on a set of entries Ω and denote the observation XΩ. Each entry in XΩ is sampled independently with probability p. We now state the main results in [1], which implies that if we observe at least order dr+1 (log d/r + log K) columns and at least O(r log2 d) entries in each column, then identification of the subspaces is possible with high probability. Although by assumption A2 the rank of each subspace is exactly r, this result can be generalized to a relax version that the dimension of each subspace is upper bounded by r. Theorem 1. [1] Suppose A1-A3 hold. Let > 0 be given. Assume the number of subspaces K ≤ 6 ed/4 , the total number of columns N ≥ (2d + 4M)/ρ∗, and p ≥ 1 d 128µ2 1rβ0 log2 (2d), β0 = 1 + log 6K 12 log(d) 2 log(2d) , M = de r + 1 r+1 (r + 1) log de r + 1 + log 8K (1) where µ2 1 := maxk d2 r UkV ∗ k 2 ∞ and UkΣkV ∗ k is the singular value decomposition of ˜X[k]2 . Then with probability at least 1 − , S can be uniquely determined from XΩ. 2.2 EM Algorithm In [4], computing the principal subspaces of a set of data vectors is formulated in a probabilistic framework and obtained by a maximum-likelihood method. Here we use the same model but extend it to the case of missing data: xo i xm i = K i=1 1zi=k Woi k Wmi k yi + µoi k µmi k + ηi (2) where 1, · · · , K zi ∼ ρ ⊥ yi ∼ N(0, I). Wk is a d × r matrix whose span is Sk, and ηi|zi ∼ N(0, σ2 zi I) is the noise in the zth i subspace. To find the Maximum Likelihood Estimate of θ = W, µ, ρ, σ2 , i.e. ˆθ = arg max θ l(θ, xo ) = arg max θ N i=1 log( K k=1 ρkP(xo i |zi = k; θ)) (3) an EM algorithm is proposed where Xo := {xo i }N i=1 is the observed data, Xm := {xm i }N i=1, Y := {yi}N i=1, Z := {zi}N i=1 are hidden variables. The iterates of the algorithm are as follows, with detailed derivation provided in Ap- pendix A. Wk = N i=1 pi,kEk xiyT i − N i=1 pi,kEk xi N i=1 pi,kEk yi T N i=1 pi,kEk yi T N i=1 pi,kEk yiyT i − N i=1 pi,kEk yi N i=1 pi,kEk yi T N i=1 pi,k −1 (4) µk = N i=1 pi,k Ek xi − WkEk yi N i=1 pi,k (5) σ2 k = 1 d N i=1 pi,k N i=1 pi,k tr(E xixT i ) − 2µT k Ek xi + µk T µk − 2tr(Ek yixT i Wk) + 2µT k WkEk yi + tr(Ek yiyT i Wk T Wk) (6) 2 ˜X[k] denote the columns of X corresponding to the kth subspace 2
• 3.
  104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 ρk = 1 N N i=1 pi,k ,pi,k := Pzi|xo i ,ˆθ(k) = ˆρkPxo i |zi=k,ˆθ(xo i ) K j=1 ˆρjPxo i |zi=j,ˆθ(xo i ) (7) where Ek · denotes the conditional expectation E·|xo i ,zi=k,ˆθ[·]. This conditional expectation can be easily calculated from its conditional distribution. Denote Mk := σ2 I + WoiT k Woi k , by Gauss-Markov theorem, xm i yi xo i ,zi=k,θ ∼ N µmi k + Wmi k M−1 k WoiT k (xo i − µoi k ) M−1 k WoiT k (xo i − µoi k ) , σ2 I + Wmi k M−1 k WmiT k Wmi k M−1 k M−1 k WmiT k M−1 k (8) Detailed derivation is provided in Appendix B. 3 Projection Residual Based K-MEANS Form Algorithm [2] proposes a form of k-means algorithm adapted to k subspaces clustering. It alternats between assigning vectors to subspaces based on the projection residuals and updates each subspace by performing an incremental gradient descent along the geodesic curve of Grassmannian 3 manifold(GROUSE[6]). 3.1 Subspace Assignment For subspace assignment, [2] proves that the incomplete data vector can be assigned to the correct subspace with high probability if the angles between the vector and the various subspaces satisfied some mild condition. For each data vector, when there is no entry missing, it’s natural to use the following v − PSi v 2 2 < v − PSj v 2 2 ∀j = i, and i, j ∈ {0, . . . , K − 1}. (9) where PSi is the projection operator onto subspace Si , to decide whether v can be assigned to Si or not. Now suppose we only observe the a subset of the entries of v, denoted as vΩ, where Ω ⊂ {1, . . . , d}, |Ω| = m < d. And define the projection operator restricted to Ω as PSΩ = (UT Ω UΩ)−1 UT Ω (10) In [7], the authors prove that UT Ω UΩ is invertible w.h.p as long as the assumptions of Theorem.2 are satisfied. Let Uk ∈ Rd×rk whose orthonormal columns span the rk dimensional subspaces Sk respectively. Then an natural question arises as under what condition we can use a similar decision rule as that of (9) for the subspace assignment for the incomplete data case. To answer this question, we will first restate a theorem in [2] to show that the projection residual will concentrated around that of the full data case by a scaling factor if the sampling size m satisfies certain condition. Let v = x + y, where x ∈ S, y ∈ S⊥ . Theorem 2. [2]4 Let δ > 0 and m ≥ 8 3 µ(S)r log 2r δ , then with probability at least 1 − 3δ, m(1 − α) − dµ(S)(1+β)2 1−γ d v − PSv 2 2 ≤ vΩ − PSΩ vΩ 2 2 (11) and with probability at least 1 − δ, vΩ − PSΩ vΩ 2 2 ≤ (1 + α) m d v − PSv 2 2 (12) where α = 2µ2(y) m log 1 δ , β = 2µ(y) log 1 δ and γ = 8rµ(S) 3m log 2r δ . 3 Grassmannian is a compact Riemannian manifold, and it’s geodesics can be explicitly computed [5] 4 In this theorem, µ(S), µ(y) is coherence parameter of subspace S and vector y, here we follow the definitions proposed in [8], µS := d r maxj PS ej 2 2 and µ(y) = d z 2 ∞ z 2 2 3
• 4.
  156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 Next we willstate the requirement on the angles between the vector and the subspaces, by which the incomplete data vector can be assigned to the correct subspace w.h.p. Define the angle between the vector v and its projection into the rk dimensional subspace Sk , k = 0, . . . , K −1 as θi = sin−1 v−PSk v 2 v 2 . And let Ck(m) = m(1−αk)−rkµ(Sk ) (1+βi)2 1−γi m(1+α0) , where αk, βk and γk are defined as that in Theorem.2 for different rk. Notice that Ck(m) < 1 and Ck(m) 1 as m → ∞. Without loss of generality, we assume θ0 < θk, ∀k = 0. Corollary 1. [2] Let m ≥ 8 3 maxi=0 riµ(Si ) log 2ri δ for fixed δ > 0. Assume that sin2 (θ0) ≤ Ci(m) sin2 (θi), ∀i = 0. Then with probability at least 1 − 4(K − 1)δ, vΩ − PS0 vΩ 2 2 < vΩ − PSi vΩ 2 2, ∀i = 0 Proof. Detail proof is available in Appendix C. 3.2 Subspace Update GROUSE [6] is a very efficient algorithm for single subspace estimation, although the theoretic support of this method is still unavailable5 . k-GROUSE [2] can be seen as a combination of k-subspaces and GROUSE for multiple subspace estimation, which is detailed in Algorithm.1. Algorithm 1 k-subspaces with the GROUSE Require: A collection of vectors vΩ(t), t = 1, . . . , T, and the observed indices Ω(t). An integer number of subspaces k and dimensions rk, k = 1, . . . , K. A maximum number of iterations, maxIter. A fixed step size η. Initialize Subspaces: Zero-fill the vectors and collect them in a mtrix V . Initialize k subspaces estimates using probabilistic farthest insertion. Calculate Orthonormal Bases Uj, k = 1, . . . , K. Let QjΩ = (UT kΩ UkΩ )−1 UT kΩ for i = 1, . . . , maxIter do Select a vector at random, vΩ. for k = 1, ..., K do Calculate weights of projection onto kth subspace: w(k) = QkΩ vΩ. Calculate Projection Residuals to kth subspace: rΩ(k) = vΩ − UkΩ w(k). end for Select min residual: k = arg mink rΩ(k) 2 2. Set w = w(k). Define p = Ukw and rΩc = 0, i.e, r is the zero filled rΩ(k) Update Subspace: Uk := Uk + (cos(ση) − 1) p p + sin(ση) r r wT w . (13) where σ = r p . end for 4 Experiment on simulated data In this section, we will explore the performance of k-GROUSE and EM algorithm on simulated data. Before we pre- sented the results we will first give a brief description about two key points of the implement of the algorithm. Firstly, we use a k-means++ like method to initialize the subspaces for both k-GROUSE and EM algorithm. Specifically, we randomly pick a point x0 ∈ X, and select it’s q 6 nearest nearest neighbors and calculate the best S0 w.r.t these selected q points. Next, we recursively initialize Sj by randomly fist pick a point x with probability proportional to 5 Local convergence of GROUSE is available in [9] 6 q is a nonnegative parameter, usually we set q = 2 maxk rk 4
• 5.
  208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 min[dist(x, S0 ), .. . , dist(x, Sj−1 )], and then find the best fit Sj of its q neighborhood. Secondly, the error is calcu- lated by examining the cluster assignments if a set of vectors who come from the same underlying subspace(which is known). For each set, we consider the cluster ID own the largest number of vectors to the true one, and the others as incorrectly clustered. 4.1 EM Algorithm The first experiment of EM algorithm is dealing with the simulated data generated by the Gaussian mixture model in section 2. We used d = 100, K = 4 and r = 5 for this part. For simulation, each of K subspaces is the span of an orthonormal basis generated from r i.i.d. standard Gaussian d-dimensional vectors. The performance of the EM algorithm strongly depends on initialization. With poor initial estimation of W and µ, estimated θ will get stuck by a local minimum of l(θ, xo ). Fig.1(a) gives an example of processing the same XΩ with different initial estimations, where we observed an obvious difference in performance. We ran 50 trials for each set of |ω|, Nk and σ2 , where |ω| is the number of observed entries in each column, Nk is the number of columns of each subspace and σ2 is the noise level. The stopping criteria is chosen as l(θ(t+1) , xo ) − l(θ(t) , xo ) < 10−4 l(θ(t) , xo ). The result are summarized in Fig.1(b)-(d). The difference between our results and results reported in [1] mainly comes from different initialization strategies. Generally, larger |ω| and Nk, and smaller σ2 gives less misclassified points, which is reasonable. For some cases, all misclassified points come from getting stuck at a local minimum. 0 50 100 150 200 250 300 350 400 −4 −2 0 2 4 6 x 10 4 Iterations LogLikelihood Good Bad (a) 0 50 100 150 200 250 0 0.2 0.4 0.6 0.8 1 N k Prop.ofmisclassifiedpoints (b) 10 15 20 25 30 35 40 0 0.2 0.4 0.6 0.8 1 |w| Prop.ofmisclassifiedpoints (c) −4 −3 −2 −1 0 0 0.2 0.4 0.6 0.8 1 log(σ 2 ) Prop.ofmisclassifiedpoints (d) Figure 1: (a) Good initial estimation: converges after 83 iterations, label correct rate = 100%. Bad initial estimation: converges after 395 iterations, label correct rate = 45.6%. (Nk = 200, ω = 24, σ2 = 10−4 ). (b)-(d) shows proportion of misclassified points: (b) as a function of Nk with |ω| = 24, σ2 = 10−4 . (c) as a function of |ω| with σ2 = 10−4 , Nk = 300. (d) as a function of σ2 with |ω| = 24, Nk = 210. 5
• 6.
  260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 4.2 k-GROUSE In thissection, we will explore the performance of k-GROUSE in two simulation scenarios. We set the ambient dimension d = 40 and the data matrix X consists of 80 vectors per subspace. In the first case, we let the number of subspaces K1 = 4 and and r = 5 for each subspace, i.e, the sum of the dimensions of the subspaces R = K k=1 rk < d. In the second case, we set the number of subspaces K2 = 5 and make one of them depended on the other four. From Fig.2(a) we can see that k-GROUSE perform well as long as the number of observations is a little bit more that r log2 d, which is within the order predicted by the theory. However, the performance of K-GROUSE is also depended on the initialization. Although k-means++ initialization can significantly improve the convergence of k-GROUSE, it can converge to some local point in the worse case. 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Sampling density probabilityoferror d = 40 dim = 5 K1 = 4 K2 = 5 (a) 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 Sampling densityprobabilityoferror d = 100 dim = 5 K = 4 k−GROUSE EM (b) Figure 2: (a) Performance of k-GROUSE with independent and dependent subspaces, obtained over 50 trials; (b) k -GROUSE vs EM Algorithm, obtained over 50 trials 4.3 k-GROUSE vs EM Algorithm Now we compare the performance of k-GROUSE and EM Algorithm in simulation data. In this setting, we set K = 4, r = 5, d = 100. From Figure.2(a) we can see that both k-GROUSE and EM Algorithm perform well in this situation, while k-GROUSE is more efficient. However, as we will see in the next section, the performance gap between the two will become obvious when we apply them to some worse conditioned real data, specifically when the conditional number of the data matrix X is large, k-GROUSE can perform poorly while EM algorithm still work under the same condition. In traditional matrix completion, the sampling density can heavily depend on the condition number of the matrix, it seems that it’s inherent here by k-GROUSE. 5 Experiment on real data 5.1 EM algorithm We test the algorithm on real data by applying it to image compression. An image of 264×536 is used, which is the same as the one used in mixture probabilistic PCA[4]. Similar with [4], the image was segmented into 8×8 non- overlapping blocks, giving a total dataset of 2211×64-dimensional vectors. In the experiment, we set the number of subspaces as 3, and test the performance of the algorithm under different missing rate (number of missing pixels in image block) and compression ratio. To make sure the resulting coded image has the same bit rate as the conventional SVD coded image, after the EM algorithm converges, the image coding was performed in a hard fashion, i.e. the coding efficient of each image block is only estimated using the subspace it has the largest probability of belonging to. We test the algorithm under different missing rate (number of missing pixels in each image block) and compression ratio. An initialization scheme that is slightly different from the simulated experiment is used: after setting all missing data to zero, a simple k-means clustering is performed on the data set. µk,Wk and σk of each subspace is initialized from the kth data cluster {xi} ∈ Rd using probabilistic PCA [4], where µk is initialized by the sample mean. Denote the sample covariance matrix as Sk = UΛV ,Wk ∈ Rd×r is initialized as Wk = Ur(Λr − σ2 I)1/2 (14) 6
• 7.
  312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 where σ2 is theinitial value of σ2 k = 1 d−r d j=r+1 λj, Ur ∈ Rd×r is composed of the leading r left singular vectors from U and Λr ∈ Λd×r is the corresponding singular value matrix. Sample images reconstructed under different missing rate (equivalently, the number of observing pixels per block ω) and compression ratio (number of components in each subspace r) are shown in Fig.3. Mean squared error (MSE) between the compressed image and the original image is plotted in Fig5. As compression ratio decreases, the quality of image improves significantly, as we can see from Fig3, the mean square error also decreases. While for different missing rate, the images appear visually the same, no significant difference is observed between missing data case and complete data case (though means square error decreases slightly). This validates the theory in [1] that as long as certain conditions are met and a reasonable number of datapoints are observed, with a high probability we can recover the true subspace from incomplete dataset. (a) ω=14,r = 4 (b) ω=64,r = 4 (c) ω=14,r = 12 (d) ω=64,r = 12 (e) ω=14,r = 32 (f) ω=64,r = 32 Figure 3: Image compression using EM algorithm 5.2 k-GROUSE algorithm We also test the k-GROUSE algorithm in image compression. We use the same block transformation scheme and number of subspaces as in the EM algorithm. The sample compressed images are shown below. 6 Conclusion From this project, we learned different subspace learning techniques, especially under missing data case. We also studied the theoretical requirement for recovering subspaces when data is incomplete. We also practiced the imple- mentation of EM algorithm and k-GROUSE. 7 Contribution Dejiao Zhang is in charge of abstract, sections 1, 2.1, 3, 4.2, 4.3 and (Appendix C). Jiabei Zheng rederived the EM algorithm for Gaussian mixture model (Appendix A), did coding and made plots in 4.1. Lianli Liu rederived the conditional distribution (Appendix B), did experiment on real dataset (section 5) and debugged the EM algorithm. 7
• 8.
  364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 (a) missing rate30%, number of modes 32 (b) missing rate 30%, number of modes 50 Figure 4: Image compression using k-GROUSE (a) MSE under different missing rate, mode = 12 (b) MSE under different compression ratio, missing rate = 78% Figure 5: Plot of mean square error References [1] D. Pimentel and R. Nowak, “On the sample complexity of subspace clustering with missing data.” [2] L. Balzano, A. Szlam, B. Recht, and R. Nowak, “k-subspaces with missing data,” in Statistical Signal Processing Workshop (SSP), 2012 IEEE. IEEE, 2012, pp. 612–615. [3] B. Eriksson, L. Balzano, and R. Nowak, “High-rank matrix completion and subspace clustering with missing data,” arXiv preprint arXiv:1112.5629, 2011. [4] M. Tipping and C. Bishop, “Mixtures of probabilistic principal component analyzers,” Neural computation, vol. 11, no. 2, pp. 443–482, 1999. [5] A. Edelman, T. A. Arias, and S. T. Smith, “The geometry of algorithms with orthogonality constraints,” SIAM journal on Matrix Analysis and Applications, vol. 20, no. 2, pp. 303–353, 1998. [6] L. Balzano, R. Nowak, and B. Recht, “Online identification and tracking of subspaces from highly incomplete information,” in Communication, Control, and Computing (Allerton), 2010 48th Annual Allerton Conference on. IEEE, 2010, pp. 704–711. [7] L. Balzano, B. Recht, and R. Nowak, “High-dimensional matched subspace detection when data are missing,” in Information Theory Proceedings (ISIT), 2010 IEEE International Symposium on. IEEE, 2010, pp. 1638–1642. [8] E. J. Cand`es and B. Recht, “Exact matrix completion via convex optimization,” Foundations of Computational mathematics, vol. 9, no. 6, pp. 717–772, 2009. [9] L. Balzano and S. J. Wright, “Local convergence of an algorithm for subspace identification from partial data,” arXiv preprint arXiv:1306.3391, 2013. 8
• 9.
  416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 Appendix A Here isthe complete derivation of the EM algorithm. The complete data probability is Pr(x, y, z; θ) = N i=1 Pr(xi, yi, zi) = N i=1 Pr(xi|yi, zi)Pr(yi|zi)Pr(zi) = N i=1 Pr(xi|yi, zi)Pr(yi)Pr(zi) = N i=1 ρkφ(xi; µk + Wkyi, σ2 kI)φ(yi; 0, I) (15) The complete data log-likelihood is l(θ; x, y, z) = N i=1 log(ρkφ(xi; µk + Wkyi, σ2 kI)φ(yi; 0, I)) = N i=1 K k=1 ∆iklog(ρkφ(xi; µk + Wkyi, σ2 kI)φ(yi; 0, I)) (16) where ∆ik = 1 if zi = k 0 else. (17) The E-step is Q(θ; ˆθ) = E[l(θ; x, y, z)|xo , ˆθ] = K k=1 N i=1 E[∆iklog(ρkφ(xi; µk + Wkyi, σ2 kI)φ(yi; 0, I))|xo , ˆθ] = K k=1 N i=1 K l=1 E[∆iklog(ρkφ(xi; µk + Wkyi, σ2 kI)φ(yi; 0, I))|xo , zi = l, ˆθ]Pr(zi = l|xo , ˆθ) = K k=1 N i=1 E[log(ρkφ(xi; µk + Wkyi, σ2 kI)φ(yi; 0, I))|xo , zi = k, ˆθ]Pr(zi = k|xo , ˆθ) = K k=1 N i=1 pik{log(ρk) − d + r 2 log(2π) − d 2 log(σ2 k) − 1 2σ2 k Ek (xi − µk − Wkyi)T (xi − µk − Wkyi) − 1 2 Ek yT i yi } = K k=1 ( N i=1 pik){log(ρk) − d + r 2 log(2π) − d 2 log(σ2 k) − 1 2σ2 k Uk − 1 2 Ek yT y } (18) where pik = Pr(zi = k|xo i , ˆθ) (19) 9
• 10.
  468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 Ek · =E[·|xo i , zi = k, ˆθ] (20) Ek u = N i=1 pikEk ui N i=1 pik (21) Ek uT v = N i=1 pikEk uT i vi N i=1 pik (22) Uk = Ek xT x + µT k µk + tr(Ek yT y WT k Wk) − 2µT k Ek x + 2µT k WkE y − 2tr(E yxT Wk) (23) The M-step is ˜θ = arg min θ Q(θ, ˆθ) (24) First derive ρ. ρ satisfies K i=1 ρk = 1. Using Lagrangian multiplier λ: ∂Q + λ K i=1 ρk ∂ρk = 0 (25) N i=1 pik + λρk = 0 (26) K k=1 N i=1 pik + λ = 0 (27) λ = −1 (28) Use this to eq(), we have: ρk = N i=1 pik (29) Then we eliminate σ2 with W fixed as W ∂Q ∂σ2 k = 0 ⇒ σk 2 = Uk d (30) σk 2 = 1 d (Ek xT x + µT k µk + tr(Ek yT y Wk T Wk) − 2µT k Ek x + 2µT k WkE y − 2tr(E yxT Wk)) (31) The elimination of µ is similar: ∂Q ∂µk = 0 ⇒ ∂Uk ∂µk = 0 (32) 10
• 11.
  520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 µk = Ekx − WkEk y (33) Then we derive expression for W. From Eq(18) we have: ∂Q ∂Wk = − d 2σ2 k ∂σ2 k ∂Wk − 1 2σ2 k ∂Uk ∂Wk + Uk 2σ4 k ∂σ2 k ∂Wk = − 1 2σ2 k ∂Uk ∂Wk = 0 (34) As a result, ∂Q ∂Wk = 0 ⇒ ∂Uk ∂Wk = 0 (35) We plug in expression of µk to Uk, We get ∂ ∂Wk {Ek xT x + (Ek x − WkEk y )T (Ek x − WkEk y ) + tr(Ek yT y WT k Wk) − 2(Ek x − WkEk y )T Ek x + 2(Ek x − WkEk y )T WkE y − 2tr(E yxT Wk)} = 0 (36) Wk = (Ek xyT − Ek x Ek yT )(Ek yyT − Ek y Ek yT )−1 (37) pik is calculated with Bayes’ rule: pik = Pr(zi = k; xo i , ˆθ) = ˆρkP(xo i |zi = k; ˆθ) K j=1 ˆρjP(xo i |zi = j; ˆθ) = ˆρkφ(xo i ; µoi k , Woi k Woi k T + σ2 kI) K j=1 ˆρjφ(xo i ; µoi j , Woi j Woi j T + σ2 j I) (38) 8 Appendix B By model definition, xo i xm i yi zi=k,θ =   Woi k Wmi k I   yi +   µoi k µmi k 0   + ηoi ηmi 0 (39) Eq(39) is a linear transform of yi ∼ N(0, I). By property of mulivariable Gaussian distribution, xo i xm i yi zi=k,θ ∼ N µoi k µmi k 0 ,   Woi k WoiT k + σ2 I Woi k WmiT k Woi k Wmi k WoiT k Wmi k WmiT k + σ2 I Wmi k WoiT k WmiT k I   (40) By Bayesian Gauss-Markov Theorem, xm i yi xo i ,zi=k,θ ∼ N(µmyi + Rmyoi R−1 oi (xo i − µoi k ), Rmyi − Rmyoi R−1 oi Romyi ) (41) where µmyi = µmi k 0 , Rmyoi = Wmi k WoiT k WoiT k , Rmyi = Wmi k WmiT k + σ2 I Wmi k WmiT k I (42) Roi = Woi k WoiT k + σ2 I, Romyi = [Woi k WmiT k Woi k ] (43) Plug Eq(42) and Eq(43) into Eq(41), Rmyoi R−1 oi = Wmi k WoiT k WoiT k (Woi k WoiT k + σ2 I)−1 (44) 11
• 12.
  572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 By matrix identities, (I+ PQ)−1 P = P(I + QP)−1 (45) Therefore WoiT k (Woi k WoiT k + σ2 I)−1 = (WoiT k Woi k + σ2 I)−1 WoiT k = (M−1 k )WoiT k (46) Put Eq(46) and Eq(44) together, Rmyoi R−1 oi = Wmi k WoiT k WoiT k [Woi k WoiT k + σ2 I]−1 = Wmi k M−1 k WoiT k M−1 k WoiT k (47) Therefore µmyi + Rmyoi R−1 oi (xo i − µoi k ) = µmi k + Wmi k M−1 k WoiT k (xo i − µoi k ) M−1 k WoiT k (xo i − µoi k ) (48) which is exactly the formula of mean given in the paper. To prove the formula of variance, by Eq(47) Rmyoi R−1 oi Romyi = Wmi k WoiT k WoiT k (Woi k WoiT k + σ2 I)−1 [Woi k WmiT k Woi k ] = Wmi k M−1 k WoiT k Woi k WmiT k Wmi k M−1 k WoiT k Woi k M−1 k WoiT k Woi k WmiT k M−1 k WoiT k Woi k (49) Thus Rmyi − Rmyoi R−1 oi Romyi = σ2 I + Wmi k (I − M−1 k WoiT k Woi k )WmiT k Wmi k (I − M−1 k WoiT k Woi k ) (I − M−1 k WoiT k Woi k )WmiT k I − M−1 k WoiT k Woi k (50) Notice that Mk − WoiT k Woi k = σ2 I ⇒ I − M−1 k WoiT k Woi k = σ2 M−1 k (51) Thus Eq(50) simplifies to Rmyi − Rmyoi R−1 oi Romyi = σ2 I + Wmi k M−1 k WmiT k Wmi k M−1 k M−1 k WmiT k M−1 k (52) which is exactly the formula of variance given in the paper. Appendix C Before we give the proof of Corollary.1, we examining the case of binary subspace assignment, i.e, K = 2. Let v ∈ Rd and S0 , S1 ⊂ Rn with dimension r0, r1 separately. Define the angle between v and its projection onto subspace S0 , S1 as θ0, θ1 : θ0 = sin−1 v − PS0 v 2 v 2 and θ1 = sin−1 v − PS1 v 2 v 2 (53) Theorem 3. [2] Let δ > 0 and m ≥ 8 3 d1µ(S1 log 2d1 δ . Assume that sin2 (θ0) < C(m) sin2 (θ1) Then with probability at least 1 − 4δ vΩ − PS0 vΩ 2 2 < vΩ − PS1 vΩ 2 2 Proof. By Theorem.2 and union bound, the following two statements hold simultaneously with probability at least 1 − 4δ, vΩ − PS0 Ω vΩ 2 2 ≤ (1 + α0) m n v − PS0 v and m(1 − α1) − r1µ(S1 )(1+β1)2 1−γ1 d v − PS1 v 2 2 ≤ vΩ − PS1 Ω vΩ 2 2 (54) 12
• 13.
  624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 Combining with (53),we have v − PS0 Ω 2 2 < C(m) v − PS1 v 2 2 will hold if and only if sin2 (θ0) < C(m) sin2 (θ1) which complete the proof Proof. of Corollary.1. The results of Theorem.3 can be generalized to the situation where there are multiple subspaces Si , i = 0, . . . , K − 1. Again without loss of generality we assume that θ0 < θi, ∀i, and define Ci(m) = m(1−αi)−riµ(Si ) (1+βi)2 1−γi m(1+α0) . Then the conclusion of Corollary.1 can be obtained by applying union bound and following the similar argument as that in the proof of Theorem.3. 13