Downloaded 17 times
![4
From Search to Recommendation
“The Web [...] is leaving the era of search and
entering one of discovery.
What's the difference?
Search is what you do when you're looking for
something. Discovery is when something wonderful
that you didn't know existed, or didn't know how to
ask for, finds you.”
– Jeffrey M. O'Brien: “The race to create a 'smart' Google”,
CNN Money / Fortune Magazine (2006)
http://archive.fortune.com/magazines/fortune/fortune_archive/2006/11/27/8394347/index.htm](https://image.slidesharecdn.com/dsep2015web-150304080455-conversion-gate01/75/Continuous-Evaluation-of-Collaborative-Recommender-Systems-in-Data-Stream-Management-Systems-4-2048.jpg)
![4
From Search to Recommendation
“The Web [...] is leaving the era of search and
entering one of discovery.
What's the difference?
Search is what you do when you're looking for
something. Discovery is when something wonderful
that you didn't know existed, or didn't know how to
ask for, finds you.”
– Jeffrey M. O'Brien: “The race to create a 'smart' Google”,
CNN Money / Fortune Magazine (2006)
http://archive.fortune.com/magazines/fortune/fortune_archive/2006/11/27/8394347/index.htm](https://crownmelresort.com/image.slidesharecdn.com/dsep2015web-150304080455-conversion-gate01/75/Continuous-Evaluation-of-Collaborative-Recommender-Systems-in-Data-Stream-Management-Systems-4-2048.jpg)
More Related Content
Similar to Continuous Evaluation of Collaborative Recommender Systems in Data Stream Management Systems
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Continuous Evaluation of Collaborative Recommender Systems in Data Stream Management Systems
- 1. Continuous Evaluation of CollaborativeRecommender Systems in Data Stream Management Systems Cornelius A. Ludmann, Marco Grawunder, Timo Michelsen, H.-Jürgen Appelrath {cornelius.ludmann, marco.grawunder, timo.michelsen, appelrath}@uni-oldenburg.de University of Oldenburg · Department of Computer Science Information Systems Group DSEP @ BTW 2015, Hamburg, Germany March 3, 2015 cornelius.ludmann.org · @ludmann
- 2. 2 Content ● Short introductionto Recommender Systems ● From static and finite rating datasets to an infinite stream of rating events ● Continuous query plan for collaborative RecSys ● Evaluation methodology for a DSMS-based RecSys ● Prototypical implementation with Odysseus
- 3. 3 Problem: Information Overload Photo:CCBY-NC2.0,https://www.flickr.com/photos/lyonora/3608656428/,UserLeonoraGiovanazzi Howto handle the flood of information?
- 4. 4 From Search toRecommendation “The Web [...] is leaving the era of search and entering one of discovery. What's the difference? Search is what you do when you're looking for something. Discovery is when something wonderful that you didn't know existed, or didn't know how to ask for, finds you.” – Jeffrey M. O'Brien: “The race to create a 'smart' Google”, CNN Money / Fortune Magazine (2006) http://archive.fortune.com/magazines/fortune/fortune_archive/2006/11/27/8394347/index.htm
- 5. 5 The Recommender Problem Estimatea utility function that predicts how a user will like an item.
- 6. 6 The Recommender Problem Estimatea utility function that predicts how a user will like an item. products
- 7. 7 The Recommender Problem Estimatea utility function that predicts how a user will like an item. movies
- 8. 8 The Recommender Problem Estimatea utility function that predicts how a user will like an item. music
- 9. 9 The Recommender Problem Estimatea utility function that predicts how a user will like an item. quantified by a rating score e. g., 1 to 5 stars:
- 10. 10 The Recommender Problem Estimatea utility function that predicts how a user will like an item. set of users set of items range of the rating score e. g.:
- 11. 11 The Recommender Problem Estimatea utility function that predicts how a user will like an item. How to estimate ? A common approach is Collaborative Filtering.
- 12. 12 Collaborative Filtering (CF) 53 3 1 5 5 4 3 1 2 4 1 4 5 ratings are given by the users: explicitly or implicitly by their behavior
- 13. 13 Collaborative Filtering (CF) 53 3 1 5 5 4 3 1 2 4 1 4 5 sparse rating matrix
- 14. 14 Collaborative Filtering (CF) Alearner “finds” an approximation to the true function : Examples for learners are: ● User similarities methods ● Matrix Factorization methods
- 15. 15 Collaborative Filtering (CF) 53 2 1 5 5 4 3 2.4 1 2 4 1 4 3 predict a rating for all unrated items recommend items with the highest predicted ratings
- 16. 16 What is about…? … different situations? … mood of the users? … (temporary) changes of user interests?
- 17. 17 Problem of TraditionalRecSys The interests of users on items can differ at different point in times. ● Depending on the situation of the user – Context awareness – Hidden contexts ● Due to changing user preferences – Concept drift Context-Aware RecSys consider context data. Time-Aware RecSys consider temporal effects.
- 18. 18 From Static RatingDatasets … Traditional RecSys learners use a static and finite set of rating data to build a RecSys model. Reoccurring model learning after a specific time span to incorporate new rating data.
- 19. 19 … to RatingEvents (Real-life RecSys) P Qx Model rating event (user, item, rating) at time t Rating events occur continuously, which leads to new learning data that potentially improves the predictions for all users.
- 20. 20 Related Work ● Time-awareCollaborative RecSys (e. g., Koren 2010) ● Incremental/Online Collaborative RecSys Algorithms (e. g., BRISMF by Gábor et al. 2009) ● Collaborative RecSys with Apache Storm by Ali et al. 2011 – Algorithm for parallel collaborative filtering ● StreamRec by Campos et al. 2011 – Based on Microsoft StreamInsight – Calculates item/user similarities with DSMS operators ● Massive Online Analysis (MOA) by Bifet et al. 2010 – Framework for evaluating machine learning algorithms – Implements BRISMF
- 21. 21 Why DSMS +RecSys? Characteristics of a (our) DSMS: ● Continuous queries on potentially infinite data streams ● Stream-based operators (one pass) / stream-based relational algebra ● Query plan as directed graph of operators ● Logical operators are transformed to physical operators ● Query optimizations, query sharing ● Time annotation of stream elements ● Operator framework
- 22. 22 Why DSMS +RecSys? What is the current interest of a user? ● Continuously processing of rating data ● Models are valid a specific time interval ● Deterministic temporal matching of model and data In general: ● Take advantages of DSMS features (like optimizations, flexible query formulation, …) ● Usage of established standard operators ● Extensibility like pre- and post-processing (e. g., context reasoning/modeling, normalization, …)
- 23. 23 RecSys Operators get unrated items predict rating train recsysmodel recommend Feedback rating event model with validity time interval request for recommendations recommendation candidates select top-k
- 24. 24 RecSys Operators get unrated items predict rating train recsysmodel recommend Feedback window Rating data is potentially infinite in size! A window operator limits the tuples in memory.
- 25. 25 RecSys Operators ● “trainrecsys model” Operator – Implements a learner to train a model. – Holds valid rating data. – Outputs models with a validity time interval. ● “get unrated items” Operator – Gets request for recommendations. – Outputs for each item that was not rated by the requesting user a tuple as recommendation candidate. ● “predict rating” Operator – Predicts for each recommendation candidate the rating. ● “recommend” Operator – Selects the recommendations for the requesting user. – Selects items with a min. rating and/or top-K items.
- 26. 26 Operators for ContinuousEvaluation get unrated items route predict rating train recsys model predict rating recommend test prediction Feedback window test data learning data prediction error
- 27. 27 Operators for ContinuousEvaluation ● “route” Operator – Distributes incoming data as learning or test data. – Implements an evaluation methodology. – e. g., Hold out: route 10 % as test data ● “predict rating” Operator – Predicts for each test tuple the rating. ● “test prediction” Operator – Calculates an error value for true and predicted . – e. g., Root Mean Square Error (RMSE)
- 28.
- 29. 29 Physical Query Plan requestsfor recommendations (RfR) rating data metadata creation (time interval) routes learning and test data joins RfR with temporal matching models joins test data with temporal matching models limits validity of learning data implements learner “now” window
- 30. 30 Physical Query Plan addspredicted rating to test tuple map operator aggregation size aggregation operator map operator outputs unrated items joins unrated items and models adds predicted rating to unrated item selects items with min rating selects top-K items
- 31. 31 Physical Operators ● “trainrecsys model” Operator – BRISMF (incremental matrix factorization) – BRISMF implementation of Massive Online Analysis (MOA) integrated in Odysseus http://moa.cms.waikato.ac.nz/details/recommender-systems/ – One model for each subset of valid learning tuples. – Exactly one valid model at every point in time. – Needs to hold all learning tuples in memory but does not build models from scratch.
- 32. 32 Physical Operators ● “interleavedtest-than-train” Operator – Physical counterpart to “route” operator – Using rating tuples for learning and testing – Sets validity interval of test tuple to [t-1, t) to ensure a matching to a model that has not used this tuple for learning ● “test prediction” Operator is implemented by a – map, → calculates square error – time window, → sets aggregation time span – aggregation, → aggregates errors (avg) – and another map operator → calculates root of error
- 33. 33 Plot of RootMean Square Error
- 34. 34 Prototype Evaluation ● Comparisonof RMSE after every learning tuple with Massive Online Analysis (MOA) – MovieLens dataset, ordered by timestamp, read line-by-line as rating data – No decay of learning tuples (unbounded window) – Aggregation of RMSE over the whole dataset – (Random users for request for recommendations) ● Same results as MOA – MOA operates sequentially (first test, than train), we ensure the correct order by the time annotation and the temporal join – Temporal matching works as expected
- 35. 35 Summary and FutureWork ● Generic, extendable and modular structure for a RecSys based on DSMS operators ● Logical operators allow different physical implementations ● Time annotations ensures deterministic temporal matching of models and data ● Prototypical implementation with BRISMF and Interleaved Test-Than- Train ● Future Work: – Implementation of learners that consider temporal aspects – Impact of decay of tuples (different windows) on accuracy, latency, throughput, memory consumption – Optimizations of algorithms, query plan, transformations … Thank you for your attention!