Large scale computing

Proprietary & Confidential 6/7/2020 1
Large Scale Computing
@ Linkedin
Bhupesh Bansal
Software Engineer
Linkedin

The Plan
• What Is Linkedin
• Large Scale problems @ Linkedin
• Scalable Architectures
• Hadoop : The mighty elephant
• Questions

What is Linkedin?
 The largest Professional social network:
 Some high level statistics:
– 50M active users, in 147 Industries and 244 Countries (580+ Regions)
– ~10M unique daily visitors
– ~25M weekly searches (150 QPS at peak)
– ~50M weekly profile views
– ~2M connections per day

Why should you care?
 Control your brand
 Market yourself
 Find inside connections
 Learn from Wisdom of the crowd

What is your Identity?
How do you want the world to see and know about you?

My Network My Strength
1. Explore opportunities before its too late
2. Make your network go far. (well really really far !!)
3. Seek expert advice
4. Stand on the shoulder of giants.

Wisdom of your network
1. Slideshare : See slides uploaded by your network
2. Answers : See expert answers & who they are
3. Groups
4. …

Large Scale Problems @ Linkedin
9

Linkedin Network Cache
• Third Degree network
• Friends of Friends of Friends
• Scales exponentially
• first degree: 100
• second-degree : first-degree 2
• third-degree : second-degree 2
• Ranges from 10,000  20 M.
• Is Unique for every user
• How do we solve it.
• Custom graph engine app.
• Cache entire third degree for logged
in members
• Graph Algorithms
• Minimize memory footprint of cache
• Bit vector optimizations
• compressions : p4Delta ..

Linkedin Search
• Linkedin Search
• 50 M documents
• Business need to filter/rank
with third degree cache
• Response time : 30-50 ms.
• How do we solve it.
• Distributed search engine.
• Documents partitioned
• 0 -5M, 5-10M , ..
• Each partition fetch network in
range
• Filter/Rank and return results.
Search & Filter on Third Degree

Data Analytics
• Data is Linkedin primary asset
• Linkedin Data has immense value
• Career trends
• Bubble bursts
• Hot keywords
• find experts/ Influencers
• Big Data VS Better Algorithm
• Big fight in machine learning
community
• personal opinion : big data rocks
if you know your data
Image1 source : http://otec.uoregon.edu/images/data-to-wisdom.gif
Image2 source : http://www.nature.com/ki/journal/v62/n5/images/4493262f1b.gif

People You May Know
• Shows People you should be
connected to ?
• For 50 M member
• Potential 50 2 = 2500,000 Billion
• Can be Narrowed down
• Triangle closing
• School/Company overlap
• .. Other factors
• secret sauce 

Scalable Architectures
• What is a scalable architecture ?
• Ability to do more if desired
• With minimum effort
• w/o software rearchitecture
• How to scale ?
• Scale Vertically (easy but
costly)
• Scale Horizontally (Harder
but very scalable if done right)
• Divide & Conquer is the only
way to scale horizontally
• Sharding/Partitioning

Hadoop Origin?
 “For the last several years, every company involved in building large
web-scale systems has faced some of the same fundamental
challenges. While nearly everyone agrees that the "divide-and-
conquer using lots of cheap hardware" approach to breaking down
large problems is the only way to scale, doing so is not easy “
 Hadoop was born to fulfill this very need

What is Hadoop?
• An Open-source Java based implementation of distributed map-
reduce paradigm
 Apache project with very active community
– heavy support from Yahoo
 Distributed Storage + Distributed Processing + Job Management
 Runs on Commodity Hardware (Heterogeneous cluster)
 Scalable, Reliable , Fault tolerant and very easy to learn.
 Yahoo currently running clusters with 10,000 nodes that processes
10TB of compressed data for their production search relevance
processing
 Written by Lucene author Doug Cutting (and others)
 Plethora of valuable sub-projects
• Pig, Hive, HBase, Mahout, Katta

What is Hadoop used for?
• Search Relevance: Yahoo, Amazon, Powerset,
Zevents
• Log Processing: Facebook, Yahoo, Joost, Last.fm
• Recommendation System: Facebook
• Data Warehouse: Facebook, AOL
• Video and Image Analysis: New york times, eyealike
• Prediction models: Quantcast, Goldman Sachs
• Genome Sequencing: University of Maryland.
• Academia : University of Washington, Cornell ,
Stanford, Carnegie Mellon,

Why Use Hadoop
• Distributed/parallel programs are very very hard to
write
• Tip of iceberg is the actual program/business logic
• Map/Reduce code
• The hidden iceberg is
• Parallelization (Divide & Conquer)
• Data Management (TB to PB)
• Fault Tolerance
• Scalability
• Data local optimizations
• Monitoring
• Job isolation

Hadoop core components
• HDFS distributed file system
• User space distributed data management
• Not a real file system (no POSIX, not in the
kernel)
• Replication
• Rebalancing
• Very high aggregate throughput
• Files are immutable
• MapReduce layer
• Very simple but powerful programming model
• Move computation to the data
• Cope with hardware failures
• Non-goal: low latency

Map Reduce
• Google paper started it all:
•“MapReduce: Simplified Data Processing on Large Clusters”, OSDI
2004
• Map = per “record” computation (extract/transform phase)
• Each map task gets a block or piece of input data
• Shuffle = copy/collect data together (hidden from users)
• Collect all values for same key together.
• Can provide custom sort implementations.
• Reduce = final computation (aggregation/summarize phase)
• All key, value pairs for one key goes to one reducer.
• Reduce is optional

Map Reduce data flow

Map Reduce Example : WordCount
Image source : http://blog.jteam.nl/wp-content/uploads/2009/08/MapReduceWordCountOverview1.png

Map Reduce Implementation
• Jobs are divided into small pieces (Tasks)
• Improves load balancing
• Faster recovery from failed nodes
• Master
• What to run?
• Where to run?
• Slaves
• One tasktracker per machine
• Can configure how many mappers / reducers per tasktracker
• Manage task failures by re-starting on different node.
• Speculative execution if a task is slow or have failed earlier
• Maps and reduces are single threaded Individual JVMs
• Kill -9
• Resource Isolation
• Complete cleanup

Map Reduce
• Job tracker
• splits input and assigns to various map tasks
• Schedules and monitors map tasks (heartbeat)
• On completion, schedules reduce tasks
• Task tracker
• Execute map tasks – call mapper for every input record
• Execute reduce tasks – call reducer for every intermediate key, list of
values pair
• Handle partitioning of map outputs
• Handle sorting and grouping of reducer input
Jobtracker
tasktrackertasktrackertasktracker
Input Job (mapper, reducer, input)
Data
transfer
Assign tasks

HDFS II
 Namenode (master)
– Maps file Name to set of blocks
– Maps blocks to list of datanodes where it resides.
– Replication engine for blocks
– Checksum based data coherency checker.
– Issues
 Metadata in memory (single point of failure)
 Secondary name node checkpoints metadata.
 Datanodes (slaves)
– Stores blocks on local file system.
– Stores checksum of blocks
– Read/write data to clients directly.
– Periodically send a report of all existing blocks to Namenode.

Hadoop @ Linkedin
• People You May Know
• 40 individual jobs:
• Graph analysis
• School & Company overlap
• Collaborative filtering
• Misc. other factors
• Regression model combines all values
• Content scoring
• Collaborative Filtering
• Sessionization and cause analysis
• Phrase analysis
• Other derived data: company data, matching fields, seniority, etc.
• Search relevance
• Peoplerank
• Link keywords to results based on user behavior
• User bucketing

Data Visualization
• Can be very very useful.
• Saves ton of time
• Very insightful
• Pictures = 1000+ words
• Challenges
• Amount of data is huge
• Need to write visualization for each problem.
• Time spent per visualization is huge
• Hard to make it distributed/scalable

References
1. Hadoop. http://hadoop.apache.org/
2. Jeffrey Dean and Sanjay Ghemawat. MapReduce:
Simplified Data Processing on Large Clusters.
http://labs.google.com/papers/mapreduce.html
3. http://code.google.com/edu/parallel/index.html
4. http://www.youtube.com/watch?v=yjPBkvYh-ss
5. http://www.youtube.com/watch?v=-vD6PUdf3Js
6. S. Ghemawat, H. Gobioff, and S. Leung. The Google
File System. http://labs.google.com/papers/gfs.html
7. Linkedin.com
8. DJ Patil Talk at Scale Unlimited 2009

Large scale computing

More Related Content

What's hot

Similar to Large scale computing

Recently uploaded

Large scale computing

Editor's Notes