Downloaded 58 times
Uploaded bySid Anand
Software Developer and Architecture @ LinkedIn (QCon SF 2014)
The document provides details about Sid Anand's career and then discusses LinkedIn's software development process and architecture when he was there. It notes that when Sid started at LinkedIn in 2011, compiling the code took a long time due to the large codebase and many dependencies. It then describes how LinkedIn scaled to support hundreds of millions of members and thousands of employees by splitting the monolithic codebase into individual Git repos, using intermediate JARs to reduce dependencies, and connecting development machines to test environments instead of deploying everything locally. It also discusses LinkedIn's use of Kafka, search federation, and not making web service calls between data centers to scale across multiple data centers.
More Related Content
Similar to Software Developer and Architecture @ LinkedIn (QCon SF 2014)
Software Developer and Architecture @ LinkedIn (QCon SF 2014)
- 1. Software Development &Arch @ LinkedIn 1 Sid Anand QCon SF 2014 @r39132
- 2. About Me 2 * Current Life… Chief Architect @ ClipMine, a video discovery company QCon SF Program Committee member Dad to a very energetic 2 year old boy Previous Life… Architect in Search and Distributed Data @ LinkedIn Cloud Data Architect @ Netflix VP Engineering at Etsy Software Developer at eBay @r39132 2
- 3. A Closer Look@ LinkedIn 3@r39132 3
- 4. LinkedIn 4 * *** Then • Created in2002 in Reid Hoffman’s living room • In its first month of operation, LinkedIn added 4500 members! @r39132 4
- 5. LinkedIn 5 * Then • Created in2002 in Reid Hoffman’s living room • In its first month of operation, LinkedIn added 4500 members! Now • 332M members in 200 countries • 2 members sign up every second • >60% of members overseas • In Q3’14, 75% of new members came from overseas @r39132 5
- 6. LinkedIn 6 * Then • Created in2002 in Reid Hoffman’s living room • In its first month of operation, LinkedIn added 4500 members! Now • 332M members in 200 countries • 2 members sign up every second • >60% of members overseas • In Q3’14, 75% of new members are coming from overseas • Fastest growing demographic is not geographic, it’s students! • > 10% of user base already and growing! @r39132 6
- 7. LinkedIn 7 * Member-growth started toramp up during 2011, when we IPO’d • 2010 : 55M • 2011 : 90M (IPO) • 2012 : 145M • Q3’14 : 332M (note : numbers reflect start of year) We added ~ same number of users in 2010 than over previous 6 years! @r39132 7
- 8. LinkedIn 8 * *** Employee-growth also startedto ramp up during 2011 • 2010 : 500 • 2011 : 1K (IPO) • 2012 : 2100 • Q3’14: 6K (25% in Engineering) (note : numbers reflect start of year) @r39132 8
- 9.
- 10. 10@r39132 10 Alan Shepard •2nd man in space • 5th person to walk on the moon! • 1st person to hit a golf ball on the moon!
- 11. LinkedIn 11@r39132 11 When askedby reporters what he thought about while awaiting liftoff, he replied: "The fact that every part of this ship was built by the lowest bidder"
- 12. How did LinkedInscale for company and member growth? 12@r39132 12
- 13.
- 14. 14 Circa 2011 • Onmy first day at LinkedIn, I felt pretty excited! Software Development : Challenges @r39132 Linux Desktop • 8 Core • 64GB Ram Mac Air
- 15. 15 Circa 2011 • Onmy first day at LinkedIn, I felt pretty excited! Software Development : Challenges @r39132 Linux Desktop • 8 Core • 64GB Ram Mac Air
- 16. 16 Circa 2011 • ThenI tried to compile the code on my laptop! Software Development : Challenges @r39132 Linux Desktop • 8 Core • 64GB Ram Mac Air
- 17. 17 Circa 2011 • 300+code projects in a single SVN Repo • SVN checkout world & go-to-lunch • Needed a server-grade machine to compile it! • Ant build (world) & go-make-espresso • Almost every WAR was built from source not intermediate JARs • To test your code locally, you needed to locally deploy every service that your code depended on! (maybe 20) • So, yes, you need a machine that typically lives in your data center! Software Development : Challenges @r39132
- 18. 18 Circa 2011 • Assumethat your code is now • Written • Compiled • Locally Tested • What Next? Software Development : Challenges @r39132
- 19. 19 Circa 2011 • 500+developers were checking code into the master branch on the single repo! • So, someone broke master every day! • So • 3 hours to write, build, and locally test code • 3 days to commit it! Software Development : Challenges @r39132
- 20. 20 Software Development :Challenges @r39132
- 21. 21 Now (Solved) • Dowhat the open-source world does with some improvements! • Break the monolithic repo into many individual Git Repos! • Have WARs depend on intermediate JARs – don’t not build the world! • Do not deploy the world for local testing – just connect your Dev machine to a test environment! • What are the improvements? Software Development : Challenges @r39132
- 22.
- 23. 23 Software Development @r39132 1. Alicecommits code to Git 2. Alice sends a Review Board request to Bob & Cathy, owners of the files! 3. Both Bob & Cathy give ship-its 4. Alice amends her commit message with : RB=<review board id> BUILD-WAR=<list of wars to build> Code Reviews
- 24. 24 Software Development @r39132 1. Alicepushes code to our Gitorious server where the following verifications: 1. Pre-push Sanity Checks! Must pass of push rejected! 1. Have all owners of the changed files given ship-its? 2. Does the code build? 2. For JAR builds, also build upstream WARs! 3. Run Integration Tests! Code Push (Git Push)
- 25. 25 Software Development @r39132 1. Assumingthat all checks passed, the WAR is now available 2. Our system automatically deploys all wars to test servers 3. QA verifies the new builds QA Test / Staging
- 26. 26 Software Development @r39132 1. Serviceowner Dave canaries the new WAR 2. Our EKG system then compares the canary machine to one control machine for 1 hour of product traffic for the following: 1. CPU, Memory increase 2. Fan-in/Fan-out increase 3. Error rate increase 4. Latency increase Production - Canary
- 27. 27 Software Development @r39132 1. Serviceowner Dave reviews the EKG report 2. If it looks acceptable, he promotes the build to the rest of the cluster in all data centers Production - Promotion
- 28. How did LinkedInscale for company and member growth? 28@r39132 28
- 29.
- 30. Web Servers Oracle LinkedIn Architecture @r39132 30 Proto-typicalUse – Case • A member updates her profile with new skills, job title, and education • She also accepts a connection request from another member Behind the scenes • Web servers commit data to Oracle • What Happens Next?
- 31. Web Servers Oracle LinkedIn Architecture @r39132 31 WhatHappens Next? Profile Updates • She should should become instantly searchable by her new skills, job title, & education! • New groups and job ads should be recommended to her Connection Updates • The news feed should instantly reflect content updates from her new connection! • Also, based on the new connection, the PYMK widget should discover a new 2nd degree neighborhood!
- 32.
- 33. 33 We also havea data pipeline to capture high-throughput events that we need to count! Databases are not a good place to do high-TP atomic counting! Kafka is! • This is typically used for ranking signals • E.g. counts member page views to determine who are “hot” LinkedIn : Architecture @r39132
- 34. Web Servers (writers) Oracle LinkedIn Architecture @r39132 34 Kafka Databus Search Systems Caches GraphSystems Recommender Systems DownstreamStreams DW
- 35. LinkedIn Architecture :Single Data Center! @r39132 35
- 36. LinkedIn : Architecture: Single Data Center! @r39132 36
- 37. LinkedIn : Architecture: Multi-data Center Project @r39132 37
- 38. LinkedIn Architecture :Rule 1 @r39132 38 Partition your user base across the data centers! e.g. using Akamai GTM
- 39. LinkedIn Architecture :Rule1 @r39132 39 Problem! User 1 (mapped to DC1) updates his profile! How will User 2 (mapped to DC2) see it?
- 40. LinkedIn Architecture :Rule 2 @r39132 40 Link your data centers together at the data fabric level! Not a new concept! Cassandra has been doing it for a few years now in the OLTP database space!
- 41. LinkedIn Architecture :Rule 2 @r39132 41 Link your data centers together at the data fabric level! Not a new concept! Cassandra has been doing it for a few years now in the OLTP database space! LinkedIn’s Sources of Truth • We have to make both work in across multiple data centers!
- 42. LinkedIn Architecture :Rule 2 @r39132 42 Link your data centers together at the data fabric level! Not a new concept! Cassandra has been doing it for a few years now in the OLTP database space! LinkedIn’s Sources of Truth • We have to make both work in across multiple data centers! • Oracle is fairly easy : we use Oracle Golden-gate! • Kafka is also pretty easy!
- 43. LinkedIn : KafkaMulti-Data Center @r39132 43 Kafka Local Producer Consumer of Local Events Kafka Data Center 1
- 44. LinkedIn : KafkaMulti-Data Center @r39132 44 Kafka Local Producer Consumer of Local Events Kafka Local Producer Consumer of Local Events Kafka Data Center 2Kafka Data Center 1
- 45. LinkedIn : KafkaMulti-Colo @r39132 45 Kafka Local Producer Consumer of Local Events Consumer of Global Events Kafka Local Producer Consumer of Local Events Kafka Data Center 2Kafka Data Center 1
- 46. LinkedIn : KafkaMulti-Colo @r39132 46 Kafka Local Producer Kafka Global Consumer of Local Events Consumer of Global Events Kafka Local Producer Consumer of Local Events Kafka Data Center 2Kafka Data Center 1
- 47. LinkedIn : KafkaMulti-Colo @r39132 47 Kafka Local Producer Kafka Global Consumer of Local Events Consumer of Global Events Kafka Local Producer Kafka Global Consumer of Local EventsConsumer of Global Events Kafka Data Center 2Kafka Data Center 1
- 48. LinkedIn Architecture :Rule 3 @r39132 48 Don’t make any web service calls between data centers! It kills latency, which kills availability!
- 49.
- 50. How did LinkedInscale for company and member growth? 50@r39132 50
- 51.
- 52. 52 LinkedIn Search @r39132 Why isSearch important to LinkedIn? • Search is a significant income driver! • 332M members that recruiters pay to find! (Recruiter Search) • 2M+ jobs that companies pay to list so you can find them! (Job Search)
- 53. What Makes LinkedInSearch Unique? 53@r39132 53
- 54. 54 LinkedIn Search :Federated @r39132
- 55. LinkedIn Search :Federated @r39132 55 • We index many entities • members, jobs, companies, groups, universities, articles, slides, etc.. • These are separate (vertical) search-engines!
- 56. LinkedIn Search :Federated @r39132 56 • We index many entities • members, jobs, companies, groups, universities, articles, slides, etc.. • These are separate (vertical) search-engines! • When a user enters “sr software engineer”, which index should we look in? • Jobs, members, groups?
- 57. LinkedIn Search :Federated @r39132 57 • We index many entities • members, jobs, companies, groups, universities, articles, slides, etc.. • These are separate (vertical) search-engines! • When a user enters “sr software engineer” , which index should we look in? • Jobs, members, groups? • Can we simply send the request to all of the search engines and then show the most relevant results? • No • Ranks (scores) are not comparable across verticals
- 58. LinkedIn Search :Federated @r39132 58 • We index many entities • members, jobs, companies, groups, universities, articles, slides, etc.. • These are separate (vertical) search-engines! • When a user enters “sr software engineer” , which index should we look in? • Jobs, members, groups? • Can we simply send the request to all of the search engines and then show the most relevant results? • No • Ranks (scores) are not comparable across verticals • What if we pick a vertical based on a user feature? • Job seeker sees jobs, recruiter sees members • Intent Detection : done by Federator
- 59. LinkedIn Search :Query Rewriting @r39132 59 • Say a recruiter searches for “sr software eng” • There are 20+ ways to represent this title • senior swe • sr swe • senior software engineer
- 60. LinkedIn Search :Query Rewriting @r39132 60 • Say a recruiter searches for “sr software eng” • There are 20+ ways to represent this title • senior swe • sr swe • senior software engineer • To solve this, we can use a title standarizer, though not every title may have a canonical form! • If a standardized title exists, we can rewrite the user query • title:sr AND title:software AND title:eng std_title:sswe234
- 61. LinkedIn Search :Query Rewriting @r39132 61 • Say a recruiter searches for “sr software eng” • There are 20+ ways to represent this title • senior swe • sr swe • senior software engineer • To solve this, we developed a title standarizer! • If a standardized title exists, we can rewrite the user query • title:sr AND title:software AND title:eng std_title:sswe234 • Query Rewriting helps by expanding the search space by methods such as synonym expansion, spell correction, etc… So we need it!
- 62. LinkedIn Search :Flexible Scoring @r39132 62 • We index many entities! • Companies, Members, Universities, etc… • We use different scoring formulas and signals for each vertical • We need a way to easily plug-in different custom scorers!
- 63. LinkedIn Search :Open Source @r39132 63 • Leading open source alternatives (e.g. Lucene, ElasticSearch, SOLR) do not offer these! • Search Federation • Pluggable Query Rewriting • Pluggable and Flexible Scoring • They DO offer some distributed system management, which we will have to re-invent unfortunately
- 64. LinkedIn Search :Open Source @r39132 64 • Leading open source alternatives (e.g. Lucene, ElasticSearch, SOLR) do not offer these! • Search Federation • Pluggable Query Rewriting • Pluggable and Flexible Scoring • They DO offer some distributed system management, which we will have to re-invent unfortunately • So, we created Galene, LinkedIn’s new search architecture! https://engineering.linkedin.com/search/did-you-mean-galene
- 65.
- 66.
- 67.
- 68. 68 Galene Architecture :Querying @r39132 Federator Frontend Browser Vertical Search Node Vertical Broker • Query Rewriting (Pluggable) • Scatter-gather across shards • Lucene (optionally sharded) • Scoring (Pluggable) • Query Intent Detection • Result Blending Other Verticals ….
- 69. 69 Galene Architecture :Indexing (Offline) @r39132 Federator Frontend Browser Vertical Search Node Hadoop Vertical Indexer Node Vertical Broker Index Distribution Service Offline Index Building and Distribution • Batch-oriented, built daily • Builds offline ranking and rewriting models • Rebuilds Indexes when new fields added
- 70. 70 Galene Architecture @r39132 Federator Frontend Browser Vertical Search Node Hadoop Vertical Indexer Node Vertical Broker Index Distribution Service Offline IndexBuilding and Distribution • Bit-Torrent-based Index Distribution Service • Pushes new indexes and models to running services
- 71. 71 Galene Architecture @r39132 Federator Frontend Browser Vertical Search Node Vertical Live Updater Hadoop Vertical Indexer Node Vertical Broker Index Distribution Service KafkaDatabus Kafka Samza Online IndexUpdates • Online (near-real-time) indexer • Updates indexes between Hadoop builds
- 72. 72 Galene Architecture @r39132 Federator Frontend Browser Vertical Search Node Vertical Live Updater Hadoop Vertical Indexer Node Vertical Broker Index Distribution Service KafkaDatabus Kafka Samza Periodic Index Optimization •Snapshots live data into a compact format • Send ss-index to search nodes over bit- torrent
- 73.
Editor's Notes
- #4 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #13 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #14 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #18 It could take 3 hours to write and test code and 3 days to get it successfully committed and built!
- #19 It could take 3 hours to write and test code and 3 days to get it successfully committed and built!
- #20 It could take 3 hours to write and test code and 3 days to get it successfully committed and built!
- #23 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #29 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #30 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #37 2 Major Reasons 2 have MDC: What happens if a DC goes down! Planned : PL/SQL (Triggers, Stored Procedures) cannot be updated online. Can’t do a rolling push for every thing! MySQL upgrade! Unplanned : Bad code push, power outage, network outage Need to scale beyond your current DC DC have a fixed max capacity! Also, sometimes it becomes important to have close proximity to users
- #38 When I think about data centers going down, I think about drinking.. Dos equis beer specifically! And then I listen the advice of this man!
- #51 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #52 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #54 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #56 We approach this by targeting specific verticals using our Intent Detection Algorithm! E.g. john smith looks like a person, only hit member index E.g. IBM looks like a company or a job title in members or in jobs, hit company, member, and jobs E.g. Stanford looks like a university that someone works at, attended —> hit university, company, member E.g. Software Engineer : if we infer user is a recruiter or hiring manager, show candidates, else show jobs!
- #57 We approach this by targeting specific verticals using our Intent Detection Algorithm! E.g. john smith looks like a person, only hit member index E.g. IBM looks like a company or a job title in members or in jobs, hit company, member, and jobs E.g. Stanford looks like a university that someone works at, attended —> hit university, company, member E.g. Software Engineer : if we infer user is a recruiter or hiring manager, show candidates, else show jobs!
- #58 We approach this by targeting specific verticals using our Intent Detection Algorithm! E.g. john smith looks like a person, only hit member index E.g. IBM looks like a company or a job title in members or in jobs, hit company, member, and jobs E.g. Stanford looks like a university that someone works at, attended —> hit university, company, member E.g. Software Engineer : if we infer user is a recruiter or hiring manager, show candidates, else show jobs!
- #59 We approach this by targeting specific verticals using our Intent Detection Algorithm! E.g. john smith looks like a person, only hit member index E.g. IBM looks like a company or a job title in members or in jobs, hit company, member, and jobs E.g. Stanford looks like a university that someone works at, attended —> hit university, company, member E.g. Software Engineer : if we infer user is a recruiter or hiring manager, show candidates, else show jobs!
- #60 R2 User types in “Senior software Engineer”, but member represent this title in 20+ different ways! sr swe, senior swe, senior software eng, etc.. To solve this, we have a title standardizer! However, it may not standardize for every title. So, we need to rewrite the query before hitting Lucene, our core search engine! title:senior AND title:software AND title:engineer if standardized title exists, stdtitle:sswe2331 This is also called synonym expansion! So, we need some smart rewriter! Rewriters also handle spelling correction! “Senior sofware Enginer” —> “Senior software Engineer” The goal is to expand the search space and to also sometimes target results for more efficient retrieval
- #61 R2 User types in “Senior software Engineer”, but member represent this title in 20+ different ways! sr swe, senior swe, senior software eng, etc.. To solve this, we have a title standardizer! However, it may not standardize for every title. So, we need to rewrite the query before hitting Lucene, our core search engine! title:senior AND title:software AND title:engineer if standardized title exists, stdtitle:sswe2331 This is also called synonym expansion! So, we need some smart rewriter! Rewriters also handle spelling correction! “Senior sofware Enginer” —> “Senior software Engineer” The goal is to expand the search space and to also sometimes target results for more efficient retrieval
- #62 R2 User types in “Senior software Engineer”, but member represent this title in 20+ different ways! sr swe, senior swe, senior software eng, etc.. To solve this, we have a title standardizer! However, it may not standardize for every title. So, we need to rewrite the query before hitting Lucene, our core search engine! title:senior AND title:software AND title:engineer if standardized title exists, stdtitle:sswe2331 This is also called synonym expansion! So, we need some smart rewriter! Rewriters also handle spelling correction! “Senior sofware Enginer” —> “Senior software Engineer” The goal is to expand the search space and to also sometimes target results for more efficient retrieval
- #63 R3 Flexible scoring/ranking per vertical! For example, if you have heard about search, then you have heard about TF-IDF, a measure of how well a document matches a search query based on text in both query and document! This is typically used for ranking in web search. However, we are doing entity search across a social graph! If you search for “Mark”, we don’t just show all people named Mark! We can only show results in your 1-2-3 degree network! We prioritize results based on a combination of social network signals and text similarlity (TF-IDF) For example, we sort results by connect weight : the % of first degree connections the searcher has in common with each retrieved member!
- #64 We approach this by targeting specific verticals using our Intent Detection Algorithm! E.g. john smith looks like a person, only hit member index E.g. IBM looks like a company or a job title in members or in jobs, hit company, member, and jobs E.g. Stanford looks like a university that someone works at, attended —> hit university, company, member E.g. Software Engineer : if we infer user is a recruiter or hiring manager, show candidates, else show jobs!
- #65 We approach this by targeting specific verticals using our Intent Detection Algorithm! E.g. john smith looks like a person, only hit member index E.g. IBM looks like a company or a job title in members or in jobs, hit company, member, and jobs E.g. Stanford looks like a university that someone works at, attended —> hit university, company, member E.g. Software Engineer : if we infer user is a recruiter or hiring manager, show candidates, else show jobs!
- #66 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #67 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #68 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.
- #74 For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in.