Downloaded 518 times
![jane_doe
{
labels:[‘Org’,’User’],
id:’jane_doe’
}
{
labels:[‘Hive’,’Table’],
id:’dim_users’
}
{
labels:[‘Tableau’,’Chart’],
id:’12345’
}
dim_users
12345](https://image.slidesharecdn.com/dataportal-170518090903/75/Democratizing-Data-at-Airbnb-88-2048.jpg)
![jane_doe
{
labels:[‘Org’,’User’],
id:’jane_doe’
}
{
labels:[‘Hive’,’Table’],
id:’dim_users’
}
{
labels:[‘Tableau’,’Chart’],
id:’12345’
}
dim_users
12345](https://crownmelresort.com/image.slidesharecdn.com/dataportal-170518090903/75/Democratizing-Data-at-Airbnb-88-2048.jpg)
Uploaded byNeo4j
Democratizing Data at Airbnb
Airbnb aims to democratize data within the company by building a graph database of all internal data resources connected by relationships. This graph is queried through a search interface to help employees explore, discover, and build trust in company data. Challenges include modeling complex data dependencies and proxy nodes, merging graph updates from different sources, and designing a data-dense interface simply. Future goals are to gamify content production, deliver recommendations, certify trusted content, and analyze the information network.
More Related Content
Similar to Democratizing Data at Airbnb
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Democratizing Data at Airbnb
- 1. Democratizing Data at Airbnb CHRISWILLIAMS / JOHN BODLEY / MAY 11, 2017
- 2. Airbnb connects peopleto unique travel experiences
- 3.
- 4. tribal knowledge |ˈtrībəlˈnäləj | noun Tribal knowledge is any unwritten information that is not commonly known by others within a company
- 5. Relying on tribalknowledge stifles productivity
- 6. As Airbnb growsso do the challenges around the volume, complexity, and obscurity of data
- 7. In a largeand complex organization, with a sea of data resources, users struggle to find the right data
- 8. Data is oftensiloed, inaccessible, or lacks context
- 9. I’m a recoveringData Scientist who wants to democratize data, automate common workflows, surface relevant information, and provide context
- 10. Tables in ourHive data warehouse 200k
- 11. > 10,000 Superset chartsand dashboards > 6,000 Experiments and metrics > 6,000 Tableau workbooks and charts > 1,500 Knowledge posts Data resources Beyond the data warehouse
- 12. With many moredata sources and data types to love
- 13.
- 14. > 3,500 Airbnbemployees
- 15. Portland San Francisco Los Angeles Toronto NewYork Miami Sao Paulo Dublin London Paris Barcelona Berlin Milan Copenhagen New Delhi Seoul Beijing Tokyo Sydney Singapore Washington, DC > 20 Offices around the world
- 16.
- 17. To democratize dataand empower Airbnb employees to be data- informed by aiding with data exploration, discovery, and trust
- 18.
- 19.
- 20. It should befairly evident what we feed into the search indices
- 22. But are wemissing something?
- 24. The relevancy ofrelationships Nodes and relationships have equal standing created consumedSpoke 3
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34. Databases 6 APIs 4 Airflow DAG 1 We leverageall these data resources to build a graph in Hive comprising of nodes and relationships The workflow is run everyday though the graph is left to soak to prevent flickering
- 35.
- 36. Addressing graph flickering Theissue is certain types of relationships are sporadic in nature causing the graph to flicker
- 37. Persistent vs. transientrelationships Persistent relationships represent a snapshot in time createdSpoke 3
- 38. Persistent vs. transientrelationships Transient relationships represent events which are somewhat sporadic in nature M Tu W Th F consumedSpoke 3
- 39. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 40. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 41. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 42. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 43. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 44. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 45. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 46. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 47. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 48. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 49. The winding datapath Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 50. Logical Given our datais represented as a graph it is logical to use a graph database to store the data Nimble Performance wins when dealing with connected data versus relational databases Popular It is the world’s leading graph database and the community edition is free Integrative It integrates well with Python and Elasticsearch Why we choose Neo4j for our database The four main reasons
- 51. The Neo4j andElasticsearch symbiotic relationship Courtesy of two GraphAware plugins Neo4j plugin Provides bi-directional integration which transparently and asynchronously replicate data from Neo4j to Elasticsearch Elasticsearch plugin Enables Elasticsearch to consult with the Neo4j database during a search query to enrich the search rankings by leveraging the graph topology
- 52. Node label hierarchy :Entity :Org :Group:User :Tableau :Workbook:Chart :Hive :Schema :Table
- 53. jane_doe (:Entity:Org:User {id: ‘jane_doe’}) (:Entity:Hive:Table{id: ‘dim_users’}) (:Entity:Tableau:Chart {id: ‘12345’}) dim_users 12345
- 54. MATCH (n:Entity:Org:User {id:’<id>’}) USING INDEX n:User(id) RETURN n
- 55. From local toglobal uniqueness A mechanism to reference nodes in an abstract manner GraphAware UUID plugin Transparently assigns a globally unique UUID property to newly created elements (nodes and relationships) which cannot be changed or deleted Globally unique Enables us to uniquely identify a single node via the Entity label and UUID property which allows for parameterized queries which leads to faster query and execution times
- 56. MATCH (n:Entity {uuid:’<uuid>’}) USING INDEX n:Entity(uuid) RETURN n
- 57.
- 58.
- 59.
- 60. Designing the interfaceand user experience of a data tool should not be an afterthought
- 61. Technical data power user;the epitome of a tribal knowledge holder Daphne Data User personas Less data literate; needs to keep tabs on her team’s resources Manager Mel New employee, new team, or new to data; has no idea what’s going on Nathan New
- 62. Designing for dataexploration, discovery, and trust Company dataSearch Resource details &metadata User data Group data
- 63. Company dataSearch Userdata Group data Resource details &metadata
- 64. Search Resource details &metadata CompanydataUser data Group data Google-esque search filters Resource details & metadata Context, context, & context
- 65. Search Resource details &metadata CompanydataUser data Group data Surface relationships, everything’s a link to promote exploration Metadata & consumption Description, external link, social
- 66. Column details &value distributions Table lineage Enrich metadata on the fly Search Resource details &metadata Company dataUser data Group data
- 67. Search Resource details &metadata CompanydataUser data Group data
- 68. User details & metadata What they make, what they consume Search Resource details &metadata Company dataUser data Group data
- 69. Former employees also hold tribal knowledge Search Resource details &metadata Company dataUser data Group data
- 70. Group overview Search Resource details &metadata Company dataUser data Group data Thumbnails for maximum context Basic organization functionality Pinterest-like curation & suggested content
- 71. We gather over15,000 thumbnails from Tableau, Superset, and the Knowledge Repo
- 72. Search Resource details &metadata CompanydataUser data Group data Pinning flow from resource page Edit mode / draggable grid
- 73. ???? ?? Employees canfeel disconnected from Company-level metrics Search Resource details &metadata Company dataUser data Group data
- 74. The technology stack Application+ dependencies DOM Testing eslint enzyme mocha chai Application state Styling khan/aphrodite
- 75.
- 76. Proxy nodes Abstracting complexity wherenecessary while accurately modeling the data ecosystem Graph merging Non-trivial Git-like merging of graph updates Data-dense design Balancing simplicity and functionality is hard; most internal design resources are not made for data-rich apps Complex dependencies An umbrella data tool is vulnerable to changes in upstream resource dependencies The challenges
- 77.
- 78. Game-ification Provide content producers witha sense of value Alerts& recommendations Move from active exploration to deliver relevant updates and content suggestions Certified content Use certification to build trust and enable users to filter through a sea of stale content Network analysis Determine obsolete nodes, critical paths, lines of communication, etc. The future
- 79.
- 80. The Dataportal team Analytics&ExperimentationProducts John Bodley Software Engineer Eli Brumbaugh Experience Designer Jeff Feng Product Manager Michelle Thomas Software Engineer Chris Williams Data Visualization
- 82.
- 83.
- 84. Naturally bidirectional relationships associated Dealingwith mutual relationships
- 85. Naturally bidirectional relationships associated Modelingboth creates an unnecessary relationship associated
- 86. Naturally bidirectional relationships associated Mostefficient solution is to use a single relationship in the many-to-one direction
- 87. CREATE TABLE nodes( labels ARRAY<STRING>, id STRING, properties STRING )
- 88.
- 89. CREATE TABLE relationships( source STRUCT<labels:ARRAY<STRING>,id:STRING>, target STRUCT<labels:ARRAY<STRING>,id:STRING>, type STRING, properties STRING )
- 90. Efficient data retrieval Solution Createan index for every label keyed by the ID and UUID properties which in addition to index hints provides optimal node retrieval Problem Indexes provide for efficient data retrieval similar to a RDBMS primary key, however they are only defined for a single label as opposed to our tuple of hierarchical labels Restrictions and workarounds with Neo4j indexes