Big Data for Healthcare:
Usage, Architecture and Technologies
Presenters

Pete Stiglich – Sr. Technical Architect
       Over 20 years IT experience

       Enterprise Data Architecture, Data Management, Data Modeling, Data Quality, DW/BI,
        MDM, Metadata Management, Data Quality, Database Administration (DBA)

       President of DAMA Phoenix, writer, speaker, former editor Real World Decision Support,
        listed expert for SearchDataManagement – Data Warehousing and Data Modeling

       Certified Data Management Professional (CDMP) and Certified Business Intelligence
        Professional (CBIP), both at master level



    Email: Pete.Stiglich@Perficient.com

    Phone: 602-284-0992

    Twitter: @pstiglich

    Blog: http://blogs.perficient.com/healthcare/blog/author/pstiglich/
Presenters

Hari Rajagopal – Sr. Solution Architect
   •   Over 15 years IT experience

   •   SOA solutions, Enterprise Service Bus technologies, Data Architecture, Algorithms

   •   Presenter at conferences, Author and Blogger

   •   IBM certified SOA solutions designer



   Email: Hari.Rajagopal@Perficient.com

   Phone: 303-517-9634
Key Takeaway Points


•   Big Data technologies represent a major paradigm shift – and is
    here to stay!

•   Big Data enables “all” the data to be leveraged for new insight–
    clinical notes, medical literature, OR videos, X-rays, consultation
    recordings, streaming medical device data, etc.

•   More intelligent enterprise – more efficient and prevalent
    advanced analytics (predictive data mining, text mining, etc.)

•   Big Data will affect application development and data
    management
Agenda


•   What is Big Data?

 How Big Data can enable better healthcare

 Types of Big Data processing

 Key technologies

 Impacts of Big Data on:

      Application Development

      Data Management

 Q&A
What is Big Data?
What is “Big Data”


•   Datasets which are too large, grow too rapidly, or are too
    varied to handle using traditional techniques

•   Volume, Velocity, Variety

•   Volume – 100’s of TB’s, petabytes, and beyond

•   Velocity – e.g., machine generated data, medical devices,
               sensors

•   Variety – unstructured data, many formats, varying
              semantics



•   Not every data problem is a “Big Data” problem!!
MPP enables Big Data


                                            100’s, 1,000’s of nodes



                    Scalability                    Scalability

                                  Cluster (homogenous) or Grid (heterogeneous)




     SMP – Symmetric                      MPP – Massively Parallel
      Multiprocessing                           Processing
    “Shared Everything”                     “Shared Nothing”
CPU, memory, disk (SAN, NAS)                Nodes do not share
                                          CPU, memory, disk (DAS)
Cost Factor


 Cost of storing and analyzing Big Data can be driven down
  by:

      Low cost commodity hardware

      Open source software

      Public Cloud? Yes, But for really massive amounts of data with many
       accesses, may be cost prohibitive

      Learning curve? You bet!
Hadoop / MapReduce


•   Hadoop and MapReduce – key Big Data technologies
    developed at Google, now open source

•   “Divide and conquer” approach

•   Highly fault tolerant – nodes are expected to fail

•   Every data block (by default) replicated on 3 nodes
    (is also rack aware)

•   MapReduce – component of Hadoop, programming
    framework for distributed processing

•   Not the only Big Data technology…
NoSQL


•   Stands for “Not only SQL” – really s/b “Not only Relational”

 New(ish) paradigms for storing and retrieving data

 Many Big Data platforms don’t use a RDBMS

        Might take too long to setup / change

        Problems with certain types of queries (e.g., social media, ragged
         hierarchies)

 Key Types of NoSQL Data Stores
          •   Key-Value Pair
          •   Wide Column
          •   Graph
          •   Document
          •   Object
          •   XML
How can “Big Data” improve Healthcare?
Healthcare “Big Data” opportunities


•   Examples of Big Data opportunities
        Patient Monitoring – inpatient, ICU, ER, home health

        Personalized Medicine

        Population health management / ACO

        Epidemiology

        Keeping abreast of medical literature

        Research

        Many more…
Healthcare “Big Data” opportunities


•   Patient Monitoring

        Big Data can enable Complex Event Processing (CEP) – dealing with
         multiple, large streams of data in real-time from medical devices,
         sensors, RFID, etc.

        Proactively address risk, improve quality, improve processes, etc.

        Data might not be persisted – Big Data can be used for distributed
         processing with the data located only in memory

        Example – an HL7 A01 message (admit a patient) received for an
         inpatient visit – but no PV1 Assigned Patient Location received within X
         hours. Is the patient on a gurney in a hallway somewhere???

        Example – home health sensor in a bed indicates patient hasn’t gotten
         out of bed for X number of hours
Healthcare “Big Data” opportunities


•   Personalized Medicine
        Genomic, proteomic, and metabolic data is large, complex, and varied

        Can have gigabytes of data for a single patient

        Use case examples - protein footprints, gene expression

        Difficult to use with a relational database, XML performance problematic

        Use wide-column stores, graphs, key-value stores (or combinations) for better
         scalability and performance




                                                                                    Source:
                                                                                    wikipedia
Healthcare “Big Data” opportunities


•   Population Management
        Preventative care for ACO – micro-segmentation of patients

              Identify most at risk patients – allocate resources wisely to help these
               patients (e.g., 1% of 100,000 patients had 30% of the costs)*

              Reduce admits/re-admits, ER visits, etc.

        Identify potential causes for infections, readmissions (e.g., which two
         materials when used together are correlated with high rates of infection)



        Even with structured data, data mining can be time consuming – distributed
         processing can speed up data mining




                                                                    * http://nyr.kr/L8o1Ag (New
                                                                    Yorker article)
Healthcare “Big Data” opportunities


•   Epidemiology
        Analysis of patterns and trends in health issues across a geography

        Tracking of the spread of disease based on streaming data

        Visualization of global outbreaks enabling the determination of ‘source’ of infection




                                                                                                 17
Healthcare “Big Data” opportunities


•   Unstructured data analysis
        Most data (80%) resides in unstructured or semi-structured sources – and a wealth
         of information might be gleaned

        One company allows dermatology patients to upload pictures on a regular basis to
         analyze moles in an automated fashion to check for melanoma based on redness,
         asymmetry, thickness, etc.

        A lot of information contained in clinical notes, but hard to extract

        Providers can’t keep abreast of medical literature – even specialists! Use Big Data
         and Semantic Web technologies to identify highly relevant literature

        Sentiment analysis – using surveys, social media

        Etc…
Poll


•   What Healthcare Big Data use case do you see as being most
    important for your organization?



    •   Patient Monitoring
    •   Personalized Medicine
    •   Population Management (e.g., for ACO)
    •   Epidemiology
    •   More effective use of medical literature
    •   Medical research
    •   Unstructured data analysis
    •   Quality Improvement
    •   Other




                                                                   19
Types of Big Data processing
Analytics


•   Big Data ideal for experimental / discovery analytics

•   Faster setup, data quality not as critical

•   Enables Data Scientists to formulate and investigate
    hypotheses more rapidly, with less expense

•   May discover useful knowledge . . . or not

•   Fail faster – so as to move on to the next hypothesis !
Unstructured Data Mining


•   Big Data can make mining unstructured sources(text, audio,
    video, image) more prevalent - more cost effective, with better
    performance

•   E.g., extract structured information, categorize documents,
    analyze shapes, coloration, how long was a video viewed, etc.

•   Text Mining capabilities
     •   Entity Extraction – extracting names, locations, dates, products, diseases, Rx,
         conditions, etc., from text

     •   Topic Tracking – track information of interest to a user

     •   Categorization – categorize a document based on wordcounts/synonyms, etc.

     •   Clustering – grouping similar documents

     •   Concept Linking – related documents based on shared concepts

     •   Question Answering – try to find best answer based on user’s environment
Data Mining

                                                                     Text
•   Can enable much faster data mining

•   Can bypass some setup and modeling                          Text Mining
    effort
                                                         Other use              Entity
                                                         cases                  Extraction
•   Data Mining is “the automatic or semi-automatic
    analysis of large quantities of data to extract
    previously unknown interesting patterns” Wikipedia                            Data
                                                             Structured
                                                             Data                 Mining
•   Examples of data mining:

     •   Association analysis - e.g., which 2 or 3                            Something
         materials when used together are correlated                          Interesting?
         with a high degree of infection

     •   Cluster analysis – e.g., patient micro-
         segmentation

     •   Anomaly / Outlier Detection –e.g., network
         breaches
Transaction Processing


•   Some Big Data platforms can be used for some types of
    transaction processing

•   Where performance is more important than consistency e.g.,
    a Facebook user updating his/her status

•   More on this later…
Poll


•   What type of Big Data use case would be most beneficial for
    your client?

     •   Complex Event Processing (using massive/numerous
         streams of real-time data)

     •   Unstructured Data Analysis

     •   Predictive Data Mining

     •   Transaction Processing (where performance more
         important than consistency)




                                                                    25
Big Data Architecture and Key Technologies
Big Data Stack
Hadoop

•   Used for batch processing – inserts/appends only – no updates

•   Single master – works across many nodes, but only a single data
    center

•   Key components

     •   HDFS – Hadoop Distributed File System

     •   MapReduce – Distributes data in key value pairs across nodes, parallel
         processing, summarize results

     •   Hbase – database built on top of Hadoop (with interactive capabilities)

     •   Hive – SQL like query tool (converts to MapReduce)

     •   Pig – Higher level execution language (vs. having to use Java, Python) –
         converts to MapRduce




                                                                                       28
Cassandra


•   Used for real-time processing / transaction processing

•   Multiple masters – works across many nodes and many data
    centers

•   Key components

     •   CFS – Cassandra File Systems

     •   CQL – Cassandra Query Language (SQL like)

•   Tunable consistency for writes or reads. E.g., option to ensure a write
    succeeds to each replica in all data centers before returning control to
    program …. or can be much less restrictive




                                                                               29
In memory processing


•   To support real-time operations, an IMDB (In-Memory Database)
    may be used

     •   Solo – or in conjunction with a disk based DBMS

•   I/O most expensive part of computing – using in memory database /cache
    reduces bottlenecks

•   Can be distributed (e.g., memcache, Terracotta, Kx)

•   Relational or non-relational

     •   E.g., for a DW, current values might reside in an IMDB, historical data on disk




                                                                                           30
MPP RDBMS


•   Have been in around for 15+ years

•   Used for large scale Data Warehousing

•   Ideal where lots of joins are needed on massive amount of data

•   Many NoSQL DB’s rely on 100% denormalization. Many do not
    support join operations (e.g., wide column stores) or updates




                                                                     31
Semantic Web


•   Semantic Web – web of data, not documents

•   Machine learning (inferencing) can be enabled via Semantic Web
    technologies. May use a graph database/triplestore (e.g.,
    Neo4j, Allegrograph, Meronymy)

•   Bridge the semantic divide (varying vocabularies) with
    ontologies – helps address the “Variety” aspect of Big Data

•   Encapsulate data values, metadata, joins, logic, business rules,
    ontologies, access methods in the data via common logical model
    (e.g., RDF triples) – very powerful for automation, federated
    queries




                                                                       32
Semantic Web
Find Jane Doe’s relatives (with machine inferencing)

           System X                            System Y                   System Z


                      a:JoeDoe                    :isInLaw

            :hasBrother          :hasBrother

                                                             :marriedTo
x:DebDoe                                       y:JohnDoe                    z:JaneDoe
                  :hasBrother



                                          :isInLaw
                                                                            Original data
                                                                            Inferred data


                                                                                        33
No One Size Fits All


 Many types of solutions will require multiple data
  paradigms

 E.g. Facebook uses MySQL (relational), Hadoop, Cassandra,
  Hive, etc., for the different types of processing required

 Be sure to have a solid use case before deciding to use Big
  Data / NoSQL technology

 Provide solid business and technical justification
What type of data store to use??
Big Data impact on Application Development
           and Data Management
ACID / CAP / BASE


 If your transaction processing application must be ACID compliant, you must
  use an RDBMS (or ODBMS)

 ACID – Atomic, Consistent, Isolated, Durable

        Atomic – All tasks in a transaction succeed – or none do
        Consistent – Adheres to db rules, no partially completed transactions
        Isolated – Transactions can’t see data from other uncommitted transactions
        Durable – Committed transaction persists even if system fails



 Not all transactions require ACID – eventual consistency may be adequate



                                       Vs..
ACID / CAP / BASE


 Brewer’s CAP theorum for distributed database

      Consistency, Availability, Partition Tolerance - Pick 2!

 For Big Data, BASE is alternative for ACID


     Basically Available – data will be available for requests, might not be consistent

     Soft state – due to eventual consistency, the system might be continually changing

     Eventually consistent – the system will eventually be consistent when input stops

•   Example: HBase every transaction will execute, but only the most recent for a
    key will persist (LILO – last in, last out) – no locking
Data Management


 Security not as mature with NoSQL – might use OS level encryption (e.g.,, IBM
  Guardium Encryption Expert, Gazzanga) - encyrpt/decrypt at IO level

 Data Governance needs to oversee Big Data – new knowledge uncovered can
  lead to risks - privacy, intellectual property, regulatory compliance, etc.

•   Physical Data Modeling less important – due to “schema-less” nature of NoSQL

     •   Conceptual Modeling still important for understanding business objects and
         relationships
     •   Semantic modeling – inform ontologies which enable inferencing
     •   Logical Data Modeling still useful for reasoning and communicating about how
         data will be organized

•   Due to schema-less nature of NoSQL – metadata management will be more
    important!
      • E.g., wide-column store with billions of records and millions of variable columns
        – useless unless you have the metadata to understand the data
Getting started


•   Data Scientist is a key role in Big Data – requires statistics, data modeling, and
    programming skills. Not many around and expect to pay $$$’s.

•   Big Data technologies represent a significant paradigm shift. Be sure to allow budget
    for training, sandbox environment, etc.

•   Start small with Big Data . Start with a single use case – allocate significant
    amount of time for learning curve, and environment setup, testing, tuning,
    management.

•   Working with open source software can present challenges. Investigate purchase of
    value added software for simplification. Tools such as IBM Big Insights, EMC
    Greenplum UAP (Unified Analytics Platform) adds analytical, administration, workflow,
    security, and other functionality.




                                                                                         40
Summary
Summary


 Big Data presents significant opportunities

 Big Data is distinguished by volume, velocity, and variety

 Big Data is not just Hadoop / MapReduce and not just NoSQL

 Key enabler for Big Data is Massively Parallel Processing (MPP)

 Using commodity hardware and open source software are options to drive
  down cost of Big Data

 Big Data and NoSQL technologies require a learning curve, and will continue to
  mature
Resources


 Perficient Healthcare: http://healthcare.perficient.com

 Perficient Healthcare IT blog: http://blogs.perficient.com/healthcare/

 Perficient Healthcare Twitter: @Perficient_HC

 Apache – download and learn more about Hadoop, Cassandra, etc.

     http://hadoop.apache.org/

     http://cassandra.apache.org/

 Comprehensive list with description of NoSQL databases: http://nosql-
  database.org/links.html

 Translational Medicine Ontology (TMO) - applying Semantic Web for
  personalized medicine: http://www.w3.org/wiki/HCLSIG/PharmaOntology
Q&A
About Perficient




Perficient is a leading information technology consulting firm serving
clients throughout North America.

We help clients implement business-driven technology solutions that
integrate business processes, improve worker productivity, increase
customer loyalty and create a more agile enterprise to better respond
to new business opportunities.
PRFT Profile
   Founded in 1997

   Public, NASDAQ: PRFT

   2011 Revenue of $260 million

   20 major market locations throughout North America
     — Atlanta, Austin, Charlotte, Chicago, Cincinnati, Cleveland,
       Columbus, Dallas, Denver, Detroit, Fairfax, Houston,
       Indianapolis, Minneapolis, New Orleans, Philadelphia, San
       Francisco, San Jose, St. Louis and Toronto


   1,800+ colleagues

   Dedicated solution practices

   600+ enterprise clients (2011) and 85% repeat business
    rate

   Alliance partnerships with major technology vendors

   Multiple vendor/industry technology and growth awards
Our Solutions Expertise & Services

Business-Driven Solutions              Perficient Services
• Enterprise Portals                    End-to-End Solution Delivery
• SOA and Business Process              IT Strategic Consulting
  Management                            IT Architecture Planning
• Business Intelligence                 Business Process & Workflow
• User-Centered Custom Applications       Consulting
• CRM Solutions                         Usability and UI Consulting
• Enterprise Performance Management     Custom Application Development
• Customer Self-Service                 Offshore Development
• eCommerce & Product Information       Package Selection, Implementation
  Management                              and Integration
• Enterprise Content Management         Architecture & Application Migrations
• Industry-Specific Solutions           Education
• Mobile Technology
• Security Assessments
                       Perficient brings deep solutions expertise and offers
                       a complete set of flexible services to help clients
                       implement business-driven IT solutions                  47

Using Big Data for Improved Healthcare Operations and Analytics

  • 1.
    Big Data forHealthcare: Usage, Architecture and Technologies
  • 2.
    Presenters Pete Stiglich –Sr. Technical Architect  Over 20 years IT experience  Enterprise Data Architecture, Data Management, Data Modeling, Data Quality, DW/BI, MDM, Metadata Management, Data Quality, Database Administration (DBA)  President of DAMA Phoenix, writer, speaker, former editor Real World Decision Support, listed expert for SearchDataManagement – Data Warehousing and Data Modeling  Certified Data Management Professional (CDMP) and Certified Business Intelligence Professional (CBIP), both at master level Email: Pete.Stiglich@Perficient.com Phone: 602-284-0992 Twitter: @pstiglich Blog: http://blogs.perficient.com/healthcare/blog/author/pstiglich/
  • 3.
    Presenters Hari Rajagopal –Sr. Solution Architect • Over 15 years IT experience • SOA solutions, Enterprise Service Bus technologies, Data Architecture, Algorithms • Presenter at conferences, Author and Blogger • IBM certified SOA solutions designer Email: Hari.Rajagopal@Perficient.com Phone: 303-517-9634
  • 4.
    Key Takeaway Points • Big Data technologies represent a major paradigm shift – and is here to stay! • Big Data enables “all” the data to be leveraged for new insight– clinical notes, medical literature, OR videos, X-rays, consultation recordings, streaming medical device data, etc. • More intelligent enterprise – more efficient and prevalent advanced analytics (predictive data mining, text mining, etc.) • Big Data will affect application development and data management
  • 5.
    Agenda • What is Big Data?  How Big Data can enable better healthcare  Types of Big Data processing  Key technologies  Impacts of Big Data on:  Application Development  Data Management  Q&A
  • 6.
  • 7.
    What is “BigData” • Datasets which are too large, grow too rapidly, or are too varied to handle using traditional techniques • Volume, Velocity, Variety • Volume – 100’s of TB’s, petabytes, and beyond • Velocity – e.g., machine generated data, medical devices, sensors • Variety – unstructured data, many formats, varying semantics • Not every data problem is a “Big Data” problem!!
  • 8.
    MPP enables BigData 100’s, 1,000’s of nodes Scalability Scalability Cluster (homogenous) or Grid (heterogeneous) SMP – Symmetric MPP – Massively Parallel Multiprocessing Processing “Shared Everything” “Shared Nothing” CPU, memory, disk (SAN, NAS) Nodes do not share CPU, memory, disk (DAS)
  • 9.
    Cost Factor  Costof storing and analyzing Big Data can be driven down by:  Low cost commodity hardware  Open source software  Public Cloud? Yes, But for really massive amounts of data with many accesses, may be cost prohibitive  Learning curve? You bet!
  • 10.
    Hadoop / MapReduce • Hadoop and MapReduce – key Big Data technologies developed at Google, now open source • “Divide and conquer” approach • Highly fault tolerant – nodes are expected to fail • Every data block (by default) replicated on 3 nodes (is also rack aware) • MapReduce – component of Hadoop, programming framework for distributed processing • Not the only Big Data technology…
  • 11.
    NoSQL • Stands for “Not only SQL” – really s/b “Not only Relational”  New(ish) paradigms for storing and retrieving data  Many Big Data platforms don’t use a RDBMS  Might take too long to setup / change  Problems with certain types of queries (e.g., social media, ragged hierarchies)  Key Types of NoSQL Data Stores • Key-Value Pair • Wide Column • Graph • Document • Object • XML
  • 12.
    How can “BigData” improve Healthcare?
  • 13.
    Healthcare “Big Data”opportunities • Examples of Big Data opportunities  Patient Monitoring – inpatient, ICU, ER, home health  Personalized Medicine  Population health management / ACO  Epidemiology  Keeping abreast of medical literature  Research  Many more…
  • 14.
    Healthcare “Big Data”opportunities • Patient Monitoring  Big Data can enable Complex Event Processing (CEP) – dealing with multiple, large streams of data in real-time from medical devices, sensors, RFID, etc.  Proactively address risk, improve quality, improve processes, etc.  Data might not be persisted – Big Data can be used for distributed processing with the data located only in memory  Example – an HL7 A01 message (admit a patient) received for an inpatient visit – but no PV1 Assigned Patient Location received within X hours. Is the patient on a gurney in a hallway somewhere???  Example – home health sensor in a bed indicates patient hasn’t gotten out of bed for X number of hours
  • 15.
    Healthcare “Big Data”opportunities • Personalized Medicine  Genomic, proteomic, and metabolic data is large, complex, and varied  Can have gigabytes of data for a single patient  Use case examples - protein footprints, gene expression  Difficult to use with a relational database, XML performance problematic  Use wide-column stores, graphs, key-value stores (or combinations) for better scalability and performance Source: wikipedia
  • 16.
    Healthcare “Big Data”opportunities • Population Management  Preventative care for ACO – micro-segmentation of patients  Identify most at risk patients – allocate resources wisely to help these patients (e.g., 1% of 100,000 patients had 30% of the costs)*  Reduce admits/re-admits, ER visits, etc.  Identify potential causes for infections, readmissions (e.g., which two materials when used together are correlated with high rates of infection)  Even with structured data, data mining can be time consuming – distributed processing can speed up data mining * http://nyr.kr/L8o1Ag (New Yorker article)
  • 17.
    Healthcare “Big Data”opportunities • Epidemiology  Analysis of patterns and trends in health issues across a geography  Tracking of the spread of disease based on streaming data  Visualization of global outbreaks enabling the determination of ‘source’ of infection 17
  • 18.
    Healthcare “Big Data”opportunities • Unstructured data analysis  Most data (80%) resides in unstructured or semi-structured sources – and a wealth of information might be gleaned  One company allows dermatology patients to upload pictures on a regular basis to analyze moles in an automated fashion to check for melanoma based on redness, asymmetry, thickness, etc.  A lot of information contained in clinical notes, but hard to extract  Providers can’t keep abreast of medical literature – even specialists! Use Big Data and Semantic Web technologies to identify highly relevant literature  Sentiment analysis – using surveys, social media  Etc…
  • 19.
    Poll • What Healthcare Big Data use case do you see as being most important for your organization? • Patient Monitoring • Personalized Medicine • Population Management (e.g., for ACO) • Epidemiology • More effective use of medical literature • Medical research • Unstructured data analysis • Quality Improvement • Other 19
  • 20.
    Types of BigData processing
  • 21.
    Analytics • Big Data ideal for experimental / discovery analytics • Faster setup, data quality not as critical • Enables Data Scientists to formulate and investigate hypotheses more rapidly, with less expense • May discover useful knowledge . . . or not • Fail faster – so as to move on to the next hypothesis !
  • 22.
    Unstructured Data Mining • Big Data can make mining unstructured sources(text, audio, video, image) more prevalent - more cost effective, with better performance • E.g., extract structured information, categorize documents, analyze shapes, coloration, how long was a video viewed, etc. • Text Mining capabilities • Entity Extraction – extracting names, locations, dates, products, diseases, Rx, conditions, etc., from text • Topic Tracking – track information of interest to a user • Categorization – categorize a document based on wordcounts/synonyms, etc. • Clustering – grouping similar documents • Concept Linking – related documents based on shared concepts • Question Answering – try to find best answer based on user’s environment
  • 23.
    Data Mining Text • Can enable much faster data mining • Can bypass some setup and modeling Text Mining effort Other use Entity cases Extraction • Data Mining is “the automatic or semi-automatic analysis of large quantities of data to extract previously unknown interesting patterns” Wikipedia Data Structured Data Mining • Examples of data mining: • Association analysis - e.g., which 2 or 3 Something materials when used together are correlated Interesting? with a high degree of infection • Cluster analysis – e.g., patient micro- segmentation • Anomaly / Outlier Detection –e.g., network breaches
  • 24.
    Transaction Processing • Some Big Data platforms can be used for some types of transaction processing • Where performance is more important than consistency e.g., a Facebook user updating his/her status • More on this later…
  • 25.
    Poll • What type of Big Data use case would be most beneficial for your client? • Complex Event Processing (using massive/numerous streams of real-time data) • Unstructured Data Analysis • Predictive Data Mining • Transaction Processing (where performance more important than consistency) 25
  • 26.
    Big Data Architectureand Key Technologies
  • 27.
  • 28.
    Hadoop • Used for batch processing – inserts/appends only – no updates • Single master – works across many nodes, but only a single data center • Key components • HDFS – Hadoop Distributed File System • MapReduce – Distributes data in key value pairs across nodes, parallel processing, summarize results • Hbase – database built on top of Hadoop (with interactive capabilities) • Hive – SQL like query tool (converts to MapReduce) • Pig – Higher level execution language (vs. having to use Java, Python) – converts to MapRduce 28
  • 29.
    Cassandra • Used for real-time processing / transaction processing • Multiple masters – works across many nodes and many data centers • Key components • CFS – Cassandra File Systems • CQL – Cassandra Query Language (SQL like) • Tunable consistency for writes or reads. E.g., option to ensure a write succeeds to each replica in all data centers before returning control to program …. or can be much less restrictive 29
  • 30.
    In memory processing • To support real-time operations, an IMDB (In-Memory Database) may be used • Solo – or in conjunction with a disk based DBMS • I/O most expensive part of computing – using in memory database /cache reduces bottlenecks • Can be distributed (e.g., memcache, Terracotta, Kx) • Relational or non-relational • E.g., for a DW, current values might reside in an IMDB, historical data on disk 30
  • 31.
    MPP RDBMS • Have been in around for 15+ years • Used for large scale Data Warehousing • Ideal where lots of joins are needed on massive amount of data • Many NoSQL DB’s rely on 100% denormalization. Many do not support join operations (e.g., wide column stores) or updates 31
  • 32.
    Semantic Web • Semantic Web – web of data, not documents • Machine learning (inferencing) can be enabled via Semantic Web technologies. May use a graph database/triplestore (e.g., Neo4j, Allegrograph, Meronymy) • Bridge the semantic divide (varying vocabularies) with ontologies – helps address the “Variety” aspect of Big Data • Encapsulate data values, metadata, joins, logic, business rules, ontologies, access methods in the data via common logical model (e.g., RDF triples) – very powerful for automation, federated queries 32
  • 33.
    Semantic Web Find JaneDoe’s relatives (with machine inferencing) System X System Y System Z a:JoeDoe :isInLaw :hasBrother :hasBrother :marriedTo x:DebDoe y:JohnDoe z:JaneDoe :hasBrother :isInLaw Original data Inferred data 33
  • 34.
    No One SizeFits All  Many types of solutions will require multiple data paradigms  E.g. Facebook uses MySQL (relational), Hadoop, Cassandra, Hive, etc., for the different types of processing required  Be sure to have a solid use case before deciding to use Big Data / NoSQL technology  Provide solid business and technical justification
  • 35.
    What type ofdata store to use??
  • 36.
    Big Data impacton Application Development and Data Management
  • 37.
    ACID / CAP/ BASE  If your transaction processing application must be ACID compliant, you must use an RDBMS (or ODBMS)  ACID – Atomic, Consistent, Isolated, Durable Atomic – All tasks in a transaction succeed – or none do Consistent – Adheres to db rules, no partially completed transactions Isolated – Transactions can’t see data from other uncommitted transactions Durable – Committed transaction persists even if system fails  Not all transactions require ACID – eventual consistency may be adequate Vs..
  • 38.
    ACID / CAP/ BASE  Brewer’s CAP theorum for distributed database  Consistency, Availability, Partition Tolerance - Pick 2!  For Big Data, BASE is alternative for ACID Basically Available – data will be available for requests, might not be consistent Soft state – due to eventual consistency, the system might be continually changing Eventually consistent – the system will eventually be consistent when input stops • Example: HBase every transaction will execute, but only the most recent for a key will persist (LILO – last in, last out) – no locking
  • 39.
    Data Management  Securitynot as mature with NoSQL – might use OS level encryption (e.g.,, IBM Guardium Encryption Expert, Gazzanga) - encyrpt/decrypt at IO level  Data Governance needs to oversee Big Data – new knowledge uncovered can lead to risks - privacy, intellectual property, regulatory compliance, etc. • Physical Data Modeling less important – due to “schema-less” nature of NoSQL • Conceptual Modeling still important for understanding business objects and relationships • Semantic modeling – inform ontologies which enable inferencing • Logical Data Modeling still useful for reasoning and communicating about how data will be organized • Due to schema-less nature of NoSQL – metadata management will be more important! • E.g., wide-column store with billions of records and millions of variable columns – useless unless you have the metadata to understand the data
  • 40.
    Getting started • Data Scientist is a key role in Big Data – requires statistics, data modeling, and programming skills. Not many around and expect to pay $$$’s. • Big Data technologies represent a significant paradigm shift. Be sure to allow budget for training, sandbox environment, etc. • Start small with Big Data . Start with a single use case – allocate significant amount of time for learning curve, and environment setup, testing, tuning, management. • Working with open source software can present challenges. Investigate purchase of value added software for simplification. Tools such as IBM Big Insights, EMC Greenplum UAP (Unified Analytics Platform) adds analytical, administration, workflow, security, and other functionality. 40
  • 41.
  • 42.
    Summary  Big Datapresents significant opportunities  Big Data is distinguished by volume, velocity, and variety  Big Data is not just Hadoop / MapReduce and not just NoSQL  Key enabler for Big Data is Massively Parallel Processing (MPP)  Using commodity hardware and open source software are options to drive down cost of Big Data  Big Data and NoSQL technologies require a learning curve, and will continue to mature
  • 43.
    Resources  Perficient Healthcare:http://healthcare.perficient.com  Perficient Healthcare IT blog: http://blogs.perficient.com/healthcare/  Perficient Healthcare Twitter: @Perficient_HC  Apache – download and learn more about Hadoop, Cassandra, etc.  http://hadoop.apache.org/  http://cassandra.apache.org/  Comprehensive list with description of NoSQL databases: http://nosql- database.org/links.html  Translational Medicine Ontology (TMO) - applying Semantic Web for personalized medicine: http://www.w3.org/wiki/HCLSIG/PharmaOntology
  • 44.
  • 45.
    About Perficient Perficient isa leading information technology consulting firm serving clients throughout North America. We help clients implement business-driven technology solutions that integrate business processes, improve worker productivity, increase customer loyalty and create a more agile enterprise to better respond to new business opportunities.
  • 46.
    PRFT Profile  Founded in 1997  Public, NASDAQ: PRFT  2011 Revenue of $260 million  20 major market locations throughout North America — Atlanta, Austin, Charlotte, Chicago, Cincinnati, Cleveland, Columbus, Dallas, Denver, Detroit, Fairfax, Houston, Indianapolis, Minneapolis, New Orleans, Philadelphia, San Francisco, San Jose, St. Louis and Toronto  1,800+ colleagues  Dedicated solution practices  600+ enterprise clients (2011) and 85% repeat business rate  Alliance partnerships with major technology vendors  Multiple vendor/industry technology and growth awards
  • 47.
    Our Solutions Expertise& Services Business-Driven Solutions Perficient Services • Enterprise Portals  End-to-End Solution Delivery • SOA and Business Process  IT Strategic Consulting Management  IT Architecture Planning • Business Intelligence  Business Process & Workflow • User-Centered Custom Applications Consulting • CRM Solutions  Usability and UI Consulting • Enterprise Performance Management  Custom Application Development • Customer Self-Service  Offshore Development • eCommerce & Product Information  Package Selection, Implementation Management and Integration • Enterprise Content Management  Architecture & Application Migrations • Industry-Specific Solutions  Education • Mobile Technology • Security Assessments Perficient brings deep solutions expertise and offers a complete set of flexible services to help clients implement business-driven IT solutions 47

Editor's Notes

  • #29 Avro – data serialization (keeps schema (JSON) with data)Kafka – real time streaming, coordination via Zookeeper. Hcatalog – metadata for all the data stored in Hadoop. Read data from Pig or Hive or HbaseOozie – scheduling system (Azkhaban – not Apache – more graphical scheduler)Flume – Log aggregation – ship to HadoopWhirr – Hadoop on Cloud – Whirr helps to automateSqoop – transfers data from RDBMS to HadoopMRUnit – unit testingMahout – Machine learning on HadoopBigTop – integrate Hadoop based software so it all works togetherCrunch – Library on top of Java Giraph – large scale distributed graph
  • #34 In this case the properties would have to be associated with rules to describe entailments (i.e., the inferences that can be drawn). These could be encoded using SWRL (Semantic web Rule Language), which also uses RDF.