Downloaded 16 times
Uploaded bySafe Software
Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB
The document discusses the use of FME Server as an Enterprise Service Bus (ESB) to facilitate data integration across various systems, addressing challenges such as data silos and complex interconnections. It highlights FME Server's capabilities in automating event-driven interactions, transforming data formats, and routing messages between systems, particularly in the context of 311 reporting and incident management. Key takeaways emphasize the importance of a well-designed ESB architecture for effective enterprise integration while allowing scalability and simplicity in managing system connections.
More Related Content
Similar to Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB
Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB
- 1. HOW TO USEFME SERVER AS AN Enterprise Service Bus
- 2. WHO WE ARE DeanHintz Senior Analyst, Strategic Experts Safe Software Neil Hellas Enterprise Solutions Architect Spatial DNA Tens of thousands of data pros use FME, the data integration platform with the best support for location data. Don Murray President, Safe Software
- 3. AGENDA ● FME ● Enterpriseintegration challenges & ESB ● FME Server as ESB ● Spatial DNA examples ● 311 ESB Demo ● Q&A
- 4. Technology is alwayschanging Innovative solutions are possible New systems are constantly being added System integration is hard CHALLENGE
- 5. Enables data translation Integratevirtually any system Powerful data transformation No coding required FME
- 6. Connect to VirtuallyAny Format or Service with FME
- 7. 400+ SUPPORTED SYSTEMSAND DATA TYPES Database servers: Oracle, SQL Server, PostGreSQL, MySQL,... Web systems: MongoDB, CouchDB, Dropbox, Box.com, Amazon S3, OData, Google Sheets, Socrata, Salesforce, Sharepoint, Azure, OGC ... Business Intelligence: Tableau, Excel, ElasticSearch, Qlik Extensibility: integration with traditional and emerging systems ● JDBC - often traditional ● HTTP - often cloud Tabular, XML, JSON, CSV, HTML ...
- 8. FME TERMINOLOGY Transformers: Usedto connect to systems, extract data, transform and remodel data and then load data. Over 400 available! Transformers use and produce Features. Features: Records or rows in spreadsheets and databases or objects in other systems. Features have Attributes. Attributes: Fields or columns attached to records, rows and objects.
- 9. Three Ways touse FME
- 10. FME Server Automate DeliverDownload * * Events / messages: Our main focus here
- 12. FME Server NotificationService ● ● ●
- 13. AUTOMATION, EVENTS ANDREAL-TIME Scheduling REST API Messaging: ○ JMS ○ SQS ○ WebSockets ○ WebHooks ○ Web Services
- 14.
- 15. Enterprise Integration Challenge ●Multiple, separate applications for different business systems: HR, CRM, BI, ERP, Accounting ● Data silos, duplicate records ● Integration is manual, ad-hoc, point to point ● The more systems that are added, the more difficult it is to manage
- 16. Enterprise Integration Challenge:311 Reporting Background: Citizens submit reports on the website about problems they notice in the community: missed garbage, potholes, etc. Those reports need to go to the service delivery team who manages related services. Challenge: The more systems that need to be connected to the 311 database, the more complex the interconnections between databases become. Specifics: Use topics within the notification service on FME Server as an ESB. All requests between client systems and the 311 database pass through the ESB topics on Server using a common messaging structure. Each system only needs to manage connections to this bus, not other systems. ESB D em o
- 17. Enterprise Integration Challenge 311DB Engineering DB Safety DB Initial architecture
- 18. 311 DB Engineering DB Safety DB Transport DB Enterprise IntegrationChallenge Architecture grows
- 19. 311 DB Engineering DB Safety DB Transport DB Contact DB EnterpriseIntegration Challenge And grows ...
- 20. 311 DB Engineering DB Safety DB Contact DB EnterpriseIntegration Challenge: ESB Approach More growth = easy 311ESB
- 21. 311 DB Engineering DB Safety DB Transport DB Contact DB EnterpriseIntegration Challenge: ESB Approach More growth = easy 311ESB
- 22. Capabilities of anESB An ESB can… has…
- 23. An ESB can…has… • Route data between systems • Testers, Filters, Notifiers Capabilities of an ESB
- 24. An ESB can…has… • Route data between systems • Transform formats and protocols • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller Capabilities of an ESB
- 25. An ESB can…has… • Route data between systems • Transform formats and protocols • Request/Response or Aynchronous • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter Capabilities of an ESB
- 26. An ESB can…has… • Route data between systems • Transform formats and protocols • Request/Response or Aynchronous • Publish/Subscribe • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter • Notifications Capabilities of an ESB
- 27. An ESB can…has… • Route data between systems • Transform formats and protocols • Request/Response or Aynchronous • Publish/Subscribe • Events and messages • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter • Notifications • Publisher/Subscriber Protocols Capabilities of an ESB
- 28. • Route databetween systems • Transform formats and protocols • Request/Response or Aynchronous • Publish/Subscribe • Events and messages • Extensible & pluggable • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter • Notifications • Publisher/Subscriber Protocols • Chaining topics An ESB can… has… Capabilities of an ESB
- 29. • Route databetween systems • Transform formats and protocols • Request/Response or Aynchronous • Publish/Subscribe • Events and messages • Extensible & pluggable • Publish Metadata • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter • Notifications • Publisher/Subscriber Protocols • Chaining topics • FME Server API An ESB can… has… Capabilities of an ESB
- 30. Spatial DNA ● ESBImplementation Examples
- 31.
- 32.
- 33. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Update DepartmentDB.fmw Engineering Update Safety Update Safety DB 311Case Update ProcessEngMsg.fmw ProcessSafetyMsg.fmw Update311DB Message Transmitter.fmw ProcessUpdate Message.fmw ESB 311 Topics
- 34. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311Case Update ProcessEngMsg.fmw ProcessSafetyMsg.fmw Update311DB Message Transmitter.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics
- 35. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em oESB311 Topics Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw ProcessSafetyMsg.fmw Update311DB Message Transmitter.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw
- 36. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB Engineering Update Safety Update Safety DB 311Case Update ProcessEngMsg.fmw Update311DB Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics ESB D em o
- 37. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311Case Update ProcessEngMsg.fmw Update311DB Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics
- 38. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311Case Update ProcessEngMsg.fmw Update311DB Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics
- 39. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311Case Update ProcessEngMsg.fmw Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics Update311DB
- 40. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311Case Update ProcessEngMsg.fmw Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics Update311DB
- 41. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Update DepartmentDB.fmw Engineering Update Safety Update Safety DB 311Case Update ProcessEngMsg.fmw ProcessSafetyMsg.fmw Update311DB Message Transmitter.fmw ProcessUpdate Message.fmw ESB 311 Topics
- 42. ESB 311 Demo:Loader, Update
- 43. ESB 311 Demo:Notifications
- 44. ESB 311 Demo:Database Summary Report
- 45. ESB Configuration onFME Server : Projects Project Content: ● Repository ● Workspaces ● Topics ● Subscriptions ● Schedules ● Shared resource ● DB connections ● Web connections ● Web socket
- 46. ESB Configuration onFME Server : Topics
- 47. ESB Configuration onFME Server : Schedules
- 48. ESB Configuration onFME Server : DB Connections
- 49.
- 50.
- 51. Update Message viaEngineering Topic on 311ESB {"Status":"Open","Category":"Traffic", "Description": "Flat tire", "Email":"dean@safe.com", "FirstName":"Dean", "ws_topic": "ESB_EngineeringUpdate", "json_featuretype":"ServiceMessage", "Time": "2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC", "Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}
- 52.
- 53.
- 54.
- 55. Update Message via311Update Topic {"Status":"Closed","Category":"Traffic", "Description":"flat tirenResolved by: Jim", "Email":"dean@safe.com","FirstName":"Dean", "ws_topic":"ESB_311DB_Update", "json_featuretype":"ServiceMessage", "Time":"2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC","Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}
- 56. FME Server asESB: Key Take-aways ● ESB - connection of many systems without increasing architecture complexity ● FME Server’s notification service provides ESB functions and components ● 311 ESB configuration tested using scheduled publication workspaces and subscription workspaces connected to a set of topics ● Initial ESB design needs to be carefully laid out so the events and message interactions are clearly mapped ● Define message structure and common data model: key elements, routing ● A good design is easily extendible: start simple, then expand
- 57. Enterprise Integration Workflows Safe Software Common Challenges ➔Field name mapping ➔ Field value mapping ➔ Lookup field mapping ➔ Validating values ➔ Table mapping ➔ Record type mapping
- 58.
- 59. Field Names andValues: AttributeManager Copy from one field to another. Provide a hard-coded default for a field. Concatenate multiple field values together into a single field. Copy part of a value from one field to another.
- 60. Value Mapping for LookupFields
- 61.
- 62. Summary ● Dilemma ofenterprise growth and complexity ● ESB approach / pattern helps manage growth ● FME Server’s notification service as an ESB ● Easy to add new systems and scale capacity without increasing complexity ● Notification service - a great way to integrate FME into your enterprise workflows whether or not you use ESB
- 63. Related Webinars ● Howto Use FME as an Enterprise Integration Platform (Aug 2017) https://www.safe.com/solutions/enterprise-data-integration ● Web Services & Web Data – Best Practices for Integration (March 2017) https://www.safe.com/webinars/web-services-fme-2017/ Upcoming: ● Deep Dive into FME Server 2018 (Feb 22, 2018) ● FME Server Automations (March 28, 2018)
- 64. Resources ● Neil’s FMEUC 2017 Talk: “Making FME Your ESB” ● FME Server Playground https://playground.fmeserver.com/ ● FME Server Notification Docs ● Tutorial: Notifications ● Tutorial: Automation Planning and Deployment ● Tutorial: Reading and Writing JSON ● FME Hub: Custom transformers, templates, coming: Server Projects ESB Demo project package will be sent to all attendees
- 65. Thank you Dean Hintz SeniorAnalyst, Strategic Experts Safe Software dean@safe.com Neil Hellas Enterprise Solutions Architect Spatial DNA neil.hellas@spatialdna.com Don Murray President, Safe Software
- 66.