Download to read offline
Uploaded byEnterprise Management Associates
Orchestrating Data Transfers in the Digital Era: Navigating Challenges and Solutions for Multi-environment Data Pipelines
The document discusses the challenges and solutions for managing multi-environment data pipelines, emphasizing the significance of file transfers in various enterprise functions. It highlights the increasing demand for secure and efficient data movement solutions, backed by insights from a sample of 264 respondents about their data transfer practices and software usage. Key findings include the necessity for improved data security, integration capabilities, and the growing adoption of iPaaS among organizations.
More Related Content
Similar to Orchestrating Data Transfers in the Digital Era: Navigating Challenges and Solutions for Multi-environment Data Pipelines
More from Enterprise Management Associates
Orchestrating Data Transfers in the Digital Era: Navigating Challenges and Solutions for Multi-environment Data Pipelines
- 1. @ema_research | @ema_research Orchestrating DataTransfers in the Digital Era: Navigating Challenges and Solutions for Multi-environment Data Pipelines Dan Twing President and COO Enterprise Management Associates (EMA) Sponsored by
- 2. @ema_research | @ema_research 2 Watchthe On-Demand Webinar • Orchestrating Data Transfers in the Digital Era On-Demand Webinar: https://info.enterprisemanagement.com/orchestrating-data- transfers-in-the-digital-era-webinar-ws • Check out upcoming webinars from EMA here: https://www.enterprisemanagement.com/freeResearch © 2024 Enterprise Management Associates, Inc.
- 3. @ema_research | @ema_research @ema_research | @ema_research WebinarSpeaker Dan joined EMA in 2005 and has over 25 years of experience in information systems, software development, and technology outsourcing. He focuses on all aspects of intelligent and automated management of IT. As President and Chief Operating Officer of EMA, Dan is responsible for developing and executing strategic market research, delivering value to IT organizations through consulting engagements, and directing product developments and marketing efforts. © 2024 Enterprise Management Associates, Inc. 3 Dan Twing President and COO EMA
- 4. @ema_research | @ema_research @ema_research | @ema_research RespondentDemographics © 2024 Enterprise Management Associates, Inc. 4 1.5% 1.9% 3.4% 4.9% 5.3% 5.7% 7.6% 8.7% 9.1% 11.4% 40.2% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% Infrastructure Engineer Data Architect Other Business Users IT Cloud Platforms/Cloud Architect Business Line Leader Business Executive IT Development IT Security IT Operations IT Service Management (ITSM) IT Executive Role in Organization Sample Size = 264 16% 30% 19% 18% 7% 10% 0% 5% 10% 15% 20% 25% 30% 35% Number of Employees NA 33% Europe 31% APAC 37% 1% 3% 5% 5% 7% 9% 9% 14% 45% 0% 10% 20% 30% 40% 50% User of dashboards Evaluate file transfer/data moving products Technical user/operator of data moving products Business IT analyst Design, build, or operate in-house-developed file… System designer Recommend and specify related products/services Financial decision-maker Technical decision-maker Role in selecting and implementing file transfer or data moving technologies/solutions
- 5. @ema_research | @ema_research @ema_research | @ema_research FileTransfer Needs Reach Across the Enterprise © 2024 Enterprise Management Associates, Inc. 5 File transfers and data movement is a key part of consistently delivering critical enterprise processes 80% IT Operations 76% Application Developers 67% Service 64% Sales 61% Supply Chain 59% Sales 58% Finance 56% Sourcing and Procurement 55% Asset Management 54% Country/Region specific Degree functional areas rely upon file transfers and data moving (High or Very High): 95% IT Operations 91% Application Developers 85% R&D/Engineering 82% Service 82% Finance 80% Sales 80% Supply Chain 80% Sourcing and Procurement 79% Asset Management 74% Country/Region specific Somewhat or Very proactive in sponsoring new file transfers or data moving initiatives: Sample Size = 264
- 6. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 6 Sample Size = 262 30% 37% 38% 39% 39% 42% 42% 46% 50% 0% 10% 20% 30% 40% 50% 60% API-centric architecture Increased emphasis on data quality Adoption of advanced analytics and AI Integrating/Supporting cloud services Increased focus on data governance Real-time data processing (streaming data) Increased data volumes Migration to the cloud Increased focus on data security How is digital transformation changing the requirements for moving data? Digital Transformation Impact on Data Movement
- 7. @ema_research | @ema_research Means ofData Movement Timing, Environments, and Methods
- 8. @ema_research | @ema_research @ema_research | @ema_research Timingof Data Movement © 2024 Enterprise Management Associates, Inc. 8 48% 48% 57% 72% 78% 0% 20% 40% 60% 80% 100% Ad hoc file sharing (on demand, spontaneous) Near-real time data integration Event-driven data movement Real-time data integration Scheduled data movement Data moving methods Sample Size = 264 16.0% 16.9% 18.7% 23.4% 25.0% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% Ad hoc file sharing Event-driven data movement Near-real time data integration Real-time data integration Scheduled data movement Distribution of data moved
- 9. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 9 36% 38% 45% 48% 58% 63% 73% 0% 20% 40% 60% 80% Third-party hosted Mainframe User devices Distributed on-premises Public cloud SaaS Private cloud Environments from which data moves Sample Size = 264 11.4% 12.5% 13.6% 14.5% 14.8% 16.1% 17.0% 0.0% 5.0% 10.0% 15.0% 20.0% User devices Third-party hosted SaaS Distributed on-premises Mainframe Public cloud Private cloud Distribution of data moved Environments Moving Data
- 10. @ema_research | @ema_research @ema_research | @ema_research SoftwareUsed to Move Data © 2024 Enterprise Management Associates, Inc. 10 33% 35% 37% 38% 42% 42% 43% 48% 48% 53% 58% 0% 10% 20% 30% 40% 50% 60% 70% Enterprise service bus (ESB) and integration platforms Middleware and message queue systems In-house-developed scripts or applications Managed file transfer (MFT) software Integration platform as a service (iPaaS) Workload automation (WLA)-native MFT Automated workflow and BPA tools API management platforms Enterprise resource planning (ERP) systems Secure file sharing and collaboration tools File transfer protocol (FTP) servers Types of data moving software in use Sample Size = 264 6.8% 8.1% 8.4% 8.6% 9.1% 9.3% 9.4% 9.4% 10.1% 10.1% 10.7% 0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% Middleware and message queue systems Enterprise service bus (ESB) and integration platforms Automated workflow and business process automation tools Enterprise resource planning (ERP) systems API management platforms In-house-developed scripts or applications File transfer protocol servers Secure file sharing and collaboration tools Workload automation (WLA) software (using native MFT features) Integration platform as a service (iPaaS) Managed file transfer (MFT) software Distribution of data moved using each method
- 11.
- 12. @ema_research | @ema_research @ema_research | @ema_research FocusingOnly On File-Based Data Movement Methods… © 2024 Enterprise Management Associates, Inc. 12 15.8% 15.9% 15.9% 17.1% 17.1% 18.1% 14.5% 15.0% 15.5% 16.0% 16.5% 17.0% 17.5% 18.0% 18.5% In-house-developed scripts or applications Secure file sharing and collaboration tools File transfer protocol servers Workload automation (WLA) software (using native MFT features) Integration platform as a service (iPaaS) Managed file transfer (MFT) software Distribution of Data Moved By File Sample Size = 264
- 13. @ema_research | @ema_research @ema_research | @ema_research Mostrespondents have 1001-5000 file transfers sent or received weekly. The weighted average is 14,308 file transfers weekly. © 2024 Enterprise Management Associates, Inc. 13 0% 3% 3% 11% 19% 28% 13% 9% 5% 3% 5% 0% 5% 10% 15% 20% 25% 30% Fewer than 10 10-50 51-100 101-500 501-1,000 1,001-5,000 5,001-10,000 10,001-20,000 20,001-50,000 50,001-100,000 More than 100,000 Weekly File Transfers Sample Size = 264
- 14. @ema_research | @ema_research @ema_research | @ema_research FileTransfers Increasing © 2024 Enterprise Management Associates, Inc. 14 1% 5% 19% 34% 22% 17% 3% 0% 5% 10% 15% 20% 25% 30% 35% 40% Increased more than 75% Increased between 50% and 75% Increased between 25% and 50% Increased between 10% and 25% Increased less than 10% Stayed the same Decreased less than 10% Decreased between 10% and 25% Decreased between 25% and 50% Decreased between 50% and 75% Decreased more than 75% Don’t know How has the number of file transfers your organization sends/receives changed in the past year? Sample Size = 264
- 15. @ema_research | @ema_research @ema_research | @ema_research 57%are sending files to other countries © 2024 Enterprise Management Associates, Inc. 15 No 42% Yes 57% Do you transfer files to other countries? No Yes Don’t know Sample Size = 264
- 16. @ema_research | @ema_research @ema_research | @ema_research Securityis the most important feature when moving files (59%), followed by Integration capabilities (49%) and High availability (47%) © 2024 Enterprise Management Associates, Inc. 16 23% 32% 33% 36% 39% 40% 40% 42% 42% 43% 47% 49% 59% 0% 10% 20% 30% 40% 50% 60% 70% Vendor support and training options Detailed reporting and monitoring tools Customization and extensibility of the solution User-friendly interface for ease of use Compliance and auditing capabilities (e.g., HIPAA, GDPR compliance) Automation of file transfer processes and workflows Cost-effectiveness and pricing flexibility Support for multiple file types and sizes Advanced encryption and data protection measures Scalability to handle growth in data volume and connections High availability and disaster recovery features Integration capabilities with existing systems (e.g., ERP, CRM) Security protocols (e.g., SSL/TLS, SFTP, FTPS) Which of the following features do you consider most important when selecting a file transfer solution for your organization? Sample Size = 264
- 17. @ema_research | @ema_research @ema_research | @ema_research WLAPlays A Significant Role In File Transfers © 2024 Enterprise Management Associates, Inc. 17 20% 20% 22% 22% 24% 0% 5% 10% 15% 20% 25% 30% Not used in file transfer activities WLA orchestrates FTP server software to transfer files WLA orchestrates MFT software to transfer files WLA orchestrates other data pipeline activities Use native WLA MFT features to transfer files Please indicate how WLA software is used in exchanging data or transferring files. Sample Size = 111
- 18.
- 19. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 19 Inbound data transfers encrypted Encrypted Unencrypted 47% Outbound data transfers encrypted Encrypted Unencrypted 49% Data at rest encrypted Encrypted Unencrypted 41% Sample Size = 264 Less Than Half Of File Transfers Are Encrypted
- 20. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 20 Sample Size = 264 25% 25% 30% 31% 34% 35% 38% 50% 50% 61% 0% 10% 20% 30% 40% 50% 60% 70% Hashing algorithms for data integrity verification Data tokenization techniques IPsec (Internet Protocol Security) Encrypted tunnels (e.g., SSH tunnels) AES (Advanced Encryption Standard) encryption Public Key Infrastructure (PKI) for digital signatures and encryption Transport Layer Security (TLS) protocols Encrypted VPN Secure Sockets Layer (SSL) protocols Secure FTP or SFTP Please select the encryption methods used during data movement between systems, networks, or cloud environments. Encryption Methods For Data Movement
- 21. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 21 24% 26% 33% 35% 39% 42% 48% 58% 0% 10% 20% 30% 40% 50% 60% 70% PGP (Pretty Good Privacy) encryption Removable media encryption Virtual machine encryption Full disk encryption (FDE) File-level encryption AES (Advanced Encryption Standard) encryption Database encryption Cloud storage encryption Please select the encryption methods used to encrypt data at rest. Sample Size = 256 Encryption Methods For Data At Rest
- 22.
- 23. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 23 Sample Size = 154 1% 1% 7% 45% 45% 1% 1% 5% 32% 61% 3% 6% 41% 50% 4% 41% 55% 0% 10% 20% 30% 40% 50% 60% 70% Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied How satisfied is your organization with the performance and reliability of your current file transfer solutions? Integration platform as a service (iPaaS) Workload automation software with native file transfer Managed file transfer software FTP servers Satisfied or Very Satisfied 96% 93% 91% 90% Satisfaction Of File Transfer Solutions
- 24. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 24 Sample Size = 154 23% 25% 27% 27% 37% 0% 10% 20% 30% 40% File size and formats Costs and licensing Integration and compatibility Monitoring and visibility Security concerns (encryption, key management) Top 3 challenges with FTP servers. 25% 26% 26% 32% 33% 0% 5% 10% 15% 20% 25% 30% 35% 40% File size and formats Integration and compatibility Complex workflows and processes Costs and licensing Security concerns (encryption, key management) Top 3 challenges with managed file transfer (MFT) software. 25% 26% 27% 28% 28% 0% 5% 10% 15% 20% 25% 30% 35% 40% Mobile and remote access Integration and compatibility Security concerns (encryption, key management) Regulatory compliance Scalability Top 3 challenges with native file transfer features of WLA software. 21% 23% 23% 28% 30% 0% 5% 10% 15% 20% 25% 30% 35% 40% User adoption and training Regulatory compliance Costs and licensing Security concerns (encryption, key management) Integration and compatibility Top 3 challenges with iPaaS software. Top Challenges For File Transfer Solutions
- 25. @ema_research | @ema_research @ema_research | @ema_research iPaaSis quite popular © 2024 Enterprise Management Associates, Inc. 25 9% 13% 17% 17% 45% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Unsure Yes, add iPaaS Yes, explore alternative solutions No, maintain current usage Yes, expand iPaaS usage Plans to add or expand iPaaS usage or explore alternative integration solutions? Sample Size = 264 45% of those using iPaaS are planning expansion of use, while 13% will add iPaaS and 17% plan to explore alternative solutions.
- 26. @ema_research | @ema_research @ema_research | @ema_research ChangingFile Transfer Solutions © 2024 Enterprise Management Associates, Inc. 26 14% 17% 13% 18% 33% 5% 0% 5% 10% 15% 20% 25% 30% 35% Yes, we are actively looking Yes, in the next 12 months Yes, in 12 to 24 months We are considering a change No, we are not considering a change Don’t know Is your organization considering switching to a new provider for file transfer solutions? Sample Size = 264 With data growth, operational challenges, and digital transformation it should be no surprise that 62% of respondents are considering switching to new file transfer software.
- 27. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 27 6.8% 9.3% 8.1% 9.1% 8.6% 10.1% 10.7% 8.4% 9.4% 10.1% 9.4% 6.1% 6.8% 7.4% 8.0% 8.5% 8.7% 9.5% 9.6% 10.9% 11.0% 13.4% 0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 14.0% 16.0% Middleware and message queue systems In-house-developed scripts or applications Enterprise service bus (ESB) and integration platforms API management platforms Enterprise resource planning (ERP) systems Workload automation (WLA) software (using native MFT features) Managed file transfer (MFT) software Automated workflow and business process automation tools Secure file sharing and collaboration tools Integration platform as a service (iPaaS) File transfer protocol servers Distribution of Data Moved in a Typical Month: 2024 vs, 2027 Expectations 2024 2027 Sample Size = 264 Forecasted Changes In Data Moving Methods 2024 vs 2027
- 28.
- 29. @ema_research | @ema_research @ema_research | @ema_research DataOpshas Grown in Popularity and Adoption Since the Term Was Coined in 2014 © 2024 Enterprise Management Associates, Inc. 29 33% 46% 20% 1% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Fully implemented and integrated Partially implemented, with ongoing integration efforts Initial stages of implementation, with plans for further integration Not implemented, but under consideration for future adoption How would you rate the level of implementation and integration of DataOps practices within your organization's data management processes? Sample Size = 264 Just 33% of respondents have DataOps fully implemented, with 46% partially implemented.
- 30. @ema_research | @ema_research @ema_research | @ema_research DataOpsIncludes Data Pipelines © 2024 Enterprise Management Associates, Inc. 30 1% 7% 43% 49% 0% 10% 20% 30% 40% 50% 60% Never heard the term "data pipeline" used in our operations Not extensively utilized or explored Used in specific departments or projects Widely adopted across various departments and use cases How extensively are data pipelines utilized within your organization for data integration and processing? Sample Size = 264 49% of respondents have widely adopted Data Pipelines. Another 43% use data pipelines for specific projects or departments.
- 31. @ema_research | @ema_research @ema_research | @ema_research ChallengesOrchestrating Data Pipelines © 2024 Enterprise Management Associates, Inc. 31 Sample Size = 235 29% 32% 34% 35% 49% 0% 10% 20% 30% 40% 50% 60% High operational costs and resource constraints Compatibility issues with legacy technologies Integration complexity with existing systems Scalability and performance issues Security and compliance concerns What are the primary challenges your organization faces in orchestrating data pipelines? Security and compliance (49%) is the biggest challenge with data pipelines.
- 32. @ema_research | @ema_research @ema_research | @ema_research Automationimproves data pipelines. © 2024 Enterprise Management Associates, Inc. 32 36% 44% 47% 49% 53% 61% 0% 10% 20% 30% 40% 50% 60% 70% Enhanced fault tolerance with built-in fault detection and recovery mechanisms Simplified workflow management, scheduling, and dependency handling Provided visibility and monitoring capabilities across the data pipeline Enhanced scalability and performance Improved reliability and reduced errors Increased efficiency and productivity How has the implementation of automation and orchestration improved your organization's data pipeline processes? Sample Size = 217
- 33. @ema_research | @ema_research @ema_research | @ema_research ©2024 Enterprise Management Associates, Inc. 33 6% 51% 41% 1% 0% 10% 20% 30% 40% 50% 60% Manual processes dominate, requiring more automation Partially automated with room for improvement Fully automated with robust orchestration capabilities I don't know To what extent are data pipelines automated and orchestrated within your organization? Sample Size = 235 Automation Is Not Fully Mature In Data Pipelines Just 41% have fully automated data pipelines, while 51% have partially automated data pipelines.
- 34. @ema_research | @ema_research @ema_research | @ema_research TheData in Motion Report is now available! Learn more and download: https://bit.ly/data-in-motion-report © 2024 Enterprise Management Associates, Inc. 34