Building world-class security response
and secure development processes
David Jorm, Senior Manager of Product Security, IIX
#ODSummit
Outline
• Introduction
• SDN attack surface
• Recent OpenDaylight vulnerabilities
• Defensive technologies
• Security response best practices
• Secure engineering best practices
• OpenDaylight security: current status
• OpenDaylight security: vision
#ODSummit
Introduction
• Software engineer for 15 years, climatology domain
• Last 5 years focusing on security, mainly Java
• Led Red Hat's Java middleware security team
• Currently manager of product security for IIX, and a founding member of the ODL security
response team
• Based in Brisbane, Australia (beautiful place, shame about the timezone)
#ODSummit
SDN Attack Surface
#ODSummit
SDN Attack Surface
• Traditional networks conflate the control and data planes on a physical device
• Software-defined networks factor the control plane out to a SDN controller
• The controller uses a protocol such as OpenFlow to control switches, which are now only
responsible for handling the data plane
• Security advantage: easy segregation of the control plane network from the production
data plane network
• Security disadvantage: the SDN controller's ability to control an entire network makes it a
very high value target
#ODSummit
SDN Attack Surface
#ODSummit
SDN Attack Surface
• SDN controllers are also exposed via the data plane
• When an OpenFlow switch encounters a packet that does not match any forwarding rules,
it passes this packet to the controller for advice
• As a result, it is possible for an attacker who is only able to send data through a switch to
exploit a vulnerability on the controller
• We will see a real-life example later in the presentation
#ODSummit
SDN Attack Surface
#ODSummit
Recent OpenDaylight Vulnerabilities
#ODSummit
Netconf XXE (CVE-2014-5035)
• Netconf (and restconf) API processes user-supplied XML
• By default, Java XML parsers do not disable external entity processing
• This led to a textbook XXE vulnerability
• Example of vulnerable code, with the patch applied:
controller/opendaylight/netconf/netconf-
util/src/main/java/org/opendaylight/controller/netconf/util/xml/XmlUtil.java
#ODSummit
Netconf XXE (CVE-2014-5035)
#ODSummit
Topology spoofing via host tracking (CVE-2015-1610)
• Most SDN controllers include host tracking, allowing hosts to migrate between different
physical locations in the network
• Host tracking is based on monitoring of Packet-In messages, and does not require any
validation, authentication, or authorization to identify the host
• An attacker can impersonate a host and make the SDN controller believe it has migrated to
a physical network location controlled by the attacker
• Data plane access is sufficient for exploitation, so long as the attacker knows the MAC
address of the target host
• Not patched in ODL l2switch
• Paper: http://www.internetsociety.org/sites/default/files/10_4_2.pdf
#ODSummit
DoS in ONOS packet deserializer (CVE-2015-1166)
• When an OpenFlow switch encounters a packet that does not match any forwarding rules,
it passes this packet to the controller for advice
• It was found that the packet deserializers in ONOS would throw exceptions when handling
malformed, truncated, or maliciously-crafted packets
• The exceptions were not caught and handled properly
• The top-level I/O thread exception handler would then disconnect the relevant switch
• Proves that attacks from the data plane are possible!
#ODSummit
Defensive Technologies
#ODSummit
Topoguard
• The same research team that reported the topology spoofing flaw developed topoguard
to mitigate it
• Doesn't add authn/authz, but instead verifies the conditions of host migrations
• A legitimate host migration would involve a Port Down signal before the host migration
finishes. The host would also be unreachable at its old physical network location after the
migration is complete.
• Currently tightly coupled to the Floodlight controller
#ODSummit
Security-mode ONOS
• A new feature in the ONOS Cardinal release
• Effectively a mandatory access control (MAC) implementation for ONOS applications
• Applications can be constrained by a policy dictating which actions they are permitted to
perform
• A vulnerability in an ONOS application could not be exploited to perform actions that are
not permitted by security-mode ONOS. This is similar to the protection SELinux provides
for applications running on Linux systems.
• Could this approach be a good model for OpenDaylight?
#ODSummit
Proposed Controller Shield Project
#ODSummit
Security Response Best Practices
#ODSummit
Open Source Security Response
• All information public
• Not just source code: bug trackers, mailing lists, etc.
• Security requires the opposite approach – information must be kept private until patches
are available
• How do you handle this in the context of an open source project?
• Good models: ASF, major OSS vendors like Red Hat and SuSE
#ODSummit
Open Source Security Response
• Dedicated mechanism for reporting security issues, separate to normal bugs
• Dedicated team with a documented process for responding to these reports
• Ability to quick build a patch asynchronous to normal release schedules
• Clear documentation of the issue in an advisory, including references to patch commits
(advantage of open source)
#ODSummit
Open Source Security Response
#ODSummit
Proprietary Security Response
#ODSummit
Secure Engineering Best Practices
#ODSummit
Open Source Secure Engineering
• No well established best practices
• Few good examples in the open source world. Proprietary software currently does a much
better job, for example Microsoft's SDLC.
• OpenStack is one good example
• Separate VMT and OSSG organizations
#ODSummit
Open Source Secure Engineering
#ODSummit
Open Source Secure Engineering
• Secure development guidelines (relies on developers to implement)
• Developer training (expensive and difficult to roll out in a virtual environment with many
contributors)
• Automated QE/CI jobs to catch known-vulnerable dependencies
• Automated QE/Ci jobs to catch security issues and enforce standards, e.g. via static
analysis
#ODSummit
OpenDaylight security: current status
#ODSummit
OpenDaylight Security Response
• Security reporting mechanism
• Dedicated team with a private mailing list and documented process for handling issues
• Security advisories page: https://wiki.opendaylight.org/view/Security_Advisories
• Advisories sent to mailing lists
#ODSummit
OpenDaylight Security Response
#ODSummit
OpenDaylight Security Response
• Scope currently limited to OpenDaylight code, not dependencies
• Handling dependencies would involve capturing a manifest, and tracking all relevant
upstreams
• Based on my experience, this would require one full time resource to be feasible
• Vulnerabilities in dependencies are sometimes handled when they are reported to the
security response team
#ODSummit
OpenDaylight Secure Engineering
• Great analysis performed in May 2014:
https://wiki.opendaylight.org/view/CrossProject:OpenDaylight_Security_Analysis
• Unfortunately quite dated now, and not much progress has been made implementing the
recommendations
• It's a lot of work to implement things, and who has time?
• Enter the OpenDaylight summer internship program!
#ODSummit
OpenDaylight Secure Engineering – Intern Project
#ODSummit
OpenDaylight Secure Engineering – Intern Project
Establish automated QE/CI jobs to catch security issues and regressions. This will involve
integrating the findsecbugs tool into Gerrit/Jenkins.
Establish automated QE/CI jobs to catch known-vulnerable dependencies. This will involve
integrating tools such as dependency-check and victims into Gerrit/Jenkins.
Document a threat model for OpenDaylight
Improve documentation to capture security best practices at installation and configuration
time
#ODSummit
✔
✔
✔
✔
✔
OpenDaylight security: vision
#ODSummit
OpenDaylight Security Vision - Reactive
High performing security response team
Equipped to handle vulnerabilities in dependencies
Able to co-ordinate disclosure and patches for issues across the community development
team and affected vendors of OpenDaylight distributions or products
Geographically distributed and able to quickly respond in all timezones
#ODSummit
✔
X
✔
✔
OpenDaylight Security Vision - Proactive
Documentation of best practices, threat model, etc.
Remove default credentials
Security hardening features applying a sandbox or MAC to the environment
Automated checks for known-vulnerable dependencies
Automated static analysis checks
Security training for developers: considering donating javapentesting.com course content
to the community
#ODSummit
X
✔
X
X
✔
✔
Questions?
#ODSummit

Building world-class security response and secure development processes

  • 2.
    Building world-class securityresponse and secure development processes David Jorm, Senior Manager of Product Security, IIX #ODSummit
  • 3.
    Outline • Introduction • SDNattack surface • Recent OpenDaylight vulnerabilities • Defensive technologies • Security response best practices • Secure engineering best practices • OpenDaylight security: current status • OpenDaylight security: vision #ODSummit
  • 4.
    Introduction • Software engineerfor 15 years, climatology domain • Last 5 years focusing on security, mainly Java • Led Red Hat's Java middleware security team • Currently manager of product security for IIX, and a founding member of the ODL security response team • Based in Brisbane, Australia (beautiful place, shame about the timezone) #ODSummit
  • 5.
  • 6.
    SDN Attack Surface •Traditional networks conflate the control and data planes on a physical device • Software-defined networks factor the control plane out to a SDN controller • The controller uses a protocol such as OpenFlow to control switches, which are now only responsible for handling the data plane • Security advantage: easy segregation of the control plane network from the production data plane network • Security disadvantage: the SDN controller's ability to control an entire network makes it a very high value target #ODSummit
  • 7.
  • 8.
    SDN Attack Surface •SDN controllers are also exposed via the data plane • When an OpenFlow switch encounters a packet that does not match any forwarding rules, it passes this packet to the controller for advice • As a result, it is possible for an attacker who is only able to send data through a switch to exploit a vulnerability on the controller • We will see a real-life example later in the presentation #ODSummit
  • 9.
  • 10.
  • 11.
    Netconf XXE (CVE-2014-5035) •Netconf (and restconf) API processes user-supplied XML • By default, Java XML parsers do not disable external entity processing • This led to a textbook XXE vulnerability • Example of vulnerable code, with the patch applied: controller/opendaylight/netconf/netconf- util/src/main/java/org/opendaylight/controller/netconf/util/xml/XmlUtil.java #ODSummit
  • 12.
  • 13.
    Topology spoofing viahost tracking (CVE-2015-1610) • Most SDN controllers include host tracking, allowing hosts to migrate between different physical locations in the network • Host tracking is based on monitoring of Packet-In messages, and does not require any validation, authentication, or authorization to identify the host • An attacker can impersonate a host and make the SDN controller believe it has migrated to a physical network location controlled by the attacker • Data plane access is sufficient for exploitation, so long as the attacker knows the MAC address of the target host • Not patched in ODL l2switch • Paper: http://www.internetsociety.org/sites/default/files/10_4_2.pdf #ODSummit
  • 14.
    DoS in ONOSpacket deserializer (CVE-2015-1166) • When an OpenFlow switch encounters a packet that does not match any forwarding rules, it passes this packet to the controller for advice • It was found that the packet deserializers in ONOS would throw exceptions when handling malformed, truncated, or maliciously-crafted packets • The exceptions were not caught and handled properly • The top-level I/O thread exception handler would then disconnect the relevant switch • Proves that attacks from the data plane are possible! #ODSummit
  • 15.
  • 16.
    Topoguard • The sameresearch team that reported the topology spoofing flaw developed topoguard to mitigate it • Doesn't add authn/authz, but instead verifies the conditions of host migrations • A legitimate host migration would involve a Port Down signal before the host migration finishes. The host would also be unreachable at its old physical network location after the migration is complete. • Currently tightly coupled to the Floodlight controller #ODSummit
  • 17.
    Security-mode ONOS • Anew feature in the ONOS Cardinal release • Effectively a mandatory access control (MAC) implementation for ONOS applications • Applications can be constrained by a policy dictating which actions they are permitted to perform • A vulnerability in an ONOS application could not be exploited to perform actions that are not permitted by security-mode ONOS. This is similar to the protection SELinux provides for applications running on Linux systems. • Could this approach be a good model for OpenDaylight? #ODSummit
  • 18.
    Proposed Controller ShieldProject #ODSummit
  • 19.
    Security Response BestPractices #ODSummit
  • 20.
    Open Source SecurityResponse • All information public • Not just source code: bug trackers, mailing lists, etc. • Security requires the opposite approach – information must be kept private until patches are available • How do you handle this in the context of an open source project? • Good models: ASF, major OSS vendors like Red Hat and SuSE #ODSummit
  • 21.
    Open Source SecurityResponse • Dedicated mechanism for reporting security issues, separate to normal bugs • Dedicated team with a documented process for responding to these reports • Ability to quick build a patch asynchronous to normal release schedules • Clear documentation of the issue in an advisory, including references to patch commits (advantage of open source) #ODSummit
  • 22.
    Open Source SecurityResponse #ODSummit
  • 23.
  • 24.
    Secure Engineering BestPractices #ODSummit
  • 25.
    Open Source SecureEngineering • No well established best practices • Few good examples in the open source world. Proprietary software currently does a much better job, for example Microsoft's SDLC. • OpenStack is one good example • Separate VMT and OSSG organizations #ODSummit
  • 26.
    Open Source SecureEngineering #ODSummit
  • 27.
    Open Source SecureEngineering • Secure development guidelines (relies on developers to implement) • Developer training (expensive and difficult to roll out in a virtual environment with many contributors) • Automated QE/CI jobs to catch known-vulnerable dependencies • Automated QE/Ci jobs to catch security issues and enforce standards, e.g. via static analysis #ODSummit
  • 28.
  • 29.
    OpenDaylight Security Response •Security reporting mechanism • Dedicated team with a private mailing list and documented process for handling issues • Security advisories page: https://wiki.opendaylight.org/view/Security_Advisories • Advisories sent to mailing lists #ODSummit
  • 30.
  • 31.
    OpenDaylight Security Response •Scope currently limited to OpenDaylight code, not dependencies • Handling dependencies would involve capturing a manifest, and tracking all relevant upstreams • Based on my experience, this would require one full time resource to be feasible • Vulnerabilities in dependencies are sometimes handled when they are reported to the security response team #ODSummit
  • 32.
    OpenDaylight Secure Engineering •Great analysis performed in May 2014: https://wiki.opendaylight.org/view/CrossProject:OpenDaylight_Security_Analysis • Unfortunately quite dated now, and not much progress has been made implementing the recommendations • It's a lot of work to implement things, and who has time? • Enter the OpenDaylight summer internship program! #ODSummit
  • 33.
    OpenDaylight Secure Engineering– Intern Project #ODSummit
  • 34.
    OpenDaylight Secure Engineering– Intern Project Establish automated QE/CI jobs to catch security issues and regressions. This will involve integrating the findsecbugs tool into Gerrit/Jenkins. Establish automated QE/CI jobs to catch known-vulnerable dependencies. This will involve integrating tools such as dependency-check and victims into Gerrit/Jenkins. Document a threat model for OpenDaylight Improve documentation to capture security best practices at installation and configuration time #ODSummit ✔ ✔ ✔ ✔ ✔
  • 35.
  • 36.
    OpenDaylight Security Vision- Reactive High performing security response team Equipped to handle vulnerabilities in dependencies Able to co-ordinate disclosure and patches for issues across the community development team and affected vendors of OpenDaylight distributions or products Geographically distributed and able to quickly respond in all timezones #ODSummit ✔ X ✔ ✔
  • 37.
    OpenDaylight Security Vision- Proactive Documentation of best practices, threat model, etc. Remove default credentials Security hardening features applying a sandbox or MAC to the environment Automated checks for known-vulnerable dependencies Automated static analysis checks Security training for developers: considering donating javapentesting.com course content to the community #ODSummit X ✔ X X ✔ ✔
  • 38.