1
DevSecOps: Essential Tooling
to Enable Continuous Security
Richard Mills
DevOps Solution Architect, Coveros Inc.
rich.mills@coveros.com
@armillz
2
Who is this guy?
●Me: Mad-Software-Developer turned Mad-Software-Engineer
turned DevOps-Solution-Architect. Pragmatist. Particular focus on
tools and automation. CI, CD, DevOps … what’s next?
○ PS: Thanks for inventing the term “DevOps” to describe what I
like to do.
○ DevSecOps, on the other hand...
○ Definitely a DevSecQaEntFinBizOps specialist
●Pays my bills: Coveros helps organizations accelerate the delivery
of secure, reliable software using agile methods.
○ Agile transformations, development, and testing
○ Dev(Sec)Ops implementations
○ Training courses in Agile, DevOps, Application Security
●Keeps me intrigued: SecureCI
○ Open-source DevOps product
○ Integrated CI/CD stack with security flavor
3
Why is he here?
● Impart some experiences (and wisdom?) on people struggling to integrate
application security assessment into their Agile development process.
○ Share some of my experiences (successes and failures)
○ How can DevSecOps enable continuous security in Agile
development
○ Visualize examples of CI/CD pipelines that include security
○ Identify essential categories of tools you need in your DevOps
pipeline
○ Anticipate challenges with integrating security tools
○ Recognize the importance of integrating security team members with
development teams
Give you a reference to walk away with
4
● DevOps/Agile processes push code continuously
● Disjoint/mysterious security teams cannot keep up
● No time for slow, manual, late-lifecycle security
● Want to be “secure enough” every day
● Need: confidence that software can defend itself
● Need: continuous security
Problem
5
● Shift security left
● Integrate into daily and sprint-ly cycles
● Touchpoints in CI/CD pipeline
● Security tools run continuously
○ Static code analysis
○ Dynamic security testing
○ Software composition analysis
○ Platform vulnerability scanning
● Break builds, reject changes
Solution: Dev (Sec) Ops
6
DevOps and Modern Pipeline
7
Pipeline defines delivery process
Continuous MonitoringContinuous Integration
BuildCode
Continuous Delivery
commit
DEV
TEST
PROD
Compile
Test
Package
Publish
Unit Test
Integration Test
Code coverage
Code Scanning
Static Analysis
Bugs
Vulnerabilities
Technical Debt
Package
Dependency
checking
Vulnerable
components
Deploy Test
Provision
Install
Configure
Vulnerability Scanning
Deployment verification
Smoke test
Manual Testing
- Exploratory
- UAT
Functional Testing
- Behavioral
- API
- Web UI
Non-Functional Testing
- Load/Performance
- Web Security
Platform
Vulnerabilities
Test Results Defects
Change Requests
Test Promote
Operate
Monitor
The software delivery process is automated through a CI/CD pipeline to deliver application
microservices into various test (and eventually production) environments
8
Poll
How many people are App Sec professionals?
How many people have a DevOps
CI/CD pipeline of some variety?
9
Levels of Platform Assurance
Application code must be assessed at multiple levels as it makes its way through the delivery process
●Application code scanned and tested for functionality, quality, and security
●Deployable docker container images scanned for security
●OpenShift cluster configured, hardened for security
●AWS instances and infrastructure configured, hardened for security to meet standards
●Entire stack monitored for behavior
AWS
Instance
OpenShift Kubernetes Cluster
Pod
AWS
Instance
AWS
Instance
Pod Pod Pod Pod
Micro-Service App Code
PodPod
10
Security Touch Points
11
Touchpoints in lifecycle
12
Pipeline quality and security gates
● From a pipeline and tooling standpoint, we want “Quality Gates”
NOTE: “Quality” means “Quality, Security, Maintainability, and every other -ility”
Goals of Quality Gates:
● Provide overall picture of health
● Stop bad code from making it through the pipeline
● Enforce standards
● Ensure code is production ready at all times
Ideally: automated with tools (always some manual)
13
Tools, tools, and more tools
https://xebialabs.com/periodic-table-of-devops-tools/
14
Essential Security Tooling Categories
● Static application scanning
○ analyze the source code, application structure, or platform as it is built to detect
defects or vulnerabilities
○ In security space: SAST, software composition analysis, vulnerability scanning
● Dynamic functional testing
○ variety of sub-categories of functional testing to verify that the software behaves
according to its functional requirements.
● Non-functional testing
○ verify software against sub-categories of cross-cutting, non-functional requirements
(security, performance, accessibility, …)
○ In security space: DAST
● Real time monitoring
○ once the software is operating, monitor its operation and look for issues. (not
necessarily a "quality gate" but it does ensure that software remains healthy)
○ In security space: may include IAST and RASP
15
Static Application Scanning
● Static application scanning - run before we launch/run software
○ Static code analysis - quality, maintainability, security (frequently referred to as Static
Application Security Testing, SAST).
○ Software Composition Analysis - performs 3rd party dependency checks
○ Platform vulnerability scanning - scan OS, middleware, configuration for known
weaknesses
○ Docker container scanning - scan container images as they are built to detect
whether vulnerable container layers are being used or misconfigured
● Tools:
○ SonarQube, FindBugs, PMD, Fortify, Veracode, …
○ OWASP Dependency Check, RetireJS, …
○ Nessus, OpenVAS, …
○ Twistlock, Falco, Cilium, Aqua, ...
16
• Code scanning and quality
dashboards
• Includes quality, security, and
maintainability scans for many
languages
• Continuous view of static code
health, unit tests, coverage, …
• Inexpensive alternative to
commercial tools such as Fortify,
Veracode, etc.
SonarQube: good place to start...
17
● Ensure that you aren’t using someone else’s vulnerable code
● Software Composition Analysis against NVD with CVE
○ OWASP (Java), RetireJS (JavaScript), ...
○ Sonatype Nexus IQ Server, JFrog Xray, ...
Dependency Checks
18
• Examine container structure and
behavior before and during
execution
• Similar to vulnerability scanning
of hosts
• Two roles:
○ Scan newly build app
container images for
vulnerabilities
○ Monitor running containers
for compliance
• Others: Falco, Cilium, Aqua, ...
Container scanning: Twistlock
19
Dynamic Functional Testing
● Unit testing - verify that code functions properly in isolation during a build (pre-deployment)
● Health Tests - quick API health check endpoint pings to ensure services are running
● API testing - REST tests divided into smoke tests, functional tests, regression tests, etc.
● UI testing - Selenium/selenified tests for UI organized as smoke, functional, etc.
With Security: test your security functions (roles, auditing, encryption, …)
Tools:
● Junit, Jest, TestNG, ...
● Selenium, Selenified, jBehave, Cucumber, ...
● REST Assured, Postman, JMeter, Taurus, …
Point: these are good places to integrate dynamic security testing
20
Non-functional testing
● Dynamic Application Security Testing (DAST) - automated web scanning, penetration
testing, database testing,
● Performance testing - automated performance tests run manually with JMeter by QA Team
● 508 Accessibility testing - executed periodically to validate that the application is usable for
all people
● Other compliance testing…
Security Tools:
● ZAP, Burp, IBM App Scan, Metasploit,
Nmap, SQLmap, ...
21
Active Security ScanningPassive Security Monitoring
Security pipeline with ZAP
OWASP Zed Attack Proxy (ZAP) is an easy to use, open-source web scanning and penetration
tool
Two primary modes: Passive and Active
ZAP Attack
Driver
Web App
security
results
ZAP Security
Proxy
Web App
BrowserDriver
security
results
Web
Tests
22
Tying it together: pipeline flow
22
23
Real time monitoring
● Various aspects
○ Log aggregation and scanning - use processing rules to
detect anomalous behavior (information leakage, high error
rates, attack detection)
○ Real-time container and host monitoring - security
monitoring of running docker containers running in test
environments for behavior, configuration
○ Container and host scanning - scan hosts against
configuration benchmarks
○ Performance monitoring - monitor system resources,
response times, etc.
● Wraps into Security Information & Event Management (SIEM)
● Tools
○ Splunk, Kibana/Logstash (ELK), Tripwire, …
○ Nessus, OpenVAS, Twistlock, …
○ Prometheus, Graphana, Hawkular, New Relic, ...
24
Bonus Round: Integrating Teams
25
Integrate your development, security, quality, and ops teams to
streamline your delivery process and enable success
● Use team structures that encourage collaboration of security
engineers with developers
○ Need engineers who understand code, build, deployment,
testing, automation
○ Can’t succeed with only compliance box checkers (yes, you
need them too)
● Half the battle: getting teams to work together, not against each
other
○ Security consultants, not security police
○ Contributors, not naysayers
Build a culture of security. Expect every build to be secure.
Integrating Dev, Sec, QA, Ops
26
Horizontal Technical Guilds
●Group of specialized professionals working together to solve cross-team problems
●Guild members in-team are focused on team-specific problems
●Dedicated guild members support cross-team needs
●Guild establishes cross-team standards and shared success
●Important: share knowledge across team members
Cross-team function (vs. cross-functional team)
Challenge: You will never have enough
security engineers for every team
Horizontal Technical Teams
Agile Vertical Team
Developers
DevOps Engr
Scrum master
Biz Analysts
DevOps EngrDevOps Lead
Security EngrSecurity EngrSecurity Lead
Test EngrTest EngrTest Lead
27
Key Takeaways
● Develop a product with security built in
● Find tools that fit each major category
○ Static analysis
○ Software Composition Analysis
○ Vulnerability scanning (platform, containers)
○ Dynamic testing
○ Monitoring
● Start with simple (free!) tools until you understand their value
and cost
● Strive for continuous assessment
● Develop a culture of security
28
rich.mills@coveros.com
@armillz
https://www.coveros.com/services/devops/
Thank You!
Questions?

DevSecOps: essential tooling to enable continuous security 2019-09-16

  • 1.
    1 DevSecOps: Essential Tooling toEnable Continuous Security Richard Mills DevOps Solution Architect, Coveros Inc. rich.mills@coveros.com @armillz
  • 2.
    2 Who is thisguy? ●Me: Mad-Software-Developer turned Mad-Software-Engineer turned DevOps-Solution-Architect. Pragmatist. Particular focus on tools and automation. CI, CD, DevOps … what’s next? ○ PS: Thanks for inventing the term “DevOps” to describe what I like to do. ○ DevSecOps, on the other hand... ○ Definitely a DevSecQaEntFinBizOps specialist ●Pays my bills: Coveros helps organizations accelerate the delivery of secure, reliable software using agile methods. ○ Agile transformations, development, and testing ○ Dev(Sec)Ops implementations ○ Training courses in Agile, DevOps, Application Security ●Keeps me intrigued: SecureCI ○ Open-source DevOps product ○ Integrated CI/CD stack with security flavor
  • 3.
    3 Why is hehere? ● Impart some experiences (and wisdom?) on people struggling to integrate application security assessment into their Agile development process. ○ Share some of my experiences (successes and failures) ○ How can DevSecOps enable continuous security in Agile development ○ Visualize examples of CI/CD pipelines that include security ○ Identify essential categories of tools you need in your DevOps pipeline ○ Anticipate challenges with integrating security tools ○ Recognize the importance of integrating security team members with development teams Give you a reference to walk away with
  • 4.
    4 ● DevOps/Agile processespush code continuously ● Disjoint/mysterious security teams cannot keep up ● No time for slow, manual, late-lifecycle security ● Want to be “secure enough” every day ● Need: confidence that software can defend itself ● Need: continuous security Problem
  • 5.
    5 ● Shift securityleft ● Integrate into daily and sprint-ly cycles ● Touchpoints in CI/CD pipeline ● Security tools run continuously ○ Static code analysis ○ Dynamic security testing ○ Software composition analysis ○ Platform vulnerability scanning ● Break builds, reject changes Solution: Dev (Sec) Ops
  • 6.
  • 7.
    7 Pipeline defines deliveryprocess Continuous MonitoringContinuous Integration BuildCode Continuous Delivery commit DEV TEST PROD Compile Test Package Publish Unit Test Integration Test Code coverage Code Scanning Static Analysis Bugs Vulnerabilities Technical Debt Package Dependency checking Vulnerable components Deploy Test Provision Install Configure Vulnerability Scanning Deployment verification Smoke test Manual Testing - Exploratory - UAT Functional Testing - Behavioral - API - Web UI Non-Functional Testing - Load/Performance - Web Security Platform Vulnerabilities Test Results Defects Change Requests Test Promote Operate Monitor The software delivery process is automated through a CI/CD pipeline to deliver application microservices into various test (and eventually production) environments
  • 8.
    8 Poll How many peopleare App Sec professionals? How many people have a DevOps CI/CD pipeline of some variety?
  • 9.
    9 Levels of PlatformAssurance Application code must be assessed at multiple levels as it makes its way through the delivery process ●Application code scanned and tested for functionality, quality, and security ●Deployable docker container images scanned for security ●OpenShift cluster configured, hardened for security ●AWS instances and infrastructure configured, hardened for security to meet standards ●Entire stack monitored for behavior AWS Instance OpenShift Kubernetes Cluster Pod AWS Instance AWS Instance Pod Pod Pod Pod Micro-Service App Code PodPod
  • 10.
  • 11.
  • 12.
    12 Pipeline quality andsecurity gates ● From a pipeline and tooling standpoint, we want “Quality Gates” NOTE: “Quality” means “Quality, Security, Maintainability, and every other -ility” Goals of Quality Gates: ● Provide overall picture of health ● Stop bad code from making it through the pipeline ● Enforce standards ● Ensure code is production ready at all times Ideally: automated with tools (always some manual)
  • 13.
    13 Tools, tools, andmore tools https://xebialabs.com/periodic-table-of-devops-tools/
  • 14.
    14 Essential Security ToolingCategories ● Static application scanning ○ analyze the source code, application structure, or platform as it is built to detect defects or vulnerabilities ○ In security space: SAST, software composition analysis, vulnerability scanning ● Dynamic functional testing ○ variety of sub-categories of functional testing to verify that the software behaves according to its functional requirements. ● Non-functional testing ○ verify software against sub-categories of cross-cutting, non-functional requirements (security, performance, accessibility, …) ○ In security space: DAST ● Real time monitoring ○ once the software is operating, monitor its operation and look for issues. (not necessarily a "quality gate" but it does ensure that software remains healthy) ○ In security space: may include IAST and RASP
  • 15.
    15 Static Application Scanning ●Static application scanning - run before we launch/run software ○ Static code analysis - quality, maintainability, security (frequently referred to as Static Application Security Testing, SAST). ○ Software Composition Analysis - performs 3rd party dependency checks ○ Platform vulnerability scanning - scan OS, middleware, configuration for known weaknesses ○ Docker container scanning - scan container images as they are built to detect whether vulnerable container layers are being used or misconfigured ● Tools: ○ SonarQube, FindBugs, PMD, Fortify, Veracode, … ○ OWASP Dependency Check, RetireJS, … ○ Nessus, OpenVAS, … ○ Twistlock, Falco, Cilium, Aqua, ...
  • 16.
    16 • Code scanningand quality dashboards • Includes quality, security, and maintainability scans for many languages • Continuous view of static code health, unit tests, coverage, … • Inexpensive alternative to commercial tools such as Fortify, Veracode, etc. SonarQube: good place to start...
  • 17.
    17 ● Ensure thatyou aren’t using someone else’s vulnerable code ● Software Composition Analysis against NVD with CVE ○ OWASP (Java), RetireJS (JavaScript), ... ○ Sonatype Nexus IQ Server, JFrog Xray, ... Dependency Checks
  • 18.
    18 • Examine containerstructure and behavior before and during execution • Similar to vulnerability scanning of hosts • Two roles: ○ Scan newly build app container images for vulnerabilities ○ Monitor running containers for compliance • Others: Falco, Cilium, Aqua, ... Container scanning: Twistlock
  • 19.
    19 Dynamic Functional Testing ●Unit testing - verify that code functions properly in isolation during a build (pre-deployment) ● Health Tests - quick API health check endpoint pings to ensure services are running ● API testing - REST tests divided into smoke tests, functional tests, regression tests, etc. ● UI testing - Selenium/selenified tests for UI organized as smoke, functional, etc. With Security: test your security functions (roles, auditing, encryption, …) Tools: ● Junit, Jest, TestNG, ... ● Selenium, Selenified, jBehave, Cucumber, ... ● REST Assured, Postman, JMeter, Taurus, … Point: these are good places to integrate dynamic security testing
  • 20.
    20 Non-functional testing ● DynamicApplication Security Testing (DAST) - automated web scanning, penetration testing, database testing, ● Performance testing - automated performance tests run manually with JMeter by QA Team ● 508 Accessibility testing - executed periodically to validate that the application is usable for all people ● Other compliance testing… Security Tools: ● ZAP, Burp, IBM App Scan, Metasploit, Nmap, SQLmap, ...
  • 21.
    21 Active Security ScanningPassiveSecurity Monitoring Security pipeline with ZAP OWASP Zed Attack Proxy (ZAP) is an easy to use, open-source web scanning and penetration tool Two primary modes: Passive and Active ZAP Attack Driver Web App security results ZAP Security Proxy Web App BrowserDriver security results Web Tests
  • 22.
    22 Tying it together:pipeline flow 22
  • 23.
    23 Real time monitoring ●Various aspects ○ Log aggregation and scanning - use processing rules to detect anomalous behavior (information leakage, high error rates, attack detection) ○ Real-time container and host monitoring - security monitoring of running docker containers running in test environments for behavior, configuration ○ Container and host scanning - scan hosts against configuration benchmarks ○ Performance monitoring - monitor system resources, response times, etc. ● Wraps into Security Information & Event Management (SIEM) ● Tools ○ Splunk, Kibana/Logstash (ELK), Tripwire, … ○ Nessus, OpenVAS, Twistlock, … ○ Prometheus, Graphana, Hawkular, New Relic, ...
  • 24.
  • 25.
    25 Integrate your development,security, quality, and ops teams to streamline your delivery process and enable success ● Use team structures that encourage collaboration of security engineers with developers ○ Need engineers who understand code, build, deployment, testing, automation ○ Can’t succeed with only compliance box checkers (yes, you need them too) ● Half the battle: getting teams to work together, not against each other ○ Security consultants, not security police ○ Contributors, not naysayers Build a culture of security. Expect every build to be secure. Integrating Dev, Sec, QA, Ops
  • 26.
    26 Horizontal Technical Guilds ●Groupof specialized professionals working together to solve cross-team problems ●Guild members in-team are focused on team-specific problems ●Dedicated guild members support cross-team needs ●Guild establishes cross-team standards and shared success ●Important: share knowledge across team members Cross-team function (vs. cross-functional team) Challenge: You will never have enough security engineers for every team Horizontal Technical Teams Agile Vertical Team Developers DevOps Engr Scrum master Biz Analysts DevOps EngrDevOps Lead Security EngrSecurity EngrSecurity Lead Test EngrTest EngrTest Lead
  • 27.
    27 Key Takeaways ● Developa product with security built in ● Find tools that fit each major category ○ Static analysis ○ Software Composition Analysis ○ Vulnerability scanning (platform, containers) ○ Dynamic testing ○ Monitoring ● Start with simple (free!) tools until you understand their value and cost ● Strive for continuous assessment ● Develop a culture of security
  • 28.