DanielStenberg-June30th
2020
for security!
Testing
Daniel Stenberg
@bagderhttps://daniel.haxx.se
Daniel Stenberg
@bagder
… ten billion installations
across the world
curl runs in...
@bagder@bagder
…… devices all of you heredevices all of you here
today own and use!today own and use!
curl runs in
@bagder@bagder
Writing safe codeWriting safe code
@bagder@bagder
Safe code is not a coincidence
Doesn’t happen randomly, we need to work on it!
We care about it!
It is about time we all address this topic!
Everything I’ll tell you today, you already know!
@bagder@bagder
Writing
Help users use the tool and APIs safelyHelp users use the tool and APIs safely
Defaults!Defaults!
Document. Everything, detailed and a lot.Document. Everything, detailed and a lot.
Get rid of warningsGet rid of warnings
UseUse commentscomments
assert()assert()
Releasing the code asReleasing the code as open sourceopen source
@bagder@bagder
Review
All codeAll code shallshall get reviewedget reviewed
CodeCode shallshall be easy to readbe easy to read and understandand understand
Use theUse the same code stylesame code style everywhereeverywhere
Commit message template andCommit message template and qualityquality
Make sure style and templates are followed!Make sure style and templates are followed!
@bagder@bagder
Test!
Unit tests
System tests
Integration tests
Test the documentation
Manual tests
Fixed a bug? Add a test. Or two.
@bagder@bagder
Torment
Valgrind, address/UB/integer sanitizersValgrind, address/UB/integer sanitizers
Static code analysersStatic code analysers
FuzzersFuzzers
@bagder@bagder
All that, all the time
For every commit
For every PR
All. The. Time
@bagder@bagder
curl CI
80+ builds and test “rounds” per commit
Tests code style, indenting etc
Thousands of tests per build
Builds and tests on tens of platforms
20-25 hours of CI per commit
@bagder@bagder
@bagder@bagder
@bagder@bagder
The curl project’s CI sponsors
Parts of the CI system run of free tiers
Teamviewer sponsors Appveyor powers
Travis gives us more power
Azure Pipelines gives us more power
@bagder@bagder
@bagder@bagder
@bagder@bagder
Writing test cases should be easy
A curl test case is a single file in a human readable well-documented
format
• conditions and features needed to run
• what command (line) to run
• what the test wants returned from a server
• how the protocol exchange should look like
• stdout and stderr contents
• expected error code
• … and more
@bagder@bagder
Custom test servers
We avoid using “real” servers for testing
Test servers are as dumb as possible
Controlled from the test case what it should send and expect
Allows “crazy” behaviors and send/receive “anything”
Makes the test servers smaller and simpler
Helps with test suite portability
TLS is done by stunnel-fronting
Servers run on random port numbers
@bagder@bagder
Tools of the trade
Valgrind
Clang address,
undefined, signed-
integer-overflow
sanitizers
Clang tidy
“torture tests”
Scan-build
Lgtm
Codacy
CodeQL
Monocle AI
Deepcode AI
Coverity
Travis CI
Appveyor
Cirrus CI
Github Actions
Azure Pipelines
Buildbots
OSS-Fuzz
CI-fuzz
@bagder@bagder
Torture tests – error injection
Build with a “debug” option
Add wrapper functions for
common “fallible” functions (that
allocate and free resources)
Each wrapper function logs what it
does and can optionally return
error.
The complete individual test case
is first run once
count how many fallible functions
rerun the test case that number of
times and for each iteration make
next fallible function return error
Verify nothing crashed and no
memory leaked
Repeat for all tests
@bagder@bagder
Source code policy
Fix all warnings (eye roll)Fix all warnings (eye roll)
No defects leftNo defects left
Use the strictest and most picky optionsUse the strictest and most picky options
As many tests as possibleAs many tests as possible
Fix security issues as soon as possibleqFix security issues as soon as possibleq
@bagder@bagder
A million build combos, 70+ OSes and 20 CPUs
Testing all combinations is simply not possible
Test the common setups
Test on as many platforms as possible
Test on several different CPU architectures
“white spots” in test coverage handled by review
Users keep finding untested areas and build combinations
@bagder@bagder
The security of curl is the sum of all
components combined.
- Lean on the right set!
@bagder@bagder
32 third party dependencies
I/O layer
libcurl
URL parser libidn2winidn
HTTPHTTPS
OpenSSL
Mesalink
gskit
mbedTLS
wolfSSL
Schannel
SecureTransport
GnuTLS
NSS
boringssl
libressl
AmiSSL
SFTP SCP LDAP
WinLDAP
OpenLDAP
RTMP
librtmp
Name resolver c-ares
compression
libz brotli
cookies
libpsl
IMAP SMTP POP3
HTTP/2
nghttp2
authentication
winsspi Heimdal MIT kerberos
HTTP/3
quiche
ngtcp2
HTTP/1
SSH
wolfSSH
libssh2
libssh
@bagder@bagder
BearSSL
nghttp3
Fuzz Testing in wolfSSL
● Best Tested TLS due to our extensive internal
testing process
● Fuzz testing in our internal testing process is
one of the reasons why
● Fuzz testing bombards the program with
invalid, unexpected, and random data that
then allows for observing if there is
potential memory leaks or logic errors  
● Allows us to catch bugs that could
turn into potential vulnerabilities
● We only use highly respected external
testers:
● Guido Vranken in Holland  
● Robert Horr of T-Systemsin Germany
● Internally, we test using several different
software fuzzers such as:
● in-memory fuzzer
● a network fuzzer 
● OSS-fuzz 
● libfuzzer 
● tlsfuzzer 
● AFL
● To maximize security we run 7 fuzz testers
internally every night on our latest code on
over 20 platforms!  
@bagder@bagder
The curl bug-bounty
https://hackerone.com/curl
Pays rewards using donated money
Rewards up to 12,000 USD for a critical flaw
Has paid 2,300 USD so far
Increasing bounty amounts
@bagder@bagder
@bagder@bagder
University researchers
looking for a class of bugs
Get involved!
Corporate security researchers
who’s companies consume curl
Pen testers
who want to find something big and widespread
Security enthusiasts
looking for a reward and “fame”
@bagder@bagder
There’s one more thing curl has
@bagder@bagder
Full-time curl maintainer
@bagder@bagder
Bad things will happen
We are not immune, no matter how hard we try!
Act immediately
Own the problem
Fix it and announce it
Learn from it
Make it harder to do the same mistake again
@bagder@bagder
But does it work?
10 billion installations is no proof
✔ Decreasing number of CVEs over time
✔ Decreasing number of OSS-Fuzz reports over time
✔ Increasing bounty rewards
@bagder@bagder
Reported security vulnerabilities per year
@bagder@bagder
CRTTA
@bagder@bagder
Clean code
Review
Test
Torment
Act on mistakes
@bagder@bagder
Daniel Stenberg
@bagder
https://daniel.haxx.se/
Thank you!Thank you!
Questions?Questions?
@bagder@bagder
License
This presentation and its contents are
licensed under the Creative Commons
Attribution 4.0 license:
http://creativecommons.org/licenses/by/4.0/
@bagder@bagder

Testing curl for security

  • 1.
  • 2.
  • 3.
  • 4.
    … ten billioninstallations across the world curl runs in... @bagder@bagder
  • 5.
    …… devices allof you heredevices all of you here today own and use!today own and use! curl runs in @bagder@bagder
  • 6.
    Writing safe codeWritingsafe code @bagder@bagder
  • 7.
    Safe code isnot a coincidence Doesn’t happen randomly, we need to work on it! We care about it! It is about time we all address this topic! Everything I’ll tell you today, you already know! @bagder@bagder
  • 8.
    Writing Help users usethe tool and APIs safelyHelp users use the tool and APIs safely Defaults!Defaults! Document. Everything, detailed and a lot.Document. Everything, detailed and a lot. Get rid of warningsGet rid of warnings UseUse commentscomments assert()assert() Releasing the code asReleasing the code as open sourceopen source @bagder@bagder
  • 9.
    Review All codeAll codeshallshall get reviewedget reviewed CodeCode shallshall be easy to readbe easy to read and understandand understand Use theUse the same code stylesame code style everywhereeverywhere Commit message template andCommit message template and qualityquality Make sure style and templates are followed!Make sure style and templates are followed! @bagder@bagder
  • 10.
    Test! Unit tests System tests Integrationtests Test the documentation Manual tests Fixed a bug? Add a test. Or two. @bagder@bagder
  • 11.
    Torment Valgrind, address/UB/integer sanitizersValgrind,address/UB/integer sanitizers Static code analysersStatic code analysers FuzzersFuzzers @bagder@bagder
  • 12.
    All that, allthe time For every commit For every PR All. The. Time @bagder@bagder
  • 13.
    curl CI 80+ buildsand test “rounds” per commit Tests code style, indenting etc Thousands of tests per build Builds and tests on tens of platforms 20-25 hours of CI per commit @bagder@bagder
  • 14.
  • 15.
  • 16.
    The curl project’sCI sponsors Parts of the CI system run of free tiers Teamviewer sponsors Appveyor powers Travis gives us more power Azure Pipelines gives us more power @bagder@bagder
  • 17.
  • 18.
  • 19.
    Writing test casesshould be easy A curl test case is a single file in a human readable well-documented format • conditions and features needed to run • what command (line) to run • what the test wants returned from a server • how the protocol exchange should look like • stdout and stderr contents • expected error code • … and more @bagder@bagder
  • 20.
    Custom test servers Weavoid using “real” servers for testing Test servers are as dumb as possible Controlled from the test case what it should send and expect Allows “crazy” behaviors and send/receive “anything” Makes the test servers smaller and simpler Helps with test suite portability TLS is done by stunnel-fronting Servers run on random port numbers @bagder@bagder
  • 21.
    Tools of thetrade Valgrind Clang address, undefined, signed- integer-overflow sanitizers Clang tidy “torture tests” Scan-build Lgtm Codacy CodeQL Monocle AI Deepcode AI Coverity Travis CI Appveyor Cirrus CI Github Actions Azure Pipelines Buildbots OSS-Fuzz CI-fuzz @bagder@bagder
  • 22.
    Torture tests –error injection Build with a “debug” option Add wrapper functions for common “fallible” functions (that allocate and free resources) Each wrapper function logs what it does and can optionally return error. The complete individual test case is first run once count how many fallible functions rerun the test case that number of times and for each iteration make next fallible function return error Verify nothing crashed and no memory leaked Repeat for all tests @bagder@bagder
  • 23.
    Source code policy Fixall warnings (eye roll)Fix all warnings (eye roll) No defects leftNo defects left Use the strictest and most picky optionsUse the strictest and most picky options As many tests as possibleAs many tests as possible Fix security issues as soon as possibleqFix security issues as soon as possibleq @bagder@bagder
  • 24.
    A million buildcombos, 70+ OSes and 20 CPUs Testing all combinations is simply not possible Test the common setups Test on as many platforms as possible Test on several different CPU architectures “white spots” in test coverage handled by review Users keep finding untested areas and build combinations @bagder@bagder
  • 25.
    The security ofcurl is the sum of all components combined. - Lean on the right set! @bagder@bagder
  • 26.
    32 third partydependencies I/O layer libcurl URL parser libidn2winidn HTTPHTTPS OpenSSL Mesalink gskit mbedTLS wolfSSL Schannel SecureTransport GnuTLS NSS boringssl libressl AmiSSL SFTP SCP LDAP WinLDAP OpenLDAP RTMP librtmp Name resolver c-ares compression libz brotli cookies libpsl IMAP SMTP POP3 HTTP/2 nghttp2 authentication winsspi Heimdal MIT kerberos HTTP/3 quiche ngtcp2 HTTP/1 SSH wolfSSH libssh2 libssh @bagder@bagder BearSSL nghttp3
  • 27.
    Fuzz Testing inwolfSSL ● Best Tested TLS due to our extensive internal testing process ● Fuzz testing in our internal testing process is one of the reasons why ● Fuzz testing bombards the program with invalid, unexpected, and random data that then allows for observing if there is potential memory leaks or logic errors   ● Allows us to catch bugs that could turn into potential vulnerabilities ● We only use highly respected external testers: ● Guido Vranken in Holland   ● Robert Horr of T-Systemsin Germany ● Internally, we test using several different software fuzzers such as: ● in-memory fuzzer ● a network fuzzer  ● OSS-fuzz  ● libfuzzer  ● tlsfuzzer  ● AFL ● To maximize security we run 7 fuzz testers internally every night on our latest code on over 20 platforms!   @bagder@bagder
  • 28.
    The curl bug-bounty https://hackerone.com/curl Paysrewards using donated money Rewards up to 12,000 USD for a critical flaw Has paid 2,300 USD so far Increasing bounty amounts @bagder@bagder
  • 29.
  • 30.
    University researchers looking fora class of bugs Get involved! Corporate security researchers who’s companies consume curl Pen testers who want to find something big and widespread Security enthusiasts looking for a reward and “fame” @bagder@bagder
  • 31.
    There’s one morething curl has @bagder@bagder
  • 32.
  • 33.
    Bad things willhappen We are not immune, no matter how hard we try! Act immediately Own the problem Fix it and announce it Learn from it Make it harder to do the same mistake again @bagder@bagder
  • 34.
    But does itwork? 10 billion installations is no proof ✔ Decreasing number of CVEs over time ✔ Decreasing number of OSS-Fuzz reports over time ✔ Increasing bounty rewards @bagder@bagder
  • 35.
    Reported security vulnerabilitiesper year @bagder@bagder
  • 36.
  • 37.
  • 38.
  • 39.
    License This presentation andits contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder