Designing and Implementing Effective Logging Strategies

July 21th, 2022
2nd Cosenza MuleSoft Meetup
Designing and Implementing Effective Logging
Strategies
Andrea Capolei

2
● Principal Solutions Architect at Cap4 Lab Italy (Cosenza)
● Software and Integration Engineer for Web Apis
● Research Engineer for Embedded Systems applications. Inventor.
● Bsc in Computer Science, Master and PhD in Applied Mathematics
from La Sapienza and DTU. Then Postdoc on HPC at DTU.
About me

3
● Feedback from previous meetups
● Blog post and Code repository
● Next meetups
Meetup Group Status

5
We log mainly to debug and diagnose issues quicky
Logging is useful for a lot of purposes
– Recording user actions
– Recording an application’s activities
– Recording system activities
Use cases for logging
– Auditing (recording domain-level events for inspection)
– Tracking progress/activities of headless or otherwise unreachable systems or applications
– Debugging

6
Observability is about internal state
Software System
Outputs
Inputs
Observability is a measure of how well internal states of a system can be inferred from knowledge of its external
outputs. (Wikipedia)
It answers to “what can be observed?”
- In general, there is a part of a system that can’t be observed and a part that we are not observing because we missed to
sense it

7
Observability is not another synonym for telemetry
• Outside this room you will find vendors claiming that Observability is about three pillars aspects. That they can (and
will) sell!: metrics, logs, and traces
• Instead, we should see Observability as an opportunity to rethink how we gather data to debug effectively
"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements." Brian W.
Kernighan (C/Unix pioneer)

8
Observability is not another synonym for telemetry
• Outside this room you will find vendors claiming that Observability is about three pillars aspects. That they can (and
will) sell!: metrics, logs, and traces
• Instead, we should see Observability as an opportunity to rethink how we gather data to debug effectively
"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements." Brian W.
Kernighan (C/Unix pioneer)
Event Logs, metrics, and traces are just different measurement functions:
Event Logs record discrete events in one application behavior.
Metrics harness the power of mathematical modeling and prediction to derive knowledge
Traces encode the end-to-end request flow through a distributed system

9
Shifting to Observability imply to gather a rich context
• We want to be able to make explorative questions about our system (query our measured data in many different ways)
without deploying new code.
• Developers are encouraged to gather a rich context* to allow this
*Security warning

10
In the OWASP top ten security risks list
A01:2021 – Broken Access Control
CWE-532: Insertion of Sensitive Information into Log File
Information written to log files can be of a sensitive nature and give valuable guidance to an attacker or expose
sensitive user information. While logging all information may be helpful during development stages, it is
important that logging levels be set appropriately before a product ships so that sensitive user data and system
information are not accidentally exposed to potential attackers.
A09:2021-Security Logging and Monitoring Failures
“Without logging and monitoring, breaches cannot be detected. Insufficient logging, detection, monitoring, and active response occurs any time”
“You are vulnerable to information leakage by making logging and alerting events visible to a user or an attacker”

11
OWASP Example attack scenarios
Scenario #1: A children's health plan provider's website operator couldn't detect a breach due to a lack of monitoring and
logging. An external party informed the health plan provider that an attacker had accessed and modified thousands of sensitive
health records of more than 3.5 million children. A post-incident review found that the website developers had not addressed
significant vulnerabilities. As there was no logging or monitoring of the system, the data breach could have been in progress
since 2013, a period of more than seven years.
Scenario #2: A major Indian airline had a data breach involving more than ten years' worth of personal data of millions of
passengers, including passport and credit card data. The data breach occurred at a third-party cloud hosting provider, who
notified the airline of the breach after some time.
Scenario #3: A major European airline suffered a GDPR reportable breach. The breach was reportedly caused by payment
application security vulnerabilities exploited by attackers, who harvested more than 400,000 customer payment records. The
airline was fined 20 million pounds as a result by the privacy regulator.

12
Explicit design for confidentiality
• You need to recognize that almost any business data is potentially sensitive when enriched with data from other
contexts. Then, almost never include business data in logging. There are some exceptions:
1. We are sure that some Business Data is not sensitive
2. We are logging data of non prod environments (i.e. it is ok to log test/fictious data)
3. We are doing auditing. Audit logs require to be placed under strict access and authorization control.
• Logging levels (ERROR, INFO, …) tend to focus on response actions rather than confidentiality of data
• We should explicitly design for sensitive information. This requires extensive business knowledge.
• With domain driven logging (domain probes) we minimize the risk of logging incorrect data (information encapsulation)
E.g.
logger.cancelBooking(bookingId, user)
logger.bookingCancellFailing(bookingId, result, user)
• Include category distinction among Audit, Behavior, Error (Use a different logger for sensitive data to avoid accidental
information leakage)

13
Logging should aim to keep the behaviour easy to decipher
• Use a normalized, structured format for logging e.g. JSON. This allows to effectively aggregate data from different
sources.
• Using normalized data you can also digitally sign the payload to ensure integrity (you know that it has not being tampered
e.g.during a copy, you should have a single source of truth for the data).
• Limit the log message length to a certain size.
• Ensure availability, traceability, and auditability
• Avoid Second order attacks: Never log unchecked user input, use domain primitives

14
Think in advance at all the states you want to observe
• Log’s source (application, system)
• Logger’s Type (mule runtime, custom logger)
• Business vs Techinical data (process name, number of retries)
• Sensitive informations (Audit, Behavior, Error)
• Transactions
• Tracepoints (e.g. before/after an external request, a transform)
• Runtime’s properties (e.g. thread name)
• Implementation’s details (e.g. flow name)
• All the additional information helpful to diagnose the system
• Alert/Priority level (ERROR, INFO, DEBUG,…)

15
Priority Configuration Levels
Event Level Logger Configuration Level
TRACE DEBUG INFO WARN ERROR
TRACE Yes No No No No
DEBUG Yes Yes No No No
INFO Yes Yes Yes No No
WARN Yes Yes Yes Yes No
ERROR Yes Yes Yes Yes Yes

16
Data format example
"log_timestamp": "2022-03-15T17:01:13.314Z",
"loggerName": "org.mule.extension.jsonlogger.JsonLogger",
"system" : "azure-net-on-prem",
"hostId" : "sv-mule-001",
"applicationName" : "json-logger-test-app",
"applicationVersion" : "1.0.0-SNAPSHOT",
"priority" : "INFO",
"threadName" : "[json-logger-test-app].api-httpListenerConfig.worker.03",
"environment" : "dev",
"correlationId" : "99825dc0-ccc9-11e7-87ad-f45c898c34ab",
"logger_content": {
"logger_version" : "1.0.0",
"businessProcess" : "payments",
"processName" : "s-payment-implementationFlow",
"tracePoint" : "START",
"processCorrelationId": "12da85c6-ad22-11e7-abc4-cec278b6b58b",
“externalRequestId": "93da85c6-ad22-11e7-abc4-cec278b6b50a",
"message" : "Paying orderId EC1245",
"content" : "{"businessData": {"country": "IT" },“techinicalData": {"retry_num": "2"}}",
“additionalData" : {
"businessData" : {
"customerId" : "1234",
"orderId" : "EC1245",
"country" : "IT",
"paid": "100.00",
"currency": "eur"
}}}
additionalData is sensitive and it is logged only
in non prod environments for testing/debug
purposes.

17
Additional informations
Application Error Code User-defined error code provided by operations.
Severity Severity of the error (e.g. Sev1, Sev2)
Error Code HTTP Status Code associated with error (400, 404, 500,
503, etc.)
Error Description Human-readable description of the error.
{
“additionalData”: {"OriginatingApplication": "hybris",
"DestinationApplication": "Melissa"},
“applicationErrorCode": "MS-ADD-404",
“severity": "3",
“errorCode": "404",
“errorDescription": "Customer address not found."
}
Originating Application/Queue If known, application name of calling application or the queue name that the
current message was pulled.
Destination Application/Queue If known, application name of target application or the queue name that the
message will be pushed.
Content field
Recommendation for Error Logging

18
Log asynchronously, synchronously when needed
MuleSoft extensively uses Log4j2 for all logging requirements
(from Mulesoft logging’s documentation)
“ asynchronous logging substantially improves the throughput and lowers the latency of
message processing”
“If you use app logs as audit trails, configure your app to always
use synchronous logging. This prevents the loss of any log messages.”

19
External High performance, reliable Logging
Providing Proces
s
Qualifyin
g
Store Displa
y

20
Custom Logger Connector (XML SDK)

21
Custom Logger Connector
Implemented using Mule Java SDK
github.com/mulesoft-consulting/json-logger

Showing a working implementation of a demo strategy using a custom mule connector
Live demo

● Domain-Oriented Observability by Pete Hodgson
● Mulesoft consulting’s Java SDK based logger, by Andres Ramirez
● Mulesoft catalyst ’s XML SDK based logger
● Mulesoft logging’s documentation
● Tracing in Mulesoft with Zipkin connector.
● OWASP Top 10 Vulnerabilities
23
References

24
● Share:
○ Tweet using the hashtag #MuleSoftMeetups #MuleSoftMeetupsCosenza
○ Invite your network to join: https://meetups.mulesoft.com/cosenza/
● Feedback:
○ Fill out the survey feedback and suggest topics for upcoming events
○ Contact MuleSoft at meetups@mulesoft.com for ways to improve the program
What’s next?

Thank you
and see you next time!

Designing and Implementing Effective Logging Strategies

More Related Content

Similar to Designing and Implementing Effective Logging Strategies

Recently uploaded

Designing and Implementing Effective Logging Strategies