[SPARK-53325] Support Prometheus 2.0 text-based-format and best pract… #9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bumps JOSDK version to 5.1.2 contains an important fix for operator-framework/java-operator-sdk#2869 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? default test suit ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#297 from csviri/bump-josdk. Authored-by: Attila Mészáros <a_meszaros@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to use `reporting.baseDirectory` instead of `reporting.baseDir`. ### Why are the changes needed? `reporting.baseDir` was deprecated at Gradle 8 and removed at Gradle 9. - https://docs.gradle.org/current/userguide/upgrading_version_8.html#reporting-base-dir > ReportingExtension.getBaseDir(), `ReportingExtension.setBaseDir(File), and ReportingExtension.setBaseDir(Object) were deprecated. They should be replaced with ReportingExtension.getBaseDirectory() property. - https://docs.gradle.org/current/dsl/org.gradle.api.reporting.ReportingExtension.html ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#300 from dongjoon-hyun/SPARK-53407. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to upgrade `Gradle` to 9.0.0. ### Why are the changes needed? `Gradle 9` is a new major release based on Java 17. We had better start with `Gradle 9` at `Spark K8s Operator v1.0.0`. - https://docs.gradle.org/9.0.0/release-notes.html ### Does this PR introduce _any_ user-facing change? No, this is a dev-only change. ### How was this patch tested? Pass the CIs and check the following log. ``` Welcome to Gradle 9.0.0! Here are the highlights of this release: - Configuration Cache is the recommended execution mode - Gradle requires JVM 17 or higher to run - Build scripts use Kotlin 2.2 and Groovy 4.0 - Improved Kotlin DSL script compilation avoidance ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#301 from dongjoon-hyun/SPARK-53409. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…LI argument ### What changes were proposed in this pull request? This PR aims to improve `merge_spark_pr.py` to accept PR numbers as a CLI argument. ### Why are the changes needed? To provide an alternative shortcut. **BEFORE** ``` $ dev/merge_spark_pr.py git rev-parse --abbrev-ref HEAD Which pull request would you like to merge? (e.g. 34): 302 ================================================================================ ``` **AFTER (Additional usage)** ``` $ dev/merge_spark_pr.py 302 git rev-parse --abbrev-ref HEAD Start to merge pull request #52020 ================================================================================ ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual test because this is irrelevant to the CI results. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#302 from dongjoon-hyun/SPARK-53410. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to upgrade the minimum K8s version to v1.32 in K8s test environment. ### Why are the changes needed? K8s 1.31 entered maintenance mode Today (August 28, 2025) and will reach the `End of Life` date on Oct 28, 2025. We had better focus on K8s v1.32+. - https://kubernetes.io/releases/patch-releases/#1-31 In addition, the default K8s versions of public cloud providers are already moving to K8s 1.32+ like the following. - EKS: v1.32 (Default), v1.33 (Available) - AKS: v1.32 (Default), v1.33 (GA) - GKE: v1.32 (Stable), v1.32 (Regular), v1.33 (Rapid) ### Does this PR introduce _any_ user-facing change? No because this is a test infra change. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#303 from dongjoon-hyun/SPARK-53411. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…ices for metrics naming
### What changes were proposed in this pull request?
This PR adds support for Prometheus text-based-format and best practices for metrics naming
Existing format
```
metrics_jvm_bufferPool_direct_capacity_Number{type="gauges"} 98348
metrics_jvm_bufferPool_direct_capacity_Value{type="gauges"} 98348
metrics_jvm_bufferPool_direct_count_Number{type="gauges"} 41
metrics_jvm_bufferPool_direct_count_Value{type="gauges"} 41
metrics_kubernetes_client_http_response_latency_nanos_Count{type="histograms"} 26910
metrics_kubernetes_client_http_response_latency_nanos_Max{type="histograms"} 232417143
metrics_kubernetes_client_http_response_latency_nanos_Mean{type="histograms"} 1.1410164260725182E7
metrics_kubernetes_client_http_response_latency_nanos_Min{type="histograms"} 2931711
metrics_kubernetes_client_http_response_latency_nanos_50thPercentile{type="histograms"} 7559152.0
metrics_kubernetes_client_http_response_latency_nanos_75thPercentile{type="histograms"} 9440850.0
metrics_kubernetes_client_http_response_latency_nanos_95thPercentile{type="histograms"} 1.2576766E7
metrics_kubernetes_client_http_response_latency_nanos_98thPercentile{type="histograms"} 1.34034482E8
metrics_kubernetes_client_http_response_latency_nanos_99thPercentile{type="histograms"} 1.34034482E8
metrics_kubernetes_client_http_response_latency_nanos_999thPercentile{type="histograms"} 1.34034482E8
metrics_kubernetes_client_http_response_latency_nanos_StdDev{type="histograms"} 2.177784612259799E7
metrics_kubernetes_client_pods_get_Count{type="counters"} 8967
metrics_kubernetes_client_pods_get_MeanRate{type="counters"} 0.02678169644780033
metrics_kubernetes_client_pods_get_OneMinuteRate{type="counters"} 0.049758750361204154
metrics_kubernetes_client_pods_get_FiveMinuteRate{type="counters"} 0.035255140329213855
metrics_kubernetes_client_pods_get_FifteenMinuteRate{type="counters"} 0.02931221844089468
```
with this patch, operator would be able to export format matching Prometheus 2.0 recommended practice like
```
# HELP jvm_bufferpool_direct_capacity Gauge metric
# TYPE jvm_bufferpool_direct_capacity gauge
jvm_bufferpool_direct_capacity 256092
# HELP jvm_bufferpool_direct_count Gauge metric
# TYPE jvm_bufferpool_direct_count gauge
jvm_bufferpool_direct_count 44
# HELP kubernetes_client_2xx_total Meter count
# TYPE kubernetes_client_2xx_total counter
kubernetes_client_2xx_total 130
# HELP kubernetes_client_http_response_latency Histogram metric
# TYPE kubernetes_client_http_response_latency histogram
kubernetes_client_http_response_latency_seconds_bucket{le="0.5"} 0.000104422
kubernetes_client_http_response_latency_seconds_bucket{le="0.75"} 0.000128422
kubernetes_client_http_response_latency_seconds_bucket{le="0.95"} 0.000139544
kubernetes_client_http_response_latency_seconds_bucket{le="0.98"} 0.000169124
kubernetes_client_http_response_latency_seconds_bucket{le="0.99"} 0.066452639
kubernetes_client_http_response_latency_seconds_count 2000
kubernetes_client_http_response_latency_seconds_sum 0.456670434
```
### Why are the changes needed?
It's Prometheus 2.0 best practice for using the next format with necessary comments.
Also, some common scrapers (like Datadog) rely on these metadata (e.g. # HELP and # TYPE) to parse metrics correctly. They may skip metrics if these are missing.
### Does this PR introduce _any_ user-facing change?
New functionalities becomes available (for metrics format)
### How was this patch tested?
CIs / curl on :19090/prometheus to validate the format
### Was this patch authored or co-authored using generative AI tooling?
No
fe2f023 to
a4ddc93
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ices for metrics naming
What changes were proposed in this pull request?
This PR adds support for Prometheus text-based-format and best practices for metrics naming
Existing format
with this patch, operator would be able to export format matching Prometheus 2.0 recommended practice like
Why are the changes needed?
It's Prometheus 2.0 best practice for using the next format with necessary comments. Also, some common scrapers (like Datadog) rely on these metadata (e.g. # HELP and # TYPE) to parse metrics correctly. They may skip metrics if these are missing.
Does this PR introduce any user-facing change?
New functionalities becomes available (for metrics format)
How was this patch tested?
CIs / curl on :19090/prometheus to validate the format
Was this patch authored or co-authored using generative AI tooling?
No
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?