So we should provide the ability to turn this feature on/off and ideally also allow for sampling of metrics and extrapolating out those numbers so that we can monitor these things only for a …

I didn't find any discussion about the semantic change in sampling strategy. No matter we implement class weights or instance weights, the simple random sampling should become …

2024-08-06 06:42:20,852 ERROR [Timer-Driven Process Thread-17] o.a.n.processors.standard.ConvertRecord ConvertRecord [id=2f9e8b7a-045c-301d-f79c …

This approach suffers in performance if sampling data is expensive. For instance, when the data to be sampled is the output of an expensive computation, sampling the data would cause the …

Nice-to-have: support additional common operators, including limit and sampling. Note that both 1 and 2 are problems that the current data source API (v1) suffers.

Nov 24, 2023 · This feature is to add a low overhead sampling profiler like async-profiler as a built in capability to the Spark job that can be turned on using only user configurable parameters …

Spark SPARK-22012 CLONE - Spark Streaming, Kafka receiver, "Failed to get records for ... after polling for 512"

Jun 17, 2024 · Hive HIVE-27831 Set hive.cbo.fallback.strategy to NEVER by default HIVE-28333 Skip CBO for UNIQUEJOIN and TABLESAMPLE

Spark SPARK-27558 NPE in TaskCompletionListener due to Spark OOM in UnsafeExternalSorter causing tasks to hang

Jan 29, 2024 · Currently all the options about state and checkpointing are reorganized and categorized by prefixes as listed below: 1. execution.checkpointing: all configurations …