🚀 vectorizedio/redpanda - Release Notes

## Bug Fixes
* Addresses startup behavior in SR that may result in inconsistent SR state between nodes by @michael-redpanda in [#25406](https://github.com/redpanda-data/redpanda/pull/25406)
* Fixes a bug in which urgent garbage collection of multiple partitions may only consider one partition in the set. by @WillemKauf in [#25275](https://github.com/redpanda-data/redpanda/pull/25275)
* This fixes a bug where kafka responses could get out-of-order with respect to request order on the same connection. This would surface as CorrelationIdMismatchException exceptions at the kafka client side. by @pgellert in [#25665](https://github.com/redpanda-data/redpanda/pull/25665)
* [#25284](https://github.com/redpanda-data/redpanda/issues/25284) Fix a bug in group management behaviour when the last member of a group expires while there are pending
members in the group. by @pgellert in [#25302](https://github.com/redpanda-data/redpanda/pull/25302)
* [#25523](https://github.com/redpanda-data/redpanda/issues/25523) kafka/server/group_manager: Prevent metrics::double_registration during group management. by @BenPope in [#25525](https://github.com/redpanda-data/redpanda/pull/25525)
* fixes a very rare situation in which Raft leader can enter into infinite loop trying to recover follower.  by @mmaslankaprv in [#25377](https://github.com/redpanda-data/redpanda/pull/25377)
* PR [#25417](https://github.com/redpanda-data/redpanda/pull/25417) [v24.2.x] pandaproxy/sr: Fix normalization not being applied on stored schemas by @IoannisRP
* PR [#25445](https://github.com/redpanda-data/redpanda/pull/25445) [v24.2.x] c/rm_stm: correct error codes by @bashtanov
* PR [#25466](https://github.com/redpanda-data/redpanda/pull/25466) [v24.2.x] pandaproxy/sr: Relax schema registry startup validity requirements by @IoannisRP
* PR [#25477](https://github.com/redpanda-data/redpanda/pull/25477) [v24.2.x] c/backend: fix copy pasta error in learner offset calculation by @bharathv
* PR [#25528](https://github.com/redpanda-data/redpanda/pull/25528) [v24.2.x] pandaproxy/sr: Re-purpose schema_registry_normalize_on_startup config by @IoannisRP
* PR [#25543](https://github.com/redpanda-data/redpanda/pull/25543) [v24.2.x] s3: Add timeout to credentials fetching operation by @Lazin
* PR [#25551](https://github.com/redpanda-data/redpanda/pull/25551) [v24.2.x] cluster/partition: Reword description for kafka_request_bytes_total. by @BenPope
* PR [#25576](https://github.com/redpanda-data/redpanda/pull/25576) [v24.2.x] Conservative validation in  by @mmaslankaprv
* PR [#25584](https://github.com/redpanda-data/redpanda/pull/25584) [v24.2.x] config: relax schema_registry_always_normalize to not need restart by @IoannisRP

## Improvements
* admin: Disallow a superuser from removing themself. by @BenPope in [#25547](https://github.com/redpanda-data/redpanda/pull/25547)
* PR [#25154](https://github.com/redpanda-data/redpanda/pull/25154) [v24.2.x] kafka/server: fix case of use after move by @IoannisRP

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.20...v24.2.21

## Features
* Redpanda now supports selecting which format the client certificate's subject DN will be parsed in.  The new cluster config `tls_certificate_name_format` can take two options: `legacy` or `rfc2253`.  See documentation of the cluster config to understand the difference in how the an X.509 certificate's subject DN will be formatted. by @michael-redpanda in [#25448](https://github.com/redpanda-data/redpanda/pull/25448)
## Bug Fixes
* [#25512](https://github.com/redpanda-data/redpanda/issues/25512) Rust target has changed and this updates it by @dhpollack in [#25513](https://github.com/redpanda-data/redpanda/pull/25513)
* [#25522](https://github.com/redpanda-data/redpanda/issues/25522) kafka/server/group_manager: Prevent metrics::double_registration during group management. by @BenPope in [#25524](https://github.com/redpanda-data/redpanda/pull/25524)
## Improvements
* Introduce lazy evaluation in schema registry during startup. by @IoannisRP in [#25464](https://github.com/redpanda-data/redpanda/pull/25464)
* `rpk transform` now uses the tinygo v37 to compile golang to Wasm. by @rockwotj in [#25538](https://github.com/redpanda-data/redpanda/pull/25538)
* admin: Disallow a superuser from removing themself. by @BenPope in [#25548](https://github.com/redpanda-data/redpanda/pull/25548)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.3.8...v24.3.9

## Features
* [#25384](https://github.com/redpanda-data/redpanda/issues/25384) A new metric, vectorized_kafka_produced_bytes, is added to the /metrics endpoint which breaks down bytes produced to Redpanda at the Kafka API by batch compression type.  by @travisdowns in [#25391](https://github.com/redpanda-data/redpanda/pull/25391)
## Bug Fixes
* Addresses startup behavior in SR that may result in inconsistent SR state between nodes by @michael-redpanda in [#25408](https://github.com/redpanda-data/redpanda/pull/25408)
* PR [#25414](https://github.com/redpanda-data/redpanda/pull/25414) [v24.3.x] r/stm_manager: do not populate batch cache when background apply happens by @mmaslankaprv
* PR [#25416](https://github.com/redpanda-data/redpanda/pull/25416) [v24.3.x] pandaproxy/sr: Fix normalization not being applied on stored schemas by @IoannisRP
* PR [#25421](https://github.com/redpanda-data/redpanda/pull/25421) [v24.3.x] Do not enable translation stm when datalake is disabled by @mmaslankaprv

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.3.7...v24.3.8

## Bug Fixes
* Fixes a bug in which urgent garbage collection of multiple partitions may only consider one partition in the set. by @WillemKauf in [#25274](https://github.com/redpanda-data/redpanda/pull/25274)
* [#25283](https://github.com/redpanda-data/redpanda/issues/25283) Fix a bug in group management behaviour when the last member of a group expires while there are pending
members in the group. by @pgellert in [#25301](https://github.com/redpanda-data/redpanda/pull/25301)
* fixes a very rare situation in which Raft leader can enter into infinite loop trying to recover follower.  by @mmaslankaprv in [#25338](https://github.com/redpanda-data/redpanda/pull/25338)
* rpk: fixes a bug that prevented serverless cloud profiles from using `rpk security user` commands. by @r-vasquez in [#25386](https://github.com/redpanda-data/redpanda/pull/25386)
* PR [#25300](https://github.com/redpanda-data/redpanda/pull/25300) [v24.3.x] k/topic_utils: do not iterate over the assignments set by @mmaslankaprv

## Improvements
* [#25125](https://github.com/redpanda-data/redpanda/issues/25125) schema_registry/protobuf: Rewrite protobuf normalization to improve compatibility with Java client by @BenPope in [#25128](https://github.com/redpanda-data/redpanda/pull/25128)
* PR [#25355](https://github.com/redpanda-data/redpanda/pull/25355) [v24.3.x] rpk: make CLI and cluster versions explicit by @daisukebe
* PR [#25362](https://github.com/redpanda-data/redpanda/pull/25362) [v24.3.x] rpk: bump dependencies to include security fixes by @r-vasquez

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.3.6...v24.3.7

## Improvements
* [#25124](https://github.com/redpanda-data/redpanda/issues/25124) [#25129](https://github.com/redpanda-data/redpanda/issues/25129) schema_registry/protobuf: Rewrite protobuf normalization to improve compatibility with Java client by @BenPope in [#25146](https://github.com/redpanda-data/redpanda/pull/25146)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.19...v24.2.20

## Bug Fixes
* Fix the endianness of `snappy_java_compressor` headers to match that of `snappy-java`. by @WillemKauf in [#25110](https://github.com/redpanda-data/redpanda/pull/25110)
* Fixed a bug over-restricting bundled json schemas to treat both "id" and "$id" as restricted keywords, regardless of the active draft.  by @IoannisRP in [#25060](https://github.com/redpanda-data/redpanda/pull/25060)
* Updated libxml2 to v2.10.12 to remediate known security vulnerabilities
* [#25007](https://github.com/redpanda-data/redpanda/issues/25007) Schema Registry/Protobuf: Fix a regression with maps. by @BenPope in [#25009](https://github.com/redpanda-data/redpanda/pull/25009)
* [#25026](https://github.com/redpanda-data/redpanda/issues/25026) rpk: fixes a bug where rpk incorrectly handles IPv6 by adding extra brackets. by @r-vasquez in [#25027](https://github.com/redpanda-data/redpanda/pull/25027)
## Improvements
* Redpanda will now schedule local segment merges of compacted topics, even when windowed compaction has occurred in a given housekeeping round. This ensures progress in reducing segment count in compacted topics with high produce traffic. by @andrwng in [#24956](https://github.com/redpanda-data/redpanda/pull/24956)
* [#25082](https://github.com/redpanda-data/redpanda/issues/25082) Use streaming parsing of transaction range manifests in tiered storage to avoid large allocations which lead to OOMs with heavy use of transactions. by @nvartolomei in [#25084](https://github.com/redpanda-data/redpanda/pull/25084)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.3.5...v24.3.6

## Bug Fixes
* Fix the endianness of `snappy_java_compressor` headers to match that of `snappy-java`. by @WillemKauf in [#25112](https://github.com/redpanda-data/redpanda/pull/25112)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.18...v24.2.19

## Bug Fixes
* Avoid large allocations for the kafka response sequencing map. by @pgellert in [#24741](https://github.com/redpanda-data/redpanda/pull/24741)
* Fix the endianness of `snappy_java_compressor` headers to match that of `snappy-java`. by @WillemKauf in [#25137](https://github.com/redpanda-data/redpanda/pull/25137)
* Fixes a bug where `rpk` wasn't parsing `--help` when used alongside `--redpanda-id` in `rpk cloud  byoc apply` by @r-vasquez in [#24398](https://github.com/redpanda-data/redpanda/pull/24398)
* Fixes a bug where failing to audit an authentication event could lead to a broker crash. by @pgellert in [#24740](https://github.com/redpanda-data/redpanda/pull/24740)
* If a discrete disk is used for cloud storage cache Redpanda previously rejected writes if that disk (cache disk) was full (in degraded state). This is incorrect since the cache disk isn't in the way of writes. From now on, reject writes only if the data disk is full (in degraded state). by @nvartolomei in [#24485](https://github.com/redpanda-data/redpanda/pull/24485)
* Remove partial kvstore snapshots at startup. by @ztlpn in [#24844](https://github.com/redpanda-data/redpanda/pull/24844)
* [#24427](https://github.com/redpanda-data/redpanda/issues/24427) Schema Registry: fixes a bug in the Avro compatibility check reader_field_missing_default_value where it was too lenient for missing default values of null-able types. by @pgellert in [#24429](https://github.com/redpanda-data/redpanda/pull/24429)
## Improvements
* Added metrics for pandaproxy resource usage.  by @IoannisRP in [#24605](https://github.com/redpanda-data/redpanda/pull/24605)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.1.18...v24.1.19

## Bug Fixes
* [#23363](https://github.com/redpanda-data/redpanda/issues/23363) [#25028](https://github.com/redpanda-data/redpanda/issues/25028) rpk: fixes a bug where rpk incorrectly handles IPv6 by adding extra brackets. by @r-vasquez in [#25067](https://github.com/redpanda-data/redpanda/pull/25067)
* [#25008](https://github.com/redpanda-data/redpanda/issues/25008) Schema Registry/Protobuf: Fix a regression with maps. by @BenPope in [#25010](https://github.com/redpanda-data/redpanda/pull/25010)
## Improvements
* [#25081](https://github.com/redpanda-data/redpanda/issues/25081) Use streaming parsing of transaction range manifests in tiered storage to avoid large allocations which lead to OOMs with heavy use of transactions. by @nvartolomei in [#25083](https://github.com/redpanda-data/redpanda/pull/25083)
* PR [#25013](https://github.com/redpanda-data/redpanda/pull/25013) [v24.2.x] k/fetch: fix selecting replica with highest high watermark by @mmaslankaprv
* PR [#25086](https://github.com/redpanda-data/redpanda/pull/25086) [v24.2.x] pandaproxy/sr: Improve schema lookup by @IoannisRP

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.17...v24.2.18

## Features
* rpk: `rpk group describe` now supports printing instance IDs by @daisukebe in [#24908](https://github.com/redpanda-data/redpanda/pull/24908)
## Bug Fixes
* Schema Registry: Fix a regression with protobuf map fields by @BenPope in [#24975](https://github.com/redpanda-data/redpanda/pull/24975)
## Improvements
* `rpk security user` is now available for users with Cloud profiles. by @r-vasquez in [#24951](https://github.com/redpanda-data/redpanda/pull/24951)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.3.4...v24.3.5

## Bug Fixes
* Fixes a crash during partition shutdown. This can happen during partition moves (cross core/broker) or at broker shutdown. by @bharathv in [#24938](https://github.com/redpanda-data/redpanda/pull/24938)
* Schema Registry: Fix a regression with protobuf map fields by @BenPope in [#24974](https://github.com/redpanda-data/redpanda/pull/24974)
* [#24915](https://github.com/redpanda-data/redpanda/issues/24915) Fixes integer overflow issues when given a schema via the `POST /subject/{subject}/version` where version was > INT_MAX or a negative value was provided. by @michael-redpanda in [#24917](https://github.com/redpanda-data/redpanda/pull/24917)
* [#25008](https://github.com/redpanda-data/redpanda/issues/25008) Schema Registry/Protobuf: Fix a regression with maps. by @BenPope in [#25010](https://github.com/redpanda-data/redpanda/pull/25010)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.16...v24.2.17

## Features
* Introduces the node config `crash_loop_sleep_sec`, which sets the time the broker sleeps before terminating the process when the limit on the number of consecutive times a broker can crash has been reached. This is most useful in Kubernetes environments where setting this value allows customers to have ssh access into a crash looping pod for a short window of time. by @pgellert in [#24825](https://github.com/redpanda-data/redpanda/pull/24825)
## Bug Fixes
* Avoid large allocations for the kafka response sequencing map. by @pgellert in [#24743](https://github.com/redpanda-data/redpanda/pull/24743)
* Fixes Iceberg metadata serialization to avoid writing an extraneous empty Avro block. This would previously prevent some query engines (e.g. BigQuery) from reading tables created by Iceberg Topics. by @andrwng in [#24920](https://github.com/redpanda-data/redpanda/pull/24920)
* Fixes a bug where failing to audit an authentication event could lead to a broker crash. by @pgellert in [#24738](https://github.com/redpanda-data/redpanda/pull/24738)
* Fixes a crash during partition shutdown. This can happen during partition moves (cross core/broker) or at broker shutdown. by @bharathv in [#24939](https://github.com/redpanda-data/redpanda/pull/24939)
* Fixes an issue where transactions incorrectly timeout due incorrect cleanup of evicted producers. by @bharathv in [#24879](https://github.com/redpanda-data/redpanda/pull/24879)
* Remove partial kvstore snapshots at startup. by @ztlpn in [#24845](https://github.com/redpanda-data/redpanda/pull/24845)
* [#24914](https://github.com/redpanda-data/redpanda/issues/24914) Fixes integer overflow issues when given a schema via the `POST /subject/{subject}/version` where version was > INT_MAX or a negative value was provided. by @michael-redpanda in [#24916](https://github.com/redpanda-data/redpanda/pull/24916)
## Improvements
* stable leadership under load by @mmaslankaprv in [#24773](https://github.com/redpanda-data/redpanda/pull/24773)
* PR [#24672](https://github.com/redpanda-data/redpanda/pull/24672) [v24.3.x] raft: Make load_snapshot exception safe by @Lazin
* PR [#24747](https://github.com/redpanda-data/redpanda/pull/24747) [v24.3.x] [CORE-8485] Reset translation state on snapshot by @mmaslankaprv
* PR [#24764](https://github.com/redpanda-data/redpanda/pull/24764) [v24.3.x] [CORE-8787] Schema Registry: Support normalize=true for protobuf by @BenPope
* PR [#24765](https://github.com/redpanda-data/redpanda/pull/24765) [v24.3.x] [CORE-8754] Handle new TLS error code by @michael-redpanda
* PR [#24783](https://github.com/redpanda-data/redpanda/pull/24783) [CORE-8787] [v24.3.x] schema_registry: Normalization improvements by @BenPope
* PR [#24785](https://github.com/redpanda-data/redpanda/pull/24785) [v24.3.x] bazel: Update protobuf to v29.0 by @BenPope
* PR [#24786](https://github.com/redpanda-data/redpanda/pull/24786) [v24.3.x] bazel: add rp_util and compat by @rockwotj
* PR [#24790](https://github.com/redpanda-data/redpanda/pull/24790) [v24.3.x] c/topic_table: notify ntp delta waiters in batches by @ztlpn
* PR [#24799](https://github.com/redpanda-data/redpanda/pull/24799) [v24.3.x] [CORE-8450] schema_registry/protobuf: Optimize construction of iobuf by @BenPope
* PR [#24805](https://github.com/redpanda-data/redpanda/pull/24805) [v24.3.x] rm_stm: improved logging related to producer eviction by @bharathv
* PR [#24809](https://github.com/redpanda-data/redpanda/pull/24809) Manual backport 24772 v24.3.x 935 by @michael-redpanda
* PR [#24836](https://github.com/redpanda-data/redpanda/pull/24836) [v24.3.x] ducktape: bump spark version by @ztlpn
* PR [#24841](https://github.com/redpanda-data/redpanda/pull/24841) [v24.3.x] r/stm_manager: stop state machines before waiting for gate by @mmaslankaprv
* PR [#24872](https://github.com/redpanda-data/redpanda/pull/24872) [v24.3.x] pandaproxy/sr: Fix normalized rendering for custom options by @IoannisRP
* PR [#24903](https://github.com/redpanda-data/redpanda/pull/24903) [v24.3.x] CORE-8804 Make brokers field no longer required by @michael-redpanda
* PR [#24906](https://github.com/redpanda-data/redpanda/pull/24906) [v24.3.x] Fix partitions local summary by @bashtanov
* PR [#24910](https://github.com/redpanda-data/redpanda/pull/24910) [v24.3.x] pandaproxy/sr: Add rendering support for extension range options by @IoannisRP

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.3.3...v24.3.4

## Features
* [#24826](https://github.com/redpanda-data/redpanda/issues/24826) Introduces the node config `crash_loop_sleep_sec`, which sets the time the broker sleeps before terminating the process when the limit on the number of consecutive times a broker can crash has been reached. This is most useful in Kubernetes environments where setting this value allows customers to have ssh access into a crash looping pod for a short window of time. by @pgellert in [#24846](https://github.com/redpanda-data/redpanda/pull/24846)
## Bug Fixes
* Avoid large allocations for the kafka response sequencing map. by @pgellert in [#24742](https://github.com/redpanda-data/redpanda/pull/24742)
* Fixes a bug where failing to audit an authentication event could lead to a broker crash. by @pgellert in [#24739](https://github.com/redpanda-data/redpanda/pull/24739)
* Fixes an issue where transactions incorrectly timeout due incorrect cleanup of evicted producers. by @bharathv in [#24878](https://github.com/redpanda-data/redpanda/pull/24878)
* Remove partial kvstore snapshots at startup. by @ztlpn in [#24843](https://github.com/redpanda-data/redpanda/pull/24843)
* [#24915](https://github.com/redpanda-data/redpanda/issues/24915) Fixes integer overflow issues when given a schema via the `POST /subject/{subject}/version` where version was > INT_MAX or a negative value was provided. by @michael-redpanda in [#24917](https://github.com/redpanda-data/redpanda/pull/24917)
## Improvements
* [#24668](https://github.com/redpanda-data/redpanda/issues/24668) stable leadership under load by @mmaslankaprv in [#24708](https://github.com/redpanda-data/redpanda/pull/24708)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.15...v24.2.16

## Bug Fixes
* Fixes a bug in Redpanda's Iceberg manifest list Avro definition that previously resulted in an end-of-file (EOF) error when reading manifest list Avro files written by other engines. This could previously crash Redpanda or block Redpanda from appending Iceberg data, and could also prevent certain query engines from successfully reading Iceberg data written by Redpanda. by @andrwng in [#24650](https://github.com/redpanda-data/redpanda/pull/24650)
* Fixes a bug which may lead to `archival_metadata_stm` inconsistencies when reconfiguring clusters with recovered compacted topics. by @mmaslankaprv in [#24678](https://github.com/redpanda-data/redpanda/pull/24678)
* [#24684](https://github.com/redpanda-data/redpanda/issues/24684) Fixes an issue that blocked the compaction of consumer offsets with group transactions. by @bharathv in [#24688](https://github.com/redpanda-data/redpanda/pull/24688)
* fixes rare bug leading to offset translation inconsistency in recovered topics by @mmaslankaprv in [#24628](https://github.com/redpanda-data/redpanda/pull/24628)

## Improvements
* Added metrics for pandaproxy resource usage.  by @IoannisRP in [#24603](https://github.com/redpanda-data/redpanda/pull/24603)
* Adds logging to mention data removed by compaction. by @andrwng in [#24736](https://github.com/redpanda-data/redpanda/pull/24736)
* Move failed authorization log statements from the `kafka` logger to a new `kafka/authz` logger, allowing for fine grained control over log statements for failed authorization. by @rockwotj in [#24718](https://github.com/redpanda-data/redpanda/pull/24718)
* rpk now supports well-known protobuf types when encoding/decoding records using Schema Registry. by @r-vasquez in [#24699](https://github.com/redpanda-data/redpanda/pull/24699)
* PR [#24591](https://github.com/redpanda-data/redpanda/pull/24591) [v24.3.x] pandaproxy: add missing internal metrics by @IoannisRP
* PR [#24608](https://github.com/redpanda-data/redpanda/pull/24608) [v24.3.x] `storage`: add `tombstones_removed` metric to `probe` by @WillemKauf
* PR [#24619](https://github.com/redpanda-data/redpanda/pull/24619) [v24.3.x] Offset translator consistency validation by @mmaslankaprv
* PR [#24627](https://github.com/redpanda-data/redpanda/pull/24627) [v24.3.x] rpk remote debug bundle: job-id help text change by @r-vasquez
* PR [#24705](https://github.com/redpanda-data/redpanda/pull/24705) [v24.3.x] kafka/client: replace std::vector with chunked vector by @IoannisRP
* PR [#24729](https://github.com/redpanda-data/redpanda/pull/24729) [v24.3.x] rpk bundle: Fix race condition in SASL credential redaction by @r-vasquez

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.3.2...v24.3.3

## Bug Fixes
* Fixes a bug where failing to audit an authentication event could lead to a broker crash. by @pgellert in [#24739](https://github.com/redpanda-data/redpanda/pull/24739)
* Fixes a bug which may lead to `archival_metadata_stm` inconsistencies when reconfiguring clusters with recovered compacted topics. by @mmaslankaprv in [#24680](https://github.com/redpanda-data/redpanda/pull/24680)
* [#24685](https://github.com/redpanda-data/redpanda/issues/24685) Fixes an issue that blocked the compaction of consumer offsets with group transactions. by @bharathv in [#24689](https://github.com/redpanda-data/redpanda/pull/24689)
* fixes rare bug leading to offset translation inconsistency in recovered topics by @mmaslankaprv in [#24629](https://github.com/redpanda-data/redpanda/pull/24629)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.14...v24.2.15

## Bug Fixes
* Fixes a bug in which a segment  being rolled and closed could race, leading to a triggered `vassert`. by @WillemKauf in [#24559](https://github.com/redpanda-data/redpanda/pull/24559)
## Improvements
* Added metrics for pandaproxy resource usage.  by @IoannisRP in [#24604](https://github.com/redpanda-data/redpanda/pull/24604)
* Show leader id in `/v1/cluster/partitions` response. by @ztlpn in [#24584](https://github.com/redpanda-data/redpanda/pull/24584)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.13...v24.2.14

## Features
* Improve the user messages when the
`topic_partitions_reserve_shard0` cluster config is used and a user tries to create a topic with more partitions than the core-based partition limit. by @pgellert in [#24461](https://github.com/redpanda-data/redpanda/pull/24461)
## Bug Fixes
* Ensure `redpanda_cloud_storage_cloud_log_size` metric consistent across all replicas. We used to update it seldomly from the leader replica only which lead to inconsistent/stale values. by @nvartolomei in [#24364](https://github.com/redpanda-data/redpanda/pull/24364)
* Fixed a bug in which sliding window compaction may become stuck on failing to build an index map for a single segment. by @WillemKauf in [#24424](https://github.com/redpanda-data/redpanda/pull/24424)
* Fixes a bug in which a segment  being rolled and closed could race, leading to a triggered `vassert`. by @WillemKauf in [#24560](https://github.com/redpanda-data/redpanda/pull/24560)
* Fixes a bug in which segments which may have tombstones in them were not considered eligible for self-compaction. by @WillemKauf in [#24500](https://github.com/redpanda-data/redpanda/pull/24500)
* Fixes a bug that could prevent topic recovery on ABS object storage when there are objects in a bucket from multiple clusters (e.g. following a whole cluster restore). by @andrwng in [#24455](https://github.com/redpanda-data/redpanda/pull/24455)
* Fixes a bug where `rpk` wasn't parsing `--help` when used alongside `--redpanda-id` in `rpk cloud  byoc apply` by @r-vasquez in [#24396](https://github.com/redpanda-data/redpanda/pull/24396)
* Fixes a bug where serializing manifests for Iceberg topics with decimal fields could cause Redpanda to crash or upload invalid manifests by @oleiman in [#24467](https://github.com/redpanda-data/redpanda/pull/24467)
* Fixes a crash resulting from incorrect cleanup of log readers used for iceberg translation. by @bharathv in [#24576](https://github.com/redpanda-data/redpanda/pull/24576)
* Fixes a race that could prevent Iceberg translation from happening following a leadership change. by @andrwng in [#24562](https://github.com/redpanda-data/redpanda/pull/24562)
* Fixes accounting of iceberg commit lag metric that can remain erroneously high in some cases even though the translation if fully caught up. Additionally the change ensures that only partition leaders emit lag metrics while followers emit 0 lag. by @bharathv in [#24575](https://github.com/redpanda-data/redpanda/pull/24575)
* If a discrete disk is used for cloud storage cache Redpanda previously rejected writes if that disk (cache disk) was full (in degraded state). This is incorrect since the cache disk isn't in the way of writes. From now on, reject writes only if the data disk is full (in degraded state). by @nvartolomei in [#24486](https://github.com/redpanda-data/redpanda/pull/24486)
* [#24428](https://github.com/redpanda-data/redpanda/issues/24428) Schema Registry: fixes a bug in the Avro compatibility check reader_field_missing_default_value where it was too lenient for missing default values of null-able types. by @pgellert in [#24430](https://github.com/redpanda-data/redpanda/pull/24430)
* [#24587](https://github.com/redpanda-data/redpanda/issues/24587) Redpanda will now permit topics to be created with `redpanda.remote.[read|write]` set to `true` when a license is expired or missing provided that the cluster config `cloud_storage_enabled` is set to `false`. by @michael-redpanda in [#24588](https://github.com/redpanda-data/redpanda/pull/24588)
## Improvements
* Adds additional debug log messages in the datalake coordinator regarding files to be committed to Iceberg. by @andrwng in [#24563](https://github.com/redpanda-data/redpanda/pull/24563)
* Beta version of Iceberg support was incorrectly classified as "enterprise only". by @oleiman in [#24443](https://github.com/redpanda-data/redpanda/pull/24443)
* Leader balancer: don't treat each core as independent and balance total number of leaders on each node as well. by @ztlpn in [#24440](https://github.com/redpanda-data/redpanda/pull/24440)
* Show leader id in `/v1/cluster/partitions` response. by @ztlpn in [#24585](https://github.com/redpanda-data/redpanda/pull/24585)
* [#24539](https://github.com/redpanda-data/redpanda/issues/24539) Disable datalake services in recovery mode by @ztlpn in [#24549](https://github.com/redpanda-data/redpanda/pull/24549)
* `rpk topic describe`  now supports the `--format` flag to display the output in either JSON or YAML. by @r-vasquez in [#24438](https://github.com/redpanda-data/redpanda/pull/24438)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.3.1...v24.3.2

## Features
* Improve the user messages when the
`topic_partitions_reserve_shard0` cluster config is used and a user tries to create a topic with more partitions than the core-based partition limit. by @pgellert in [#24462](https://github.com/redpanda-data/redpanda/pull/24462)
## Bug Fixes
* Ensure `redpanda_cloud_storage_cloud_log_size` metric consistent across all replicas. We used to update it seldomly from the leader replica only which lead to inconsistent/stale values. by @nvartolomei in [#24365](https://github.com/redpanda-data/redpanda/pull/24365)
* Fixes a bug that could prevent topic recovery on ABS object storage when there are objects in a bucket from multiple clusters (e.g. following a whole cluster restore). by @andrwng in [#24454](https://github.com/redpanda-data/redpanda/pull/24454)
* Fixes a bug where `rpk` wasn't parsing `--help` when used alongside `--redpanda-id` in `rpk cloud  byoc apply` by @r-vasquez in [#24397](https://github.com/redpanda-data/redpanda/pull/24397)
* If a discrete disk is used for cloud storage cache Redpanda previously rejected writes if that disk (cache disk) was full (in degraded state). This is incorrect since the cache disk isn't in the way of writes. From now on, reject writes only if the data disk is full (in degraded state). by @nvartolomei in [#24484](https://github.com/redpanda-data/redpanda/pull/24484)
* [#24431](https://github.com/redpanda-data/redpanda/issues/24431) Schema Registry: fixes a bug in the Avro compatibility check reader_field_missing_default_value where it was too lenient for missing default values of null-able types. by @pgellert in [#24432](https://github.com/redpanda-data/redpanda/pull/24432)
* PR [#24200](https://github.com/redpanda-data/redpanda/pull/24200) [v24.2.x] cst/cache: fix use-after-move caused by calling get_exception twice by @nvartolomei
* PR [#24329](https://github.com/redpanda-data/redpanda/pull/24329) [v24.2.x] Fixed race condition between appends and prefix truncation by @mmaslankaprv
* PR [#24335](https://github.com/redpanda-data/redpanda/pull/24335) rm_stm: remove always true assert on transaction_ga feature by @bharathv
* PR [#24349](https://github.com/redpanda-data/redpanda/pull/24349) [v24.2.x] c/balancer_planner: check if topic exists in node count map by @mmaslankaprv
* PR [#24372](https://github.com/redpanda-data/redpanda/pull/24372) [v24.2.x] c/controller_backend: allow `shutdown_partition` to fail on app shutdown by @bashtanov
* PR [#24459](https://github.com/redpanda-data/redpanda/pull/24459) [v24.2.x] raft/c: fix an indefinite hang in transfer leadership by @bharathv

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.12...v24.2.13

## Features
* Added support for Iceberg Topics (various improvements below)
* New REST API for mounting/unmounting topics by @mmaslankaprv in [#23167](https://github.com/redpanda-data/redpanda/pull/23167)
* adds rpk cluster storage topic mount, unmount, list-mount, status-mount, cancel-mount by @gene-redpanda in [#23575](https://github.com/redpanda-data/redpanda/pull/23575)
* Add leadership pinning: ability to set preferred racks for topic partition leaders. To configure, set `redpanda.leaders.preference` topic config property or `default_leaders_preference` cluster config property. by @ztlpn in [#23691](https://github.com/redpanda-data/redpanda/pull/23691)
* Enable `node_local_core_assignment` feature by default by @ztlpn in [#23453](https://github.com/redpanda-data/redpanda/pull/23453)
* Adds Schema Registry support for the JavaScript Data Transforms SDK by @oleiman in [#21491](https://github.com/redpanda-data/redpanda/pull/21491)
* Adds list-mountable to allow listing mountable topics  by @gene-redpanda in [#23924](https://github.com/redpanda-data/redpanda/pull/23924)
* Adds the topic property `delete.retention.ms`, as well as the cluster property `tombstone_retention_ms`. Configuring these allow for the removal of tombstone records in compacted topics with tiered storage disabled in `redpanda`. by @WillemKauf in [#23662](https://github.com/redpanda-data/redpanda/pull/23662)
* Schema Registry: Support `normalize=true` by @BenPope in [#22519](https://github.com/redpanda-data/redpanda/pull/22519)
* Schema Registry: added support for the "verbose" query parameter on the schema compatibility checker endpoint by @pgellert in [#22877](https://github.com/redpanda-data/redpanda/pull/22877)
* Schema Registry: verbose compatibility error reporting is now supported for JSON as well by @pgellert in [#23208](https://github.com/redpanda-data/redpanda/pull/23208)
* [#17984](https://github.com/redpanda-data/redpanda/issues/17984) Adds a new broker configuration transaction_max_timeout_ms. The configuration controls the maximum allowed user set timeout for transactions. If a client requested transaction timeout exceeds this configuration, the broker will return
an error during transactional producer initialization. This guardrail prevents hanging transactions from blocking consumer progress. The default value is 15mins. by @bharathv in [#21504](https://github.com/redpanda-data/redpanda/pull/21504)
* rpk: Add `rpk registry mode` to manage the schema registry mode. by @r-vasquez in [#22675](https://github.com/redpanda-data/redpanda/pull/22675)
* rpk: supports triggering on-demand partition balancer by @daisukebe in [#22855](https://github.com/redpanda-data/redpanda/pull/22855)
* Added support for using PKCS#12 files for TLS services by @michael-redpanda in [#21313](https://github.com/redpanda-data/redpanda/pull/21313)
* Adds admin API endpoint for enterprise feature info `GET /v1/features/enterprise` by @oleiman in [#23314](https://github.com/redpanda-data/redpanda/pull/23314)
* A new metric (*cluster_features_enterprise_license_expiry_sec*) is added for easier monitoring of the enterprise license's expiry time. by @pgellert in [#23367](https://github.com/redpanda-data/redpanda/pull/23367)
* After the cluster is first formed, a trial license is automatically loaded to provide an evaluation period of enterprise features. by @pgellert in [#23893](https://github.com/redpanda-data/redpanda/pull/23893)

## Improvements
* --regex flag in `rpk topic describe` now supports internal topics. by @r-vasquez in [#23487](https://github.com/redpanda-data/redpanda/pull/23487)
* A number of optimizations to local storage compaction. by @WillemKauf in [#23380](https://github.com/redpanda-data/redpanda/pull/23380)
* Add an LRU caching layer to Rust transform SDK Schema Registry client by @oleiman in [#19859](https://github.com/redpanda-data/redpanda/pull/19859)
* Add support for differentiating tombstone records from empty-string value records in `rpk produce` and `rpk consume`.  by @WillemKauf in [#23264](https://github.com/redpanda-data/redpanda/pull/23264)
* Added support for Metadata API v8 by @michael-redpanda in [#22669](https://github.com/redpanda-data/redpanda/pull/22669)
* Added vectorized_kafka_rpc_connections_rejected_rate_limit metric which counts incoming Kafka connections rejected due to the connection rate limit (if set), analogously to the existing vectorized_kafka_rpc_connections_rejected metric which counts rejected connections due to the hitting the open connection limit. by @travisdowns in [#22803](https://github.com/redpanda-data/redpanda/pull/22803)
* Adds a shard label to some consumer group metrics. by @ballard26 in [#23339](https://github.com/redpanda-data/redpanda/pull/23339)
* Adds support for setting schema registry connection parameters in the `rpk` stanza of `redpanda.yaml`. by @andrewstucki in [#24017](https://github.com/redpanda-data/redpanda/pull/24017)
* Adds the `cloud_storage_backend::oracle` value, and helps the `s3_client` properly configure for OCI storage. by @WillemKauf in [#22902](https://github.com/redpanda-data/redpanda/pull/22902)
* Adds the ability to configure Node UUID and ID overrides at broker startup. by @oleiman in [#22972](https://github.com/redpanda-data/redpanda/pull/22972)
* Allow `rpk cluster self-test start` to run, even in a cluster with mixed versions of `redpanda` (before and after `cloudcheck` addition in `24.2.x`). by @WillemKauf in [#21370](https://github.com/redpanda-data/redpanda/pull/21370)
* Allows `DeleteRecords` requests from Kafka clients or `rpk topic trim-prefix` to be called with `truncation_offset <= start_offset` without returning an error. The request is instead treated as a no-op. by @WillemKauf in [#22905](https://github.com/redpanda-data/redpanda/pull/22905)
* Allows the self-test to be completely compatible with a mixed version cluster, in the case of a rolling upgrade. by @WillemKauf in [#22831](https://github.com/redpanda-data/redpanda/pull/22831)
* Deprecate `leader_balancer_mode` cluster config property. by @ztlpn in [#23780](https://github.com/redpanda-data/redpanda/pull/23780)
* Implements `@redpanda-data/transform-sdk-sr.SchemaFormat` for the WASM Transforms JS module by @oleiman in [#23164](https://github.com/redpanda-data/redpanda/pull/23164)
* Improve handling of boolean property values during a `CreateTopics` request by making parsing case-insensitive.  by @WillemKauf in [#23682](https://github.com/redpanda-data/redpanda/pull/23682)
* Improve handling of boolean values during a `CreateTopics` request by no longer silently ignoring an invalid value, instead throwing a configuration error. by @WillemKauf in [#23682](https://github.com/redpanda-data/redpanda/pull/23682)
* Improve handling of certain invalid topic configuration parameters that would lead to a timeout failure instead of a graceful error code during a `CreateTopics` request. by @WillemKauf in [#23682](https://github.com/redpanda-data/redpanda/pull/23682)
* Improve property configuration descriptions.  by @Deflaimun in [#23347](https://github.com/redpanda-data/redpanda/pull/23347)
* Minimizes data loss in recovery scenarios by @mmaslankaprv in [#24071](https://github.com/redpanda-data/redpanda/pull/24071)
* Reduce the memory overhead of many small segments. by @rockwotj in [#22962](https://github.com/redpanda-data/redpanda/pull/22962)
* Return core assignments from health report in `/v1/cluster/partitions` admin API output. by @ztlpn in [#22695](https://github.com/redpanda-data/redpanda/pull/22695)
* Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in [#22798](https://github.com/redpanda-data/redpanda/pull/22798)
* Schema Registry: Improve AVRO Normalization by @BenPope in [#22519](https://github.com/redpanda-data/redpanda/pull/22519)
* Schema Registry: now reports more specific error messages for Avro and Protobuf schemas when they are incompatible with earlier schemas. by @pgellert in [#22958](https://github.com/redpanda-data/redpanda/pull/22958)
* Set the default value of `topic_partitions_reserve_shard0` to zero. This means that we no longer weight shard 0 as if it has 2 more partitions than it actually has, leading to more even partition distribution in cases where the total number of partitions is close to the vCPU count. by @travisdowns in [#22841](https://github.com/redpanda-data/redpanda/pull/22841)
* The command line is now printed to the log at startup by the Redpanda process. by @travisdowns in [#22826](https://github.com/redpanda-data/redpanda/pull/22826)
* Upgrade data transforms tinygo compiler to version 0.34.0 by @rockwotj in [#23969](https://github.com/redpanda-data/redpanda/pull/23969)
* [#17682](https://github.com/redpanda-data/redpanda/issues/17682) Schema Registry: Remove spurious log entry: `No syntax specified for the proto file` by @BenPope in [#22633](https://github.com/redpanda-data/redpanda/pull/22633)
* [#21536](https://github.com/redpanda-data/redpanda/issues/21536) `rpk topic describe-storage` can be used now with internal topics. by @r-vasquez in [#22338](https://github.com/redpanda-data/redpanda/pull/22338)
* [#22333](https://github.com/redpanda-data/redpanda/issues/22333) rpk debug bundle: include the result of `uname -a` by @JFlath in [#22334](https://github.com/redpanda-data/redpanda/pull/22334)
* [#22666](https://github.com/redpanda-data/redpanda/issues/22666) Allows users to query the value of a cluster property with `rpk cluster config get` using either the original property name, or any of its aliases. Whereas before, `rpk cluster config get` using a property's aliased name would return a `Property {} not found` result. by @WillemKauf in [#22674](https://github.com/redpanda-data/redpanda/pull/22674)
* [#23038](https://github.com/redpanda-data/redpanda/issues/23038) Schema Registry: Tolerate `null` `metadata` and `ruleSet` when registering or querying a schema by @BenPope in [#23044](https://github.com/redpanda-data/redpanda/pull/23044)
* `rpk registry schema get` will now fail if is incorrectly invoked using the `--print-schema` and `--format` flag. by @r-vasquez in [#23886](https://github.com/redpanda-data/redpanda/pull/23886)
* fixed large allocation in Raft implementation  by @mmaslankaprv in [#24002](https://github.com/redpanda-data/redpanda/pull/24002)
* fixed spurious warning log entries that may be altering users by @mmaslankaprv in [#22752](https://github.com/redpanda-data/redpanda/pull/22752)
* optimized memory usage with large core count nodes by @mmaslankaprv in [#22977](https://github.com/redpanda-data/redpanda/pull/22977)
* rpk cluster license info now shows if your loaded license is expired by @r-vasquez in [#23636](https://github.com/redpanda-data/redpanda/pull/23636)
* rpk cluster license info: now includes more information about your license status and possible violations. by @r-vasquez in [#23744](https://github.com/redpanda-data/redpanda/pull/23744)
* rpk connect will be downloaded on the first run. by @r-vasquez in [#23117](https://github.com/redpanda-data/redpanda/pull/23117)
* rpk will now warn you when you are running enterprise features without a license. by @r-vasquez in [#23472](https://github.com/redpanda-data/redpanda/pull/23472)
* rpk: `redpanda admin brokers list` exposes Host/Port/Rack/UUID additionally by @daisukebe in [#23595](https://github.com/redpanda-data/redpanda/pull/23595)
* rpk: add `--format` JSON/YAML support to `rpk topic list`. by @gene-redpanda in [#23806](https://github.com/redpanda-data/redpanda/pull/23806)
* rpk: introduce `rpk connect install`, `rpk connect upgrade`, and `rpk connect uninstall`, manual ways to manage `rpk connect` versioning.  by @r-vasquez in [#23117](https://github.com/redpanda-data/redpanda/pull/23117)
* tbd by @nvartolomei in [#23381](https://github.com/redpanda-data/redpanda/pull/23381)
* with this improvement the force reconfigured partitions will be backfilled from Tiered Storage even if they lost all local data by @mmaslankaprv in [#22908](https://github.com/redpanda-data/redpanda/pull/22908)

## Bug Fixes
* Fixed an issue where creating a topic with a huge number of partitions could lead to a crash. by @IoannisRP in [#24232](https://github.com/redpanda-data/redpanda/pull/24232)
## Improvements
* Schema Registry: Add Some metrics for resource usage taken by in-memory schemas  by @BenPope in [#24270](https://github.com/redpanda-data/redpanda/pull/24270)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.11...v24.2.12

## Bug Fixes
* Construct audit metrics probe during service initialization to prevent null pointer access. by @michael-redpanda in [#24127](https://github.com/redpanda-data/redpanda/pull/24127)
* Fixed an issue where creating a topic with a huge number of partitions could lead to a crash. by @IoannisRP in [#24232](https://github.com/redpanda-data/redpanda/pull/24232)
* Fixes a bug in which upload candidates made from segments with missing batches would trigger metadata related errors in the `ntp_archiver_service`, due to assigned start offsets being lower than they should be. by @WillemKauf in [#24106](https://github.com/redpanda-data/redpanda/pull/24106)
* [#24076](https://github.com/redpanda-data/redpanda/issues/24076) Fixes a rare bug during remote partition manifest downloads where broken pipe exceptions weren't retried in an edge case. by @pgellert in [#24080](https://github.com/redpanda-data/redpanda/pull/24080)
* [#24144](https://github.com/redpanda-data/redpanda/issues/24144) This fixes a bug in the audit client where if the cluster config value `kafka_batch_max_bytes` was greater than `audit_client_max_buffer_size`, the audit client ends up not producing any messages and becomes stuck filling up the audit log buffers. by @pgellert in [#24148](https://github.com/redpanda-data/redpanda/pull/24148)
* [#24207](https://github.com/redpanda-data/redpanda/issues/24207) Redpanda neglected to include ECDSA based ciphers in the cipher strings used for TLSv1.2 and below.  This caused TLS connections that used ECDSA based certificates to fail cipher negotiation when using TLSv1.2 and below.  ECDSA ciphers are now in the list of supported ciphers. by @michael-redpanda in [#24209](https://github.com/redpanda-data/redpanda/pull/24209)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.10...v24.2.11

## Features
* [#23454](https://github.com/redpanda-data/redpanda/issues/23454) A new metric (*cluster_features_enterprise_license_expiry_sec*) is added for easier monitoring of the enterprise license's expiry time. by @pgellert in [#23467](https://github.com/redpanda-data/redpanda/pull/23467)
* [#23760](https://github.com/redpanda-data/redpanda/issues/23760) Adds admin API endpoint for enterprise feature info `GET /v1/features/enterprise` by @oleiman in [#23761](https://github.com/redpanda-data/redpanda/pull/23761)
## Bug Fixes
* Construct audit metrics probe during service initialization to prevent null pointer access. by @michael-redpanda in [#24128](https://github.com/redpanda-data/redpanda/pull/24128)
* Fixes a bug in which upload candidates made from segments with missing batches would trigger metadata related errors in the `ntp_archiver_service`, due to assigned start offsets being lower than they should be. by @WillemKauf in [#24105](https://github.com/redpanda-data/redpanda/pull/24105)
* Fixes a bug where only a group static member's protocols would be updated on rejoin, even if more properties had been passed to the rejoin command by @IoannisRP in [#23733](https://github.com/redpanda-data/redpanda/pull/23733)
* [#23863](https://github.com/redpanda-data/redpanda/issues/23863) Fixes a bug where audit log manager would retry a bad request forever, causing buffers to fill up, blocking audit log appends and preventing authZ. by @oleiman in [#23868](https://github.com/redpanda-data/redpanda/pull/23868)
* [#23930](https://github.com/redpanda-data/redpanda/issues/23930) Ignore heartbeat requests/replies to/from unexpected node ids. by @ztlpn in [#23934](https://github.com/redpanda-data/redpanda/pull/23934)
* [#24056](https://github.com/redpanda-data/redpanda/issues/24056) Cleanup tiered storage temporary cache file if exceptions are thrown during download. by @nvartolomei in [#24064](https://github.com/redpanda-data/redpanda/pull/24064)
* [#24077](https://github.com/redpanda-data/redpanda/issues/24077) Fixes a rare bug during remote partition manifest downloads where broken pipe exceptions weren't retried in an edge case. by @pgellert in [#24079](https://github.com/redpanda-data/redpanda/pull/24079)
* [#24143](https://github.com/redpanda-data/redpanda/issues/24143) This fixes a bug in the audit client where if the cluster config value `kafka_batch_max_bytes` was greater than `audit_client_max_buffer_size`, the audit client ends up not producing any messages and becomes stuck filling up the audit log buffers. by @pgellert in [#24149](https://github.com/redpanda-data/redpanda/pull/24149)

## Improvements
* --regex flag in `rpk topic describe` now supports internal topics. by @r-vasquez in [#23605](https://github.com/redpanda-data/redpanda/pull/23605)
* Adds a shard label to some consumer group metrics. by @ballard26 in [#23626](https://github.com/redpanda-data/redpanda/pull/23626)
* [#23404](https://github.com/redpanda-data/redpanda/issues/23404) Adds the ability to configure Node UUID and ID overrides at broker startup. by @oleiman in [#23412](https://github.com/redpanda-data/redpanda/pull/23412)
* fixed large allocation in Raft implementation  by @mmaslankaprv in [#24009](https://github.com/redpanda-data/redpanda/pull/24009)
* rpk: `redpanda admin brokers list` exposes Host/Port/Rack/UUID additionally by @daisukebe in [#23688](https://github.com/redpanda-data/redpanda/pull/23688)
* PR [#23414](https://github.com/redpanda-data/redpanda/pull/23414) [v24.1.x] `archival`: use `log_level_for_error()` for failed reupload candidates by @WillemKauf
* PR [#23450](https://github.com/redpanda-data/redpanda/pull/23450) [v24.1.x] `storage`: catch `ss::gate_closed_exception` in `log_manager` (manual backport) by @WillemKauf
* PR [#23501](https://github.com/redpanda-data/redpanda/pull/23501) [v24.1.x] tests/failure_injector: undo the failures on exit by @bashtanov
* PR [#23506](https://github.com/redpanda-data/redpanda/pull/23506) [v24.1.x] cluster_recovery_backend_test: only reset relevant config by @andrwng
* PR [#23515](https://github.com/redpanda-data/redpanda/pull/23515) [v24.1.x] rptest: produce more data in FullDiskReclaimTest to trigger gc conditions by @nvartolomei
* PR [#23527](https://github.com/redpanda-data/redpanda/pull/23527) [v24.1.x] CORE-7689 dt/rp_installer: ensure cache directory exists by @pgellert
* PR [#23537](https://github.com/redpanda-data/redpanda/pull/23537) [v24.1.x] rptest: do not expect cached segment readers at the end of the test by @nvartolomei
* PR [#23555](https://github.com/redpanda-data/redpanda/pull/23555) [v24.1.x] ssx: exit early from sleep_abortable if already aborted by @nvartolomei
* PR [#23608](https://github.com/redpanda-data/redpanda/pull/23608) [v24.1.x] tests: fix rpk generate test by @r-vasquez
* PR [#23646](https://github.com/redpanda-data/redpanda/pull/23646) [24.1.x] tests: bump ducktape to latest of 0.11.x by @ivotron
* PR [#23674](https://github.com/redpanda-data/redpanda/pull/23674) [v24.1.x] tests: test legacy dashboard in rpk generate by @r-vasquez
* PR [#23708](https://github.com/redpanda-data/redpanda/pull/23708) [v24.1.x] gha: rm use of rp_storage_tool_uploader by @andrewhsu
* PR [#23746](https://github.com/redpanda-data/redpanda/pull/23746) [v24.1.x] Keep producer inflight requests queue bounded by @mmaslankaprv
* PR [#23766](https://github.com/redpanda-data/redpanda/pull/23766) [v24.1.x] gha: fix pip install on python actions by @ivotron
* PR [#23818](https://github.com/redpanda-data/redpanda/pull/23818) [v24.1.x] rpk: debug bundle collecting broker UUIDs by @daisukebe
* PR [#23831](https://github.com/redpanda-data/redpanda/pull/23831) [v24.1.x] rpk: introduce license warnings messages by @r-vasquez
* PR [#23849](https://github.com/redpanda-data/redpanda/pull/23849) [v24.1.x] [CORE-7957] tests: wait for license information comparisons by @r-vasquez
* PR [#23861](https://github.com/redpanda-data/redpanda/pull/23861) [v24.1.x] kafka: oversized alloc in list_offsets_topic by @IoannisRP
* PR [#23866](https://github.com/redpanda-data/redpanda/pull/23866) [v24.1.x] [CORE-7719] Add has_valid_license & has_enterprise_features to phone-home metrics by @oleiman
* PR [#23923](https://github.com/redpanda-data/redpanda/pull/23923) [v24.1.x] rpk: fill schema registry information in cloud profiles by @r-vasquez
* PR [#23965](https://github.com/redpanda-data/redpanda/pull/23965) [v24.1.x] [DEVEX-36] rpk: change expiry check for free_trial  by @r-vasquez
* PR [#23988](https://github.com/redpanda-data/redpanda/pull/23988) [v24.1.x] rpk: fix printing new lines by @r-vasquez
* PR [#24015](https://github.com/redpanda-data/redpanda/pull/24015) [v24.1.x] metrics: Add list of enterprise features to call-home POST by @oleiman
* PR [#24025](https://github.com/redpanda-data/redpanda/pull/24025) [v24.1.x] transform-sdk/go/tests: remove -quiet flag by @rockwotj
* PR [#24044](https://github.com/redpanda-data/redpanda/pull/24044) [v24.1.x] [CORE-8141] Add host information to metrics report by @michael-redpanda
* PR [#24051](https://github.com/redpanda-data/redpanda/pull/24051) [v24.1.x] storage: housekeeping metrics by @nvartolomei
* PR [#24062](https://github.com/redpanda-data/redpanda/pull/24062) [v24.1.x] [CORE-1478] rptest: fix retention value in `archive_retention_test` by @WillemKauf
* PR [#24070](https://github.com/redpanda-data/redpanda/pull/24070) [v24.1.x] rptest: reduce cache eviction throttling for space leak test by @nvartolomei
* PR [#24089](https://github.com/redpanda-data/redpanda/pull/24089) [v24.1.x] storage: remove assertion on `is_cloud_retention_active` by @ballard26

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.1.17...v24.1.18

## Bug Fixes
* [#24057](https://github.com/redpanda-data/redpanda/issues/24057) Cleanup tiered storage temporary cache file if exceptions are thrown during download. by @nvartolomei in [#24063](https://github.com/redpanda-data/redpanda/pull/24063)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.9...v24.2.10

## Improvements
* Adds a shard label to some consumer group metrics. by @ballard26 in [#23627](https://github.com/redpanda-data/redpanda/pull/23627)
* fixed large allocation in Raft implementation  by @mmaslankaprv in [#24010](https://github.com/redpanda-data/redpanda/pull/24010)
* PR [#24012](https://github.com/redpanda-data/redpanda/pull/24012) [v24.2.x] metrics: Add list of enterprise features to call-home POST by @oleiman
* PR [#24027](https://github.com/redpanda-data/redpanda/pull/24027) [v24.2.x] transform-sdk/go/tests: remove -quiet flag by @rockwotj
* PR [#24045](https://github.com/redpanda-data/redpanda/pull/24045) [v24.2.x] [CORE-8141] Add host information to metrics report by @michael-redpanda
* PR [#24048](https://github.com/redpanda-data/redpanda/pull/24048) [v24.2.x] storage: housekeeping metrics by @nvartolomei

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.8...v24.2.9

## Bug Fixes
* [#23846](https://github.com/redpanda-data/redpanda/issues/23846) fixes the term of recovered topic moving backward by @mmaslankaprv in [#23879](https://github.com/redpanda-data/redpanda/pull/23879)
* [#23864](https://github.com/redpanda-data/redpanda/issues/23864) Fixes a bug where audit log manager would retry a bad request forever, causing buffers to fill up, blocking audit log appends and preventing authZ. by @oleiman in [#23867](https://github.com/redpanda-data/redpanda/pull/23867)
* [#23928](https://github.com/redpanda-data/redpanda/issues/23928) Ignore heartbeat requests/replies to/from unexpected node ids. by @ztlpn in [#23933](https://github.com/redpanda-data/redpanda/pull/23933)
## Improvements
* `rpk registry schema get` will now fail if is incorrectly invoked using the `--print-schema` and `--format` flag. by @r-vasquez in [#23901](https://github.com/redpanda-data/redpanda/pull/23901)
* `rpk cluster license info` now includes more details about the license, including:
  * Whether your loaded license is expired. By @r-vasquez in https://github.com/redpanda-data/redpanda/pull/23636
  * License status and possible violations. By @r-vasquez in https://github.com/redpanda-data/redpanda/pull/23744

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.7...v24.2.8

## Bug Fixes
* Fixes a bug where only a group static member's protocols would be updated on rejoin, even if more properties had been passed to the rejoin command by @IoannisRP in https://github.com/redpanda-data/redpanda/pull/23729
* Fixes possible high CPU utilisation caused by transactional coordinator leadership changes by @mmaslankaprv in https://github.com/redpanda-data/redpanda/pull/23617

## Improvements
* rpk: `--regex` flag in rpk topic describe now supports internal topics. by @r-vasquez in https://github.com/redpanda-data/redpanda/pull/23604
* rpk: cluster license info now shows if your loaded license is expired by @r-vasquez in https://github.com/redpanda-data/redpanda/pull/23665
* rpk: redpanda admin brokers list exposes Host/Port/Rack/UUID additionally by @daisukebe in https://github.com/redpanda-data/redpanda/pull/23686

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v24.2.6...v24.2.7

## Features
* [#23455](https://github.com/redpanda-data/redpanda/issues/23455) A new metric (*cluster_features_enterprise_license_expiry_sec*) is added for easier monitoring of the enterprise license's expiry time. by @pgellert in [#23466](https://github.com/redpanda-data/redpanda/pull/23466)
* rpk: supports triggering on-demand partition balancer by @daisukebe in [#23216](https://github.com/redpanda-data/redpanda/pull/23216)
## Bug Fixes
* Fixes a bug where URL path params to `PandaProxy` and `SchemaRegistry` would be URL decoded twice, destroying escape sequences present in the original request. by @oleiman in [#23547](https://github.com/redpanda-data/redpanda/pull/23547)

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.5...v24.2.6

## Features
* [#22957](https://github.com/redpanda-data/redpanda/issues/22957) Schema Registry: added support for the "verbose" query parameter on the schema compatibility checker endpoint by @pgellert in [#23028](https://github.com/redpanda-data/redpanda/pull/23028)
## Bug Fixes
* Fixed a segfault that will occur if mTLS is in use and the auditing client is not configured correctly by @michael-redpanda in [#23289](https://github.com/redpanda-data/redpanda/pull/23289)
* Fixed possible oversized alloc in the DescribeLogsDir API handler by @michael-redpanda in [#23292](https://github.com/redpanda-data/redpanda/pull/23292)
* Fixes a bug that would cause retention to be applied to compact-only tiered storage topics. by @andrwng in [#23361](https://github.com/redpanda-data/redpanda/pull/23361)
* Fixes a regression in schema registry where concurrent requests could lead to memory faults by @oleiman in [#23137](https://github.com/redpanda-data/redpanda/pull/23137)
* Return `NOT_COORDINATOR` immediately in group transactional operations if the broker is aware that it is not the group coordinator. by @ztlpn in [#23187](https://github.com/redpanda-data/redpanda/pull/23187)
* [#23108](https://github.com/redpanda-data/redpanda/issues/23108) Schema Registry: fixes a bug in the protobuf compatibility checks of nested fields where nested paths were incorrectly handled. by @pgellert in [#23170](https://github.com/redpanda-data/redpanda/pull/23170)
* [#23222](https://github.com/redpanda-data/redpanda/issues/23222) Fixes a bug that resulted in persisting an invalid `topic_properties` configuration in the `topic_table`. by @WillemKauf in [#23227](https://github.com/redpanda-data/redpanda/pull/23227)
## Improvements
* [#23079](https://github.com/redpanda-data/redpanda/issues/23079) Schema Registry: Tolerate `null` `metadata` and `ruleSet` when registering or querying a schema by @BenPope in [#23081](https://github.com/redpanda-data/redpanda/pull/23081)
* [#23108](https://github.com/redpanda-data/redpanda/issues/23108) Schema Registry: now reports more specific error messages for Avro and Protobuf schemas when they are incompatible with earlier schemas. by @pgellert in [#23170](https://github.com/redpanda-data/redpanda/pull/23170)
* PR [#23238](https://github.com/redpanda-data/redpanda/pull/23238) [v23.3.x] `config`: mark `cloud_storage_enabled()` as `needs_restart::yes` by @WillemKauf

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v23.3.20...v23.3.21

## Features
* Adds admin API endpoint for enterprise feature info `GET /v1/features/enterprise` by @oleiman in [#23368](https://github.com/redpanda-data/redpanda/pull/23368)
* New REST API for mounting/unmounting topics by @bashtanov in [#23209](https://github.com/redpanda-data/redpanda/pull/23209)
* Schema Registry: verbose compatibility error reporting is now supported for JSON as well by @pgellert in [#23299](https://github.com/redpanda-data/redpanda/pull/23299)
## Bug Fixes
* Fixed a segfault that will occur if mTLS is in use and the auditing client is not configured correctly by @michael-redpanda in [#23288](https://github.com/redpanda-data/redpanda/pull/23288)
* Fixed possible oversized alloc in the DescribeLogsDir API handler by @michael-redpanda in [#23291](https://github.com/redpanda-data/redpanda/pull/23291)
* Fixes a bug that resulted in persisting an invalid `topic_properties` configuration in the `topic_table`. by @WillemKauf in [#23223](https://github.com/redpanda-data/redpanda/pull/23223)
* Fixes a bug that would cause retention to be applied to compact-only tiered storage topics. by @andrwng in [#23359](https://github.com/redpanda-data/redpanda/pull/23359)
* Return `NOT_COORDINATOR` immediately in group transactional operations if the broker is aware that it is not the group coordinator. by @ztlpn in [#23189](https://github.com/redpanda-data/redpanda/pull/23189)
* [#23316](https://github.com/redpanda-data/redpanda/issues/23316) Fixes crash when consuming from topic partitions with negative indexes by @bharathv in [#23317](https://github.com/redpanda-data/redpanda/pull/23317)
## Improvements
* `rpk connect` now will be downloaded on the first run. by @r-vasquez in [#23351](https://github.com/redpanda-data/redpanda/pull/23351)
* rpk: introduce rpk connect install, rpk connect upgrade, and rpk connect uninstall, manual ways to manage rpk connect versioning. by @r-vasquez in [#23351](https://github.com/redpanda-data/redpanda/pull/23351)
* PR [#23127](https://github.com/redpanda-data/redpanda/pull/23127) [v24.2.x] c/members_table: include members table version in the snapshot by @mmaslankaprv
* PR [#23155](https://github.com/redpanda-data/redpanda/pull/23155) [v24.2.x] c/topic_config: Explicitly initialize non-class fields by @oleiman
* PR [#23175](https://github.com/redpanda-data/redpanda/pull/23175) [v24.2.x] c/tm_stm: fix log format strings by @ztlpn
* PR [#23236](https://github.com/redpanda-data/redpanda/pull/23236) [v24.2.x] `config`: mark `cloud_storage_enabled()` as `needs_restart::yes` by @WillemKauf
* PR [#23241](https://github.com/redpanda-data/redpanda/pull/23241) [v24.2.x] Migrations storage improvements by @bashtanov
* PR [#23242](https://github.com/redpanda-data/redpanda/pull/23242) [v24.2.x] cst/inv: Minor fixes for issues found during scrubber integration by @abhijat
* PR [#23325](https://github.com/redpanda-data/redpanda/pull/23325) [v24.2.x] CORE-7268 fix migrated resources raft snapshot application by @bashtanov
* PR [#23357](https://github.com/redpanda-data/redpanda/pull/23357) [v24.2.x] `admin`: add `cloud_storage_cache_size` check to `config_multi_property_validation()` by @WillemKauf


**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.2.4...v24.2.5

## Features
* rpk: supports triggering on-demand partition balancer by @daisukebe in [#23395](https://github.com/redpanda-data/redpanda/pull/23395)
## Bug Fixes
* Fixed a segfault that will occur if mTLS is in use and the auditing client is not configured correctly by @michael-redpanda in [#23290](https://github.com/redpanda-data/redpanda/pull/23290)
* Fixed possible oversized alloc in the DescribeLogsDir API handler by @michael-redpanda in [#23293](https://github.com/redpanda-data/redpanda/pull/23293)
* Fixes a bug that would cause retention to be applied to compact-only tiered storage topics. by @andrwng in [#23360](https://github.com/redpanda-data/redpanda/pull/23360)
* Return `NOT_COORDINATOR` immediately in group transactional operations if the broker is aware that it is not the group coordinator. by @ztlpn in [#23188](https://github.com/redpanda-data/redpanda/pull/23188)
* [#23221](https://github.com/redpanda-data/redpanda/issues/23221) Fixes a bug that resulted in persisting an invalid `topic_properties` configuration in the `topic_table`. by @WillemKauf in [#23226](https://github.com/redpanda-data/redpanda/pull/23226)
## Improvements
* `rpk connect` now will be downloaded on the first run. by @r-vasquez in [#23352](https://github.com/redpanda-data/redpanda/pull/23352)
* rpk: introduce rpk connect install, rpk connect upgrade, and rpk connect uninstall, manual ways to manage rpk connect versioning. by @r-vasquez in [#23352](https://github.com/redpanda-data/redpanda/pull/23352)
* PR [#23128](https://github.com/redpanda-data/redpanda/pull/23128) [v24.1.x] c/members_table: include members table version in the snapshot by @mmaslankaprv
* PR [#23235](https://github.com/redpanda-data/redpanda/pull/23235) [v24.1.x] CORE-7049 dt/rbac: wait for healthy cluster after restart by @pgellert
* PR [#23237](https://github.com/redpanda-data/redpanda/pull/23237) [v24.1.x] `config`: mark `cloud_storage_enabled()` as `needs_restart::yes` by @WillemKauf
* PR [#23362](https://github.com/redpanda-data/redpanda/pull/23362) [v24.1.x] `admin`: add `cloud_storage_cache_size` check to `config_multi_property_validation()` (manual backport) by @WillemKauf
* PR [#23377](https://github.com/redpanda-data/redpanda/pull/23377) [v24.1.x] Remove some warnings and ignored errors from ducktape log by @ivotron

**Full Changelog**: https://github.com/redpanda-data/redpanda/compare/v24.1.16...v24.1.17