release-5779

What's Changed

• storcon: handle entire cluster going unavailable correctly by @VladLazar in #8060
• CI(check-build-tools-image): change build-tools image persistent tag by @bayandin in #8059
• Fix log rates by @MMeent in #8035
• [v2] Include openssl and ICU statically linked by @problame in #8074
• pageserver: eliminate CalculateSyntheticSizeError::LsnNotFound (test_metric_collection flake) by @jcsp in #8065
• safekeeper: streaming pull_timeline by @arssher in #7944
• Add support to specifying storage account in AzureConfig by @arpad-m in #8090
• build(deps): bump ws from 8.16.0 to 8.17.1 in /test_runner/pg_clients/typescript/serverless-driver by @dependabot in #8087
• build(deps): bump urllib3 from 1.26.18 to 1.26.19 by @dependabot in #8086
• Update document on Postgres core changes by @hlinnaka in #7416
• feat(pageserver): use leases to temporarily block gc by @yliang412 in #8084
• Remove dead code from walproposer_pg.c by @arssher in #8092
• feat(pageserver): add iterator API for btree reader by @skyzh in #8083
• Copy release images to prod ECR by @SergeyMelnikov in #8101
• storcon: add drain and fill background operations for graceful cluster restarts by @VladLazar in #8014
• test: fix tenant duplication utility generation numbers by @VladLazar in #8096
• neon_local: background_process: launch all processes in repo dir (or datadir) by @problame in #8058
• tests: make Endpoint.stop() thread safe (occasional flakes in test_multi_attach) by @jcsp in #8110
• proxy: reduce some per-task memory usage by @conradludgate in #8095
• allow storage_controller error during pagebench by @Bodobolero in #8109
• Make sure we can handle temporarily offline PS when we first connect by @MMeent in #8094
• tests: use semaphore instead of lock for Endpoint.running by @jcsp in #8112
• CI(test-images): add dockerhub auth by @bayandin in #8115
• CI: miscellaneous cleanups by @bayandin in #8073
• remove materialized page cache by @problame in #8105
• fix: preinitialize pageserver_basebackup_query_seconds metric by @problame in #8121
• storcon: use attached shard counts for initial shard placement by @VladLazar in #8061
• Add tracing for shared locks in id_lock_map by @jbajic in #7618
• storcon: avoid promoting too many shards of the same tenant by @VladLazar in #8099
• pageserver: remove unused load/ignore APIs by @jcsp in #8122
• Allow longer timeout for starting pageserver, safe keeper and storage controller in test cases to make test cases less flaky by @Bodobolero in #8079
• Add eviction_state to control file by @petuhovskiy in #8125
• pageserver: always detach before deleting by @jcsp in #8082
• storcon: implement endpoints for cancellation of drain and fill operations by @VladLazar in #8029
• tests: allow-list neon_local endpoint errors from storage controller by @jcsp in #8123
• Remove kq_imcx extension from public compute image by @lubennikovaav in #5201
• storcon: improve drain and fill shard placement by @VladLazar in #8119
• Use serde for RemoteStorageConfig parsing by @arpad-m in #8126
• Storage & Compute release 2024-06-24 by @vipvap in #8138

Full Changelog: release-5733...release-5779

release-5733

What's Changed

• test(pageserver): quantify compaction outcome by @skyzh in #7867
• Revert "Include openssl and ICU statically linked" by @problame in #8003
• Simplify scanning compute logs in tests by @hlinnaka in #7997
• fix: allow layer flushes more often by @koivunej in #7927
• docs: highlight neon env comes with an initial timeline by @skyzh in #7995
• refactor: Timeline layer flushing by @koivunej in #7993
• Add testing for extensions by @a-masterov in #7818
• fix: stop storing TimelineMetadata in index_part.json as bytes by @koivunej in #7699
• test: fix duplicated harness name by @koivunej in #8010
• feat(pageserver): initial code sketch & test case for combined gc+compaction at gc_horizon by @skyzh in #7948
• storcon: track number of attached shards for each node by @VladLazar in #8011
• storcon_cli: add 'drain' command by @VladLazar in #8007
• Copy editor config for the neon extension from PostgreSQL by @hlinnaka in #8009
• Rename S3 scrubber to storage scrubber by @arpad-m in #8013
• Add On-demand WAL Download to logicalfuncs by @save-buffer in #7960
• Another attempt at making test_vm_bits less flaky by @hlinnaka in #7989
• Update default Postgres version in docker-compose.yml by @hlinnaka in #8019
• Update documentation on running locally with Docker by @hlinnaka in #8020
• Resolve the problem the docker compose caused by the extensions tests by @a-masterov in #8024
• storcon_cli: do not drain to undesirable nodes by @VladLazar in #8027
• Reactivate page bench test in CI after ignoring CopyFail error in pageserver by @Bodobolero in #8023
• Add the image version to the neon-test-extensions image by @a-masterov in #8032
• test(pageserver): add test keyspace into collect_keyspace by @skyzh in #8016
• Fix query error in vm-image-spec.yaml by @MMeent in #8028
• Fix on-demand SLRU download on standby starting at WAL segment boundary by @hlinnaka in #8031
• Proxy process updated errors by @khanova in #8026
• test(pageserver): add test wal record for unit testing by @skyzh in #8015
• fix: vectored get returns incorrect result on inexact materialized page cache hit by @problame in #8050
• Set application_name for internal connections to computes by @tristan957 in #8047
• extensions: pgvector-0.7.2 by @jamesbroadhead in #8037
• pageserver: refine shutdown handling in secondary download by @jcsp in #8052
• Fix test_replica_query_race flakiness by @hlinnaka in #8038
• CI: Update outdated GitHub Actions by @bayandin in #8042
• storage controller: always wait for tenant detach before delete by @jcsp in #8049
• pageserver: improved synthetic size & find_gc_cutoff error handling by @jcsp in #8051
• chore: update to rust 1.79 by @conradludgate in #8048
• Fix test_segment_init_failure. by @arssher in #8056
• CI: downgrade docker/setup-buildx-action by @bayandin in #8062
• chore(pageserver): vectored get target_keyspace directly accums by @skyzh in #8055
• add halfvec indexing and queries to periodic pgvector performance tests by @Bodobolero in #8057
• Install rust binaries before running rust tests. by @arssher in #8054
• Release 2024-06-17 by @vipvap in #8069

New Contributors

• @jamesbroadhead made their first contribution in #8037

Full Changelog: release-5691...release-5733

release-5691

What's Changed

• test: allow no vectored gets happening by @koivunej in #7939
• feat(pageserver): collect aux file tombstones by @skyzh in #7900
• Move keyspace utils to inherent impls by @arpad-m in #7929
• s3_scrubber: add pageserver-physical-gc by @jcsp in #7925
• Store logical replication origin in KV storage by @knizhnik in #7099
• storcon_cli: add 'drop' and eviction interval utilities by @jcsp in #7938
• pageserver: don't squash all image layer generation errors into anyhow::Error by @jcsp in #7943
• [proxy] Fix dynamic rate limiter by @khanova in #7950
• Set page LSN when reconstructing VM in page server by @knizhnik in #7935
• test: no missed wakeups, cancellation and timeout flow to downloads by @koivunej in #7863
• build(Dockerfile.compute-node): do not log tar contents by @koivunej in #7953
• feat(rtc): maintain dirty and uploaded IndexPart by @koivunej in #7833
• pageserver: include heatmap in tenant deletion by @jcsp in #7928
• fix(pageserver): duplicate layers can cause corruption by @problame in #7799
• feat: support changing IndexPart::metadata_bytes to json in future release by @koivunej in #7693
• feat(pageserver): compute aux file size on initial logical size calculation by @skyzh in #7958
• upgrade pgvector from 0.7.0 to 0.7.1 by @Bodobolero in #7954
• chore!: remove walredo_process_kind config option & kind type by @problame in #7756
• pageserver/controller: enable tenant deletion without attachment by @jcsp in #7957
• chore(walredo): avoid duplicate tenant_id and shard_slug fields by @problame in #7977
• refactor(pageserver): designated api error type for cancelled request by @yliang412 in #7949
• fix: do TimelineMetrics::shutdown only once by @koivunej in #7983
• Perform retries on azure bulk deletion by @arpad-m in #7964
• fix(pageserver): more flexible layer size test by @skyzh in #7945
• fix(pageserver): skip metadata compaction if LSN is not accumulated enough by @skyzh in #7962
• raise the warning for oversized L0 to 2*target by @koivunej in #7985
• Remove random sleep in partial backup by @petuhovskiy in #7982
• Downgrade No response from reading prefetch entry WARNING to LOG by @lubennikovaav in #7988
• test_local_only_layers_after_crash: various fixes by @koivunej in #7986
• CI: copy run-* labels from external contributors' PRs by @a-masterov in #7915
• Move SQL migrations out of line by @tristan957 in #7991
• Include openssl and ICU statically linked by @rahulinux in #7956
• Release 2024-06-10 by @vipvap in #7999

New Contributors

• @yliang412 made their first contribution in #7949

Full Changelog: release-5657...release-5691

release-5657

What's Changed

• Fix connect to PS on MacOS/X by @knizhnik in #7885
• Add safekeeper test truncating WAL. by @arssher in #7891
• Performance test for pgvector HNSW index build and queries by @Bodobolero in #7873
• clarify how to load the dbpedia vector embeddings into a postgres database by @Bodobolero in #7894
• pageserver: fix a warning on secondary mode downloads after evictions by @jcsp in #7877
• Drop postgres-native-tls in favour of tokio-postgres-rustls by @arpad-m in #7883
• proxy: reintroduce dynamic limiter for compute lock by @conradludgate in #7737
• Do not produce error if gin page is not restored in redo by @knizhnik in #7876
• CI: switch ubuntu-latest with ubuntu-22.04 (#7256) by @a-masterov in #7901
• proxy fix wake compute rate limit by @conradludgate in #7902
• proxy: upload postgres connection options as json in the parquet upload by @conradludgate in #7903
• proxy: lazily parse startup pg params by @conradludgate in #7905
• refacter : VirtualFile::open uses AsRef by @SeinoYuki in #7908
• fix(pageserver): ensure to_i128 works for metadata keys by @skyzh in #7895
• perf(pageserver): postpone vectored get fringe keyspace construction by @skyzh in #7904
• chore(pageserver): warn on delete non-existing file by @skyzh in #7847
• Update tokio-epoll-uring for linux-raw-sys by @arpad-m in #7918
• pageserver: quieten some shutdown logs around logical size and flush by @jcsp in #7907
• neon_walreader: check after local read that the segment still exists. by @arssher in #7898
• Add test checking term change during pull_timeline by @arssher in #7893
• safekeeper: rename epoch to last_log_term. by @arssher in #7892
• pageserver: avoid spurious "bad state" logs/errors during shutdown by @jcsp in #7912
• Add FullAccessTimeline guard in safekeepers by @petuhovskiy in #7887
• test(pageserver): add test interface to create artificial layers by @skyzh in #7899
• pageserver: remove AncestorStopping error variants by @jcsp in #7916
• pagectl: key command for dumping what we know about the key by @koivunej in #7890
• pageserver: add GcError type by @jcsp in #7917
• Run gc on too many partial backup segments by @petuhovskiy in #7700
• Fix span for WAL removal task by @petuhovskiy in #7930
• Add metrics for Azure blob storage by @arpad-m in #7933
• Release 2024-06-03 by @vipvap in #7936

New Contributors

• @a-masterov made their first contribution in #7901
• @SeinoYuki made their first contribution in #7908

Full Changelog: release-5625...release-5657

release-5625

What's Changed

• Compute release 2024-05-31 by @bayandin in #7920

Full Changelog: release-5623...release-5625

release-5623

What's Changed

• refactor(test): duplication with fullbackup, tar content hashing by @koivunej in #7828
• CI(report-benchmarks-failures): fix condition by @bayandin in #7820
• openapi: document timeline ancestor detach by @koivunej in #7650
• Fix some action typos by @tristan957 in #7834
• CI(promote-images): simplify & fix the job by @bayandin in #7826
• CI: build multi-arch images by @bayandin in #7696
• safekeeper: use CancellationToken instead of watch channel by @jcsp in #7836
• fix(Layer): carry gate until eviction is complete by @koivunej in #7838
• Fix and add test case for issue 7791 by @hlinnaka in #7795
• Make 'neon.protocol_version = 2' the default, take two by @hlinnaka in #7819
• feat(pageserver): auto-detect previous aux file policy by @skyzh in #7841
• proxy password threadpool by @conradludgate in #7806
• chore(pageserver): use kebab case for aux file flag by @skyzh in #7840
• tests: refine test_secondary_background_downloads by @jcsp in #7829
• tests: add a couple of allow lists for shutdown cases by @jcsp in #7844
• chore(pageserver): add force aux file policy switch handler by @skyzh in #7842
• chore(pageserver): use kebab case for compaction algorithms by @skyzh in #7845
• Bump vm-builder v0.28.1 -> v0.29.3 by @Omrigan in #7849
• pageserver: fix API-driven secondary downloads possibly colliding with background downloads by @jcsp in #7848
• test_attach_tenant_config: add allowed error by @koivunej in #7839
• Skip unnecessary shared state updates in safekeepers by @petuhovskiy in #7851
• [proxy] Do not fail after parquet upload error by @khanova in #7858
• tests: fix an allow list entry by @jcsp in #7856
• remove march=native from pgvector Makefile's OPTFLAGS by @Bodobolero in #7854
• test_import_from_pageserver_small: try to make less flaky by @koivunej in #7843
• Warn if a blob in an image is larger than 256 MiB by @arpad-m in #7852
• fix(pageserver): make wal connstr a connstr by @skyzh in #7846
• Use Timeline::create_image_layer_for_rel_blocks in tiered compaction by @arpad-m in #7850
• chore(pageserver): reduce logging related to image layers by @skyzh in #7864
• refactor(rtc): remove the duplicate IndexLayerMetadata by @koivunej in #7860
• Remove apostrophe by @save-buffer in #7868
• Rework PageStream connection state handling: by @MMeent in #7611
• Export db size metrics for 10 user databases by @zaynetro in #7857
• pageserver: post-shard-split layer rewrites (2/2) by @jcsp in #7531
• pageserver: revert concurrent secondary downloads by @jcsp in #7866
• chore: lower gate guard drop logging threshold to 100ms by @koivunej in #7862
• CI(release): tune Storage & Compute release PR title by @bayandin in #7870
• proxy: fix websocket buffering by @conradludgate in #7878
• safekeeper: test pull_timeline with WAL gc. by @arssher in #7812
• Release 2024-05-27 by @vipvap in #7888

New Contributors

• @Omrigan made their first contribution in #7849

Full Changelog: release-5578...release-5623

release-5579

What's Changed

• Metrics hotfix release by @save-buffer in #7869

Full Changelog: release-5577...release-5579

release-5578

What's Changed

• Metrics hotfix release by @save-buffer in #7869

Full Changelog: release-5577...release-5578

release-5577

What's Changed

• pageserver: use the heatmap upload interval to set the secondary download interval by @jcsp in #7793
• Upgrade of pgvector to 0.7.0 by @Bodobolero in #7726
• safekeeper: log LSNs on connection exit by @arssher in #7728
• feat(pageserver): generate image layers for sparse keyspace by @skyzh in #7567
• Update banner image in Readme by @ruf-io in #7801
• feat(pageserver): do not read past image layers for vectored get by @skyzh in #7773
• Add some more replication slot metrics by @save-buffer in #7761
• build(deps): bump requests from 2.31.0 to 2.32.0 by @dependabot in #7816
• pageserver: fix secondary progress stats when layers are 404 by @jcsp in #7814
• Propagate apply_lsn from SK to PS to prevent GC from collecting objects which may be still requested by replica by @knizhnik in #7368
• Postgres 14.12, 15.7, and 16.3 updates by @tristan957 in #7688
• neon_local: run controller's postgres with fsync=off by @jcsp in #7817
• test(detach_ancestor): ensure L0 compaction in history is ok by @koivunej in #7813
• chore(pageserver): remove metrics for in-memory ingestion by @skyzh in #7823
• Add dummy lsn lease http and page service APIs by @arpad-m in #7815
• compute_ctl: Fix handling of missing /neonvm/bin/resize-swap by @sharnoff in #7832
• Implement timeline_manager for safekeeper background tasks by @petuhovskiy in #7768
• Refactor PageServerHandler::process_query parsing by @arpad-m in #7835
• Another iteration on making walcraft more robust by @arssher in #7588
• Fix zenith_test_evict mode and clear_buffer_cache() function by @hlinnaka in #7803
• Compute-Only Release 2024-05-22 by @vipvap in #7837

New Contributors

• @ruf-io made their first contribution in #7801

Full Changelog: release-5545...release-5577

release-5545

What's Changed

• test(ancestor detach): verify with fullbackup by @koivunej in #7706
• test: fix confusing limit and logging by @koivunej in #7589
• test: allow vectored get validation failure during shutdown by @koivunej in #7716
• chore(pageserver): plumb through RequestContext to VirtualFile read methods by @problame in #7720
• pageserver: refuse to run without remote storage by @jcsp in #7722
• python: allow using allowed_errors.py by @koivunej in #7719
• Fix test_lock_time_tracing flakiness by @jbajic in #7712
• Revert protocol version upgrade by @VladLazar in #7727
• chore(pageserver): plumb through RequestContext to VirtualFile open methods by @problame in #7725
• CI(report-benchmarks-failures): report benchmarks failures to slack by @bayandin in #7678
• fix: do not create metrics contention from background task permit by @koivunej in #7730
• feat(pageserver): add metrics for aux file size by @skyzh in #7623
• pageserver: improve API for invoking GC by @jcsp in #7655
• pagectl: fix diagrams generation for paths containing generations by @VladLazar in #7739
• pageserver: use adaptive concurrency in secondary layer downloads by @jcsp in #7675
• fix(test): ensure compatibility test uses the correct compute node by @skyzh in #7741
• Tiered compaction: cut deltas along lsn as well if needed by @arpad-m in #7671
• Always use Lsn::MAX as the request LSN in the primary by @hlinnaka in #7708
• pageserver: do fewer heatmap uploads for tiny tenants by @jcsp in #7731
• pageserver: switch on new-style local layer paths by @jcsp in #7660
• CI(report-benchmarks-failures): fix condition by @bayandin in #7745
• chore(test): add version check for forward compat test by @skyzh in #7685
• pageserver: skip waiting for logical size on shard >0 by @jcsp in #7744
• Update mold to 2.31 by @arpad-m in #7757
• test suite: allow overriding default compaction algorithm via env var by @problame in #7747
• Add restart_lsn metric for logical slots. by @arssher in #7698
• CI(neon-extra-builds): Use small-arm64 runners instead of large-arm64 by @bayandin in #7740
• pageserver: remove Option<> around remote storage, clean up metadata file refs by @jcsp in #7752
• chore(deps): use upstream svg_fmt after they merged our PR by @problame in #7764
• chore!: always use async walredo, warn if sync is configured by @problame in #7754
• Add test_uploads_and_deletions test by @arpad-m in #7758
• fixup #7747: actually use the fixture for neon_env_builder by @problame in #7767
• Add performance regress test_ondemand_download_churn.py by @jbajic in #7242
• feat(pageserver): use fnv hash for aux file encoding by @skyzh in #7742
• fix(test): ensure fixtures are correctly used for pageserver_aux_file_policy by @skyzh in #7769
• pageserver: refine tenant_id->shard lookup by @jcsp in #7762
• compute_ctl: catalog API endpoints by @prepor in #7575
• proxy: swap tungstenite for a simpler impl by @conradludgate in #7353
• tiered compaction: introduce PAGE_SZ constant and use it by @arpad-m in #7785
• Remove SSO_ACCOUNT_ID from scrubber docs and BucketConfig by @arpad-m in #7774
• Tiered compaction: improvements to the windows by @arpad-m in #7787
• test_suite: patch, don't replace, the tenant_config field, where appropriate by @problame in #7771
• chore: update defaults for timeline_detach_ancestor by @koivunej in #7779
• pageserver: fixes for layer path changes by @jcsp in #7786
• storage controller: use SERIALIZABLE isolation level by @jcsp in #7792
• pageserver: implement auto-splitting by @jcsp in #7681
• feat(pageserver): persist aux file policy in index part by @skyzh in #7668
• feat(pagebench): add aux file bench by @skyzh in #7746
• Fix calculation in test by @hlinnaka in #7805
• proxy: bump parquet by @conradludgate in #7782
• Release 2024-05-20 by @vipvap in #7807

Full Changelog: release-5493...release-5545