Upgrade path is probably wrong

[pschonmann]

Self-Hosted Version

23.6.2

CPU Architecture

x86_64

Docker Version

Docker version 27.1.1, build 6312585

Docker Compose Version

Docker Compose version v2.29.1

Steps to Reproduce

Ive tried to update by upgrade path from https://develop.sentry.dev/self-hosted/releases/
23.6.2 > Latest Version, but get

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c61ff37 in /usr/bin/clickhouse
1. DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x00000000074a9bcd in /usr/bin/clickhouse
2. DB::checkUnsupportedVersion(std::shared_ptr<DB::Context>, String const&) @ 0x00000000122dd350 in /usr/bin/clickhouse
3. DB::loadSystemDatabaseImpl(std::shared_ptr<DB::Context>, String const&, String const&) @ 0x00000000122e540b in /usr/bin/clickhouse
4. DB::loadMetadataSystem(std::shared_ptr<DB::Context>) @ 0x00000000122e43de in /usr/bin/clickhouse
5. DB::Server::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000c6b7f1d in /usr/bin/clickhouse
6. Poco::Util::Application::run() @ 0x0000000015b50a66 in /usr/bin/clickhouse
7. DB::Server::run() @ 0x000000000c6a6dde in /usr/bin/clickhouse
8. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000015b5fb79 in /usr/bin/clickhouse
9. mainEntryClickHouseServer(int, char**) @ 0x000000000c6a3baa in /usr/bin/clickhouse
10. main @ 0x000000000712c42f in /usr/bin/clickhouse
11. ? @ 0x00007f207616d083 in ?
12. _start @ 0x000000000635186e in /usr/bin/clickhouse
 (version 23.8.11.29.altinitystable (altinity build))
2024.08.09 23:11:51.837063 [ 1 ] {} <Error> Application: DB::Exception: Data directory for system database exists, but metadata file does not. Probably you are trying to upgrade from version older than 20.7. If so, you should upgrade through intermediate version.
2024.08.09 23:11:54.235332 [ 1 ] {} <Warning> Context: Linux transparent hugepages are set to "always". Check /sys/kernel/mm/transparent_hugepage/enabled
2024.08.09 23:11:54.468694 [ 1 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 23.8.11.29.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.08.09 23:11:54.487492 [ 1 ] {} <Error> Application: Caught exception while loading metadata: Code: 48. DB::Exception: Data directory for system database exists, but metadata file does not. Probably you are trying to upgrade from version older than 20.7. If so, you should upgrade through intermediate version. (NOT_IMPLEMENTED), Stack trace (when copying this message, always include the lines below):

Ive upgraded trought 23.6.2 > 23.11.0 > Latest Version and seems sentry is working

Expected Result

Smooth upgrade 23.6.2 > Latest Version, or fixed upgrade path in docs

Actual Result

.
.
.
▶ Bootstrapping and migrating Snuba ...
 Container sentry-self-hosted-zookeeper-1  Creating
 Container sentry-self-hosted-redis-1  Creating
 Container sentry-self-hosted-clickhouse-1  Creating
 Container sentry-self-hosted-clickhouse-1  Created
 Container sentry-self-hosted-redis-1  Created
 Container sentry-self-hosted-zookeeper-1  Created
 Container sentry-self-hosted-kafka-1  Creating
 Container sentry-self-hosted-kafka-1  Created
 Container sentry-self-hosted-redis-1  Starting
 Container sentry-self-hosted-clickhouse-1  Starting
 Container sentry-self-hosted-zookeeper-1  Starting
 Container sentry-self-hosted-clickhouse-1  Started
 Container sentry-self-hosted-redis-1  Started
 Container sentry-self-hosted-zookeeper-1  Started
 Container sentry-self-hosted-zookeeper-1  Waiting
 Container sentry-self-hosted-zookeeper-1  Healthy
 Container sentry-self-hosted-kafka-1  Starting
 Container sentry-self-hosted-kafka-1  Started
dependency failed to start: container sentry-self-hosted-clickhouse-1 is unhealthy
Error in install/bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:36
--> install/bootstrap-snuba.sh:source:3

Cleaning up...

now with my upgrade path i have working instance but see every second this error in
/meh/volumes/sentry-self-hosted_sentry-clickhouse-log/_data/clickhouse-server.err.log

2024.08.10 07:23:37.961594 [ 13931 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000015b3dbf2 in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::peerAddress() @ 0x0000000015b40376 in /usr/bin/clickhouse
2. DB::ReadBufferFromPocoSocket::ReadBufferFromPocoSocket(Poco::Net::Socket&, unsigned long) @ 0x000000000c896cc6 in /usr/bin/clickhouse
3. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptr<DB::IHTTPContext>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x000000001315451b in /usr/bin/clickhouse
4. DB::HTTPServerConnection::run() @ 0x0000000013152ba4 in /usr/bin/clickhouse
5. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in /usr/bin/clickhouse
6. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in /usr/bin/clickhouse
7. Poco::PooledThread::run() @ 0x0000000015c7a667 in /usr/bin/clickhouse
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in /usr/bin/clickhouse
9. ? @ 0x00007f2eef0f1609 in ?
10. ? @ 0x00007f2eef016353 in ?
 (version 23.8.11.29.altinitystable (altinity build))

Event ID

No response

[aldy505]

Thanks for raising the upgrade path problem. We'll fix the docs somehow.

The ClickHouse error of "Socket not connected" is a very well known error, and it's being tracked here getsentry/snuba#5707

We'll keep this issue open until we updated the docs. Thanks again!

[joshuarli]

23.6.2 to latest is a pretty big jump, that's about a year. I don't think we can realistically support that expectation (going forward). That said I'll update the docs to recommend smaller incremental upgrades.

[pschonmann]

Yeap, do upgrade path docs or something like that https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/ Dne po 12. 8. 2024 22:38 uživatel joshuarli ***@***.***> napsal:

…

23.6.2 to latest is a pretty big jump, that's about a year. I don't think we can realistically support that expectation (going forward). That said I'll update the docs to recommend smaller incremental upgrades. — Reply to this email directly, view it on GitHub <#3257 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQKLFBAQVLF3YAAIK75DDDZREMN3AVCNFSM6AAAAABMJTOLYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBUHA3DKNBQGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[joshuarli]

I was able to reproduce this on Mac x86_64, same error. Clickhouse was happy with 23.11.0 as an intermediate upgrade before 24.7.1. I've updated the hard stop recommendations in the docs here: getsentry/develop#1345

Thanks for the bug report!

[iulian0512]

i get this on 24.8.0 fresh install :(