fediverse => Bluesky bridging currently paused

[wbz0100]

Its delayed so much. Is this normal?
I think it's been lagging a lot since the last time it was disconnected.

Instead of being posted sequentially. And In severe cases, there is a delay of up to 10 hours.

[snarfed]

Yes! Not normal, we're badly backed up right now. Started around midnight PT last night, our send task queue peaked at over 3M tasks (!). Sorry for the trouble. It's down to 2.5M ish now, but still. 😕

I'm not sure what's going on yet. Coincidentally, I got pretty sick last night, and I'm still out of it, so I haven't been able to look yet. Hopefully soon.

[snarfed]

For my own notes, here's the repl code I'm using to clear out the send queue:

from google.cloud import tasks_v2
tasks_client = tasks_v2.CloudTasksClient()

from common import TASKS_LOCATION
from google.cloud.tasks_v2.types import Task

request = tasks_v2.ListTasksRequest(
    parent=tasks_client.queue_path(appengine_info.APP_ID, TASKS_LOCATION, 'send'),
    response_view=Task.View.FULL,
)

deleted = 0
for resp in tasks_client.list_tasks(request=request):
  if urllib.parse.parse_qs(resp.app_engine_http_request.body).get(b'user') == [b'ahBicmlkZ3ktZmVkZXJhdGVkci0LEgdBVFByb3RvIiBkaWQ6cGxjOm92eWFqYm5lZ3JpcjJkZHR3ZmhpamlvYgw']:
    deleted += 1
    print(deleted)
    tasks_client.delete_task(request=tasks_v2.DeleteTaskRequest(name=resp.name))

[snarfed]

We're almost entirely caught up, send task queue is under 10k now. Thanks for your patience all!

[imsodin]

Thanks for dealing with this so quickly!

For my own notes, here's the repl code I'm using to clear out the send queue:

Does that mean some posts during the incident were not replicated?
I am just asking to get an idea if some problems I just heared about might be related or not. I had a chat with someone who unfortunately started trying out the bridge more or less at the same time as these troubles happened.

[snarfed]

Hopefully we didn't drop any posts. Let me know if you see any that are missing and pass the checks in https://fed.brid.gy/docs#troubleshooting , happy to look.

[fadookie]

@snarfed thanks for handling this. Is it possible this is happening again? In my fedi -> bsky bridged profile I haven't seen any posts updating in over 9 hours (I'm not sure what the usual delay is): https://bsky.app/profile/Eliot-L.social.coop.ap.brid.gy

[wbz0100]

Umm... Looks like it's happening now.
A post from about 2 hours ago is publishing now.

This seems to happen constantly, and I'm not sure if it's because the post speed can't keep up with the processing speed, or if there's something wrong in the processing.

[EmilJacobs]

Can confirm, this issue isn't resolved.

[avocado1969]

I’m also not seeing fedi actions populate onto bsky since yesterday evening, even though my profile on fed.brid.gy shows the changes going though with the green check

[dakota-marshall]

Not to pile on, but I can confirm this is still happening aswell. I have severl public posts and replies that all seem to have the green check, but are not appearing on the Bluesky account.

I actually haven't had anything sync over since I bridged my account like 24 hours ago

[Ralph1798]

I followed a newly bridged account and posts from 350d+ started to appear in my Mastodon-timeline.
Interesting enough: no current posts, just the old ones.

[snarfed]

Not to pile on, but I can confirm this is still happening aswell. I have severl public posts and replies that all seem to have the green check, but are not appearing on the Bluesky account.

I actually haven't had anything sync over since I bridged my account like 24 hours ago

Yes! Sorry, reopening. The current Bluesky => fediverse delay is on us, fediverse => Bluesky delay is on Bluesky team. We're (all) working on it. 😬

[wbz0100]

I know you've been sick, so please take it easy and work slowly.

I think one of the causes may have been a delay in moving posts from X (formerly Twitter) to Bluesky.

[dakota-marshall]

Not to pile on, but I can confirm this is still happening aswell. I have severl public posts and replies that all seem to have the green check, but are not appearing on the Bluesky account.
I actually haven't had anything sync over since I bridged my account like 24 hours ago

Yes! Sorry, reopening. The current Bluesky => fediverse delay is on us, fediverse => Bluesky delay is on Bluesky team. We're (all) working on it. 😬

Thanks for the response! No one is dying, so this is such a non-issue in the grand scheme of things. Hope you feel better :) Thanks for such a cool tool :D

[Daft-Freak]

How much delay should I be expecting on the bluesky side? (I haven't seen a post I've made get through in the last 24 hours and am wondering if I have an actual problem...)

[snarfed]

@Daft-Freak you mean fediverse => Bluesky? They're doing a lot of work on their relay, bluesky-social/atproto#3036 , and they've temporarily paused ingesting commits from federated PDSes like Bridgy Fed. I don't have an ETA for when that will be lifted, sadly.

[Daft-Freak]

Ah okay, good to know!

[snarfed]

https://bsky.app/profile/bnewbold.net/post/3lbimsarmw22g
https://bsky.app/profile/bnewbold.net/post/3lbipqvij7k2m
https://bsky.app/profile/bnewbold.net/post/3lbirrqme2s2w

[snarfed]

@jlahtela hopefully, once this is fixed on Bluesky's side, everything that's happened over the last four days will show up in Bluesky, including bridging new accounts. 🤞

[snarfed]

Filed bluesky-social/atproto#3094 to track on Bluesky's side.

[andrewrgross]

This is very interesting to read. I was wondering why I was seeing posts from a few days ago but nothing since. And this makes sense. I think a lot of stuff is popping right now (in a good way!), and I'm not surprised that troubleshooting will be needed.

Thanks for running this and the communication! I honestly feel like this could be a big step on the way to a less crappy internet.

[AtiusAmy]

One weird thing that's happening. My boyfriend runs a bot that posts Bangkok rainstorm every hour (It should've been 3 hours, but that's going off on a tangent).

https://bsky.app/profile/bkkraindarbot.misskey.winscloud.net.ap.brid.gy

If you look at the time (22.00 or 18.00) you'll see it has been repeating for a while now, not sure what's going on, if this is the reason I'll tell him to unplug from the bridge (and change it to 3 hours damnit!)

[snarfed]

Hmm! I don't see that...? I see a gap between 11/21 03:52 and 11/22 13:02, but otherwise it looks ok...?

[dos1]

@AtiusAmy It's just old posts from the queue finally being processed - look inside the posts and see their "archived" times. It's actually a brilliant account to look at to assess where were are in the queue right now (looks like we're 24h behind and progressing at about 1h per 2-3 minutes) 😄

[edit] ...and looks like it's up-to-date now

[AtiusAmy]

I think it works now!

http://bsky.app/profile/amyiscoolz.atiusamy.com

Also my username now changes to the custom domain I set yay!! (Now, I just wish I could interact with more bsky account 😔)

[wbz0100]

There's still a delay, but it seems to be decreasing, up to about 5 minutes (maybe because it's processing the backlog?).

[lucajet]

@snarfed

i think you can remove the red alert warning 🙂

[foamplugger]

This might be related to the recent pause, but posts on Mastodon that were boosted and then unboosted still appear on the bridged Bluesky account.

[snarfed]

Thanks all!

Afaict Bluesky hasn't yet backfilled all activity that happened during their outage, there's still a period of time when bridged activities didn't take effect, and I don't know yet whether they will or not. Still trying to track that down so I know what to tell peope. bluesky-social/atproto#3094 (reply in thread)

[jlahtela]

I see as well that my account is created for Bluesky now. It did post my messages there and processed those.

There may still be some delays. The post that I made ~3 minutes ago is not there. I don't know what is except delay on normal operation.

[dos1]

It's normal for new posts to show up after a few minutes.

[wbz0100]

It used to post within seconds before the bridge broke, so it's not normal yet.

[wbz0100]

It used to post within seconds before the bridge broke, so it's not normal yet.

[snarfed]

Bridging is often immediate ish, you're right, but not always. Sometimes Bridgy Fed is backed up, sometimes (like now) Bluesky itself may be backed up, which I don't really have control over.

[wbz0100]

I know there's no other way to do it at this point!

Before it stopped, I had quite a few records where I was replying to my opponents within 20 seconds or so of each other, so it's possible that it could be faster, but I'll have to wait and see.

[AtiusAmy]

Perhaps it would be better to use a messaging platform you two can agree on to communicate live instead, social media in general just wasn't made for that

[lucajet]

I greatly appreciated the effort they put to help fix the issue.
Now if we can just convince them to increase up to 500 characters 🙂

There's no other way but to report the link of the full message, but each time i write a 301 char post, i feel like i'm 'advertising' mastodon, while i just want to talk with the people there.

[AtiusAmy]

I greatly appreciated the effort they put to help fix the issue. Now if we can just convince them to increase up to 500 characters 🙂

There's no other way but to report the link of the full message, but each time i write a 301 char post, i feel like i'm 'advertising' mastodon, while i just want to talk with the people there.

500 character is still far too little, there should be like 5000 character limits, reading threads is jarring as hell. Markdown + 5000 character support (at least just rendering it, so it bridges properly (vanilla masto renders it properly despite not being able to do it))

[lucajet]

Coming from pleroma, when i post on fediverse, i always have 5000 char and, occasionally markdown (which i thought is rendered fine on bs, isn't it ?)
Most of mastodon instances default to 500, so i think is a compromise.

I don't write long essays all the time though, my practical point is that once in a while, an external link is accepted by a normie.
If it happens most of the time, maybe not.

We don't have control on this, but kinda surprised me that many paid engineers worked so hard to fix this interoperability issue, and they keep insisting with this extremely low limit, that harm their users

[snarfed]

Re-closing. Feel free to open new issues for individual accounts or posts!