Ensure WebExtensions finish startup task before allowing the platform to submit the first request

[englehardt]

In #221 we started the move over to WebExtensions. Unfortunately, WebExtensions do not block the first page load on extensions startup, meaning that page content may load before the extension injects instrumentation. See Bug 1378459. We've thus had to account for this in our tests, see: 643e457.

This is a problem for us since most of our crawls are stateless and will experience some delay on every load, meaning we'll miss a small amount of data at the start of each page.

Let's see if we can block on startup within the browser manager until the extension has fully loaded.

[englehardt]

Telemetry on startup time: https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-07-04&include_spill=0&keys=__none__!__none__!__none__&max_channel_version=nightly%252F56&measure=WEBEXT_BACKGROUND_PAGE_LOAD_MS&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-06-12&table=0&trim=1&use_submission_date=0

[englehardt]

Possibly related: gbaptista/luminous#55 and EFForg/privacybadger#1865.

[ghostwords]

There may be two separate kinds of issues here:

• Pages loading before the extension finished registering listeners or reading its config from extension storage on extension startup (tracked under no Trackers detected in specified startup page (aka home page) in Firefox EFForg/privacybadger#1845 for Privacy Badger)
• Pages loading before content scripts have finished receiving their instructions (such as whether to continue to run).

[englehardt]

@ghostwords thanks for pointing that out. I updated the title and filed #242 to make it clear there are two potential issues.

[englehardt]

It's now possible to block requests on startup until webextensions are ready. See mozilla/contain-facebook#206 and mdn/sprints#1015 for a description.

[nhnt11]

It's now possible to block requests on startup until webextensions are ready. See mozilla/contain-facebook#206 and mdn/sprints#1015 for a description.

Alright, I looked at the code landed in bug 1447551 and it's promising. It indeed allows us to write a blocking WebRequest listener that stalls page loads till the instrumentation is ready. I'm going to try and write a patch for this today.

[jallmann]

@nhnt11 Have you made any progress with this patch? Or otherwise, would you mind if I pick it up?

[nhnt11]

@jallmann ugh, I didn't see this. Please feel free to pick it up and let me know if I can help.

[nhnt11]

@englehardt Can we close this?

[englehardt]

@nhnt11 If I remember correctly from a few weeks back you mentioned that you had tested this manually by adding in a big delay into startup and didn't see any missed requests? Is that right?

[nhnt11]

Yes. Specifically, we verified that the extension content script is guaranteed to run before page scripts. I'm not sure if this issue includes the idea of making the extension send an ack to the platform before the platform starts issuing requests.

Edit: Ah, that was #242! My bad, sorry.

[nhnt11]

I think what we verified was that we can indeed use the blocking WebRequest listener to achieve this but I can't remember if we landed it. It doesn't seem like it.

[vringar]

The second part of the issue (the ack to the plattform) is being tracked in #135 and the first part was successfully implemented so I'm closing this