PB blocks avatars[0123].githubusercontent.com #1466

jeremyn · 2017-06-30T16:05:05Z

PB blocks avatars[0123].githubusercontent.com (red slider). I just noticed this in the past couple days. The problem description is similar to fixed issue #443.

The text was updated successfully, but these errors were encountered:

jeremyn · 2017-06-30T16:09:29Z

I'm not sure if it's related but another site I visit has a cloudfront.net domain blocked which breaks CSS. I think this problem started around the same time as the avatars problem. Please let me know if you want a separate issue for the CloudFront problem. Issue #513 is similar.

ghostwords · 2017-06-30T16:28:54Z

Hello, could you run the following in Badger's background page console? This should print the decisions your Badger reached for all "githubusercontent" domains, and the site domains Badger saw "githubusercontent" domains perform tracking on.

(function () {
  const STR = "githubusercontent";
  console.log("**** ACTION_MAP for", STR);
  _.each(badger.storage.getBadgerStorageObject('action_map').getItemClones(), (obj, domain) => {
    if (domain.indexOf(STR) != -1) console.log(domain, JSON.stringify(obj, null, 2));
  });
  console.log("**** SNITCH_MAP for", STR);
  _.each(badger.storage.getBadgerStorageObject('snitch_map').getItemClones(), (sites, domain) => {
    if (domain.indexOf(STR) != -1) console.log(domain, JSON.stringify(sites, null, 2));
  });
}());

To get to the background page console in Chrome, visit chrome://extensions, make sure "Developer mode" is checked, click on the "background page" link in Privacy Badger's row, and select the Console tab.

In Firefox, visit about:debugging, enable add-on debugging, click Debug next to Privacy Badger, click the OK button on the popup warning about remote debugging, and enter the above script into the console after the >>.

ghostwords · 2017-06-30T16:30:03Z

CloudFront: Yes please! A new issue would be great. You should specify the website where the issue happened and include the debugging information I asked for above, replacing "githubusercontent" on the second line of the code with "cloudfront".

jeremyn · 2017-06-30T16:42:00Z

I notice the debug output contains a small bit of my internet history. I'd rather not post that publicly, but I don't know what I can redact, if anything, without making the data useless to you. Can you narrow down what you're looking for?

ghostwords · 2017-06-30T17:07:27Z

Could you send it just to me?

jeremyn · 2017-06-30T23:40:59Z

I'd rather not send it out at all, sorry.

I've tried to reproduce this myself by installing PB into a new Firefox profile and browsing around, but so far the avatars domains are still green. I'm curious how the short cookieblocklist reported in #443 (comment) revealed the problem in that issue. Are you looking for something similar here and if so, what exactly?

ghostwords · 2017-07-01T02:29:39Z

I am looking for the decisions your Badger reached for all "githubusercontent" domains, and the site domains Badger saw "githubusercontent" domains perform tracking on. I am more interested in the site domains, the "snitch_map" stuff.

adjagu · 2017-07-01T14:47:40Z

@ghostwords All avatars on github.com are broken for me as well. Is this the information you require?

socialwidgetloader.js:80 [Deprecation] Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.
getFileContents @ socialwidgetloader.js:80
loadJSONFromFile @ socialwidgetloader.js:60
loadSocialWidgetsFromFile @ socialwidgetloader.js:97
(anonymous) @ background.js:30
background.js:782 Loading badgers into the pen.
background.js:793 Privacy badger is ready to rock!
background.js:794 Set DEBUG=1 to view console messages.
(function () {
  const STR = "githubusercontent";
  console.log("**** ACTION_MAP for", STR);
  _.each(badger.storage.getBadgerStorageObject('action_map').getItemClones(), (obj, domain) => {
    if (domain.indexOf(STR) != -1) console.log(domain, JSON.stringify(obj, null, 2));
  });
  console.log("**** SNITCH_MAP for", STR);
  _.each(badger.storage.getBadgerStorageObject('snitch_map').getItemClones(), (sites, domain) => {
    if (domain.indexOf(STR) != -1) console.log(domain, JSON.stringify(sites, null, 2));
  });
}());
VM290:3 **** ACTION_MAP for githubusercontent
VM290:5 avatars0.githubusercontent.com {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1498527943655,
  "userAction": ""
}
VM290:5 avatars1.githubusercontent.com {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1498506079577,
  "userAction": ""
}
VM290:5 avatars2.githubusercontent.com {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1498397193773,
  "userAction": ""
}
VM290:5 avatars3.githubusercontent.com {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1498734598584,
  "userAction": ""
}
VM290:5 camo.githubusercontent.com {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1498272232464,
  "userAction": ""
}
VM290:5 cloud.githubusercontent.com {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1498428002015,
  "userAction": ""
}
VM290:5 raw.githubusercontent.com {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1498561286436,
  "userAction": ""
}
VM290:7 **** SNITCH_MAP for githubusercontent
undefined

ghostwords · 2017-07-01T15:29:21Z

Yes, thank you. Feel free to mark them as allowed to fix the issue for yourself.

Something isn't right here ... Your Badger decided to block githubusercontent subdomains while not seeing them track on any sites.

adjagu · 2017-07-01T16:29:02Z

@ghostwords Will do and thank you for the quick reply.

jeremyn · 2017-07-01T16:45:09Z

Thanks @adjagu . For what it's worth, my own SNITCH_MAP is not undefined.

I'm a little confused here actually. I'm not really sure how PB decides what to block. Is githubusercontent whitelisted or otherwise specially handled in the code? Maybe PB actually should block githubusercontent stuff.

adjagu · 2017-07-01T17:23:27Z

@jeremyn I'm not sure as to how PB chooses what it blocks either and usually don't notice when it does. This time however it was noticeable since pictures were being blocked. All I did to see avatars again was to move the sliders (of the blocked content) from red to yellow.

As to the undefined bit after going through some of the issues here I did notice that I wasn't getting the same information back from PB that other people were and I ended up removing it and reinstalling it.

Edit:
After uninstalling and reinstalling PB my SNITCH_MAP is still undefined.

socialwidgetloader.js:80 [Deprecation] Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.
getFileContents @ socialwidgetloader.js:80
background.js:782 Loading badgers into the pen.
background.js:793 Privacy badger is ready to rock!
background.js:794 Set DEBUG=1 to view console messages.
migrations.js:27 changing privacy settings
migrations.js:70 fixing DNT check times
migrations.js:80 Running migration to forget mistakenly flagged domains ...
code.jquery.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 ()
ajax.googleapis.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 ()
emailoctopus.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 ()
cdn.embedly.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 403 (Forbidden)
tedium.imgix.net/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 ()
activate.bloglovin.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (Not Found)
netdna.bootstrapcdn.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 403 ()
assets.pinterest.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 403 (Forbidden)
migrations.js:41 MIGRATING BLOCKED SUBDOMAINS THAT ARE ON COOKIE BLOCK LIST
giphy.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 ()
dl.dropboxusercontent.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 ()
shortformblog.disqus.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (NOT FOUND)
media.giphy.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 403 ()
disqus.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (NOT FOUND)
c.disquscdn.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 ()
a.disquscdn.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (Not Found)
www.facebook.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 ()
scontent-ort2-1.xx.fbcdn.net/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 502 ()
staticxx.facebook.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 403 ()
external-ort2-1.xx.fbcdn.net/.well-known/dnt-policy.txt Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
3-edge-chat.facebook.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 400 ()
video-ort2-1.xx.fbcdn.net/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 502 ()
avatars2.githubusercontent.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (Not Found)
camo.githubusercontent.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (Not Found)
avatars3.githubusercontent.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (Not Found)
avatars0.githubusercontent.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (Not Found)
avatars1.githubusercontent.com/.well-known/dnt-policy.txt Failed to load resource: the server responded with a status of 404 (Not Found)
(function () {
  const STR = "githubusercontent";
  console.log("**** ACTION_MAP for", STR);
  _.each(badger.storage.getBadgerStorageObject('action_map').getItemClones(), (obj, domain) => {
    if (domain.indexOf(STR) != -1) console.log(domain, JSON.stringify(obj, null, 2));
  });
  console.log("**** SNITCH_MAP for", STR);
  _.each(badger.storage.getBadgerStorageObject('snitch_map').getItemClones(), (sites, domain) => {
    if (domain.indexOf(STR) != -1) console.log(domain, JSON.stringify(sites, null, 2));
  });
}());
VM106:3 **** ACTION_MAP for githubusercontent
VM106:5 avatars2.githubusercontent.com {
  "userAction": "",
  "dnt": false,
  "heuristicAction": "",
  "nextUpdateTime": 1499376693100
}
VM106:5 avatars3.githubusercontent.com {
  "userAction": "",
  "dnt": false,
  "heuristicAction": "",
  "nextUpdateTime": 1499457742750
}
VM106:5 camo.githubusercontent.com {
  "userAction": "",
  "dnt": false,
  "heuristicAction": "",
  "nextUpdateTime": 1499250766867
}
VM106:5 avatars0.githubusercontent.com {
  "userAction": "",
  "dnt": false,
  "heuristicAction": "",
  "nextUpdateTime": 1499152161015
}
VM106:5 avatars1.githubusercontent.com {
  "userAction": "",
  "dnt": false,
  "heuristicAction": "",
  "nextUpdateTime": 1499466043975
}
VM106:7 **** SNITCH_MAP for githubusercontent
undefined

jeremyn · 2017-07-01T17:42:52Z

@adjagu I agree the avatar images are blocked, that's what I'm getting and why I reported the problem. What I'm saying that is maybe PB is correct in deciding that the avatars should be blocked, by its own logic.

In other words I want to understand whether the problem is

PB is incorrectly deciding to block githubusercontent by its own logic, or
PB correctly blocks githubusercontent, but githubusercontent should be in some whitelist, and there is a bug with the whitelist, or
PB correctly blocks githubusercontent, and there is no whitelist or githubusercontent is correctly left out of the whitelist, and so there's no bug.

ghostwords · 2017-07-01T19:21:54Z

Uninstalling clears all of Privacy Badger's data, so it has to relearn what to block (which goes in action_map) and where tracking was seen (which goes in snitch_map).

ghostwords · 2017-07-01T19:24:47Z

@jeremyn Could you share the size of your yellowlist? Run the following in your Badger's background page console:

console.log(Object.keys(badger.storage.getBadgerStorageObject('cookieblock_list')._store).length);

ghostwords · 2017-07-05T14:43:09Z

Let us know if this comes up again. I think we know what caused this issue to happen, and will take steps to prevent it from happening again.

jeremyn · 2017-07-05T17:53:17Z

@ghostwords Sorry for totally ignoring your last comment 😦 I'm not sure what you mean "let us know if this comes up again" though. Did you push a fix?

ghostwords · 2017-07-05T19:34:15Z

Sorry, yes, let me explain.

I think what happened was that for a period of time last week, people's Badgers were getting their "yellowlists" (a list of domains Badger should treat specially to avoid breaking sites) erroneously wiped out thanks to an eff.org issue. Once the website issue was resolved, this problem should have been fixing itself whenever the browser was restarted (or just left open for longer than 24 hours), or Privacy Badger was re-installed. I opened #1473 to make sure this sort of failed update doesn't happen again.

jeremyn · 2017-07-17T15:35:25Z

I'm getting camo.githubusercontent.com blocked by PB, which breaks things like for example the Travis-CI graphical icon at https://github.com/EFForg/privacybadger .

jeremyn · 2017-07-17T15:39:10Z

Also, and I'm sorry to just throw out criticism without a PR, but I don't like the design where PB makes regular calls to an eff.org site. I expect this would be an unpleasant surprise to many users who disapprove of any sort of phoning home for any reason. Also, it means things can break when there hasn't been an update or really any indication that anything has changed.

Please consider rewriting the add-on so that the whitelist and anything similar is included in the add-on itself.

ghostwords · 2017-07-17T17:38:52Z

I'm sorry our updating system is so buggy, but we are working on fixing it. Here are some related issues (I'm about to open a new one to fix what I see is still an open issue here): #1487, #1474, #1473.

I don't think remotely fetching assets from the same organization that publishes the add-on should be considered "phoning home". If you trust the EFF to install the add-on, you should trust the EFF to serve add-on assets in a privacy-respectful manner.

jeremyn · 2017-07-17T18:38:50Z

I appreciate your response and acknowledge that you feel bad about the breakages. However there is an endless list of software creators who want to build in quiet, uncontrollable update processes to their software. They always think they are justified. Users complain because they feel a loss of control over their system and because it causes unexpected breakage. This is no different. The EFF heavily criticized Microsoft for this sort of thing during the Windows 10 rollout.

In HTTPS Everywhere, ruleset changes are bundled with the code, and the add-on is updated through the usual browser add-on update process and the add-on version can be checked in the browser. I can revert or block updates if I want. I can't do that with PB.

ghostwords · 2017-07-17T20:58:35Z

#1487 is about shipping assets with the add-on. Perhaps in the future we could have a switch to disable asset updates altogether.

I believe HTTPS Everywhere is moving to having rule sets become updateable outside of add-on upgrades.

jeremyn · 2017-07-17T22:33:34Z

I haven't heard anything about HTTPS Everywhere rulesets being updateable outside of the regular add-on update process. There is talk of moving HTTPS Everywhere ruleset work into a separate repository from the code. Maybe that's what you're thinking of?

ghostwords · 2017-07-18T20:04:25Z

Here is a WIP HTTPS Everywhere branch for updateable rulesets: https://github.com/EFForg/https-everywhere/compare/sign-rulesets

jeremyn · 2017-07-18T23:32:21Z

Thanks @ghostwords .

@Hainish Is there an issue/pull request against HTTPS Everywhere for the changes @ghostwords linked in their previous commit?

jeremyn · 2017-07-21T17:56:24Z

This issue was closed but just to be clear, PB still blocks the githubusercontent subdomains.

ghostwords · 2017-07-21T19:04:44Z

Sorry, it's the disconnect between GitHub as a developer platform and GitHub as a support platform. We fixed it in development but haven't released the fix yet. There should be a release the coming Monday that fixes this and a bunch of other mis-blocked domains for everybody.

ghostwords · 2017-07-24T21:18:46Z

This should now be fixed in production with Privacy Badger 2017.7.24.

ghostwords · 2019-06-17T17:51:23Z

I opened a PR for an admin policy flag to disable remote asset updates: #2394

ghostwords added the broken site label Jun 30, 2017

andresbase assigned ghostwords Jun 30, 2017

ghostwords added the important label Jul 1, 2017

ghostwords added the mystery label Jul 1, 2017

andresbase added the in progress label Jul 3, 2017

ghostwords mentioned this issue Jul 5, 2017

Guard against updating to blank/invalid yellowlists #1473

Closed

ghostwords closed this as completed Jul 5, 2017

SwartzCr removed the in progress label Jul 5, 2017

ghostwords reopened this Jul 17, 2017

ghostwords mentioned this issue Jul 17, 2017

Add migration to fix yellowlisted yet blocked domains #1508

Merged

cowlicks closed this as completed in #1508 Jul 19, 2017

jeremyn mentioned this issue Aug 25, 2017

Use Ed25519 instead of RSA for ruleset signatures EFForg/https-everywhere#12087

Closed

jeremyn mentioned this issue Sep 17, 2017

Discussion about out-of-band ruleset updates EFForg/https-everywhere#12606

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PB blocks avatars[0123].githubusercontent.com #1466

PB blocks avatars[0123].githubusercontent.com #1466

jeremyn commented Jun 30, 2017

jeremyn commented Jun 30, 2017

ghostwords commented Jun 30, 2017 •

edited

Loading

ghostwords commented Jun 30, 2017

jeremyn commented Jun 30, 2017

ghostwords commented Jun 30, 2017 •

edited

Loading

jeremyn commented Jun 30, 2017

ghostwords commented Jul 1, 2017

adjagu commented Jul 1, 2017

ghostwords commented Jul 1, 2017 •

edited

Loading

adjagu commented Jul 1, 2017

jeremyn commented Jul 1, 2017

adjagu commented Jul 1, 2017 •

edited

Loading

jeremyn commented Jul 1, 2017

ghostwords commented Jul 1, 2017 •

edited

Loading

ghostwords commented Jul 1, 2017

ghostwords commented Jul 5, 2017

jeremyn commented Jul 5, 2017

ghostwords commented Jul 5, 2017 •

edited

Loading

jeremyn commented Jul 17, 2017

jeremyn commented Jul 17, 2017

ghostwords commented Jul 17, 2017 •

edited

Loading

jeremyn commented Jul 17, 2017

ghostwords commented Jul 17, 2017

jeremyn commented Jul 17, 2017

ghostwords commented Jul 18, 2017 •

edited

Loading

jeremyn commented Jul 18, 2017

jeremyn commented Jul 21, 2017

ghostwords commented Jul 21, 2017

ghostwords commented Jul 24, 2017

ghostwords commented Jun 17, 2019

PB blocks avatars[0123].githubusercontent.com #1466

PB blocks avatars[0123].githubusercontent.com #1466

Comments

jeremyn commented Jun 30, 2017

jeremyn commented Jun 30, 2017

ghostwords commented Jun 30, 2017 • edited Loading

ghostwords commented Jun 30, 2017

jeremyn commented Jun 30, 2017

ghostwords commented Jun 30, 2017 • edited Loading

jeremyn commented Jun 30, 2017

ghostwords commented Jul 1, 2017

adjagu commented Jul 1, 2017

ghostwords commented Jul 1, 2017 • edited Loading

adjagu commented Jul 1, 2017

jeremyn commented Jul 1, 2017

adjagu commented Jul 1, 2017 • edited Loading

jeremyn commented Jul 1, 2017

ghostwords commented Jul 1, 2017 • edited Loading

ghostwords commented Jul 1, 2017

ghostwords commented Jul 5, 2017

jeremyn commented Jul 5, 2017

ghostwords commented Jul 5, 2017 • edited Loading

jeremyn commented Jul 17, 2017

jeremyn commented Jul 17, 2017

ghostwords commented Jul 17, 2017 • edited Loading

jeremyn commented Jul 17, 2017

ghostwords commented Jul 17, 2017

jeremyn commented Jul 17, 2017

ghostwords commented Jul 18, 2017 • edited Loading

jeremyn commented Jul 18, 2017

jeremyn commented Jul 21, 2017

ghostwords commented Jul 21, 2017

ghostwords commented Jul 24, 2017

ghostwords commented Jun 17, 2019

ghostwords commented Jun 30, 2017 •

edited

Loading

ghostwords commented Jun 30, 2017 •

edited

Loading

ghostwords commented Jul 1, 2017 •

edited

Loading

adjagu commented Jul 1, 2017 •

edited

Loading

ghostwords commented Jul 1, 2017 •

edited

Loading

ghostwords commented Jul 5, 2017 •

edited

Loading

ghostwords commented Jul 17, 2017 •

edited

Loading

ghostwords commented Jul 18, 2017 •

edited

Loading