All Chicago Scrapers #16
Conversation
| e.add_media_link(note='Recording', | ||
| url = events['Video']['url'], | ||
| type="recording", | ||
| media_type = '???') |
There was a problem hiding this comment.
In legistar systems, the Video's link to pages like: http://chicago.granicus.com/MediaPlayer.php?view_id=2&clip_id=401
Through some hackery, we can get a media url out from this, but I'm not sure the best way to proceed.
There was a problem hiding this comment.
hurm, I'd just use application/octet-stream
There was a problem hiding this comment.
This doesn't seem right because if you follow that link you get an html page.
| elif datetime.datetime.utcnow().replace(tzinfo = pytz.utc) > when : | ||
| status = 'confirmed' | ||
| else : | ||
| status = 'passed' |
There was a problem hiding this comment.
What's the point of the 'passed' status? Shouldn't this be calculated on the fly by Imago?
There was a problem hiding this comment.
Yeah, for passed, totally. Let's get a bug open on that, and leave it at confirmed
| e.add_document(note= events[doc_type]['label'], | ||
| url = events[doc_type]['url'], | ||
| media_type="application/pdf") | ||
| except ValueError : |
There was a problem hiding this comment.
Sometimes 'Notice' and 'Agenda' are the same file. Sometimes not. Pupa doesn't allow the same file to appear more than once in document list (throws an ValueError). What's the way we want to handle this?
There was a problem hiding this comment.
Hurm.
We have an on_duplicate="ignore" with Billy, we could implement something similar with Pupa, I guess. That or make the checking explicit or something
|
Okay, as soon as opencivicdata/python-opencivicdata#22 I think this is ready to go. |
…hold decrease warning threshold for sentry notifications from critical to warn
Starting on events scraper.