Reports: Additional logic for when to show a report

[shrayshray]

In #345 , we established logic for determining whether to show a report on the site. A new step should be added to account for Motions which are made during a meeting, rather than submitted in advance and included on the published Agenda. Steps 1 and 4 below are already in place. Step 2 should be revised as noted below. Step 3 should be added.

1. Check whether "Not Viewable via InSite" is True or False. If True, do not scrape it. Handled by: Skip bills with restricted view opencivicdata/scrapers-us-municipal#251

2. Check bill_type - always display "Board Correspondence" (was previously entered as ""Board Box") - this is the "local_classification" in the extras field or bill_type in the Bill model

3. Check whether it has Minutes History - Display if it does.

4. Check whether the report is on a published agenda, by querying the database for the Bill, its related EventAgendaItem, and the related Event. If the status is "passed," then the Bill is on a published agenda.

An example of a report which should display on the site, but isn't right now because it was submitted during a meeting and not published on an Agenda: 2016-0950

[hancush]

@shrayshray I assumed we'd be able to get at this via the matter history, but I'm not seeing any histories associated with that matter: https://webapi.legistar.com/v1/metro/matters/3746/histories

Is the minutes history a different thing? If so, how do we get at it via the API?

[shrayshray]

@hancush I don't remember why I sent that particular matter initially, but you're right - it doesn't have any history. Here is an example that does! 2015-0812

[hancush]

Thanks, @shrayshray! Looks like that bill is appearing, as expected.

Meanwhile, I deployed revised display logic to the staging site!

Here's an example of a bill that has associated history but doesn't appear on a published agenda (i.e., criterion 3) shown with the revised logic on staging (https://lametro-upgrade.datamade.us/board-report/2019-0388/) but not with the original logic on the live site (https://boardagendas.metro.net/board-report/2019-0388/).

Doesn't look like we have any board reports with the Board Correspondence local classification currently, but the logic is in place to show reports classified as Board Box or Board Correspondence.

Overall, do the visible bills on staging look ok to you? If so, I can deploy this change.

[hancush]

The example I linked above looks weird! Holding on this for further guidance.

[shrayshray]

@hancush could you remind me - what addition guidance are you waiting on from Metro for this?

[hancush]

@shrayshray I don't remember specifically, only that the change in logic revealed some "strange looking board reports" (didn't do myself any favors with that one, haha) that you weren't sure you wanted to display. Perhaps I was referring to "a bill that has associated history but doesn't appear on a published agenda"?

[shrayshray]

Another report which has minutes history but did not appear on an agenda (it was a motion made during a meeting and added to Legistar after the meeting. We do want it to show on the site) - 2020-0745. It's showing on InSite: https://metro.legistar.com/LegislationDetail.aspx?ID=4676674&GUID=2655B012-441B-4549-BD42-040307D50F4B&Options=ID|Text|&Search=2020-0745
But not on boardagendas.metro.net, or on the staging site.

[hancush]

Hi, @shrayshray, the logic to show these kinds of board reports was drafted in #617, but we held off on deploying because there were some board reports that met the conditions that you didn't want to show. (I have a very fuzzy memory that some draft or test board reports became visible with the revised logic, but I don't have a specific link.) Are there any additional attributes of 2020-0745 that we could add to the filter to reduce the likelihood of exposing a board report preliminarily?

[shrayshray]

@hancush one additional criteria for the scenario we're trying to accommodate would be board report type="Motion / Motion Response"
So, display if it has minutes history and report type="Motion / Motion Response".

[hancush]

@fatima3558 This issue is mostly done over in #617. We want to update this filter, specifically the condition to return a report if it has a minutes history, to only return reports with a minutes history if they are of the classification "Motion / Motion Response".

One important note: The OCD bill has a classification attribute, but there is a "controlled taxonomy" for values (that is, it can only be one of the values on this list), so we store Metro's classification in the extras dictionary for each bill, under the key local_classification. Here's an example of how to query the extras dictionary using the ORM.

Related links:

• Definition of the extras field
• Django docs on working with JSONFields

[hancush]

@shrayshray This is live on the staging site. See 2020-0745: https://lametro-upgrade.datamade.us/board-report/2020-0745/.

Want to take a look around the staging site and confirm the bills you expect to be live on the staging site, are?

[shrayshray]

@hancush I think this looks great. Yet another @hancush miracle 🥇 The reports which where showing before under the initial new logic are no longer showing, and he ones we expect to see are. Hurrah!

[hancush]

Yay, we're live! This one's all @fatima3558, I'm just the coach. :-)

[hancush]

I will look to see what criteria is to enter test board reports and coordinate a time to test board correspondence entry with @shrayshray.

[hancush]

@shrayshray We don't have logic to expose test bills on staging but hide them on production, the way we do with events. We could add it, if it's very important not to expose test bills on the live site. Otherwise, we could coordinate a time to test and remove the test bills to limit the time they are live on the site. What do you think? And if we add that logic, what would be a good condition to test for? (For events, we look for a location name of TEST.)

[shrayshray]

@hancush I think we could just coordinate a time to test and remove - either Wednesday 1/13, Thursday 1/14, or after this months meetings - between 2/1 to 2/11. Let me know when is a good date/time time for you.
During our test, I'll create 4 different reports with the following attributes, and add the file ID's here once they're created:

1. MatterTypeName: Board Correspondence, MatterStatusName: Agenda Ready, MatterRestrictViewViaWeb: False
2. MatterTypeName: Board Correspondence, MatterStatusName: Draft, MatterRestrictViewViaWeb: False
3. MatterTypeName: Board Correspondence, MatterStatusName: Agenda Ready, MatterRestrictViewViaWeb: True
4. MatterTypeName: Board Correspondence, MatterStatusName: Draft, MatterRestrictViewViaWeb: True

[hancush]

Hi, @shrayshray, I'm generally available today. I'll be away from my computer for lunch from noon to 1 p.m. Central, and sign off at 4 p.m. Central, but any other time would work. Your test bills should come in no more than 15 minutes after you add them to Legistar. Let me know if there are any issues, or otherwise, when you've completed your test so I can go in and remove.

[shrayshray]

@hancush great - I'll get started on this after your lunch break at 1pm Central and update the issue with the matter IDs once they're created.

[shrayshray]

@hancush these have all been created:

1. File ID: 2020-0898, MatterID: 7340, MatterTypeName: Board Correspondence, MatterStatusName: Agenda Ready, MatterRestrictViewViaWeb: False

2. File ID: 2020-0899, MatterID: 7341, MatterTypeName: Board Correspondence, MatterStatusName: Draft, MatterRestrictViewViaWeb: False

3. File ID: 2020-0900, MatterID: 7342, MatterTypeName: Board Correspondence, MatterStatusName: Agenda Ready, MatterRestrictViewViaWeb: True

4. File ID: 2020-0901, MatterID: 7343, MatterTypeName: Board Correspondence, MatterStatusName: Draft, MatterRestrictViewViaWeb: True

[shrayshray]

Update - File ID: 2020-0898 / MatterID: 7340 is showing on metro.legistar.com, the others are not. 2020-0898 is the only one we want to show.

[hancush]

Ok, this is going to sound wacky, but I think our logic is working.

2020-0898 was imported into the database, and navigating to https://boardagendas.metro.net/board-report/2020-0898/ throws an internal server error. In our log for that error, we can see that the site is trying to render the page. We are seeing a similar error when trying to add the bill to the search index, which explains why it is not appearing in search results. The error has to do with how we create a friendly name for the bill. Specifically, we try to combine the file number with the subject from the board report full text, however no subject is returned when the subject is [PROJECT OR SERVICE NAME], [DESCRIPTION], or [CONTRACT NUMBER].

I've gone ahead and made a blunt fix for this and deployed it to the staging site, so you can see that the display logic is allowing that board report to come through: https://lametro-upgrade.datamade.us/board-report/2020-0898/

Conversely, the remaining files resolve to page not found messages, which is the expected behavior for private board reports:

• https://boardagendas.metro.net/board-report/2020-0899/
• https://boardagendas.metro.net/board-report/2020-0900/
• https://boardagendas.metro.net/board-report/2020-0901/

[shrayshray]

@hancush doh! That just happened to be the only one of the 4 on which I neglected to replace the bracket text from the template. If I go back in and add a subject, would that confirm your hypothesis? Or should I create a brand new test report with the same attributes but also with a subject?

[hancush]

Haha! Life tends to be that way, eh, @shrayshray? Can you create a new test report with a subject? I would guess that updating the subject would toggle the last modified timestamp in Legistar and trigger a rescrape but I'm not certain – a new report would remove any doubt.

[shrayshray]

@hancush okay, new file info:
File ID: 2020-0903, MatterID: 7345, MatterTypeName: Board Correspondence, MatterStatusName: Agenda Ready, MatterRestrictViewViaWeb: False

[shrayshray]

update - it's showing on metro.legistar.com

[hancush]

Nice, thank you! Next scrape slated for 20 past the hour.

[hancush]

@shrayshray Here we are: https://boardagendas.metro.net/board-report/2020-0903/

[shrayshray]

@hancush woohoo! I'll go ahead and remove the agenda dates, switch the status to draft, and change the in control body to "TO BE REMOVED" on all 5 test reports.

[hancush]

Awesome, all test bills are now private / removed from the site.

[shrayshray]

Perfect - thank you, @hancush!