synced with main, cleaned outputs

[rahulkulhalli]

@Abby-Wheelis @shankari Please disregard the previous draft PR. I have synced this branch with the current main implementation.

[Abby-Wheelis]

A few questions for now, I'll return to this when I get to run the notebooks myself

[Abby-Wheelis]

I haven't needed to do this before, why do you need the path?

[Abby-Wheelis]

Does this differ from the one in scaffolding.py? Could we import it instead

[rahulkulhalli]

Yes, we could import it directly. I'll make the changes to just use the code from scaffolding instead. I wanted to see how the code works, which is why I had kept everything in one notebook.

[Abby-Wheelis]

ditto

[rahulkulhalli]

Same as above! I'll make the changes to use the code from scaffolding.py instead

[Abby-Wheelis]

what does this comment mean? Was there a data issue? Did it get resolved?

[rahulkulhalli]

For some observations in the allCEO demographic data, there was a user who had input 9999 as values for n_residence_members, n_residents_u18, and n_residents_with_license. There were also 2 users who had provided a n_residence_members input as 50. I dealt with these instances by dropping these users.

[Abby-Wheelis]

Oooh, ok, they were individual outlier responses, maybe you can update the comment

[Abby-Wheelis]

For my clarification, is this "flattening" out, for example, all of the occupational categories so that every user has a 0/1 for each occupation (and the column with 1 is their occupation)?

[rahulkulhalli]

Yes, that is correct. Given a set of alternatives, if an observation lies within one of the alternatives, we activate that bit and keep the bits for the remaining alternatives zero. It is indeed a way of flattening-out categorical data.

[Abby-Wheelis]

Does "unwanted" here just mean those who don't have both sensed trip AND demographic survey data? It might be good to clarify that in the comment.

[rahulkulhalli]

Oh, that was a misplaced comment. I resolved it. Sorry!

[Abby-Wheelis]

does this remove trips that fall outside of these? That should be ok, but if it removes trips that have any segments faster than these speeds I'd worry about that - anecdotally I've broken both of those speeds several times, but maybe not for a trip on the whole (I think I have sustained walking around 4mph though, back when I was walking in a mostly flat area)

[rahulkulhalli]

Yes, it removes trips that fall outside given range of percentiles. The default arguments are configured to be the 10th and 90th percentile, but in the subsequent cell, I call the method by specifying the 1st and 99th percentile as the outlier ranges. It is fairly standard preprocessing practice to remove these outliers, especially because such outliers can easily dominate/influence any aggregation-based analyses.

[Abby-Wheelis]

So they're removed based on their position as outliers (by percentile) not by these hard cutoff values? I was looking at the 4mph walking and 15mph biking cutoffs

[Abby-Wheelis]

Following up on this based on our call today - it seems that yes, we are cutting off sections over these levels right now, but plan to re-evaluate this decision in the future -- maybe remove outliers since we have data to determine what those are rather than using the outside source.

[Abby-Wheelis]

Why is this written as a loop if you only ever want it to execute once?

[rahulkulhalli]

We do so create one chunk of train and test sets with 80% and 20% of the stratified data respectively. sklearn's reguar train_test_split method unfortunately does not provide a way to create training and testing splits with groups. There may be other ways to achieve this, but this is what I went with for now.

For clarification, StratifiedGroupKFold creates splits by stratifying w.r.t. the number of trips per user. The algorithm then attempts to preserve the percentage of groups in each split. I purposely mention n=5 splits, because that is the information that the algorithm uses to determine the percentage of data in each split.

n=5 => 20% of test data. Therefore we create one split, get the training and testing data, and break the iteration.

[Abby-Wheelis]

Does this mean the summary ends up being just one segment of the trip? Or is the total of all segments with the most represented mode?

[rahulkulhalli]

For every user, the trip summary is a combination of the aggregation performed over the 'argmax' section information and the entire trip-level distance and duration aggregates.

[Abby-Wheelis]

This line (and the other ones like it in this file) are throwing an error for me

ValueError: 'stat' must be one of ['count', 'density', 'probability', 'frequency'], but percent was passed.

I wonder if it could be something like my seaborn version? I'll check what is installed against what you have in the readme

[Abby-Wheelis]

Yep! Updating (with the instructions in the readme you included 🎉 ) fixed the issue!

[Abby-Wheelis]

With a few exceptions, I am able to run these now:

• still need to run the models in notebook 03, I'll let those run this evening while I'm at home
• getting an error with script 02 that there is no key transit - this is confusing since the script has not changed and it ran earlier, I will try this one again tonight as well

[Abby-Wheelis]

The models in 03 run for me, though they output all 1's, is this a consequence of the dataset (nicr) that I'm using and how small it is?

[rahulkulhalli]

With a few exceptions, I am able to run these now:

* still need to run the models in notebook 03, I'll let those run this evening while I'm at home

* getting an error with script 02 that there is no key `transit` - this is confusing since the script has not changed and it ran earlier, I will try this one again tonight as well

02 was not updated yesterday. it seems the preprocessing fix might have made introduced some changes in that code. I shall fix it today

[rahulkulhalli]

The models in 03 run for me, though they output all 1's, is this a consequence of the dataset (nicr) that I'm using and how small it is?

Yes, due to the small data size in NICR, they are overfitting. If you look at the cosine similarities, even they are very high.