How does Peril trigger an installation event? #362

jlengstorf · 2018-08-27T15:48:06Z

Hi, team!

I'm trying to use the scheduler feature, but when I log the installations variable here:

peril/source/scheduler/startRepeatedTaskScheduler.ts

Lines 11 to 26 in ecd41ac

    
           export const runSchedule = async () => { 
        
             const db = getDB() 
        
             const installations = await db.getSchedulableInstallations() 
        
             installations.forEach(installation => { 
        
               if (installation && Object.keys(installation.scheduler)) { 
        
                 // Loop through the object's properties and set up the scheduler 
        
                 for (const cronTask in installation.scheduler) { 
        
                   if (installation.scheduler.hasOwnProperty(cronTask)) { 
        
                     const task = installation.scheduler[cronTask] 
        
                     scheduleJob(cronTask, () => runTaskForInstallation(installation, task, {})) 
        
                   } 
        
                 } 
        
               } 
        
             }) 
        
           }

I get [ null ], which I'm assuming means an installation was never created.

Looking at how installations are created, I see this:

peril/source/plugins/installationLifeCycle.ts

Lines 16 to 22 in 2591f3d

    
           // Create a db entry for any new installation 
        
           if (action === "created") { 
        
             logger.info("") 
        
             logger.info(`## Creating new installation for ${request.installation.account.login}`) 
        
             createInstallation(installation, req, res) 
        
             return true 
        
           }

But I'm not super clear on how to send the created action to get an installation created. Did I miss a step somewhere, or is there a doc that explains how to manually trigger an installation creation?

It would also be great to get an idea of the general lifecycle: should the installation be checked/created on every Peril run, or only during a very specific phase?

Thanks for all your help!

The text was updated successfully, but these errors were encountered:

orta · 2018-08-28T12:00:09Z

I think you're running Peril entirely via the JSON repo settings, instead of in mongo - which means the installation should only be null during the setup call on launch. As to get all schedulable installations should just return your singleton installation.

This is likely a concurrency issue, which should get solved this week.

TBH, I'd wait it out on using this feature for maybe a week or two, it's next up on my list for pretty drastic refactoring - #363 #361 because I want to avoid keeping every installation's scheduler settings in memory.

That said, I am interested in how often you were planning to schedule something, in #361 I spec'd out a few of the examples that I think I wanted but I'd love to know If that covers your cases too

jlengstorf · 2018-08-28T15:35:27Z

@orta I have a MongoDB setup in the Heroku instance, and the startup shows me that everything is running:

2018-08-28T00:18:36.659527+00:00 app[web.1]: info: ☢️  Starting up Peril
2018-08-28T00:18:36.659665+00:00 app[web.1]: info: ⅹ Papertrail
2018-08-28T00:18:36.659756+00:00 app[web.1]: info: ✓ Clustering
2018-08-28T00:18:36.659856+00:00 app[web.1]: info: ✓ Mongo at ds133152.mlab.com:33152
2018-08-28T00:18:36.659945+00:00 app[web.1]: info: ✓ Task Scheduler
2018-08-28T00:18:36.660033+00:00 app[web.1]: info: ✓ Server:
2018-08-28T00:18:36.660182+00:00 app[web.1]: info:   - Local: http://localhost:38743
2018-08-28T00:18:36.754191+00:00 app[web.1]: info:   - ✓ Agenda Task Scheduler

I can't figure out where the installation is actually intended to be created, but it seems like I either did something out of sequence or the installation event never fires.

I'll add our use cases to #361 as well.

Thanks!

orta · 2018-08-28T16:52:18Z

OK - yeah, so those installation create/updates get triggered right at the beginning of setup

You can go to something like: http://github.com/organizations/gatsbyjs/settings/apps/ - find peril, find you installation of peril the app uninstall and re-install it

the page should look something like this:

That will trigger the right web hooks from GH to create the installation. I think I should add some better logs for that. Is it possible that your MongoDB instance is empty for installations completely?

jlengstorf · 2018-08-28T19:54:48Z

@orta Yeah, there's nothing in the DB.

I'll do the uninstall/reinstall here, but maybe this would make a good check going forward, because I think what happened is I set up Peril, installed it for Gatsby, then realized scheduling was possible and added the MongoDB add-on. I imagine that'll be fairly common for people as they try out, then expand their Peril setup.

I'm not exactly sure how this would work, but in pseudo-code, something like:

if (!validInstallation && hasMongoDB) {
  // try to create an installation
}

I'm making a lot of assumptions about how that could work, so take that with a grain of salt. 😄

orta · 2018-08-28T19:57:00Z

Hah - yeah, I'm in this weird state where I can start making a single peril instance for everyone to use any one of these days, I just need to feel like my ideas are all complete - so hopefully all that faff can be managed by me.

Agree though - that's a pretty easy check I'll add - kinda surprised you got this far without it, but I guess if you aren't using the separate docker-based runner it doesn't need to do a lot of the harder installation stuff

jlengstorf · 2018-08-28T20:06:07Z

Our installation is currently just running a few rules, but I do see occasional crashes that probably roll up to having that data missing. I'll let you know if I keep seeing those after this uninstall/reinstall.

jlengstorf · 2018-08-28T20:11:54Z

I'm seeing 404s when I try to uninstall/reinstall.

The error is:

{"error":"WIP - not built out support for non-repo related events - sorry","iID":310133}

Did I miss a step somewhere?

orta · 2018-08-28T20:13:43Z

I don't think so, I refactored that a few months back and maybe it's possible that installation events are getting through into the danger runner - likely a case I didn't cover, it should still make the installation in mongo as that's before that code

jlengstorf · 2018-08-28T20:15:28Z

My mLab instance is still showing no collections except an empty agendaJobs. Is there something I can log or dump that'll help identify what's not connecting?

orta · 2018-08-28T20:19:21Z

The event it hooks up with is called "installation" in master - above you're referencing integration_installation - maybe GitHub changed their webhook naming structure

orta · 2018-08-28T20:21:56Z

OK, looks like Peril should be ignoring integration_installation webhooks but not installation

jlengstorf · 2018-08-28T20:28:47Z

Ah, okay — both were fired and I got distracted by the 404. There's an installation event as well. Now I'm seeing a different error:

2018-08-28T20:23:39.606260+00:00 app[web.1]: info: ## Creating new installation for gatsbyjs
2018-08-28T20:23:39.613751+00:00 app[web.1]: Tue, 28 Aug 2018 20:23:39 GMT express deprecated res.send(status, body): Use res.status(status).send(body) instead at out/github/events/create_installation.js:14:13
2018-08-28T20:23:39.627429+00:00 app[web.1]: error: UnhandledRejection Error:  Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
2018-08-28T20:23:39.627433+00:00 app[web.1]: at validateHeader (_http_outgoing.js:503:11)
2018-08-28T20:23:39.627435+00:00 app[web.1]: at ServerResponse.setHeader (_http_outgoing.js:510:3)
2018-08-28T20:23:39.627438+00:00 app[web.1]: at ServerResponse.header (/app/node_modules/express/lib/response.js:767:10)
2018-08-28T20:23:39.627439+00:00 app[web.1]: at ServerResponse.json (/app/node_modules/express/lib/response.js:264:10)
2018-08-28T20:23:39.627441+00:00 app[web.1]: at ServerResponse.send (/app/node_modules/express/lib/response.js:158:21)
2018-08-28T20:23:39.627450+00:00 app[web.1]: at Object.exports.githubDangerRunner (/app/out/github/events/github_runner.js:92:14)
2018-08-28T20:23:39.627452+00:00 app[web.1]: at <anonymous>
2018-08-28T20:23:39.627454+00:00 app[web.1]: at process._tickCallback (internal/process/next_tick.js:182:7)
2018-08-28T20:23:39.632648+00:00 app[web.1]: /app/out/peril.js:25
2018-08-28T20:23:39.632651+00:00 app[web.1]: throw reason;
2018-08-28T20:23:39.632653+00:00 app[web.1]: ^
2018-08-28T20:23:39.632654+00:00 app[web.1]:
2018-08-28T20:23:39.632656+00:00 app[web.1]: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
2018-08-28T20:23:39.632658+00:00 app[web.1]: at validateHeader (_http_outgoing.js:503:11)
2018-08-28T20:23:39.632660+00:00 app[web.1]: at ServerResponse.setHeader (_http_outgoing.js:510:3)
2018-08-28T20:23:39.632662+00:00 app[web.1]: at ServerResponse.header (/app/node_modules/express/lib/response.js:767:10)
2018-08-28T20:23:39.632663+00:00 app[web.1]: at ServerResponse.json (/app/node_modules/express/lib/response.js:264:10)
2018-08-28T20:23:39.632665+00:00 app[web.1]: at ServerResponse.send (/app/node_modules/express/lib/response.js:158:21)
2018-08-28T20:23:39.632666+00:00 app[web.1]: at Object.exports.githubDangerRunner (/app/out/github/events/github_runner.js:92:14)
2018-08-28T20:23:39.632668+00:00 app[web.1]: at <anonymous>
2018-08-28T20:23:39.632669+00:00 app[web.1]: at process._tickCallback (internal/process/next_tick.js:182:7)

Presumably this is happening here?

peril/source/github/events/create_installation.ts

Lines 17 to 22 in d981dd7

    
           if (existingInstallation) { 
        
             res.send(204, "Did not create new installation, it already existed.") 
        
           } else { 
        
             await db.saveInstallation(installation) 
        
             res.send(200, "Creating new installation.") 
        
           }

I'm pushing up some logging to figure out if I'm right.

orta · 2018-08-28T20:32:53Z

I think so - yeah, I think #365 might fix that crash. It returns right after the calls res.sends now

jlengstorf · 2018-08-28T20:43:12Z

My logging is coming back confusing.

2018-08-28T20:31:15.669854+00:00 app[web.1]: info: ☢️  Starting up Peril
2018-08-28T20:31:15.670070+00:00 app[web.1]: info: ⅹ Papertrail
2018-08-28T20:31:15.670183+00:00 app[web.1]: info: ✓ Clustering
2018-08-28T20:31:15.670314+00:00 app[web.1]: info: ✓ Mongo at ds133152.mlab.com:33152
2018-08-28T20:31:15.670424+00:00 app[web.1]: info: ✓ Task Scheduler
2018-08-28T20:31:15.670530+00:00 app[web.1]: info: ✓ Server:
2018-08-28T20:31:15.670698+00:00 app[web.1]: info:   - Local: http://localhost:8179
2018-08-28T20:31:15.808861+00:00 app[web.1]: info:   - ✓ Agenda Task Scheduler
2018-08-28T20:31:15.762925+00:00 heroku[web.1]: State changed from starting to up
2018-08-28T20:33:37.879656+00:00 heroku[router]: at=info method=POST path="/webhook" host=gatsby-peril.herokuapp.com request_id=64f7c0a2-f3db-4cb3-bd43-93930c7ee97b fwd="192.30.252.34" dyno=web.1 connect=1ms service=14ms status=204 bytes=148 protocol=https
2018-08-28T20:33:37.869504+00:00 app[web.1]: info:
2018-08-28T20:33:37.869729+00:00 app[web.1]: info: ## Creating new installation for gatsbyjs
2018-08-28T20:33:37.872189+00:00 app[web.1]: Installation already exists.
2018-08-28T20:33:37.876218+00:00 app[web.1]: { iID: 310133,
2018-08-28T20:33:37.876221+00:00 app[web.1]: login: 'unknown',
2018-08-28T20:33:37.876223+00:00 app[web.1]: avatarURL: '',
2018-08-28T20:33:37.876225+00:00 app[web.1]: repos: {},
2018-08-28T20:33:37.876226+00:00 app[web.1]: rules:
2018-08-28T20:33:37.876228+00:00 app[web.1]: { 'issues.opened': [ 'rules/emptybody.js', 'rules/labeler.ts' ],
2018-08-28T20:33:37.876230+00:00 app[web.1]: 'pull_request.opened': [ 'rules/emptybody.js' ],
2018-08-28T20:33:37.876235+00:00 app[web.1]: 'pull_request.closed (pull_request.merged == true)': [ 'rules/invite-collaborator.js' ] },
2018-08-28T20:33:37.876237+00:00 app[web.1]: scheduler: { '30 * * * * ?': 'slack-test' },
2018-08-28T20:33:37.876239+00:00 app[web.1]: tasks: { 'slack-test': 'gatsbyjs/peril-gatsbyjs@tasks/slacktest.js' },
2018-08-28T20:33:37.876240+00:00 app[web.1]: perilSettingsJSONURL: 'gatsbyjs/[email protected]',
2018-08-28T20:33:37.876242+00:00 app[web.1]: installationSlackUpdateWebhookURL: undefined,
2018-08-28T20:33:37.876243+00:00 app[web.1]: settings:
2018-08-28T20:33:37.876245+00:00 app[web.1]: { env_vars: [ 'SLACK_WEBHOOK_URL' ],
2018-08-28T20:33:37.876247+00:00 app[web.1]: ignored_repos: [],
2018-08-28T20:33:37.876248+00:00 app[web.1]: modules: [ '@slack/client' ] } }
2018-08-28T20:33:37.877584+00:00 app[web.1]: Tue, 28 Aug 2018 20:33:37 GMT express deprecated res.send(status, body): Use res.status(status).send(body) instead at out/github/events/create_installation.js:16:13
2018-08-28T20:33:37.880605+00:00 app[web.1]: error: UnhandledRejection Error:  Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
2018-08-28T20:33:37.880608+00:00 app[web.1]: at validateHeader (_http_outgoing.js:503:11)
2018-08-28T20:33:37.880609+00:00 app[web.1]: at ServerResponse.setHeader (_http_outgoing.js:510:3)
2018-08-28T20:33:37.880611+00:00 app[web.1]: at ServerResponse.header (/app/node_modules/express/lib/response.js:767:10)
2018-08-28T20:33:37.880613+00:00 app[web.1]: at ServerResponse.json (/app/node_modules/express/lib/response.js:264:10)
2018-08-28T20:33:37.880614+00:00 app[web.1]: at ServerResponse.send (/app/node_modules/express/lib/response.js:158:21)
2018-08-28T20:33:37.880616+00:00 app[web.1]: at Object.exports.githubDangerRunner (/app/out/github/events/github_runner.js:92:14)
2018-08-28T20:33:37.880618+00:00 app[web.1]: at <anonymous>
2018-08-28T20:33:37.880619+00:00 app[web.1]: at process._tickCallback (internal/process/next_tick.js:182:7)
2018-08-28T20:33:37.881006+00:00 app[web.1]: /app/out/peril.js:25
2018-08-28T20:33:37.881009+00:00 app[web.1]: throw reason;
2018-08-28T20:33:37.881011+00:00 app[web.1]: ^
2018-08-28T20:33:37.881012+00:00 app[web.1]:
2018-08-28T20:33:37.881014+00:00 app[web.1]: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
2018-08-28T20:33:37.881016+00:00 app[web.1]: at validateHeader (_http_outgoing.js:503:11)
2018-08-28T20:33:37.881017+00:00 app[web.1]: at ServerResponse.setHeader (_http_outgoing.js:510:3)
2018-08-28T20:33:37.881019+00:00 app[web.1]: at ServerResponse.header (/app/node_modules/express/lib/response.js:767:10)
2018-08-28T20:33:37.881021+00:00 app[web.1]: at ServerResponse.json (/app/node_modules/express/lib/response.js:264:10)
2018-08-28T20:33:37.881023+00:00 app[web.1]: at ServerResponse.send (/app/node_modules/express/lib/response.js:158:21)
2018-08-28T20:33:37.881024+00:00 app[web.1]: at Object.exports.githubDangerRunner (/app/out/github/events/github_runner.js:92:14)
2018-08-28T20:33:37.881026+00:00 app[web.1]: at <anonymous>
2018-08-28T20:33:37.881028+00:00 app[web.1]: at process._tickCallback (internal/process/next_tick.js:182:7)
2018-08-28T20:33:41.769066+00:00 app[web.1]: info:
2018-08-28T20:33:41.771829+00:00 app[web.1]: (node:36) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.

This says the installation already exists, but it's missing the repos and other details, plus I can't find it in the MongoDB. I've verified that the MongoDB ID in the logs matches up with what I'm looking at, so I don't know what's going on over here.

jlengstorf · 2018-08-28T21:00:50Z

The changes in #365 fixed the crash. Still trying to figure out why my installation isn't in Mongo, but says it already exists.

orta · 2018-08-28T21:21:12Z

I think it's possible/likely that DATABASE_JSON_FILE is set in the env, which triggers you to be running from the JSON db instead of stored in mongo.

You can remove that, re-trigger the integration, then edit the JSON for the installation in Mongo to have a settings URL - but all of that feels bike-sheddy. It's an interesting problem, and I don't want crashes, but I'm reasonably sure that it's off-topic

I still feel like the original problem lays in the async-ness of the launch setup between grabbing the JSON and storing it in-memory and the scheduler starting up. I'll fix this, but it'll be part of #361 which is a bit more involved but will work across my staging/prod builds and self-hosted

The reasonable quick hack is just to throw a 5 second setInterval before launching the scheduler check :D

jlengstorf · 2018-08-28T21:52:12Z

Does that mean that if you have a MongoDB instance you can't configure the rules/tasks using a Peril settings file in GitHub?

I'm happy to hold off scheduler stuff if this is on your roadmap, but to be clear, the workflow I'm hoping we can follow is:

Peril is hosted on Heroku
Our settings are configured at https://github.com/gatsbyjs/peril-gatsbyjs/ in peril.settings.json
The tasks configured under scheduler will be picked up and run by our Peril instance

Am I correct that this currently won't work due to the requirement of DATABASE_JSON_FILE to load the peril.settings.json, and the necessity of relying on MongoDB for the scheduler? Using these is currently mutually exclusive?

Thanks for all your help today!

orta · 2018-08-29T01:06:24Z

Am I correct that this currently won't work due to the requirement of DATABASE_JSON_FILE to load the peril.settings.json, and the necessity of relying on MongoDB for the scheduler? Using these is currently mutually exclusive?

Nope, I think this isn't working because:

This is likely a concurrency issue, which should get solved this week.

Just dodgy hacky code - the rest of it was just answering your questions and trying to not make peril crash.

That setup should work (if you change "30 * * * * ?" to "daily") with the refactors, it's how I anticipated folks to also use it

orta mentioned this issue Aug 28, 2018

Reshuffles the routing around in Peril #365

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How does Peril trigger an installation event? #362

How does Peril trigger an installation event? #362

jlengstorf commented Aug 27, 2018

orta commented Aug 28, 2018 •

edited

Loading

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 29, 2018

How does Peril trigger an installation event? #362

How does Peril trigger an installation event? #362

Comments

jlengstorf commented Aug 27, 2018

orta commented Aug 28, 2018 • edited Loading

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 28, 2018

jlengstorf commented Aug 28, 2018

orta commented Aug 29, 2018

orta commented Aug 28, 2018 •

edited

Loading