Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cb ceiling middle tier plots #1050

Closed
wants to merge 13 commits into from
Closed

Cb ceiling middle tier plots #1050

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
9c6caba
valid time station plot is working
Aug 10, 2023
f2801b7
ValidTime all stations for 1 month in 33 seconds
Aug 10, 2023
e377187
Merge branch 'development' into cb_ceiling_middle_tier_plots
Aug 14, 2023
671e053
fixing template issue for countour, countour-diff perf
Aug 15, 2023
557a856
Map - all stations for 1 month in 2 minutes
Aug 29, 2023
bc2fb09
subDocType = CEILING to templates
Sep 5, 2023
e348096
partial merge - merge middleware code for multiple plot types #1036
Sep 6, 2023
0946b69
updating subrepo pointers
Sep 6, 2023
09c1ff1
update subrepo
Sep 6, 2023
6498517
run format
Sep 6, 2023
af3ad5b
updating sub-repo pointers
Sep 7, 2023
1ab0f09
removing memoryUsage import
Sep 8, 2023
ad17c43
MATS submodule conflict resolution
mollybsmith-noaa Sep 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion MATScommon
Open in desktop
Submodule MATScommon updated 9 files
+55 −47 meteor_packages/mats-common/imports/startup/server/data_query_util.js
+48 −2 meteor_packages/mats-common/imports/startup/server/matsMiddle_common.js
+356 −0 meteor_packages/mats-common/imports/startup/server/matsMiddle_dailyModelCycle.js
+20 −51 meteor_packages/mats-common/imports/startup/server/matsMiddle_dieoff.js
+436 −0 meteor_packages/mats-common/imports/startup/server/matsMiddle_map.js
+25 −49 meteor_packages/mats-common/imports/startup/server/matsMiddle_timeSeries.js
+20 −72 meteor_packages/mats-common/imports/startup/server/matsMiddle_ts_dieoff.js
+392 −0 meteor_packages/mats-common/imports/startup/server/matsMiddle_validTime.js
+6 −0 meteor_packages/mats-common/package.js
24 changes: 12 additions & 12 deletions apps/cb-ceiling/server/dataFunctions/data_contour.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@
} from "meteor/randyp:mats-common";
import { moment } from "meteor/momentjs:moment";

dataContour = function (plotParams, plotFunction) {

Check warning on line 15 in apps/cb-ceiling/server/dataFunctions/data_contour.js

View workflow job for this annotation

GitHub Actions / lint (cb-ceiling) 

'dataContour' is not defined
const fs = require("fs");

Check warning on line 16 in apps/cb-ceiling/server/dataFunctions/data_contour.js

View workflow job for this annotation

GitHub Actions / lint (cb-ceiling) 

Unexpected require()
// initialize variables common to all curves
const appParams = {
plotType: matsTypes.PlotTypes.contour,
Expand Down Expand Up @@ -46,7 +46,7 @@
// initialize variables specific to the curve
const curve = curves[0];

queryTemplate = fs.readFileSync("assets/app/sqlTemplates/tmpl_Contour.sql", "utf8");

Check warning on line 49 in apps/cb-ceiling/server/dataFunctions/data_contour.js

View workflow job for this annotation

GitHub Actions / lint (cb-ceiling) 

'queryTemplate' is not defined

const { label } = curve;
const { variable } = curve;
Expand All @@ -60,7 +60,7 @@
`INFO: ${label}'s threshold is undefined. Please assign it a value.`
);
}
var threshold = Object.keys(

Check warning on line 63 in apps/cb-ceiling/server/dataFunctions/data_contour.js

View workflow job for this annotation

GitHub Actions / lint (cb-ceiling) 

All 'var' declarations must be at the top of the function scope

Check warning on line 63 in apps/cb-ceiling/server/dataFunctions/data_contour.js

View workflow job for this annotation

GitHub Actions / lint (cb-ceiling) 

Unexpected var, use let or const instead
matsCollections.threshold.findOne({ name: "threshold" }).valuesMap[variable]
).find(
(key) =>
Expand All @@ -73,15 +73,15 @@
if (xAxisParam !== "Valid UTC hour" && yAxisParam !== "Valid UTC hour") {
const validTimes = curve["valid-time"] === undefined ? [] : curve["valid-time"];
if (validTimes.length !== 0 && validTimes !== matsTypes.InputTypes.unused) {
queryTemplate = queryTemplate.replace(

Check warning on line 76 in apps/cb-ceiling/server/dataFunctions/data_contour.js

View workflow job for this annotation

GitHub Actions / lint (cb-ceiling) 

'queryTemplate' is not defined

Check warning on line 76 in apps/cb-ceiling/server/dataFunctions/data_contour.js

View workflow job for this annotation

GitHub Actions / lint (cb-ceiling) 

'queryTemplate' is not defined
/vxVALID_TIMES/g,
cbPool.trfmListToCSVString(validTimes, null, false)
);
} else {
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "vxVALID_TIMES");
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "{{vxVALID_TIMES}}");
}
} else {
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "vxVALID_TIMES");
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "{{vxVALID_TIMES}}");
}

if (xAxisParam !== "Fcst lead time" && yAxisParam !== "Fcst lead time") {
Expand All @@ -91,9 +91,9 @@
`INFO: ${label}'s forecast lead time is undefined. Please assign it a value.`
);
}
queryTemplate = queryTemplate.replace(/vxFCST_LEN/g, forecastLength);
queryTemplate = queryTemplate.replace(/{{vxFCST_LEN}}/g, forecastLength);
} else {
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "vxFCST_LEN");
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "{{vxFCST_LEN}}");
}

if (
Expand All @@ -105,7 +105,7 @@
} else {
dateString = "m0.fcstValidEpoch";
}
queryTemplate = queryTemplate.replace(/vxDATE_STRING/g, dateString);
queryTemplate = queryTemplate.replace(/{{vxDATE_STRING}}/g, dateString);

const regionType = curve["region-type"];
if (regionType === "Select stations") {
Expand All @@ -126,13 +126,13 @@
{ optionsMap: 1 }
).optionsMap;

queryTemplate = queryTemplate.replace(/vxFROM_SECS/g, fromSecs);
queryTemplate = queryTemplate.replace(/vxTO_SECS/g, toSecs);
queryTemplate = queryTemplate.replace(/vxTHRESHOLD/g, threshold);
queryTemplate = queryTemplate.replace(/vxREGION/g, region);
queryTemplate = queryTemplate.replace(/vxMODEL/g, model);
queryTemplate = queryTemplate.replace(/vxXVAL_CLAUSE/g, xValClause);
queryTemplate = queryTemplate.replace(/vxYVAL_CLAUSE/g, yValClause);
queryTemplate = queryTemplate.replace(/{{vxFROM_SECS}}/g, fromSecs);
queryTemplate = queryTemplate.replace(/{{vxTO_SECS}}/g, toSecs);
queryTemplate = queryTemplate.replace(/{{vxTHRESHOLD}}/g, threshold);
queryTemplate = queryTemplate.replace(/{{vxREGION}}/g, region);
queryTemplate = queryTemplate.replace(/{{vxMODEL}}/g, model);
queryTemplate = queryTemplate.replace(/{{vxXVAL_CLAUSE}}/g, xValClause);
queryTemplate = queryTemplate.replace(/{{vxYVAL_CLAUSE}}/g, yValClause);

// For contours, this functions as the colorbar label.
const statType = statisticOptionsMap[statisticSelect][0];
Expand Down
26 changes: 13 additions & 13 deletions apps/cb-ceiling/server/dataFunctions/data_contour_diff.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,14 +79,14 @@ dataContourDiff = function (plotParams, plotFunction) {
const validTimes = curve["valid-time"] === undefined ? [] : curve["valid-time"];
if (validTimes.length !== 0 && validTimes !== matsTypes.InputTypes.unused) {
queryTemplate = queryTemplate.replace(
/vxVALID_TIMES/g,
/{{vxVALID_TIMES}}/g,
cbPool.trfmListToCSVString(validTimes, null, false)
);
} else {
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "vxVALID_TIMES");
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "{{vxVALID_TIMES}}");
}
} else {
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "vxVALID_TIMES");
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "{{vxVALID_TIMES}}");
}

if (xAxisParam !== "Fcst lead time" && yAxisParam !== "Fcst lead time") {
Expand All @@ -96,9 +96,9 @@ dataContourDiff = function (plotParams, plotFunction) {
`INFO: ${label}'s forecast lead time is undefined. Please assign it a value.`
);
}
queryTemplate = queryTemplate.replace(/vxFCST_LEN/g, forecastLength);
queryTemplate = queryTemplate.replace(/{{vxFCST_LEN}}/g, forecastLength);
} else {
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "vxFCST_LEN");
queryTemplate = cbPool.trfmSQLRemoveClause(queryTemplate, "{{vxFCST_LEN}}");
}

if (
Expand All @@ -110,7 +110,7 @@ dataContourDiff = function (plotParams, plotFunction) {
} else {
dateString = "m0.fcstValidEpoch";
}
queryTemplate = queryTemplate.replace(/vxDATE_STRING/g, dateString);
queryTemplate = queryTemplate.replace(/{{vxDATE_STRING}}/g, dateString);

const regionType = curve["region-type"];
if (regionType === "Select stations") {
Expand All @@ -131,13 +131,13 @@ dataContourDiff = function (plotParams, plotFunction) {
{ optionsMap: 1 }
).optionsMap;

queryTemplate = queryTemplate.replace(/vxFROM_SECS/g, fromSecs);
queryTemplate = queryTemplate.replace(/vxTO_SECS/g, toSecs);
queryTemplate = queryTemplate.replace(/vxTHRESHOLD/g, threshold);
queryTemplate = queryTemplate.replace(/vxREGION/g, region);
queryTemplate = queryTemplate.replace(/vxMODEL/g, model);
queryTemplate = queryTemplate.replace(/vxXVAL_CLAUSE/g, xValClause);
queryTemplate = queryTemplate.replace(/vxYVAL_CLAUSE/g, yValClause);
queryTemplate = queryTemplate.replace(/{{vxFROM_SECS}}/g, fromSecs);
queryTemplate = queryTemplate.replace(/{{vxTO_SECS}}/g, toSecs);
queryTemplate = queryTemplate.replace(/{{vxTHRESHOLD}}/g, threshold);
queryTemplate = queryTemplate.replace(/{{vxREGION}}/g, region);
queryTemplate = queryTemplate.replace(/{{vxMODEL}}/g, model);
queryTemplate = queryTemplate.replace(/{{vxXVAL_CLAUSE}}/g, xValClause);
queryTemplate = queryTemplate.replace(/{{vxYVAL_CLAUSE}}/g, yValClause);

// For contours, this functions as the colorbar label.
var statType = statisticOptionsMap[statisticSelect][0];
Expand Down
84 changes: 46 additions & 38 deletions apps/cb-ceiling/server/dataFunctions/data_dailymodelcycle.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
matsDataDiffUtils,
matsDataCurveOpsUtils,
matsDataProcessUtils,
matsMiddleValidTime,
} from "meteor/randyp:mats-common";
import { moment } from "meteor/momentjs:moment";

Expand Down Expand Up @@ -42,6 +43,7 @@ dataDailyModelCycle = function (plotParams, plotFunction) {
let xmin = Number.MAX_VALUE;
let ymin = Number.MAX_VALUE;
const idealValues = [];
let statement = "";

for (let curveIndex = 0; curveIndex < curvesLength; curveIndex++) {
// initialize variables specific to each curve
Expand All @@ -54,23 +56,13 @@ dataDailyModelCycle = function (plotParams, plotFunction) {
"assets/app/sqlTemplates/tmpl_DailyModelCycle_region.sql",
"utf8"
);
} else {
queryTemplate = fs.readFileSync(
"assets/app/sqlTemplates/tmpl_DailyModelCycle_stations.sql",
"utf8"
);
}
console.log(`\nqueryTemplate:\n${queryTemplate}`);

queryTemplate = queryTemplate.replace(/{{vxFROM_SECS}}/g, fromSecs);
queryTemplate = queryTemplate.replace(/{{vxTO_SECS}}/g, toSecs);

const { diffFrom } = curve;
const { label } = curve;
var { variable } = curve;
const model = matsCollections["data-source"].findOne({ name: "data-source" })
.optionsMap[variable][curve["data-source"]][0];
queryTemplate = queryTemplate.replace(/{{vxMODEL}}/g, model);
var thresholdStr = curve.threshold;
let threshold = Object.keys(
matsCollections.threshold.findOne({ name: "threshold" }).valuesMap[variable]
Expand All @@ -81,18 +73,13 @@ dataDailyModelCycle = function (plotParams, plotFunction) {
] === thresholdStr
);
threshold = threshold.replace(/_/g, ".");
queryTemplate = queryTemplate.replace(/{{vxTHRESHOLD}}/g, threshold);
if (curve["utc-cycle-start"].length !== 1) {
throw new Error(
"INFO: Please select exactly one UTC Cycle Init Hour for this plot type."
);
}
const utcCycleStart = Number(curve["utc-cycle-start"][0]);
utcCycleStarts[curveIndex] = utcCycleStart;
queryTemplate = queryTemplate.replace(
/{{vxUTC_CYCLE_START}}/g,
cbPool.trfmListToCSVString(utcCycleStart, null, false)
);
const statisticSelect = curve.statistic;
const statisticOptionsMap = matsCollections.statistic.findOne(
{ name: "statistic" },
Expand All @@ -109,18 +96,17 @@ dataDailyModelCycle = function (plotParams, plotFunction) {
regionStr
);
queryTemplate = queryTemplate.replace(/{{vxREGION}}/g, region);
queryTemplate = queryTemplate.replace(/{{vxFROM_SECS}}/g, fromSecs);
queryTemplate = queryTemplate.replace(/{{vxTO_SECS}}/g, toSecs);
queryTemplate = queryTemplate.replace(/{{vxMODEL}}/g, model);
queryTemplate = queryTemplate.replace(/{{vxTHRESHOLD}}/g, threshold);
queryTemplate = queryTemplate.replace(
/{{vxUTC_CYCLE_START}}/g,
cbPool.trfmListToCSVString(utcCycleStart, null, false)
);
} else {
const sitesList = curve.sites === undefined ? [] : curve.sites;
if (sitesList.length > 0 && sitesList !== matsTypes.InputTypes.unused) {
queryTemplate = queryTemplate.replace(
/{{vxSITES_LIST_OBS}}/g,
cbPool.trfmListToCSVString(sitesList, "obs.data.", false)
);
queryTemplate = queryTemplate.replace(
/{{vxSITES_LIST_MODELS}}/g,
cbPool.trfmListToCSVString(sitesList, "models.data.", false)
);
} else {
sitesList = curve.sites === undefined ? [] : curve.sites;
if (sitesList.length === 0 && sitesList === matsTypes.InputTypes.unused) {
throw new Error(
"INFO: Please add sites in order to get a single/multi station plot."
);
Expand All @@ -140,23 +126,45 @@ dataDailyModelCycle = function (plotParams, plotFunction) {

var d;
if (!diffFrom) {
// this is a database driven curve, not a difference curve
// prepare the query from the above parameters
statement = cbPool.trfmSQLForDbTarget(queryTemplate);

dataRequests[label] = statement;

var queryResult;
const startMoment = moment();
var finishMoment;

try {
// send the query statement to the query function
queryResult = matsDataQueryUtils.queryDBSpecialtyCurve(
cbPool,
statement,
appParams,
statisticSelect
);
if (regionType === "Predefined region") {
// this is a database driven curve, not a difference curve
// send the query statement to the query function
statement = cbPool.trfmSQLForDbTarget(queryTemplate);
queryResult = matsDataQueryUtils.queryDBSpecialtyCurve(
cbPool,
statement,
appParams,
statisticSelect
);
} else {
// send to matsMiddle
const tss = new matsMiddleDailyModelCycle.MatsMiddleDailyModelCycle(cbPool);

const rows = tss.processStationQuery(
"Ceiling",
sitesList,
model,
threshold,
fromSecs,
toSecs,
utcCycleStart
);

// send the query statement to the query function
queryResult = matsDataQueryUtils.queryDBSpecialtyCurve(
cbPool,
rows,
appParams,
statisticSelect
);
}

finishMoment = moment();
dataRequests[`data retrieval (query) time - ${label}`] = {
begin: startMoment.format(),
Expand Down
23 changes: 19 additions & 4 deletions apps/cb-ceiling/server/dataFunctions/data_map.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
matsDataQueryUtils,
matsDataCurveOpsUtils,
matsDataPlotOpsUtils,
matsMiddleMap,
} from "meteor/randyp:mats-common";
import { moment } from "meteor/momentjs:moment";

Expand Down Expand Up @@ -88,23 +89,37 @@ dataMap = function (plotParams, plotFunction) {
"INFO: Please add sites in order to get a single/multi station plot."
);
}
statement = cbPool.trfmSQLForDbTarget(queryTemplate);

dataRequests[label] = statement;

let queryResult;
const startMoment = moment();
let finishMoment;
const statement = "";

try {
// send to matsMiddle
const tss = new matsMiddleMap.MatsMiddleMap(cbPool);

const rows = tss.processStationQuery(
"Ceiling",
sitesList,
model,
forecastLength,
threshold,
fromSecs,
toSecs,
validTimes
);

// send the query statement to the query function
queryResult = matsDataQueryUtils.queryDBMapCTC(
cbPool,
statement,
rows,
model,
statistic,
siteMap,
appParams
);

finishMoment = moment();
dataRequests[`data retrieval (query) time - ${label}`] = {
begin: startMoment.format(),
Expand Down
2 changes: 2 additions & 0 deletions apps/cb-ceiling/server/dataFunctions/data_series.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ dataSeries = function (plotParams, plotFunction) {
try {
if (regionType === "Predefined region") {
statement = cbPool.trfmSQLForDbTarget(queryTemplate);

// send the query statement to the query function
queryResult = matsDataQueryUtils.queryDBTimeSeries(
cbPool,
Expand All @@ -166,6 +167,7 @@ dataSeries = function (plotParams, plotFunction) {
} else {
// send to matsMiddle
const tss = new matsMiddleTimeSeries.MatsMiddleTimeSeries(cbPool);

const rows = tss.processStationQuery(
"Ceiling",
sitesList,
Expand Down
Loading
Loading