mintable.js Refactor (#26)

* Fix progress indicator for account setup * Refactor Google & Plaid code to support multiple months & pagination * Refactor Google & Plaid code to support multiple months & pagination * Batch format & clear requests for sheets * Documentation improvements * Documentation improvements * Documentation improvements * Documentation improvements * Documentation improvements * Bump Version
kevinschaich · Apr 16, 2019 · 1a1bcb0 · 1a1bcb0
1 parent 51a87a3
commit 1a1bcb0
Show file tree

Hide file tree

Showing 13 changed files with 500 additions and 288 deletions.
diff --git a/README.md b/README.md
@@ -56,11 +56,11 @@ You can see a full list of options in the **[Config Docs](./docs/CONFIG.md)**.
 **How is this different from [build-your-own-mint](https://github.com/yyx990803/build-your-own-mint)?**
 
 - **[build-your-own-mint](https://github.com/yyx990803/build-your-own-mint)** is a set of scripts which solely facilitates the integration between Plaid and Google Sheets. It makes no assumptions about what you want your spreadsheet to look like, and you have to define your own logic to map transactions to spreadsheet updates.
-- **[Mintable](#)** is and end-to-end system that works out of the box. It comes with a setup wizard, a web-based configuration server, pluggable providers (you're not limited to just Plaid & Google Sheets), and a spreadsheet template.
+- **[Mintable](#)** is and end-to-end system that works out of the box. It comes with a setup wizard, a web-based configuration server, [pluggable providers](./docs/PROVIDERS.md) (you're not limited to just Plaid & Google Sheets), and a spreadsheet template.
 
 **Do I have to give my data to Plaid and Google? Are there any completely self-hosted alternatives I can use?**
 
-- It's pluggable! Plaid & Google Sheets are working right now – contributions are welcome for other providers!
+- It's [pluggable](./docs/PROVIDERS.md)! Plaid & Google Sheets are working right now – contributions are welcome for [other providers](./docs/PROVIDERS.md)!
 
 **Do I have to manually run this every time I want new transactions in my spreadsheet?**
 

diff --git a/docs/CONFIG.md b/docs/CONFIG.md
@@ -1,12 +1,12 @@
-## Configuration
+# Configuration
 
 All configurations below can be made using the web configuration framework or by editing `mintable.config.json`.
 
-`mintable.config.json` is the secret sauce – it contains all of your private tokens and is never sent to third-party servers. This file is ignored by Git – keep a backup somewhere safe as you only have 100 Plaid accounts on the free version.
+`mintable.config.json` is the secret sauce – it contains all of your private tokens and is never sent to third-party servers. This file is ignored by Git – keep a backup somewhere safe.
 
 > **Pro Tip:** You can use Dropbox or another trusted service to sync `mintable.config.json` across your machines. Run `ln -s <path_to_cloud_folder>/mintable.config.json .` from the repo root to symlink Mintable to the cloud version.
 
-#### Automate Updates with a CI Provider
+### Automate Updates with a CI Provider
 
 This repo includes config files for both [CircleCI](https://circleci.com/) and [Travis CI](https://travis-ci.com) to run builds automatically.
 
@@ -18,53 +18,156 @@ yarn export
 
 Run this command and paste the result into an environment variable called `MINTABLE_CONFIG` in your CI provider of choice. Mintable will handle the rest.
 
-> **Note:** Some CI providers (like Travis) require you to wrap this variable in single quotes, i.e. `'{ PLAID_TOKEN: "" ...}'`. If you get an error similar to `Unable to parse JSON...` when you run your CI build, give this a try.
+> **Note:** Some CI providers (like Travis) require you to wrap this variable in single quotes, i.e. `'{ "ACCOUNT_PROVIDER": "plaid", ...}'`. If you get an error similar to `Unable to parse JSON...` when you run your CI build, give this a try.
 
 > **Warning:** If you choose to use CircleCI, you should turn off **Pass secrets to builds from forked pull requests** under **Build Settings** > **Advanced Settings**.
 
-#### Transaction Columns
+### Start Date
 
-`TRANSACTION_COLUMNS` specifies a list of [Plaid transaction properties](https://plaid.com/docs/#transactions) (using [`_.get()` syntax](https://lodash.com/docs/4.17.11#get)) to override the default automated columns. Each time you run Mintable, all the contents of these columns will be cleared and overwritten.
+`START_DATE` specifies the lower bound for fetching transactions in `YYYY.MM.DD` format.
 
-For example, if you only want to auto-populate the name and amount for each transaction, you could add the following line to your `mintable.config.json` file:
+**Default:**
+
+```javascript
+"START_DATE": undefined // If end date is not specified, Mintable will fetch the last 2 months of transactions
+```
+
+For example, if you only want to fetch transactions which occur after or on December 1, 2018, you could add the following line to your `mintable.config.json` file:
+
+```javascript
+"START_DATE": "2018.12.01"
+```
+
+### End Date
+
+`END_DATE` specifies the upper bound for fetching transactions in `YYYY.MM.DD` format.
+
+**Default:**
+
+```javascript
+"END_DATE": undefined // If end date is not specified, Mintable will fetch up until the current date
+```
+
+For example, if you only want to fetch transactions which occur before or on December 1, 2018, you could add the following line to your `mintable.config.json` file:
 
+```javascript
+"END_DATE": "2018.12.01"
 ```
-TRANSACTION_COLUMNS=["name", "amount"]
+
+### Transaction Columns
+
+`TRANSACTION_COLUMNS` specifies a list of transaction properties (using [`_.get()` syntax](https://lodash.com/docs/4.17.11#get)) to automatically update in your spreadsheet. All the contents of these columns will be cleared and overwritten each time you run Mintable.
+
+**Default:** 
+
+```javascript
+"TRANSACTION_COLUMNS": [ 'date', 'amount', 'name', 'account_details.official_name', 'category.0', 'category.1', 'pending' ]
 ```
 
-> **Warning:** Your mileage may vary if you choose to use additional properties outside the tested defaults (`date`, `amount`, `name`, `account`, `category.0`, `category.1`, `pending`). Proceed at your own risk, you're in uncharted territory.
+For example, if you only want to auto-populate the name and amount for each transaction, you could add the following line to your `mintable.config.json` file:
+
+```javascript
+"TRANSACTION_COLUMNS": ["name", "amount"]
+```
 
-#### Reference Columns
+### Reference Columns
 
 `REFERENCE_COLUMNS` specifies a list of additional, non-automated columns for your reference/bookkeeping purposes. Each time you run Mintable, the contents of these columns will be preserved.
 
-For example, if you want to add one column to track work expenses, and another to track joint expenses shared with a partner, you could add the following line to your `mintable.config.json` file:
+**Default:** 
 
+```javascript
+"REFERENCE_COLUMNS": ['notes', 'work', 'joint']
 ```
-REFERENCE_COLUMNS=["work", "joint"]
+
+For example, if you want to add one column to track work expenses, and another to track joint expenses shared with a partner, you could add the following line to your `mintable.config.json` file:
+
+```javascript
+"REFERENCE_COLUMNS": ["work", "joint"]
 ```
 
 > **Warning:** Since reference columns are not automated by Mintable, they have the potential to get out of sync with transaction data (for example, if your bank deletes a transaction, causing a row to get removed in `TRANSACTION_COLUMNS`)
 
-#### Category Overrides
+### Account Provider
+
+`ACCOUNT_PROVIDER` specifies which service to use to fetch transactions.
 
-`CATEGORY_OVERRIDES` specifies a list of overrides to handle transactions that are routinely miscategorized by Plaid's servers. Overrides take the following format:
+**Default:** 
+
+```javascript
+"ACCOUNT_PROVIDER": "plaid"
+```
+
+### Spreadsheet Provider
+
+`SHEET_PROVIDER` specifies which service to use to automate spreadsheet updates.
+
+**Default:** 
+
+```javascript
+"SHEET_PROVIDER": "sheets" // "sheets" = Google Sheets
+```
+
+# Provider-Specific Configuration
+
+You can see the API definitions for account & spreadsheet providers in the **[provider docs](./docs/PROVIDERS.md)**.
+
+## Plaid
+
+### Category Overrides
+
+`CATEGORY_OVERRIDES` specifies a list of overrides to handle transactions that are routinely miscategorized by Plaid's servers.
+
+**Default:** 
+
+```javascript
+"CATEGORY_OVERRIDES": []
+```
+
+Overrides take the following format:
 
 * `pattern`: [JavaScript Regular Expression](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp#Syntax) to test transaction names against
 * `flags`: [JavaScript Regular Expression flags](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp#Syntax) (i.e. `i` for case insensitive)
 * `category.0`: Override for first (top-level) category
 * `category.1`: Override for second (level-2) category
 
-For example, if you want anything matching `autopay` or `e-payment` to get categorized as `Credit Card Payment`, you could add the following line to your `mintable.config.json` file:
-
-```
-CATEGORY_OVERRIDES=[{ "pattern": ".*(autopay|e.payment).*", "flags": "i", "category.0": "Transfer", "category.1": "Credit Card Payments" }]
+For example, if you want anything matching `autopay` or `e-payment` to get categorized as `Credit Card Payment`, you could add the following lines to your `mintable.config.json` file:
+
+```javascript
+"CATEGORY_OVERRIDES": [
+    {
+        "pattern": ".*(autopay|e.payment).*",
+        "flags": "i",
+        "category.0": "Transfer",
+        "category.1": "Credit Card Payments"
+    }
+]
 ```
 
-#### Transaction Provider
+## Google Sheets
 
-`ACCOUNT_PROVIDER` specifies which service to use to fetch transactions. At this time, the only possible value is `"plaid"`, but we plan to add other providers in the future.
+### Template Sheet
 
-#### Spreadsheet Provider
+`TEMPLATE_SHEET` specifies the template spreadsheet to use when creating a _new_ sheet for a month.
 
-`SHEET_PROVIDER` specifies which service to use to automate spreadsheet updates. At this time, the only possible value is `"sheets"`, but we plan to add other providers in the future.
+**Default:** 
+
+```javascript
+"TEMPLATE_SHEET": {
+     // Public template: https://docs.google.com/spreadsheets/d/10fYhPJzABd8KlgAzxtiyFN-L_SebTvM8SaAK_wHk-Fw
+    "SHEET_ID": "10fYhPJzABd8KlgAzxtiyFN-L_SebTvM8SaAK_wHk-Fw",
+    "SHEET_TITLE": "Template"
+}
+```
+
+* `SHEET_ID`: Google Sheets spreadsheet ID (from the URL: `docs.google.com/spreadsheets/d/`**`sheet_id`**`/edit`)
+* `SHEET_TITLE`: Title of the sheet (along the bottom row of the document)
+
+For example, you could add the following lines to your `mintable.config.json` file:
+
+```javascript
+"TEMPLATE_SHEET": {
+    "SHEET_ID": "10fYhPJzABd8KasbqiyFN-L_SebTvM8SaAK_wHk-Fw",
+    "SHEET_TITLE": "My Template Sheet"
+}
+```
diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md
@@ -0,0 +1,47 @@
+# Providers APIs
+
+Mintable is designed to be pluggable, i.e. you can swap out Plaid or Google Sheets for another service of your choice.
+
+This document outlines what Mintable expects of a provider, and what constraints/functionality you need to adhere to/implement if you want to add a new provider.
+
+## Transactions (`ACCOUNT_PROVIDER`) API
+
+Account providers should provide an exported function which takes in a startDate and endDate, and returns a `Promise` of a raw list of transaction objects, i.e.:
+
+```javascript
+transactions = await require('../lib/providerName').fetchTransactions(startDate, endDate)
+```
+
+At minimum, we expect `name`, `date`, and `amount` to be defined.
+
+For example, the following would an acceptable response (after promise resolution):
+
+```javascript
+[
+    { "name": "Amazon.com", "date": "2019-04-16T07:00:00.000Z", "amount": -40.22 },
+    { "name": "United Airlines", "date": "2019-04-06T07:00:00.000Z", "amount": -500 },
+    { "name": "Uber", "date": "2019-04-04T07:00:00.000Z", "amount": -6.33 }
+]
+```
+
+## Spreadsheets (`SHEET_PROVIDER`) API
+
+Spreadsheet providers should provide an exported function which takes in a map of sheet name to a list of transactions for that sheet, and return a `Promise` which resolves when all necessary operations to update that sheet are complete, i.e.:
+
+```javascript
+await require('../lib/providerName').updateSheets(updates, options)
+```
+
+where `updates` come in the following format:
+
+```javascript
+{
+    "2019.04": [
+        { "name": "Amazon.com", "date": "2019.04.16", "amount": -40.22 },
+        { "name": "United Airlines", "date": "2019.04.26", "amount": -500 }
+    ],
+    "2019.05": [
+        { "name": "Uber", "date": "2019.05.11", "amount": -6.33 }
+    ]
+}
+```
diff --git a/package.json b/package.json
@@ -4,7 +4,7 @@
   "main": "mintable.js",
   "author": "Kevin Schaich",
   "license": "MIT",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "scripts": {
     "circleci:dry-run": "circleci local execute -e MINTABLE_CONFIG=$MINTABLE_CONFIG",
     "export": "node ./src/scripts/export.js",
@@ -18,18 +18,19 @@
     "@zeit/next-sass": "^1.0.1",
     "body-parser": "^1.18.3",
     "clipboardy": "^1.2.3",
+    "date-fns": "^1.30.1",
     "dotenv": "^7.0.0",
     "express": "^4.16.4",
     "googleapis": "27",
     "indent-string": "^3.2.0",
     "isomorphic-unfetch": "^3.0.0",
     "lodash": "^4.17.11",
     "log-symbols": "^2.2.0",
-    "moment": "^2.23.0",
     "next": "^8.0.3",
     "node-sass": "^4.11.0",
     "opn": "^6.0.0",
     "ora": "^3.4.0",
+    "p-each-series": "^2.1.0",
     "p-map-series": "^2.1.0",
     "plaid": "^2.10.0",
     "promise-fs": "^2.1.0",

diff --git a/src/components/configPropertyInput.jsx b/src/components/configPropertyInput.jsx
@@ -2,10 +2,10 @@ import { FiCheckCircle, FiHelpCircle } from 'react-icons/fi'
 
 const ConfigPropertyInput = props => (
   <div className='config-property-input'>
-    <code title={props.tooltip} >
+    <code title={props.tooltip}>
       <FiCheckCircle className='icon' style={{ color: `#FFFFFF${props.modified ? 'FF' : '00'}` }} />
       {props.displayName}
-      {props.tooltip && <FiHelpCircle className='icon' style={{paddingBottom: '4px'}}/>}
+      {props.tooltip && <FiHelpCircle className='icon' style={{ paddingBottom: '4px' }} />}
     </code>
     <input
       type='text'

diff --git a/src/components/progressSidebar.jsx b/src/components/progressSidebar.jsx
@@ -4,24 +4,23 @@ import * as _ from 'lodash'
 const ProgressSidebar = props => {
   return (
     <div className='progress-sidebar'>
-      <ProgressItem tab={props.tab} number={1} title='Welcome' href='/' />
+      <ProgressItem tab={props.tab} number={1} title='Welcome' href='/welcome' />
       <ProgressItem
-        completed={props.config.accountsSetupCompleted}
+        completed={props.config.accountProviderSetupComplete}
         tab={props.tab}
         number={2}
         title={`${_.startCase(_.lowerCase(props.config.ACCOUNT_PROVIDER))} Setup`}
         href='/account-provider-setup'
       />
       <ProgressItem
-        // completed={_.some(props.config, (value, key).plaid} TODO
-        completed={false}
+        completed={props.config.accountSetupComplete}
         tab={props.tab}
         number={3}
         title='Account Setup'
         href='/account-setup'
       />
       <ProgressItem
-        completed={props.config.sheetsSetupCompleted}
+        completed={props.config.sheetProviderSetupComplete}
         tab={props.tab}
         number={4}
         title={`${_.startCase(_.lowerCase(props.config.SHEET_PROVIDER))} Setup`}